Is there a direct or automated way to automatically refresh the newly appended data in hdfs folder path to be visible in physical dataset,
When i fire a sql query “alter table <> refresh metadata;” it refreshes the data…
but i want a direct or automated way… Is there any property or any API to do this?
PS- I am using community version
You can set the background refresh to a frequency that will suit your data freshness needs
Thanks
Bali
@balaji.ramaswamy
My data is on HDFS,so i am using HDFS data source… The above caching source metadata does not support to HDFS data source… it is specified in the link which you mentioned in the reply
Is there any other option?
You could use the SQL API to submit the “alter table <> refresh metadata” command.
And, as @balaji.ramaswamy mentioned, you can adjust the background refresh interval for your HDFS source in Dremio.
@ben,
thanks for you response… but i am unable to find metadata option in my HDFS source, is it because i am using community edition?
@ankitaKamat, you should have this feature with Community Edition. If you are not and Admin in Dremio, then you will not be able to see it:
