We are working with 2 datasources like NAS and S3. We are using refresh policy for NAS as 1 min,for S3 as 5 min.I am using this 2 datasource in data transformation works like joins.Here my question is when we got new data inserted to these 2 datasources,how the refresh takes place whether it will be happen at 1min or 5 min
The refresh policy applies to Dremio Reflections, which are part of Dremio’s Data Acceleration capabilities. You can read more about Reflections here:
http://docs.dremio.com/acceleration/
Have you created reflections for both of these sources?
If you don’t have any reflections then queries are always executed against the live data.
Thanks for reply.I have gone through the link which you have given.Now I have created reflections for both the sources and now refresh policy is 5 mins.but data is getting refreshed very quickly within secs. So as per the refresh policy,data should get refreshed by 5 mins,instead it is getting refreshed in secs.So please explain how the refresh policy is working here.
If you are talking about refreshing data while running some select queries or listing new files - it is pretty much instantaneous. Meaning you will always query fresh data directly from datasource. If you enabled raw reflections or aggregate reflections to speed up your queries than it is different matter and data in the reflections will be updated based on the “data freshness policy”
Please see following for the reference
