As we add new data to an incrementally refreshed data set it was originally taking 10 minutes to build and now its up to 30 minutes. We are triggering the refresh either by hitting Refresh Now on the source or via the catalog refresh api endpoint. Do both of these methods trigger a full refresh instead of an incremental one? If so, then how do we trigger an incremental refresh?
Currently we do not support incremental refreshes if there are joins in the Virtual data sets and would fall back to full automatically. Please see documentation for what is supported with Incremental refreshes
There are no joins in this dataset, it is reading straight from the source.
Kindly send us the profile of the refresh reflection job that keeps falling back to FULL REFRESH
Thanks
@balaji.ramaswamy