I’ve configured a promoted physical dataset to have incremental refresh which is checking every few hours. This seems to be working as intended and the refreshes are basically instant due to no new files being detected… However this is still triggering all virtual datasets downstream to refresh. I understand the downstream (with joins) would need to do a full refresh but in this case no data has changed. Is there a way to change this to not do a downstream refresh unless something actually changed?
One approach could be for us to manually kick off a refresh of the pds when we know something is added… but this in itself will only affect one partition (ie. added a new date). Some of our downstreams are filtering on dates and so would also never need to be refreshed (except manually if something changes). Is there any way these could be made smarter such that they know no partitions they care about have actually changed so they dont need to refresh? Or is there a plan for this in the future?