Ok so I figured out the issue (repeating what you said mostly here @doron)
- I was under the impression that incremental refresh actually updated the reflection as some function of the new files arriving but it still refreshes hourly, just uses incremental refresh to make it more efficient.
- The refresh api is for the physical data source, it was hard to get the physical data source had to use url encoding and then tweak that some, but eventually found it. Once I found that the refresh worked although it was very slow, up to 20 -30 seconds for a simple data set that just consisted of id’s under 10,000
One issue that I am noticing that may be a big one. I was under the impression that when an impression is stale that it simply affects the speed with which the query is run and not its accuracy, but I’m finding on testing that the query actually produces stale results when reflections are not up to date. Does this match your understanding?