My use case involves providing near-real time updates of data with very low latency requirements. I figured out that careful configuration of raw and aggregate reflections could meet the requirements. However, they are not updateable near-real time.
Therefore, given a timestamp field, I wanted to use raw reflection for data before x timestamp and use external reflection with manual metadata refresh for data after x timestamp. Union of the two will give the latest and great results with the fastest query.
However, it seems that once raw reflection is created, even on a Dremio view off the source iceberg table, the raw reflection is used by default. Is there a way that I could achieve my use case?
@kyleahn Depends on costing, send us the profile when it is not picking the right reflection, turn on planner.verbose_profile, then run the query then send the profile
There’s a feature coming out in the upcoming release, where you can specify reflection hints to the query planner. There are some notes about how “first view with a hint wins” during planning, but I’m unsure whether it applies when doing a union all. You could give that a go?
The release notes are up, but the image usually lags a week or so.
Nah, it seems we’re all waiting. Earlier, the actual release (git push, tarball, Docker image) used to follow about a week after the release notes. That doesn’t seems to hold true anymore - it’s more weeks if not a month(s).
