I plan to use incremental refresh policy for reflections on MongoDB collections. However, instead of incrementing the reflection key for every document, I’d like to publish a complete set of new documents for a new date - by recreating a MongoDB view.
Will this approach work with the incremental reflections? Would it be a problem that the key field (in my case DATE) is not unique?
https://docs.dremio.com/acceleration/updating-reflections.html?h=Incremental
I have used both incremental update options BIGINT and DATE/TIME
Date has duplicates and works based on what you are describing in your use case.
In my use cases, I have only used a auto incremental BIGINT Never tested duplication or that field not being unique. Would be interesting if it worked though. Just not sure of a use case for that.
@dealercrm, sorry, confused with your answer: did you try using DATE with duplicates as the key for incremental reflections and it worked? You later say “I have only used an auto-incremental BIGINT” which is the confusing part.
Yes.
Used DATE/TIME with duplicates and it worked for me.
Thanks, I’ll try it.
keep in mind that Dremio uses the key field to filter out any value already encountered in the previous refreshes. So if you add some records with the same key field value before and after an incremental refresh, the reflection won’t include the values that appeared after the refresh.