Incremental refresh

1

The docs for refreshing Data Reflections include the following:

All other datasets (physical and virtual): An administrator specifies a monotonically increasing field such as a created_at timestamp or auto-incrementing key.

Yet the Refresh Reflection page in the UI states:

Incremental updating is not available for datasets without any BigInt fields.

It appears that the UI message is correct, since it doesn’t recognize timestamp fields as being usable for refresh.

Also, Oracle does not have a native BIGINT datatype. The closest type is NUMBER(19), but a field of this type is not usable for incremental refresh.

My questions are the following:

  • What’s the reason for the discrepancy in the documentation? Will timestamp fields be usable for incremental refreshes in the future?
  • Will other numeric datatypes be supported?
  • How to do an incremental refresh on a source that does not support BIGINT, such as Oracle?
1 Like
2

Thanks Joe. We’ll look into the documentation discrepancy. As for future datatype support I know date/timestamp is important, but what other numeric datatypes do you think would be useful?

3

Well, anything that could be used as an incremental unique identifier… So potentially any numeric type.

4

Can you help me with my question about databases that do not include BIGINT?

5

Hey @Joe, we’re thinking about improving this across the board to include applicable numeric, date/time and string types (probably in that order). We’re actively looking at this from a design perspective – however don’t have a timeline yet.

6

Ok, so that means incremental refresh for Oracle sources is not possible currently?

7

Is there a plan to add the incremental refresh capability for Oracle source? Our tables in oracle db are very big and an incremental refresh would be a good fit.

8

Hey @igreg, yes this is on our radar – we’ll reach out once timing clarifies.