Aggregate reflections always does full refresh

1

Hi,
We have noticed that aggregate reflections always does full refresh then incremental refresh. It is throwing following message in Planning phase: “Full Refresh.
Cannot do incremental update because the reflection has multiple aggregate operators”.
My usecase: I have a iceberg PDS with 30 properties of double datatype. Created a filtered-view VDS on PDS.For the filtered view created AGG reflection. We have created a following aggregates for each property (sum,count,min,max) . The agg reflection is partitioned on a _time column at day granularity.
We are seeing complete dataset is refreshed periodically. As the dataset grows to GB’s refresh will be come costly as it occupies good resources.
Any workaround or help is appreciated.

Screenshot and reflection profile is uploaded.

Thanks,
Pavan

Screenshot 2024-04-22 at 4.46.03 PM (2)
Screenshot 2024-04-22 at 4.46.03 PM (2)2560×1440 480 KB

3de52a4a-8ff8-4f25-bb2c-527161f0b65b.zip (40.7 KB)

2

@Julian_Kiryakov , @balaji.ramaswamy any help on this will be much appreciated. Not sure what is the use off agg reflections if it does full refresh each time.

3

Hi Pavan,
Thanks for reaching out. Unfortunately, incremental refresh is not supported when you have 2 aggregations on top of each other. If you look at the logical plan you will find 2 AggregateRel on top of each other. Likely one comes from the view definition and the other from the agg reflection itself.

As a workaround you can consider removing the aggregation as part of the view and only doing aggregation as part of the agg reflection.
Alternatively, you could rewrite the view definition to contain all the aggregations you need in a single select statement (no nested selects with different group by’s each) and then create a raw reflection on the view.
Thanks,
Julian

4

Thanks Julian for the quick resolutions. We will evaluate them and get back to you incase of any issues.

Thanks,
Pavan