3.1 matching algorithm : Reflection not used

After migration to 3.1.1, I’m unfortunately running into the case when an aggregation reflection is not used

I have two aggregation reflections on a parquet source dataset (one with few columns, one with more columns), and when I basically test the simple one (called basic), the reflection is not used with message “Considered : not matched”

Any insight why it is not chosen ?

Here is the query profile :1af618b8-f20a-484e-96f1-279eb96a9e09.zip (23.8 KB)

The definition os the reflection is as follow, and I simply try to do
a basic
EXPLAIN PLAN FOR SELECT wtbid,action, sum(prc)
FROM S3a.“swn-sframe”.raw_wtb_actions
group by wtbid,action

Reflection Definition: basic

Matched: 0, Chosen: 0, Match Latency: 0 ms

Reflection Id: 02d67809-bbc9-452e-9c1b-d8884a3b7238, Materialization Id: a1418aa7-c1ae-4073-9c62-2aa9c5fd71b0
Expiration: 3018-06-02T11:17:37Z
Dataset: S3a.“swn-sframe”.raw_wtb_actions
Age: 4 hours 57 minutes 17 seconds
Dimensions: distri_id, is_preview, auto, is_bot_traffic, module, is_bot, ts, wtbid, is_mobile, action, type, fmt, dir0, dir1, dir2, dir3, is_pc,
Measures:

  • prc ( SUM, COUNT, )

Sorted: ts,

I didn’t have this issue with 3.0 release.

Thanks

Same issue. Aggregated reflection not used after upgrade from 3.0.6 to 3.1.

Sometimes I wish I could force a reflection in query , like forcing an index in SQL :wink:

2 Likes

@asm did yours work with 3.0?

It works fine with 3.0.6

I was about to rollback to 3.0.6 this morning, when I checked again the query. At the moment it uses the aggregation reflection

Unfortunately in my case aggregation reflection is still not used in 3.1.1 :worried:
Pls find attached query profiles for

@dfleckinger are you saying it works on Thursdays? :slight_smile:

Yes @kelly it started to work again the day after migration to 3.1.1
As if some reflections needed to be rebuilt.

You shouldn’t need to recreate or refresh your reflections for the enhancements to apply. Not sure what happened but glad it is working now.

Personally, I find matching is much, much better now. We will always be refining the algorithms and this was a big step forward.

Today, the reflection is not used anymore when executing a query on the dataset.
Not that stable.
I will check if this could be link to the Incremental or Full refresh mode.

Yes, let’s dig in and see what’s going on here. Thanks for your help.

About my situation, it seems that the issue was indeed coming from the Incremental refresh method.
I switched the refresh method back to “Full update”, and forced the reflections to materialize again,
and I can now see my basic queries using the reflections.