3.1 matching algorithm : Reflection not used

#1

After migration to 3.1.1, I’m unfortunately running into the case when an aggregation reflection is not used

I have two aggregation reflections on a parquet source dataset (one with few columns, one with more columns), and when I basically test the simple one (called basic), the reflection is not used with message “Considered : not matched”

Any insight why it is not chosen ?

Here is the query profile :1af618b8-f20a-484e-96f1-279eb96a9e09.zip (23.8 KB)

The definition os the reflection is as follow, and I simply try to do
a basic
EXPLAIN PLAN FOR SELECT wtbid,action, sum(prc)
FROM S3a.“swn-sframe”.raw_wtb_actions
group by wtbid,action

Reflection Definition: basic

Matched: 0, Chosen: 0, Match Latency: 0 ms

Reflection Id: 02d67809-bbc9-452e-9c1b-d8884a3b7238, Materialization Id: a1418aa7-c1ae-4073-9c62-2aa9c5fd71b0
Expiration: 3018-06-02T11:17:37Z
Dataset: S3a.“swn-sframe”.raw_wtb_actions
Age: 4 hours 57 minutes 17 seconds
Dimensions: distri_id, is_preview, auto, is_bot_traffic, module, is_bot, ts, wtbid, is_mobile, action, type, fmt, dir0, dir1, dir2, dir3, is_pc,
Measures:

  • prc ( SUM, COUNT, )

Sorted: ts,

I didn’t have this issue with 3.0 release.

Thanks

#2

Same issue. Aggregated reflection not used after upgrade from 3.0.6 to 3.1.

#3

Sometimes I wish I could force a reflection in query , like forcing an index in SQL :wink:

2 Likes
#4

@asm did yours work with 3.0?

#5

It works fine with 3.0.6

#6

I was about to rollback to 3.0.6 this morning, when I checked again the query. At the moment it uses the aggregation reflection

Dremio 3.1 is out!
#7

Unfortunately in my case aggregation reflection is still not used in 3.1.1 :worried:
Pls find attached query profiles for

Dremio 3.1 is out!
#8

@dfleckinger are you saying it works on Thursdays? :slight_smile:

#9

Yes @kelly it started to work again the day after migration to 3.1.1
As if some reflections needed to be rebuilt.

#10

You shouldn’t need to recreate or refresh your reflections for the enhancements to apply. Not sure what happened but glad it is working now.

Personally, I find matching is much, much better now. We will always be refining the algorithms and this was a big step forward.

#11

Today, the reflection is not used anymore when executing a query on the dataset.
Not that stable.
I will check if this could be link to the Incremental or Full refresh mode.

#12

Yes, let’s dig in and see what’s going on here. Thanks for your help.

#13

About my situation, it seems that the issue was indeed coming from the Incremental refresh method.
I switched the refresh method back to “Full update”, and forced the reflections to materialize again,
and I can now see my basic queries using the reflections.