Dremio 23 not match any reflections with MINIOS3

@nicolas.guerra

I assume TEST.test1 should have matched? Do you have the profile when it was matched in the lower version? we will try and reproduce this issue and provide an update

@balaji.ramaswamy I have some tests from 23 version, all most of the time raw reflection not works

@rongfengliang And this was consistently working on 21.x/22.x?

@balaji.ramaswamy Yes, 21.x/22.x reflection can works well

@rongfengliang Any chance you have the 21.x/22.x working profile?

@balaji.ramaswamy Raw reflection not used to accelerate select star query - #7 by darian the same problem

1 Like

Thanks, let me check

@rongfengliang Have all the information we need, has to be reproduced now

UPDATE: dremio 23 still not working, we test again on clean installation on new servers, and dremio 22.1.1 its working.
b55446b1-7d39-48e4-8eaf-43472599cb0c.zip (18,4 KB)

Your 22.x profile does not have the filter

SELECT * FROM TEST.test1
where RUT like(‘%77%’)

Does 22.x match and choose with the above SQL?

yes, rigth now 22.x its the solution, all reflections works

Dremio 23 also doesn’t use reflection to accelerate my queries, even though the VDS has reflections with green tick (can accelerate).

select * from VDS: DOESN’T use acceleration
select count(*) from VDS: DOES use acceleration

I’m switching back to 22.1.1

I’m having the exact same problem. Since our upgrade to v23 our raw reflections never match. In the acceleration profile I see that it “caught” some exceptions during substitution.

Query profile: 10e9045d-8ca4-4c51-95ce-094375792d3a.zip (27,4 KB)

@jeffreyslort

Was this a fresh install or an upgrade from a lower version?

This was an upgrade from an lower version. Started at 20 upgraded consecutively to 21, 22 and finally 23. With runs of dremio-admin inbetween each upgrade

@jeffreyslort

Thanks, so your reflections worked in 20,21,22 and not in 23?

I did not test the reflections between upgrades. So as far as I know the reflections worked in V20 and did not work on V23.

@jeffreyslort In the v23 profile you sent, you ran:

SELECT * FROM Raw.TOS.“che_master”.“che_master_current”

And the reflection substitution failed with the error:

Caught exception during default reflection substitution for Raw.TOS.che_master.che_master_current

Is it possible for you to run the same query with the same reflection on a prior working version and give us the profile?

Thanks

@Benny_Chow I could not downgrade the dremio instance with this problem so I set up a new environment where I connected to the same azure storage account installed V22 setup the datasets and reflections. Run a “SELECT * FROM” reflection is used. Then I upgraded, ran the dremio-admin upgrade and restarted. Ran the same “SELECT * FROM” reflection is not used.

v23-job-profile.zip (19,0 KB)
v22-job-profile.zip (16,4 KB)

@jeffreyslort Thank you for the profiles. I logged an internal ticket and will try to reproduce.