Dremio 23 not match any reflections with MINIOS3

1

i have infraestructure with docker, one dremio as coordinator and 3 dremio as executors and i setting up a minio server as s3 distributed storage for reflections, downloads, etc.
we are been using dremio from dremio 18. but now we ecountered a problem with dremio 23, where any of the reflection we make for our querys match.

image
image1288×838 72.6 KB

c23c3a57-5cc4-48db-870c-317e03fe658e.zip (124,7 KB)

2

btw, we have a dremio 20 with exectuor and coordinator all in one, and this exact query with the same datasources and same reflections, and it works, only with the dremio distributed does not work.

3

Your profile shows a bunch of exceptions while matching on specific reflections. You could enable DEBUG logging to get the stack traces to see if that helps. This is pretty verbose so you only want this on to debug this problem:

<appender name="acceleration" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${dremio.log.path}/acceleration.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
    <fileNamePattern>${dremio.log.path}/archive/acceleration.%d{yyyy-MM-dd}.log.gz</fileNamePattern>
    <maxHistory>30</maxHistory>
  </rollingPolicy>
  <encoder>
    <pattern>%date{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
  </encoder>
</appender>

<logger name="com.dremio.exec.planner.acceleration" additivity="false">
    <level value="debug" />
    <appender-ref ref="acceleration" />
</logger>
<logger name="com.dremio.reflection.bup" additivity="false">
  <level value="debug" />
  <appender-ref ref="acceleration" />
</logger>
<logger name="com.dremio.reflection.ReflectionTracer" additivity="false">
  <level value="debug" />
  <appender-ref ref="acceleration" />
</logger>

I see three reflections hit the union/join limit. You could try increasing that limit with the support option: reflections.planning.algebraic_match_limit

The location that you setup for the dist storage for reflections shouldn’t affect how the query planner does reflection matching.

4

i try it putting down on logback.xml, but isn’t working. maybe its because this is a docker and docker does not have a logfiles.

i try it with new deployment with nfs-server as distributed path with test data, but i getting the same error on using reflection, the issue its seem to appear from dremio 21, because i try it dremio 20, dremio 21 dremio 22, dremio 23, with same config and only dremio 20 its capable of getting reflections from distributed path. this are my config for setting up the server.
dremio.conf.zip (1,0 KB)
i try it with nfs-server, minio-s3, and hadoop.
169f99f4-2a89-404c-8fff-418a3ef7d939.zip (48,6 KB)

5

@nicolas.guerra

I assume TEST.test1 should have matched? Do you have the profile when it was matched in the lower version? we will try and reproduce this issue and provide an update

6

@balaji.ramaswamy I have some tests from 23 version， all most of the time raw reflection not works

7

@rongfengliang And this was consistently working on 21.x/22.x?

8

@balaji.ramaswamy Yes, 21.x/22.x reflection can works well

9

@rongfengliang Any chance you have the 21.x/22.x working profile?

10

@balaji.ramaswamy Raw reflection not used to accelerate select star query - #7 by darian the same problem

1 Like
11

Thanks, let me check

12

@rongfengliang Have all the information we need, has to be reproduced now

13

UPDATE: dremio 23 still not working, we test again on clean installation on new servers, and dremio 22.1.1 its working.
b55446b1-7d39-48e4-8eaf-43472599cb0c.zip (18,4 KB)

14

Your 22.x profile does not have the filter

SELECT * FROM TEST.test1
where RUT like(‘%77%’)

Does 22.x match and choose with the above SQL?

15

yes, rigth now 22.x its the solution, all reflections works

16

Dremio 23 also doesn’t use reflection to accelerate my queries, even though the VDS has reflections with green tick (can accelerate).

select * from VDS: DOESN’T use acceleration
select count(*) from VDS: DOES use acceleration

I’m switching back to 22.1.1

17

I’m having the exact same problem. Since our upgrade to v23 our raw reflections never match. In the acceleration profile I see that it “caught” some exceptions during substitution.

Query profile: 10e9045d-8ca4-4c51-95ce-094375792d3a.zip (27,4 KB)

18

@jeffreyslort

Was this a fresh install or an upgrade from a lower version?

19

This was an upgrade from an lower version. Started at 20 upgraded consecutively to 21, 22 and finally 23. With runs of dremio-admin inbetween each upgrade

20

@jeffreyslort

Thanks, so your reflections worked in 20,21,22 and not in 23?