Dremio 23 not match any reflections with MINIOS3

21

I did not test the reflections between upgrades. So as far as I know the reflections worked in V20 and did not work on V23.

22

@jeffreyslort In the v23 profile you sent, you ran:

SELECT * FROM Raw.TOS.“che_master”.“che_master_current”

And the reflection substitution failed with the error:

Caught exception during default reflection substitution for Raw.TOS.che_master.che_master_current

Is it possible for you to run the same query with the same reflection on a prior working version and give us the profile?

Thanks

23

@Benny_Chow I could not downgrade the dremio instance with this problem so I set up a new environment where I connected to the same azure storage account installed V22 setup the datasets and reflections. Run a “SELECT * FROM” reflection is used. Then I upgraded, ran the dremio-admin upgrade and restarted. Ran the same “SELECT * FROM” reflection is not used.

v23-job-profile.zip (19,0 KB)
v22-job-profile.zip (16,4 KB)

24

@jeffreyslort Thank you for the profiles. I logged an internal ticket and will try to reproduce.

25

@jeffreyslort I took your v22 profile, extracted the PDS and VDS definitions, re-created them on a 22 server as Iceberg tables in S3. Then, I created a raw reflection on the VDS (also stored as Iceberg in S3) and verified that the “select * on VDS” was accelerated. Then, I shutdown the server, upgraded to 23.1.0 and re-ran the query on the 23.1.0 server. For me, the “select * on VDS” continues to get accelerated with the 22 reflection.

So, right now, I still can’t reproduce the substitution error. :upside_down_face:

Could you add the following to your v23 logback.xml, re-run the “select *” and then send me reflection.log, acceleration.log, server.log and dremio.conf? Thanks

<appender name="reflection" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${dremio.log.path}/reflection.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
    <fileNamePattern>${dremio.log.path}/archive/reflection.%d{yyyy-MM-dd}.log.gz</fileNamePattern>
    <maxHistory>30</maxHistory>
  </rollingPolicy>
  <encoder>
    <pattern>%date{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
  </encoder>
</appender>
  
<logger name="com.dremio.service.reflection" additivity="false">
  <level value="DEBUG"/>
    <appender-ref ref="reflection"/>
</logger>

<appender name="acceleration" class="ch.qos.logback.core.rolling.RollingFileAppender">
  <file>${dremio.log.path}/acceleration.log</file>
  <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
    <fileNamePattern>${dremio.log.path}/archive/acceleration.%d{yyyy-MM-dd}.log.gz</fileNamePattern>
    <maxHistory>30</maxHistory>
  </rollingPolicy>
  <encoder>
    <pattern>%date{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
  </encoder>
</appender>

<logger name="com.dremio.exec.planner.acceleration" additivity="false">
    <level value="debug" />
    <appender-ref ref="acceleration" />
</logger>
<logger name="com.dremio.reflection.bup" additivity="false">
  <level value="debug" />
  <appender-ref ref="acceleration" />
</logger>
<logger name="com.dremio.reflection.ReflectionTracer" additivity="false">
  <level value="debug" />
  <appender-ref ref="acceleration" />
</logger>
26

@Benny_Chow, So for me the files are stored on a Azure storage account and the files are just Parquet files, not Iceberg.

See the zip for an extract of my log files :slight_smile:

logs.zip (6,6 KB)

1 Like
27

Hi @jeffreyslort , just want to give you an update. We were able to reproduce and fix the problem with reflections not matching starting on V23. I would recommend waiting until V24 comes out and upgrade to that.

28

@Benny_Chow alright that sounds promising. Any idea when v24 will be released ?

29

Very soon! Also, I want to reiterate the appreciation for you and others in this group for sharing the profiles and/or logs so that we could finally get to the bottom of this and fix this issue. This problem was also related to substitution errors, upgrade and matching issues in V23.

1 Like
30

@jeffreyslort v24 is released. The issue with reflections containing VARCHAR columns not matching has been fixed. Index of community-server/24.0.0-202302100528110223-3a169b7c/

2 Likes
31

@Benny_Chow Thanks for the headsup will test it right away :slight_smile:

32

Alright I can confirm that it works with dremio v24

1 Like