I did not test the reflections between upgrades. So as far as I know the reflections worked in V20 and did not work on V23.
@jeffreyslort In the v23 profile you sent, you ran:
SELECT * FROM Raw.TOS.“che_master”.“che_master_current”
And the reflection substitution failed with the error:
Caught exception during default reflection substitution for Raw.TOS.che_master.che_master_current
Is it possible for you to run the same query with the same reflection on a prior working version and give us the profile?
Thanks
@Benny_Chow I could not downgrade the dremio instance with this problem so I set up a new environment where I connected to the same azure storage account installed V22 setup the datasets and reflections. Run a “SELECT * FROM” reflection is used. Then I upgraded, ran the dremio-admin upgrade and restarted. Ran the same “SELECT * FROM” reflection is not used.
v23-job-profile.zip (19,0 KB)
v22-job-profile.zip (16,4 KB)
@jeffreyslort I took your v22 profile, extracted the PDS and VDS definitions, re-created them on a 22 server as Iceberg tables in S3. Then, I created a raw reflection on the VDS (also stored as Iceberg in S3) and verified that the “select * on VDS” was accelerated. Then, I shutdown the server, upgraded to 23.1.0 and re-ran the query on the 23.1.0 server. For me, the “select * on VDS” continues to get accelerated with the 22 reflection.
So, right now, I still can’t reproduce the substitution error. 
Could you add the following to your v23 logback.xml, re-run the “select *” and then send me reflection.log, acceleration.log, server.log and dremio.conf? Thanks
<appender name="reflection" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${dremio.log.path}/reflection.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${dremio.log.path}/archive/reflection.%d{yyyy-MM-dd}.log.gz</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%date{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="com.dremio.service.reflection" additivity="false">
<level value="DEBUG"/>
<appender-ref ref="reflection"/>
</logger>
<appender name="acceleration" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${dremio.log.path}/acceleration.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${dremio.log.path}/archive/acceleration.%d{yyyy-MM-dd}.log.gz</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder>
<pattern>%date{ISO8601} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="com.dremio.exec.planner.acceleration" additivity="false">
<level value="debug" />
<appender-ref ref="acceleration" />
</logger>
<logger name="com.dremio.reflection.bup" additivity="false">
<level value="debug" />
<appender-ref ref="acceleration" />
</logger>
<logger name="com.dremio.reflection.ReflectionTracer" additivity="false">
<level value="debug" />
<appender-ref ref="acceleration" />
</logger>
@Benny_Chow, So for me the files are stored on a Azure storage account and the files are just Parquet files, not Iceberg.
See the zip for an extract of my log files 
logs.zip (6,6 KB)
1 Like
Hi @jeffreyslort , just want to give you an update. We were able to reproduce and fix the problem with reflections not matching starting on V23. I would recommend waiting until V24 comes out and upgrade to that.
Very soon! Also, I want to reiterate the appreciation for you and others in this group for sharing the profiles and/or logs so that we could finally get to the bottom of this and fix this issue. This problem was also related to substitution errors, upgrade and matching issues in V23.
1 Like
@jeffreyslort v24 is released. The issue with reflections containing VARCHAR columns not matching has been fixed. Index of community-server/24.0.0-202302100528110223-3a169b7c/
2 Likes
Alright I can confirm that it works with dremio v24
1 Like