Failure during post-substitution processing

1

Hi Dremio Team,
we are seeing a lot of Failure during post-substitution processing. errors when executing a query. this makes reflections being not used, Thanks in advance.

dremio version: 3.2.4-201906051751050278-1bcce62

Error: Failure during post-substitution processing. Used reflections: [ce4624dd-644f-4169-bc3a-e551be9fd8b9]

same error multiple times

query profile attached 1450523e-e589-486c-b9f4-149be198f83c.zip (513.5 KB)

2

@smora

Which reflection in the below list you were expecting to be accelerated?

Query was accelerated

  • fx_pivot (RWH_HDFS)
    • Raw Reflection (raw): considered, matched, chosen.
  • fx_pivot (RWH)
    • VDS_RAW_ALL (raw): considered, not matched.
  • fx_pivot (RWH_LOCAL."_helper")
    • Raw Reflection (raw): considered, not matched.
  • rdv_dynamic_part (RWH_HDFS)
    • Raw Reflection (raw): considered, matched, not chosen.
  • rdv_dynamic_part_20190708 (RWH_LOACL2."_helper")
    • Raw Reflection (raw): considered, not matched.
3

Hi Balaji

this one, its matched but not chosen due to substitution error.

rdv_dynamic_part (RWH_HDFS)

  • Raw Reflection (raw): considered, matched, not chosen.

Thanks

4

@smora

Would be good to get the server.log and see the excpetion

5

@balaji.ramaswamy

we are running in a multi node cluster, is there a way to find out on which executor this query was run? so that i can get log from the specific executor

6

Exception should be on the coordinator log

7

Attached log. Thanks.syslog.zip (13.4 KB)

8

@balaji.ramaswamy
you got a chance to look at the log?

Thanks

9

@balaji.ramaswamy

Any update on this issue?

10

@smora

Sorry did not get a chance to look yet, Can you please get the reflection id for this reflection from sys.reflections and the materialization_id for the same reflection_id from sys.materializations and search for those in this logfile. Would it be possible to put the error stack here?

Thanks
@balaji.ramaswamy