Dremio 19.0.0 Regression - "java.lang.UnsupportedOperationException) Unknown type to copy."

1

Starting with Dremio 19.0.0, When I’m trying to run a simple query over a (big) Json file read from S3, I’m getting the following error if I add a WHERE filter over a BIGINT field:

  (java.lang.UnsupportedOperationException) Unknown type to copy.
    com.dremio.sabot.op.copier.FieldBufferCopier.addValueCopier():999
    com.dremio.sabot.op.copier.FieldBufferCopier.getCopiers():1014
    com.dremio.sabot.op.copier.FieldBufferCopier$ListCopier.<init>():722
    com.dremio.sabot.op.copier.FieldBufferCopier.addValueCopier():987
    com.dremio.sabot.op.copier.FieldBufferCopier.getCopiers():1014
    com.dremio.sabot.op.copier.FieldBufferCopier$StructCopier.<init>():632
    com.dremio.sabot.op.copier.FieldBufferCopier.addValueCopier():978
    com.dremio.sabot.op.copier.FieldBufferCopier.getCopiers():1014
    com.dremio.sabot.op.copier.FieldBufferCopier$StructCopier.<init>():632
    com.dremio.sabot.op.copier.FieldBufferCopier.addValueCopier():978
    com.dremio.sabot.op.copier.FieldBufferCopier.getCopiers():1014
    com.dremio.sabot.op.copier.FieldBufferCopier$StructCopier.<init>():632
    com.dremio.sabot.op.copier.FieldBufferCopier.addValueCopier():978
    com.dremio.sabot.op.copier.FieldBufferCopier.getCopiers():1014
    com.dremio.sabot.op.copier.FieldBufferCopier$ListCopier.<init>():722
    com.dremio.sabot.op.copier.FieldBufferCopier.addValueCopier():987
    com.dremio.sabot.op.copier.FieldBufferCopier.getCopiers():1014
    com.dremio.sabot.op.copier.VectorizedCopyOperator.setup():101
    com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup():255
    com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():73
    com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():63
    com.dremio.sabot.driver.SmartOp$SmartSingleInput.accept():200
    com.dremio.sabot.driver.StraightPipe.setup():103
    com.dremio.sabot.driver.StraightPipe.setup():102
    com.dremio.sabot.driver.StraightPipe.setup():102
    com.dremio.sabot.driver.Pipeline.setup():68
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():405
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():269
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$1600():94
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():747
    com.dremio.sabot.task.AsyncTaskWrapper.run():112
    com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():243
    com.dremio.sabot.task.slicing.SlicingThread.run():171

Following the stacktrace to com.dremio.sabot.op.copier.FieldBufferCopier and git blaminng, it seems this class changed the way it calls addValueCopier since 19.0.0. This behavior was not spotted in earlier versions

2

@sheinbergon Are you able to provide the failed job profile?

3

Yes, Please allow for a day or so

4

Hi @balaji.ramaswamy

I sent you a private message containing the failed job profile.

Please confirm you have received it

5

@balaji.ramaswamy Did you receive the failed job profile?

6

Could you please check if turning off the following support key helps?
exec.operator.copier.complex.vectorize
This will turn off the support for complex copiers that was added in 19.0.

To diagnose this issue further, we need info on what is the data type that gets mapped to ‘ANY’ (as seen in the profile). This datatype is seen nested inside this structure Array<RecordType<RecordType<RecordType<Array<ANY>>>>>

Also, if possible please provide all the datatypes of the columns that are being used.

7

@Suresh_Jayaraman No, I hadn’t. Which value does it require in order to turn off?

8

From the Dremio UI
Settings > Support > Support Keys
search for the key exec.operator.copier.complex.vectorize and disable this option.

9

I think It did the trick

10x