UnsupportedOperationException

Dremio University
1

Dremio throws “UnsupportedOperationException” when I try joining JSON files in ADLS Storage with another dataset local to Dremio - here is the complete error. The query runs fine until I include the join condition and the table.

SYSTEM ERROR: UnsupportedOperationException

SqlOperatorImpl HASH_PARTITION_SENDER
Location 2:8:0
SqlOperatorImpl HASH_PARTITION_SENDER
Location 2:8:0
Fragment 2:0

[Error Id: fdf75a3e-54b8-4e9b-bd72-3f80f7bf0778 on qa-dremio-instance001-executor-default-0.qa-dremio-instance001-headless.qa-dremio-instance001.svc.cluster.local:0]

(java.lang.UnsupportedOperationException) null
org.apache.arrow.vector.complex.UnionVector.getValidityBufferAddress():241
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.():63
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.():41
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier$GenericCopier.():372
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.addValueCopier():449
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.getCopiers():477
com.dremio.sabot.op.sender.partition.vectorized.VectorizedPartitionSenderOperator.setup():185
com.dremio.sabot.driver.SmartOp$SmartTerminal.setup():346
com.dremio.sabot.driver.Pipe$SetupVisitor.visitTerminalOperator():85
com.dremio.sabot.driver.Pipe$SetupVisitor.visitTerminalOperator():63
com.dremio.sabot.driver.SmartOp$SmartTerminal.accept():290
com.dremio.sabot.driver.StraightPipe.setup():103
com.dremio.sabot.driver.Pipeline.setup():68
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():401
com.dremio.sabot.exec.fragment.FragmentExecutor.run():265
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1400():94
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():733
com.dremio.sabot.task.AsyncTaskWrapper.run():112
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():243
com.dremio.sabot.task.slicing.SlicingThread.run():171

1 Like
2

@priyanka It seems like you are having mixed types for certain columns, can you please send profiles of the 2 datasets before joining

3

@balaji.ramaswamy I have created a Dremio Support ticket and attached all the requested information. I have requested that the ticket be routed to you. Appreciate the help!

4

Hi @priyanka Will take care of it,

5

Any updates on this? I’m facing a similar error trying to create a reflection that joins two tables to these CTAs created from JSONs:

WITH numbers_30 AS (
    SELECT
        ROW_NUMBER() OVER() -1 AS num
    FROM 
        (SELECT FLATTEN(CONVERT_FROM('[0,0,0,0,0,0,0,0,0,0]' ,'JSON')))
        , (SELECT FLATTEN(CONVERT_FROM('[0,0,0]' ,'JSON')))
)

I can and will upload full query as well as profile if useful. Just want to first be sure there isnt an update or work around on this issue other than not using JSON

Thanks!

6

Hello @balaji.ramaswamy,
AsyncTimeoutException issue shows during query, are you familiar with this type of error. I’m currently running Dremio 21.4.1 …need to know the likely cause & what to do next

7

@josephpavao Please send us the profile

8

@Victor Seems like a separate issue, are you able to send me the profile in a new post?