UnsupportedOperationException: Gandiva code generation is handled during build

Ken · May 1, 2024, 2:57pm

I am new to Dremio and I have some files stored in Minio in the Iceberg format.

I have a table which captures the Market Capitalisation of some stocks.

I have a column called adj_market_capitalisation, which is essentially market capitalisation in HKD.

I want to create a dimension column which classifies market capitalisation size of a stock with a case-logic.

It worked previously in postgresql, but in dremio, I have getting the error

UnsupportedOperationException: Gandiva code generation is handled during build.

I read other posts that it might be due to mixed type.

but my adj_market_capitalisation has the type of DECIMAL.

========

Query Profile:
ed4aecf3-90d5-4d46-bb3b-0646a4ebe1ac.zip (28.7 KB)

Ken · May 1, 2024, 2:58pm

Screenshot with error message.

Ken · May 1, 2024, 3:05pm

Here are the logs from the kubernetes pod running the Dremio Executor.

   at com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup(SmartOp.java:317)
    at com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput(Pipe.java:83)
    at com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput(Pipe.java:70)
    at com.dremio.sabot.driver.SmartOp$SmartSingleInput.accept(SmartOp.java:258)
    at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:101)
    at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:100)
    at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:100)
    at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:100)
    at com.dremio.sabot.driver.Pipeline.setup(Pipeline.java:79)
    at com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution(FragmentExecutor.java:723)
    at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:510)
    at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:1213)
    at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:130)
    at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:278)
    at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.UnsupportedOperationException: Gandiva code generation is handled during build.
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2087)
    at com.google.common.cache.LocalCache.get(LocalCache.java:4036)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4059)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5041)
    at com.dremio.exec.compile.CodeCompiler.getImplementationClassFromExpToCompiledClazzCache(CodeCompiler.java:102)
    at com.dremio.exec.compile.CodeCompiler.getImplementationClass(CodeCompiler.java:80)
    at com.dremio.exec.compile.CodeCompiler.getImplementationClass(CodeCompiler.java:69)
    at com.dremio.exec.expr.CodeGenerator.getImplementationClass(CodeGenerator.java:165)
    at com.dremio.exec.expr.SplitStageExecutor.setupFinish(SplitStageExecutor.java:204)
    at com.dremio.exec.expr.SplitStageExecutor.setupProjector(SplitStageExecutor.java:240)
    at com.dremio.exec.expr.ExpressionSplitter.projectorSetup(ExpressionSplitter.java:408)
    at com.dremio.exec.expr.ExpressionSplitter.setupProjector(ExpressionSplitter.java:690)
    at com.dremio.sabot.op.project.ProjectOperator.setup(ProjectOperator.java:164)
    at com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup(SmartOp.java:313)
    ... 14 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Gandiva code generation is handled during build.
    at com.dremio.exec.expr.fn.GandivaFunctionHolder.renderStart(GandivaFunctionHolder.java:52)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:248)
    at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitFunctionHolderExpression(EvaluationVisitor.java:1409)
    at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1153)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.lambda$visitFunctionHolderExpression$0(EvaluationVisitor.java:1583)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.preProcessIfConstant(EvaluationVisitor.java:1567)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1582)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1527)
    at com.dremio.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:53)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:256)
    at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitFunctionHolderExpression(EvaluationVisitor.java:1409)
    at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1153)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.lambda$visitFunctionHolderExpression$0(EvaluationVisitor.java:1583)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.preProcessIfConstant(EvaluationVisitor.java:1567)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1582)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1527)
    at com.dremio.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:53)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitIfExpression(EvaluationVisitor.java:276)
    at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitIfExpression(EvaluationVisitor.java:1434)
    at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitIfExpression(EvaluationVisitor.java:1164)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.lambda$visitIfExpression$3(EvaluationVisitor.java:1613)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.preProcessIfConstant(EvaluationVisitor.java:1567)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitIfExpression(EvaluationVisitor.java:1613)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitIfExpression(EvaluationVisitor.java:1527)
    at com.dremio.common.expression.IfExpression.accept(IfExpression.java:50)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitIfExpression(EvaluationVisitor.java:348)
    at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitIfExpression(EvaluationVisitor.java:1434)
    at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitIfExpression(EvaluationVisitor.java:1164)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.lambda$visitIfExpression$3(EvaluationVisitor.java:1613)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.preProcessIfConstant(EvaluationVisitor.java:1567)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitIfExpression(EvaluationVisitor.java:1613)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitIfExpression(EvaluationVisitor.java:1527)
    at com.dremio.common.expression.IfExpression.accept(IfExpression.java:50)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitFunctionHolderExpression(EvaluationVisitor.java:256)
    at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitFunctionHolderExpression(EvaluationVisitor.java:1409)
    at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1153)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.lambda$visitFunctionHolderExpression$0(EvaluationVisitor.java:1583)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.preProcessIfConstant(EvaluationVisitor.java:1567)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1582)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitFunctionHolderExpression(EvaluationVisitor.java:1527)
    at com.dremio.common.expression.FunctionHolderExpression.accept(FunctionHolderExpression.java:53)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitCaseExpression(EvaluationVisitor.java:439)
    at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitCaseExpression(EvaluationVisitor.java:1459)
    at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitCaseExpression(EvaluationVisitor.java:1176)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.lambda$visitCaseExpression$4(EvaluationVisitor.java:1620)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.preProcessIfConstant(EvaluationVisitor.java:1567)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitCaseExpression(EvaluationVisitor.java:1619)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitCaseExpression(EvaluationVisitor.java:1527)
    at com.dremio.common.expression.CaseExpression.accept(CaseExpression.java:48)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitValueVectorWriteExpression(EvaluationVisitor.java:706)
    at com.dremio.exec.expr.EvaluationVisitor$EvalVisitor.visitUnknown(EvaluationVisitor.java:625)
    at com.dremio.exec.expr.EvaluationVisitor$InnerMethodNester.visitUnknown(EvaluationVisitor.java:1494)
    at com.dremio.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:1316)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.lambda$visitUnknown$16(EvaluationVisitor.java:1704)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.preProcessIfConstant(EvaluationVisitor.java:1567)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitUnknown(EvaluationVisitor.java:1704)
    at com.dremio.exec.expr.EvaluationVisitor$ConstantFilter.visitUnknown(EvaluationVisitor.java:1527)
    at com.dremio.exec.expr.ValueVectorWriteExpression.accept(ValueVectorWriteExpression.java:63)
    at com.dremio.exec.expr.EvaluationVisitor.addExpr(EvaluationVisitor.java:125)
    at com.dremio.exec.expr.ClassGenerator.addExpr(ClassGenerator.java:394)
    at com.dremio.exec.expr.ClassGenerator.evaluateAllLazyExps(ClassGenerator.java:449)
    at com.dremio.exec.compile.CodeCompiler$ExpressionsToCompiledClazzCacheLoader.load(CodeCompiler.java:151)
    at com.dremio.exec.compile.CodeCompiler$ExpressionsToCompiledClazzCacheLoader.load(CodeCompiler.java:139)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2328)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2187)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
    ... 27 common frames omitted

Ken · May 1, 2024, 3:10pm

The screenshot below shows that the field adj_market_capitalisation is a decimal field.

Simon_Ashley · May 2, 2024, 2:14pm

Thanks for sharing the profile.

Does the query work if you run the following in the same session

ALTER SESSION SET “exec.preferred.codegenerator” = ‘java’;
SELECT * FROM common.l1.“etnet_detail_quote”

Ken · May 3, 2024, 12:02pm

Hi @Simon_Ashley

Thanks for your advise, but it still does not work.

=======

However, I have a workaround which avoids multi-case statements.

I put my critera in a separate table (via dbt seed).

Then I left join that table with some conditions, and it works perfectly.

Below is my code.

with clean_data as (

....

)      select ...
     q."size" as market_capitalisation_size

     from  clean_data p
     left outer join {{ ref('market_capitalisation_size')}} q
     on p.adj_market_capitalisation >= q.minimum_hkd
     and (
             (q.maximum_hkd is not null and p.adj_market_capitalisation < q.maximum_hkd) or
             (q.maximum_hkd is null)
         )

My seed file market_capitalisation_size looks like this

size,minimum_hkd,maximum_hkd
MEGA_CAP,200000000000,
LARGE_CAP,10000000000,200000000000
MID_CAP,2000000000,10000000000
SMALL_CAP,250000000,2000000000
MICRO_CAP,0,250000000

It works perfectly now

balaji.ramaswamy · May 5, 2024, 4:42pm

@Ken Looks like there was an issue fixed recently when Nested CASE WHEN with CAST was used. Let me check if you have the fix

balaji.ramaswamy · May 5, 2024, 5:39pm

@Ken Seems like a different issue, we have an internal ticket with a repro to investigate

Topic		Replies	Views
UnsupportedOperationException Dremio University	7	2527	September 29, 2022
Re-Opening: Dremio 19.0.0 Regression - "java.lang.UnsupportedOperationException) Unknown type to copy."	1	881	December 8, 2021
SystemError - "value" is neither a method, a field, nor a member class of "org.apache.arrow.vector.holders.NullableDecimalHolder"	3	1030	May 4, 2021
Dremio 19.0.0 Regression - "java.lang.UnsupportedOperationException) Unknown type to copy."	8	1356	December 14, 2021
GandivaException: offsets buffer too small for _split_expr0 minimum required 15980 actual 15880	0	1153	February 2, 2021

UnsupportedOperationException: Gandiva code generation is handled during build

Related topics