Problems with mapping "number" datatype from Snowflake to Dremio

Hi all,

I’m using the Dremio 4.0.0 community, and encounter datatype transform error (number to decimal) from Snowflake. I also have the same datatype in Oracle, but Oracle does not have this problem…

Datatype:

  • Snowflake: NUMBER(38,0)
  • Dremio: Decimal

Here is part of my snowflake-arp.yaml, I follow the config that Dremio provide: snowflake-arp
- source:
name: “NUMBER”
max_precision: 38
max_scale: 37
dremio:
name: “DECIMAL”

Error log:
(java.lang.IllegalArgumentException) Gandiva only supports decimals of upto 38 precision. Input precision : 0
org.apache.arrow.util.Preconditions.checkArgument():136
org.apache.arrow.gandiva.expression.ArrowTypeHelper.initArrowTypeDecimal():118
org.apache.arrow.gandiva.expression.ArrowTypeHelper.arrowTypeToProtobuf():246
org.apache.arrow.gandiva.expression.FunctionNode.toProtobuf():44
org.apache.arrow.gandiva.expression.ExpressionTree.toProtobuf():42
org.apache.arrow.gandiva.evaluator.Projector.make():119
org.apache.arrow.gandiva.evaluator.Projector.make():79
com.dremio.sabot.op.llvm.NativeProjector.build():64
com.dremio.sabot.op.llvm.NativeProjectorBuilder.build():69
com.dremio.exec.expr.SplitStageExecutor.setupFinish():178
com.dremio.exec.expr.SplitStageExecutor.setupProjector():211
com.dremio.exec.expr.ExpressionSplitter.projectorSetup():295
com.dremio.exec.expr.ExpressionSplitter.setupProjector():396
com.dremio.exec.store.CoercionReader.newSchema():155
com.dremio.exec.store.CoercionReader.setup():119
com.dremio.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser():214
com.dremio.sabot.op.scan.ScanOperator.setupReader():187
com.dremio.sabot.op.scan.ScanOperator.setup():173
com.dremio.sabot.driver.SmartOp$SmartProducer.setup():563
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():79
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():63
com.dremio.sabot.driver.SmartOp$SmartProducer.accept():533
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.Pipeline.setup():68
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():372
com.dremio.sabot.exec.fragment.FragmentExecutor.run():258
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():87
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():658
com.dremio.sabot.task.AsyncTaskWrapper.run():104
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226
com.dremio.sabot.task.slicing.SlicingThread.run():156

Where should I go to make changes so that Dremio can catch the right precision?
Please give me some suggestion, any inputs will be appreciated!

1 Like

@carterlin - can you verify the precision returned for the column metadata for the NUMBER column in question? It appears that a precision of 0 is being used here.

Thanks for suggestion.
I fix this issue using the latest dremio community and dremio-snowflake-plungin.

Thanks again for your help!

1 Like