I am getting below error when I run reflection refresh is triggered intermittently in AKS cluster deployment of dremio. When I encounter this I have to drop the reflection and create it again. Then the same job works. Seems like this is due to some issue related to Iceberg. This happens if I run reflection refresh after doing any update / delete operations on the table, which triggers incremental refresh.
Dremio version - community 24.2.2
Deployment - AKS cluster
Table used - Iceberg table
Reflection type - Raw reflection
Refresh method - Incremental update
Refresh policy - Never refresh, Never expire.
Spark version used to create table - 3.2
Spark Iceberg jar used to create table - iceberg-spark-runtime-3.2_2.12-1.3.1
Error in dremio Job profile
SYSTEM ERROR: NoSuchElementException: No value present
SqlOperatorImpl WRITER_COMMITTER
Location 0:0:5
SqlOperatorImpl WRITER_COMMITTER
Location 0:0:5
Fragment 0:0
[Error Id: 64b670ea-5754-4cf4-abc0-d7907177449f on dremio-executor-dremio-qasprint-0-2.dremio-cluster-pod-dremio-qasprint-0.qa.svc.cluster.local:0]
(java.util.NoSuchElementException) No value present
java.util.Optional.get():148
com.dremio.exec.store.iceberg.manifestwriter.IcebergCommitOpHelper.setup():171
com.dremio.sabot.op.writer.WriterCommitterOperator.setup():148
com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup():282
com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():74
com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():64
com.dremio.sabot.driver.SmartOp$SmartSingleInput.accept():227
com.dremio.sabot.driver.StraightPipe.setup():103
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.Pipeline.setup():71
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():621
com.dremio.sabot.exec.fragment.FragmentExecutor.run():443
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1700():108
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1007
com.dremio.sabot.task.AsyncTaskWrapper.run():122
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():249
com.dremio.sabot.task.slicing.SlicingThread.run():171
Error in dremio executor node
2023-12-14 14:56:23,557 [e1 - 1a84e868-3ed1-537b-b999-afd8214ff000:frag:0:0] ERROR com.dremio.sabot.driver.SmartOp - NoSuchElementException: No value present
com.dremio.common.exceptions.UserException: NoSuchElementException: No value present
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:926)
at com.dremio.sabot.driver.SmartOp.contextualize(SmartOp.java:164)
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup(SmartOp.java:286)
at com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput(Pipe.java:74)
at com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput(Pipe.java:64)
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.accept(SmartOp.java:227)
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:103)
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:102)
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:102)
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:102)
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:102)
at com.dremio.sabot.driver.StraightPipe.setup(StraightPipe.java:102)
at com.dremio.sabot.driver.Pipeline.setup(Pipeline.java:71)
at com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution(FragmentExecutor.java:621)
at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:443)
at com.dremio.sabot.exec.fragment.FragmentExecutor.access$1700(FragmentExecutor.java:108)
at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:1007)
at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:122)
at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:249)
at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:171)
Caused by: java.util.NoSuchElementException: No value present
at java.base/java.util.Optional.get(Optional.java:148)
at com.dremio.exec.store.iceberg.manifestwriter.IcebergCommitOpHelper.setup(IcebergCommitOpHelper.java:171)
at com.dremio.sabot.op.writer.WriterCommitterOperator.setup(WriterCommitterOperator.java:148)
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup(SmartOp.java:282)
... 17 common frames omitted
[2023-12-14T14:56:23.557+0000][info][stacktrace ] java.lang.IllegalArgumentException, 25
[2023-12-14T14:56:23.557+0000][info][stacktrace ] java.lang.IllegalArgumentException, 25
[2023-12-14T14:56:23.557+0000][info][oopstorage,ref ] StringTable weak: allocated 0x00005569d5ed41e8
[2023-12-14T14:56:23.557+0000][info][exceptions ] Exception <a 'java/lang/IllegalArgumentException'{0x0000000739f39cd8}: No enum constant com.dremio.common.expression.SupportedEngines.CodeGenOption.gandiva>
thrown in C1 compiled method <{method} {0x00007f250cba8490} 'getCodeGenOption' '(Ljava/lang/String;)Lcom/dremio/common/expression/SupportedEngines$CodeGenOption;' in 'com/dremio/common/expression/SupportedEngines$CodeGenOption'>
at PC0x00007f25271d5e3c for thread 0x00007f2500d1a800
[2023-12-14T14:56:23.557+0000][info][oopstorage,ref ] StringTable weak: allocated 0x00007f253f2e5130
[2023-12-14T14:56:23.557+0000][info][exceptions ] Exception <a 'java/lang/IllegalArgumentException'{0x000000073bad8f18}: No enum constant com.dremio.common.expression.SupportedEngines.CodeGenOption.gandiva>
thrown in C1 compiled method <{method} {0x00007f250cba8490} 'getCodeGenOption' '(Ljava/lang/String;)Lcom/dremio/common/expression/SupportedEngines$CodeGenOption;' in 'com/dremio/common/expression/SupportedEngines$CodeGenOption'>
at PC0x00007f25271d5e3c for thread 0x00007f2500286000