Hi,
When executing the following query:
ALTER TABLE "rn3-dataset"."rn3-dataset"."df-0000140"."collections"."dc-0004071"."current"."Axle" REFRESH METADATA AUTO PROMOTION
it fails for some tables with
FileNotFoundException: File not found reportnet3/metadata/a2697eb5-a07e-4d9a-8db7-82da80fd920b/metadata/6e0b572b-c148-438a-830e-5aaac917cc1d.avro
reportnet3/metadata… is the metadata folder in the distributed storage bucket. Checking and the file does not exist in the distributed storage s3.
The logs from the executor kubernetes pod:
2024-10-29 12:21:14,460 [e17 - 18df2cc4-a530-0ddf-38c5-27649a004600:frag:0:0] ERROR com.dremio.sabot.driver.SmartOp - FileNotFoundException: File not found reportnet3/metadata/ca131808-5326-48d0-9793-cf9518fc87b9/metadata/633e69d0-3fd5-4cef-b6d2-d411658b55ce.avro
com.dremio.common.exceptions.UserException: FileNotFoundException: File not found reportnet3/metadata/ca131808-5326-48d0-9793-cf9518fc87b9/metadata/633e69d0-3fd5-4cef-b6d2-d411658b55ce.avro
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:979)
at com.dremio.sabot.driver.SmartOp.contextualize(SmartOp.java:189)
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.outputData(SmartOp.java:248)
at com.dremio.sabot.driver.StraightPipe.pump(StraightPipe.java:55)
at com.dremio.sabot.driver.Pipeline.doPump(Pipeline.java:134)
at com.dremio.sabot.driver.Pipeline.pumpOnce(Pipeline.java:124)
at com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run(FragmentExecutor.java:647)
at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:556)
at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:1213)
at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:130)
at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:278)
at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:185)
Caused by: org.apache.iceberg.exceptions.RuntimeIOException: Failed to open file: com.dremio.exec.store.iceberg.DremioInputFile@4f6a4811
at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:102)
at org.apache.iceberg.avro.AvroIterable.iterator(AvroIterable.java:76)
at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:188)
at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:187)
at org.apache.iceberg.io.CloseableIterable.lambda$filter$0(CloseableIterable.java:109)
at org.apache.iceberg.io.CloseableIterable$2.iterator(CloseableIterable.java:72)
at org.apache.iceberg.io.CloseableIterable$7$1.<init>(CloseableIterable.java:188)
at org.apache.iceberg.io.CloseableIterable$7.iterator(CloseableIterable.java:187)
at com.dremio.exec.store.iceberg.ManifestFileProcessor.setupManifestFile(ManifestFileProcessor.java:119)
at com.dremio.exec.store.iceberg.ManifestScanTableFunction.startRow(ManifestScanTableFunction.java:72)
at com.dremio.sabot.op.tablefunction.TableFunctionOperator.outputData(TableFunctionOperator.java:154)
at com.dremio.sabot.driver.SmartOp$SmartSingleInput.outputData(SmartOp.java:243)
... 9 common frames omitted
Caused by: java.io.FileNotFoundException: File not found reportnet3/metadata/ca131808-5326-48d0-9793-cf9518fc87b9/metadata/633e69d0-3fd5-4cef-b6d2-d411658b55ce.avro
What could be the reason and how can we troubleshoot further?
We just upgraded our Dremio to version 25 in kubernetes. Could the upgrade cause this or does the latest Dremio version have any changes that could cause this error?