Iceberg table has updated. Expected metadataRootPointer

1

Hi all,

I’m getting the following error on a partitioned metadata refresh and trying to understand what it means.

Iceberg table has updated. Expected metadataRootPointer: wasbs://dremio@spicecachepremiumbloblrs.blob.core.windows.net/metadata/1e1c6540-3411-4019-9ec3-f41410098b40/metadata/01477-483ab7ed-8580-4360-ae6e-670d2b5a575d.metadata.json, Found metadataRootPointer: wasbs://dremio@spicecachepremiumbloblrs.blob.core.windows.net/metadata/1e1c6540-3411-4019-9ec3-f41410098b40/metadata/01478-d1446bd0-630d-4626-a4d3-9789b5000b45.metadata.json

Here is the job profile:
bb2725f5-a852-4a5d-88ed-201f9b8fa465.zip (23.3 KB)

Inside of the verbose logs I can see there is a CONCURRENT_MODIFICATION ERROR - does this mean that two concurrent partitioned metadata refreshes (on separate partitions) can conflict?

CONCURRENT_MODIFICATION ERROR: Iceberg table has updated. Expected metadataRootPointer: wasbs://dremio@spicecachepremiumbloblrs.blob.core.windows.net/metadata/1e1c6540-3411-4019-9ec3-f41410098b40/metadata/01477-483ab7ed-8580-4360-ae6e-670d2b5a575d.metadata.json, Found metadataRootPointer: wasbs://dremio@spicecachepremiumbloblrs.blob.core.windows.net/metadata/1e1c6540-3411-4019-9ec3-f41410098b40/metadata/01478-d1446bd0-630d-4626-a4d3-9789b5000b45.metadata.json

SqlOperatorImpl WRITER_COMMITTER
Location 0:0:2
Fragment 0:0

[Error Id: ca8251ef-995a-4409-8ad4-8314d6ea566a on dremio-executor-1.dremio-cluster-pod.ethereum.svc.cluster.local:0]



SqlOperatorImpl WRITER_COMMITTER
Location 0:0:2
Fragment 0:0
2

@phillip This happens when two metadata refresh queries ran on same table, one of them succeeded and the second one failed because the iceberg table got modified since the query start. Can we please check the jobs page to confirm that? Make sure to enable internal jobs too

1 Like