Concurrent DML operation on Iceberg table

1

Hi Team,

We are posting dremio jobs through Dremio’s SQL API. We are trigerring subsequent SQL jobs that update an Iceberg table, however some of the jobs are failing with the below error:

Concurrent DML operation has updated the table, please retry. (Attached screenshot for your reference)

Please let us know if we can issue multiple DML commands on an Iceberg table concurrently.

Iceberg_ConcurrencyUpdate_Failed
Iceberg_ConcurrencyUpdate_Failed2689×340 25.6 KB

Thanks
Sumi

2

Hi @Sumithra

Looks like the commit is happening at the same time, if you introduce a delay between the SQL statements, does it still happen?

Thanks
Bali

3

Hi @balaji.ramaswamy ,

Yes, we tried introducing delay between the SQL statements, but still some updates coincides with the commit of other update statements and is failing some of the updates. Please let us know if there is a way around issuing multiple update statements on the same iceberg table simultaneously.

Thanks
Sumi

4

@Sumithra Currently no, is this because you have updates from different users ?

5

@Sumi et all - did you find a way to do this? We also have a similar issue in that we want to support multiple processes deleting data in the same iceberg table. We know there will be no clashes between these delete statements but they need to be executed concurrently ideally.