# Dremio Iceberg table - DELETE splits files

**URL:** https://community.dremio.com/t/dremio-iceberg-table-delete-splits-files/12689
**Category:** Uncategorized
**Created:** [January 9, 2025, 4:25pm UTC](https://community.dremio.com/t/dremio-iceberg-table-delete-splits-files/12689 "2025-01-09T16:25:18Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![vladislav-stolyarov](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vladislav-stolyarov/32/2351_2.png) [@vladislav-stolyarov](https://community.dremio.com/u/vladislav-stolyarov)
#### Post date: [January 9, 2025, 4:25pm UTC](https://community.dremio.com/t/dremio-iceberg-table-delete-splits-files/12689/1 "2025-01-09T16:25:18Z")

</div>

Hello. I ve encoutered strange(at least for me) behavior when deleting records from Iceberg table out of the box. Aws Glue Datasource.  
I ve created unpartitioned Test table using CTA with 20k rows (\>170 columns if it matters) and 5mb of total size:

```auto
CREATE TABLE Table
LOCALSORT BY (SortColumn)
AS 
SELECT ..170columns.. FROM ...

```

After that checking it’s files: `SELECT * FROM TABLE( table_files( "Table"'))`

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/a/a6672df3ebde7e58b5fe5a23ce97bf18a4e2b692.png)  
Single \<5mb file is created.

Than i ran query to delete 200 rows `DELETE FROM "Table" where SortColumn=10;`

> Rows Deleted 200

Again ran `SELECT * FROM TABLE( table_files( "Table"'))`

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/c/ccbbf747738db9b4307b470b57382afad31d4b33.png)

Now 6 files were produced, each just 1mb of size(total of \>6mb vs \<5mb before)

Running `OPTIMIZE TABLE "Table"` does what is expected: 1 file yielded again.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/7/72f630ac06fdf18040d780fde2ac08c572bd8fd9.png)

Doing similar DELETE from Athena produces such setup:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/2/2815f94e92533404d35c5f5a69db0fa74237a607.png)

Is this by design? Splitting such a small file into 6 doesn’t sound bery optimal for me. How can i avoid it.

---

<div class="post-metadata">

### Author: ![Benny\_Chow](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/benny_chow/32/5527_2.png) [@Benny\_Chow](https://community.dremio.com/u/Benny_Chow)
#### Post date: [January 9, 2025, 9:51pm UTC](https://community.dremio.com/t/dremio-iceberg-table-delete-splits-files/12689/2 "2025-01-09T21:51:16Z")

</div>

Dremio supports both COW and MOR as described here: [Lakehouse Strategies: Copy-on-Write vs. Merge-on-Read](https://www.dremio.com/blog/row-level-changes-on-the-lakehouse-copy-on-write-vs-merge-on-read-in-apache-iceberg/)

Your example above has COW with Dremio and MOR with Athena which is why POSITION\_DELETE files are being generated.

As for why COW is producing more files, it has to do with parallelism of the DELETE query to re-write the original data file with the deleted rows excluded. It’s likely more optimal to parallelize the write when the data size is much larger.

---

<div class="post-metadata">

### Author: ![vladislav-stolyarov](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vladislav-stolyarov/32/2351_2.png) [@vladislav-stolyarov](https://community.dremio.com/u/vladislav-stolyarov)
#### Post date: [January 10, 2025, 10:41am UTC](https://community.dremio.com/t/dremio-iceberg-table-delete-splits-files/12689/3 "2025-01-10T10:41:15Z")

</div>

I tried using merge-on-read.

```auto
CREATE TABLE TableMOR
LOCALSORT BY (SortColumn)
TBLPROPERTIES ('format-version' = '2', 'write.delete.mode' = 'merge-on-read', 'write.merge.mode' = 'merge-on-read', 'write.update.mode' = 'merge-on-read')
AS 
SELECT * FROM ...

```

BUt got error: `The target iceberg table's write.delete.mode table-property is set to 'merge-on-read', but dremio does not support this write property at this time. Please alter your write.delete.mode table property to 'copy-on-write' to proceed.`  
I am on AWS 25.0.7 build.

But anyway my original question is the same - is this normal that delete creates 6 files in COW? I i will issue another delete will it create +6 files?)  
I mean when i use COW with INSERT/MERGE/UPDATE it keeps 1 or 2(if new records inserted) files. Than why parallelism was not applied in such case?)

I can not test COW Delete in Athena to compare cos it uses Trino that has no COW mode.

FYI: We are doing DELETE cos MERGE doesnt fit our need, cos we need INSERT OVERWRITE logic.

---

<div class="post-metadata">

### Author: ![vladislav-stolyarov](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vladislav-stolyarov/32/2351_2.png) [@vladislav-stolyarov](https://community.dremio.com/u/vladislav-stolyarov)
#### Post date: [January 15, 2025, 12:08pm UTC](https://community.dremio.com/t/dremio-iceberg-table-delete-splits-files/12689/4 "2025-01-15T12:08:51Z")

</div>

So which write modes does dremio support at the moment?

---

<div class="post-metadata">

### Author: ![vladislav-stolyarov](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vladislav-stolyarov/32/2351_2.png) [@vladislav-stolyarov](https://community.dremio.com/u/vladislav-stolyarov)
#### Post date: [January 20, 2025, 12:13pm UTC](https://community.dremio.com/t/dremio-iceberg-table-delete-splits-files/12689/5 "2025-01-20T12:13:15Z")

</div>

Another weird behavior…  
When i run OPTIMIZE against single partition(truncate function is used to partition by col using

```auto
OPTIMIZE TABLE Table REWRITE DATA USING BIN_PACK 
-- single partition
FOR PARTITIONS col IN (173500)
(MIN_INPUT_FILES = 1)

```

It produces 5 files…

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/2/2d52ff134490d9b2b9a98af970406ce6f2cb11f6.png)  
When i run this command without Partitions clause nothing is compacted or 1 file is produced for the same modified partition as expected.  
If change `FOR PARTITIONS col IN (173500, ...)` to include multiply partitions again it works fine(either nothing happens or single file per partition is produced).  
partition total file size is small. Here is how it splits file into 5 for above example for no reason…  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/c/caa7208d3e93d5da891918ae02a35553e8ea982f.png)  
so each file is way below default Dremio target file size and even if target size would be below it wont explain why calling OPTIMIZE over all table or multiply partitions do not cause file split in this case.
