Dremio not supports reading Apache Iceberg v2 tables with positional deletes

1

I’m using latest dremio community to test query Iceberg table from spark job.I found the feature mentioned in doc not work:

image
image1024×181 17.4 KB

1.My Iceberg format version is 2
image
2.I use merge-on-read mode for delete
image
3.I can see some positional delete related file in the table info-> ‘total-delete-files’: ‘1’, ‘total-position-deletes’: ‘1’, ‘total-equality-deletes’: ‘0’
4.Spark can query out with correct result.
5.Dremio query out all data without merge the delete file

2

@fiona Which source is this?

3

What do you mean about source? My data is on S3.

4

The issue could be that merge-on-read is not supported.

5

@fiona

If it’s filesystem source (which is true in your case as S3) changes won’t be visible until either scheduled or manual refresh.

Have you tried to do an “ALTER PDS REFRESH METADATA” and then see if it works?

6

Hi Balaji, thanks for your time. I reformat the table which should have the same effect like refresh metadata, right? But still not work.

7

Hi @fiona Are you getting the same error again?

8

Hi Balaji, no system error, just not show correct data like what I describe above.