Hi all, dremio19 support for apache iceberg V2 format table.However we found that the data has been deleted in iceberg can also been query by using (select * from tableName/select count(1) from tableName). Is that a bug or has not been support for?
@littleT Have you refreshed metadata after the data got deleted?
Thanks for reply. yes,i did refreshed metadata after the data got deleted.
Now I found the reason why the deleted data can be queried . It because the iceberg table was created by flink, and the data was process by flink cdc . When I used spark3.0 everything is OK.
Now a new question is that I can not use spark to process cdc data, so I want to konow ,Is dremio plan to support Apache Iceberg using flink?
@littleT Need to check if this is supported or Dremio plans to support
@balaji.ramaswamy Hi,sorry for bothering you again.Recently I test the problem again and again,I am sure that the data dealed with flink cdc can be queried OK on my own spark3.0 ,but the dremio can also query out the deleted data,I read the dremio source code ,but can not found how the dremio use spark3.0 to deal with iceberg,no configre file ,no spark code in dremio,so I want to know how did dremio dealed the data using spark3.0. That should be useful for me
Dremio uses iceberg Java API, check this code