Upcoming support for query Apache Iceberg deleted data?

littleT · November 8, 2021, 11:52am

Hi all, dremio19 support for apache iceberg V2 format table.However we found that the data has been deleted in iceberg can also been query by using (select * from tableName/select count(1) from tableName). Is that a bug or has not been support for?

balaji.ramaswamy · November 10, 2021, 6:56am

@littleT Have you refreshed metadata after the data got deleted?

littleT · November 11, 2021, 5:38am

Thanks for reply. yes,i did refreshed metadata after the data got deleted.
Now I found the reason why the deleted data can be queried . It because the iceberg table was created by flink, and the data was process by flink cdc . When I used spark3.0 everything is OK.
Now a new question is that I can not use spark to process cdc data, so I want to konow ,Is dremio plan to support Apache Iceberg using flink?

balaji.ramaswamy · November 11, 2021, 7:07am

@littleT Need to check if this is supported or Dremio plans to support

littleT · November 15, 2021, 8:52am

@balaji.ramaswamy Hi,sorry for bothering you again.Recently I test the problem again and again,I am sure that the data dealed with flink cdc can be queried OK on my own spark3.0 ,but the dremio can also query out the deleted data,I read the dremio source code ,but can not found how the dremio use spark3.0 to deal with iceberg,no configre file ,no spark code in dremio,so I want to know how did dremio dealed the data using spark3.0. That should be useful for me

chulucninh09 · November 25, 2021, 1:57pm

Dremio uses iceberg Java API, check this code

github.com

dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/store/iceberg/model/IcebergCatalogType.java

/*
 * Copyright (C) 2017-2019 Dremio Corporation
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.dremio.exec.store.iceberg.model;

/**
 * supported Iceberg catalog types
 */

This file has been truncated. show original

Topic		Replies	Views
Dremio not supports reading Apache Iceberg v2 tables with positional deletes	9	761	December 7, 2023
Iceberg support GA	13	1760	November 7, 2021
Unable to Query Iceberg Data in Dremio - Data Written by Kafka Connector Apache Iceberg	1	21	November 19, 2024
When can it be supported iceberg v2 tables with equality delete file Dremio University	3	800	March 31, 2023
Can't read Iceberg V2 tables with equality deletes on Dremio Software v25.0 Dremio University	3	227	May 1, 2024

Upcoming support for query Apache Iceberg deleted data?

Related topics