Upcoming support for query Apache Iceberg deleted data?

littleT · November 8, 2021, 11:52am

Hi all, dremio19 support for apache iceberg V2 format table.However we found that the data has been deleted in iceberg can also been query by using (select * from tableName/select count(1) from tableName). Is that a bug or has not been support for?

balaji.ramaswamy · November 10, 2021, 6:56am

@littleT Have you refreshed metadata after the data got deleted?

littleT · November 11, 2021, 5:38am

Thanks for reply. yes,i did refreshed metadata after the data got deleted.
Now I found the reason why the deleted data can be queried . It because the iceberg table was created by flink, and the data was process by flink cdc . When I used spark3.0 everything is OK.
Now a new question is that I can not use spark to process cdc data, so I want to konow ,Is dremio plan to support Apache Iceberg using flink?

balaji.ramaswamy · November 11, 2021, 7:07am

@littleT Need to check if this is supported or Dremio plans to support

littleT · November 15, 2021, 8:52am

@balaji.ramaswamy Hi,sorry for bothering you again.Recently I test the problem again and again,I am sure that the data dealed with flink cdc can be queried OK on my own spark3.0 ,but the dremio can also query out the deleted data,I read the dremio source code ,but can not found how the dremio use spark3.0 to deal with iceberg,no configre file ,no spark code in dremio,so I want to know how did dremio dealed the data using spark3.0. That should be useful for me

chulucninh09 · November 25, 2021, 1:57pm

Dremio uses iceberg Java API, check this code

github.com

dremio/dremio-oss/blob/master/sabot/kernel/src/main/java/com/dremio/exec/store/iceberg/model/IcebergCatalogType.java

/*
 * Copyright (C) 2017-2019 Dremio Corporation
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.dremio.exec.store.iceberg.model;

/**
 * supported Iceberg catalog types
 */

This file has been truncated. show original

Topic		Replies	Views
Iceberg support GA	13	1666	November 7, 2021
When can it be supported iceberg v2 tables with equality delete file Dremio University	3	699	March 31, 2023
Iceberg Tables: DML Operations NOT supported for HDFS/Hive	6	1409	February 4, 2022
Dbt tables stored in Apache Iceberg	8	1837	August 31, 2023
Connecting to apache iceberg	2	881	September 20, 2022

Upcoming support for query Apache Iceberg deleted data?

Related Topics