Does dremio handles incremental Load?

1

Hi So right now I am using dremio with GCS as source, I want to know does dremio handles incremental load?

Now when I say incremental load that means new inserted data, updated data and deleted data, Is dremio able to handle these data?

Please let me know as soon as possible I have to handle incremental load till tomorrow, If it is not possible I will try to find other solution.

Regards,
Fenil Jain.

2

Your question is kind of generic here… are you talking about Iceberg DML operations or incremental reflection refreshes?

For incremental reflection refresh, we do support insert, update and deletes now.

3

We are using Gcs as source for dremio, we are uploading files with full data and with incremental load ( new inserted,updated and deleted) data, this is how the structure in google cloud storage:

In User folder we have created 2 files=>
1> first File with full load (let’s say 100 records)
2> with incremental load (let’s say 2 new users and one updated user (from last 100 recrods one user has updated it’s name) (total 3 records are in file)

so right now dremio is combining these 2 files with 103 records when we create pds or vds it is giving 103 records means it is handling incremental but not updated record, it should only take latest updated record and give the 102 records

And thanks for letting me know this will work in reflection
let me know what do you think for above scenario I want to handle updated and deleted file too. is it possible in dremio?

Thanks,
Fenil Jain

4

Hi Fenijl,
This would be a great use case to take advantage of Dremio’s Iceberg capabilities. By loading the data into an Iceberg table, you can get full insert, update, and delete support. Take a look at our DML Support for Apache Iceberg blog for some helpful material. Announcing DML Support for Apache Iceberg | Dremio

5

@caseykarst
Thanks, Let me check it.