# Does dremio handles incremental Load?

**URL:** <https://community.dremio.com/t/does-dremio-handles-incremental-load/11342>\
**Category:** Uncategorized\
**Created:** [January 4, 2024, 5:31am UTC](https://community.dremio.com/t/does-dremio-handles-incremental-load/11342 "2024-01-04T05:31:14Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fenilj](https://avatars.discourse-cdn.com/v4/letter/f/e9bcb4/32.png) [@fenilj](https://community.dremio.com/u/fenilj)\
**Post date:** [January 4, 2024, 5:31am UTC](https://community.dremio.com/t/does-dremio-handles-incremental-load/11342/1 "2024-01-04T05:31:14Z")

</div>

Hi So right now I am using dremio with GCS as source, I want to know does dremio handles incremental load?

Now when I say incremental load that means new inserted data, updated data and deleted data, Is dremio able to handle these data?

Please let me know as soon as possible I have to handle incremental load till tomorrow, If it is not possible I will try to find other solution.

Regards,  
Fenil Jain.

---

<div class="post-metadata">

**Author:** ![Benny\_Chow](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/benny_chow/32/5527_2.png) [@Benny\_Chow](https://community.dremio.com/u/Benny_Chow)\
**Post date:** [January 8, 2024, 7:50pm UTC](https://community.dremio.com/t/does-dremio-handles-incremental-load/11342/2 "2024-01-08T19:50:45Z")

</div>

Your question is kind of generic here… are you talking about Iceberg DML operations or incremental reflection refreshes?

> **[Intelligent Reflection Refresh | Dremio](https://www.dremio.com/blog/intelligent-reflection-refresh/)**
>
> Dremio Reflection Refresh automatically selects the fastest and mostcost-effective refresh technique depending on table actions.

For incremental reflection refresh, we do support insert, update and deletes now.

---

<div class="post-metadata">

**Author:** ![fenilj](https://avatars.discourse-cdn.com/v4/letter/f/e9bcb4/32.png) [@fenilj](https://community.dremio.com/u/fenilj)\
**Post date:** [January 11, 2024, 1:27pm UTC](https://community.dremio.com/t/does-dremio-handles-incremental-load/11342/3 "2024-01-11T13:27:35Z")

</div>

We are using Gcs as source for dremio, we are uploading files with full data and with incremental load ( new inserted,updated and deleted) data, this is how the structure in google cloud storage:

In User folder we have created 2 files=\>  
1\> first File with full load (let’s say 100 records)  
2\> with incremental load (let’s say 2 new users and one updated user (from last 100 recrods one user has updated it’s name) (total 3 records are in file)

so right now dremio is combining these 2 files with 103 records when we create pds or vds it is giving 103 records means it is handling incremental but not updated record, it should only take latest updated record and give the 102 records

And thanks for letting me know this will work in reflection  
let me know what do you think for above scenario I want to handle updated and deleted file too. is it possible in dremio?

Thanks,  
Fenil Jain

---

<div class="post-metadata">

**Author:** ![caseykarst](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/caseykarst/32/5758_2.png) [@caseykarst](https://community.dremio.com/u/caseykarst)\
**Post date:** [January 11, 2024, 6:02pm UTC](https://community.dremio.com/t/does-dremio-handles-incremental-load/11342/4 "2024-01-11T18:02:33Z")

</div>

Hi Fenijl,  
This would be a great use case to take advantage of Dremio’s Iceberg capabilities. By loading the data into an Iceberg table, you can get full insert, update, and delete support. Take a look at our DML Support for Apache Iceberg blog for some helpful material. [Announcing DML Support for Apache Iceberg | Dremio](https://www.dremio.com/blog/announcing-dml-support-for-apache-iceberg/)

---

<div class="post-metadata">

**Author:** ![fenilj](https://avatars.discourse-cdn.com/v4/letter/f/e9bcb4/32.png) [@fenilj](https://community.dremio.com/u/fenilj)\
**Post date:** [January 12, 2024, 5:26am UTC](https://community.dremio.com/t/does-dremio-handles-incremental-load/11342/5 "2024-01-12T05:26:30Z")

</div>

@caseykarst  
Thanks, Let me check it.