# Optimizing Incremental Reflections in Dremio: Handling Newly Inserted Data with Timestamp Columns
**URL:** https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625
**Category:** Uncategorized
**Created:** [December 19, 2024, 6:25pm UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625 "2024-12-19T18:25:05Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)
#### Post date: [December 19, 2024, 6:25pm UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625/1 "2024-12-19T18:25:05Z")
Hi team,
In Dremio, when I configure the `ID` column as the incremental reflection, it effectively scans only the newly inserted data based on the query-level `WHERE` condition for the `ID`. However, when using a timestamp column for the incremental reflection, a full table scan occurs even after new records are inserted and the reflection is manually refreshed.
How can Dremio be optimized to scan only the newly inserted data during a manual reflection refresh? Additionally, how does Dremio internally manage incremental updates when using a timestamp column?
Thanks in Advance
---
### Author:  [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)
#### Post date: [December 21, 2024, 4:07am UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625/2 "2024-12-21T04:07:46Z")
Hi team,
Can you give please give an update on this?
Thanks in Advance
---
### Author:  [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [December 23, 2024, 5:20am UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625/3 "2024-12-23T05:20:41Z")
@JoiceJacob Can you please send us the profile of the reflection that scanned all rows? plus the first full refresh reflection profile
---
### Author:  [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)
#### Post date: [December 23, 2024, 10:29am UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625/4 "2024-12-23T10:29:54Z")
Hi @balaji.ramaswamy
Please find the Query profile of first full refresh reflection and reflection that scanned all rows
First Full refresh reflection :
[5174de1b-3504-4f4c-8b3e-e403393cbe56.zip](https://community.dremio.com/uploads/short-url/nfXeKkLbXuTWZpDvYAzPZxefkD1.zip) (13.7 KB)
Reflection scanned full row :
[4d99ab39-9660-42be-a1eb-902170df5528.zip](https://community.dremio.com/uploads/short-url/zoSs8auays18MuvdObzYIfeVM5.zip) (13.8 KB)
---
### Author:  [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)
#### Post date: [December 24, 2024, 7:56am UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625/5 "2024-12-24T07:56:11Z")
Hi team,
Can you give please give an update on this?
Thanks in Advance
---
### Author:  [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)
#### Post date: [December 26, 2024, 5:03am UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625/6 "2024-12-26T05:03:46Z")
Hi team,
Could you please provide an update on this?
Thanks in Advance
---
### Author:  [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)
#### Post date: [January 8, 2025, 5:50am UTC](https://community.dremio.com/t/optimizing-incremental-reflections-in-dremio-handling-newly-inserted-data-with-timestamp-columns/12625/7 "2025-01-08T05:50:23Z")
@JoiceJacob If you see the second refresh that you had expected to be incremental, the message (in planning tab of the raw profile) shows “Full Refresh.
Change in refresh method, doing an initial refresh.”
Was there a change done from full to incremental after the first full refresh? Can you try below steps?
- Create a reflection and choose type incremental
- Click on refresh now
- Again click on refresh now
Check all three profiles-planning tab to see if in the third you see “Incremental Refresh”
Also, I see the source is RDBMS, the incremental refresh column should be
- **For all other datasets (such as datasets in relational or NoSQL databases):** An administrator specifies a strictly monotonically increasing field, such as an auto-incrementing key, that must be of type BigInt, Int, Timestamp, Date, Varchar, Float, Double, or Decimal. This allows Dremio to find and fetch the records that have been created since the last time the acceleration was incrementally refreshed.
> **[Refreshing Reflections | Dremio Documentation](https://docs.dremio.com/cloud/sonar/reflections/refreshing-reflections/#)**
>
> The data in a reflection can become stale and need to be refreshed. The refresh of a reflection causes two updates: