# Failed to create Incremental Reflection on ElasticSearch

**URL:** <https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281>\
**Category:** Uncategorized\
**Created:** [September 14, 2017, 12:42am UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281 "2017-09-14T00:42:05Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![allan.sene](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/allan.sene/32/144_2.png) [@allan.sene](https://community.dremio.com/u/allan.sene)\
**Post date:** [September 14, 2017, 12:42am UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/1 "2017-09-14T00:42:05Z")

</div>

Hi, guys! I’m trying to create an Incremental Reflection from an index using the `timestamp` field but I’m getting this error:

`Dremio only supports incremental column update on BIGINT types. The identified column was of type TIMESTAMP.`

My mapping on ES of this field in particular is:

```
@timestamp": {
   "type": "date"
},

```

and the data is in this format: `2017-09-01 07:24:49.767`.

Do you have some workaround? This index is append-only and grows really fast.

Dremio does recognize this field as `Date` and suggests to me using it:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/cef85b1ceca9b36f6c327d7d0ec7d3e534d04b81.jpg)Captura de Tela 2017-09-13 às 21.39.39

---

<div class="post-metadata">

**Author:** ![allan.sene](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/allan.sene/32/144_2.png) [@allan.sene](https://community.dremio.com/u/allan.sene)\
**Post date:** [September 21, 2017, 12:14am UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/2 "2017-09-21T00:14:31Z")

</div>

Any idea guys? 😔

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [September 21, 2017, 5:16am UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/3 "2017-09-21T05:16:44Z")

</div>

Hi Allan,

Would you please be able to send us the profile from the failed job?

Thanks,

---

<div class="post-metadata">

**Author:** ![allan.sene](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/allan.sene/32/144_2.png) [@allan.sene](https://community.dremio.com/u/allan.sene)\
**Post date:** [September 21, 2017, 10:27pm UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/4 "2017-09-21T22:27:53Z")

</div>

It doesn’t even have a plan 🤔, look:

![21](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/e4c166c35eb35ae0758de321969c2436d7657169.jpg)

 ![32](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/dc97aa60d4e64ebf2a1a45340143bd4511e9f08d.jpg)

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [September 21, 2017, 10:44pm UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/5 "2017-09-21T22:44:41Z")

</div>

Hi Allan,

Can you please click on jobs, download the profile for this failed job and send us the zip file downloaded? I will get back to you once I have a look.

Thanks

---

<div class="post-metadata">

**Author:** ![allan.sene](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/allan.sene/32/144_2.png) [@allan.sene](https://community.dremio.com/u/allan.sene)\
**Post date:** [September 22, 2017, 6:59pm UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/6 "2017-09-22T18:59:58Z")

</div>

Here goes, @balaji.ramaswamy.

[e6ea846a-8510-462d-bfd4-9f8e9979dad7.zip](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/1X/0b83ac5c724e95635f40b5160dbcbe7d83a8c546.zip) (2.3 KB)

---

<div class="post-metadata">

**Author:** ![muv](https://avatars.discourse-cdn.com/v4/letter/m/e9a140/32.png) [@muv](https://community.dremio.com/u/muv)\
**Post date:** [June 14, 2018, 6:34pm UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/7 "2018-06-14T18:34:24Z")

</div>

@allan.sene, did you manage to solve this? I’ve hit the same problem with MongoDB dataset. Reported it here: [Dremio only supports incremental column update on BIGINT types - why?](https://community.dremio.com/t/dremio-only-supports-incremental-column-update-on-bigint-types-why/1306). But thought you may know a solution given that someone from Dremio was trying to help you.

---

<div class="post-metadata">

**Author:** ![allan.sene](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/allan.sene/32/144_2.png) [@allan.sene](https://community.dremio.com/u/allan.sene)\
**Post date:** [June 14, 2018, 9:21pm UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/8 "2018-06-14T21:21:57Z")

</div>

No, I didn’t 😔

At the time, we decided to take a break of Dremio’s project and try other alternatives.

---

<div class="post-metadata">

**Author:** ![christy](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@christy](https://community.dremio.com/u/christy)\
**Post date:** [June 18, 2018, 9:54am UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/9 "2018-06-18T09:54:08Z")

</div>

Hey guys,

Thanks for raising this requirement. We are tracking this internally, and will provide updates as they arise.

Christy

---

<div class="post-metadata">

**Author:** ![csteckel](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/csteckel/32/753_2.png) [@csteckel](https://community.dremio.com/u/csteckel)\
**Post date:** [July 11, 2018, 8:45am UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/10 "2018-07-11T08:45:41Z")

</div>

Hi all,

the issue seems to be backend independant. I have the same issue with a Postgres DB as backend where the incremental column (in my case a column named “modifed\_at”) is defined as a TIMESTAMP.

Are there any news on the issue?

I will test a workaround by converting the timestamp to a BIGINT and see if the incremental update will work then.

Cheers  
Carsten

---

<div class="post-metadata">

**Author:** ![firefly707](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/firefly707/32/646_2.png) [@firefly707](https://community.dremio.com/u/firefly707)\
**Post date:** [July 13, 2018, 7:38pm UTC](https://community.dremio.com/t/failed-to-create-incremental-reflection-on-elasticsearch/281/11 "2018-07-13T19:38:40Z")

</div>

Hello, did converting the timestamp to BIGINT work for you?
