Failed to create Incremental Reflection on ElasticSearch

1

Hi, guys! I’m trying to create an Incremental Reflection from an index using the timestamp field but I’m getting this error:

Dremio only supports incremental column update on BIGINT types. The identified column was of type TIMESTAMP.

My mapping on ES of this field in particular is:

@timestamp": {
   "type": "date"
},

and the data is in this format: 2017-09-01 07:24:49.767.

Do you have some workaround? This index is append-only and grows really fast.

Dremio does recognize this field as Date and suggests to me using it:

image
image.jpg3360×2100 670 KB
Captura de Tela 2017-09-13 às 21.39.39

2

Any idea guys? :pensive:

3

Hi Allan,

Would you please be able to send us the profile from the failed job?

Thanks,

4

It doesn’t even have a plan :thinking:, look:

21

32
Captura de Tela 2017-09-21 às 19.24.32.jpg3360×2100 553 KB

5

Hi Allan,

Can you please click on jobs, download the profile for this failed job and send us the zip file downloaded? I will get back to you once I have a look.

Thanks

6

Here goes, @balaji.ramaswamy.

e6ea846a-8510-462d-bfd4-9f8e9979dad7.zip (2.3 KB)

7

@allan.sene, did you manage to solve this? I’ve hit the same problem with MongoDB dataset. Reported it here: Dremio only supports incremental column update on BIGINT types - why?. But thought you may know a solution given that someone from Dremio was trying to help you.

8

No, I didn’t :pensive:

At the time, we decided to take a break of Dremio’s project and try other alternatives.

1 Like
9

Hey guys,

Thanks for raising this requirement. We are tracking this internally, and will provide updates as they arise.

Christy

10

Hi all,

the issue seems to be backend independant. I have the same issue with a Postgres DB as backend where the incremental column (in my case a column named “modifed_at”) is defined as a TIMESTAMP.

Are there any news on the issue?

I will test a workaround by converting the timestamp to a BIGINT and see if the incremental update will work then.

Cheers
Carsten

11

Hello, did converting the timestamp to BIGINT work for you?