Dremio compatibility with elasticsearch 7.0.1

1

Dear dremio team,

I recently upgraded elasticsearch to version 7.0.1 and dremio stopped working (Getting “Unexpected error ocurred”, “Invalid response”, etc. when trying to execute queries).

Apparently dremio is not yet compatible with ES 7.0.1. Are there any plans to make it compatible soon? Can’t wait! :wink:

Thank you and best regards,
Marcos

1 Like
2

Hi @marcos.llama,

We do not currently test against 7.x releases of Elasticsearch, only 6 and later versions of 5, but it’s not clear why there would be a breaking change with these newer versions. Do you have query profiles for any of the jobs that fail when querying against 7.0.1?

3

Hello @ben

There is no specific query profile, it fails simply after adding the elasticsearch repository as a data source, then navigating to any of the indexes and clicking the “doc” item inside. Instead of getting the preview, I get these messages “Unexpected error ocurred”, “Invalid response”, etc.

Thank you

Marcos

4

Hello again @ben

I found this in the query log:

{“queryId”:“2312a04e-c47b-b850-6998-5e1d3dc63b00”,“schema”:"[XXX, YYY-iptables-2019.04]",“queryText”:"select * f
rom ",“start”:1559060400428,“finish”:1559060400454,“outcome”:“FAILED”,“username”:“admin”}

Same outcome for all indexes/queries

Thanks again

Marcos

5

Also happens when typing the query directly: select * from doc

image

All information entered previously was generated on a fresh dremio instance

If a try using a previous dremio installation on another computer (which was working fine prior to ES upgrade), I get a different error message:

“Failure while attempting to read metadata for XXX.“YYY-2019.05”.doc.”

6

Update: I upgraded from ES 7.0.1 to ES 7.1, same outcome from Dremio…

Marcos

7

Any movement on the compatibility with elasticsearch 7. I did an upgrade also and this completely broke connectivity with elasticsearch.

8

I encountered the same issue with DremIO 3.3.2-201908141640190085-d60145d and Elasticsearch 7.4; see job profile attached.

a93b49ae-214d-4549-bbd3-fb085d704faa.zip (4.3 KB)

Since there does not appear to be any updates to this issue, I looked through the stack trace and DremIO source code in Git, it appears that com.dremio.plugins.elastic.execution.ElasticsearchJsonReader.getScrollAndTotalSizeThenSeekToHits() on line 96 is expecting Elasticsearch’s response JSON to look like:

{
…
“hits”: {
“total”: 123,
“max_score”: 123,
…
}

whereas the response JSON from V7.4 looks something like:

{
…
“hits” : {
“total” : {
“value” : 8,
“relation” : “eq”
},
“max_score” : 1.0,
…
}

com.dremio.plugins.elastic.execution.ElasticsearchJsonReader line 95 and 96 are as follows:
final JsonToken totalSizeToken = seekForward(ElasticsearchConstants.TOTAL_HITS);
Preconditions.checkState(totalSizeToken == JsonToken.VALUE_NUMBER_INT, “Invalid response”);

As such, it failed the check because the “total” attribute is no longer an int. I found the ES 7 release document referencing changes to hits.total:

https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#hits-total-now-object-search-response

Regards,

Eric

2 Likes
9

Awesome analysis …and technical details on this compatibility issue.

10

A possible (simple) fix would make the plugin incompatible with the elastic versions prior to 7

11

Is there any way to fix it? I saw from https://github.com/atajti/dremio-oss that elasticsearch7 plugin change com.dremio.plugins.elastic.ElasticsearchConstants line 47 from String TOTAL_HITS = "total"; to String TOTAL_HITS = "total.value";. But then this only works for elasticsearch 7.