Unable to connect to local Elasticsearch

1

Hello Dremio team,

I am using 3 EC2 instances to run Elasticsearch 6.2.4 (ES6) on them and Dremio on on these servers as well for testing. I have read that ES6 is not fully supported yet but I am curious how come Dremio is able to read one simple index in ES but not another one with more data.

I get a metadata related error:
2018-04-30 18:29:46,167 [25189f65-8263-6d33-f82d-978421cd6200:foreman] INFO c.dremio.exec.catalog.DatasetManager - User Error Occurred [ErrorId: df7bb81a-1149-4ada-bafc-df1f65bfa4aa]
com.dremio.common.exceptions.UserException: Failure while attempting to read metadata for xyz
…
caused by: java.lang.NumberFormatException: For input string: “%{NCES Latitude}”
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[na:1.8.0_172]
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122) ~[na:1.8.0_172]
at java.lang.Float.parseFloat(Float.java:451) ~[na:1.8.0_172]

My ES6 mapping for this index is:
PUT xyz/_mapping/doc
{
"properties" : {
"@timestamp": {"type": "date"},
"@version": {"type": "keyword"},
"Claim Date": {
"type": "date",
"format": "M/d/yy"
},
"NCES Location" : {"type" : "geo_point", "ignore_malformed": true},
"NCES Latitude" : {"type" : "half_float"},
"NCES Longitude" : {"type" : "half_float"}
}
}

It seems like half_float might not be fully support by Dremio. Is there a workaround for this or I need to wait for ES6 support? Any timeline on this, as I was planning to make Dremio our organization’s main data connector?

Thank you in advance,
George Zoto

2

Hi George, unfortunately we don’t support ES6 at this time, but hope to have support within the next few months. Currently some things work with ES6, but many other things do not.

Sorry you have run into this issue. Thanks for your patience.

3

Thank you Kelly, I look forward to ES6 support, more than happy to help out with any testing you might need.
As for this specific issue, do you know if half_float is supported by Dremio?

Thank you again and great work on this platform,
George

4

We don’t currently support half_float (I think this is new with ES6), but plan to when we support ES6.

5

Hi, any update for ES6?

6

Hey @Naveesh_Doolhur we’re actively working on this. We’ll reach out once available.

7

Good news! support for Elasticsearch 6.0 is now available in our latest release, Dremio 2.1. More details here.

1 Like