Hello Dremio team,
I am using 3 EC2 instances to run Elasticsearch 6.2.4 (ES6) on them and Dremio on on these servers as well for testing. I have read that ES6 is not fully supported yet but I am curious how come Dremio is able to read one simple index in ES but not another one with more data.
I get a metadata related error:
2018-04-30 18:29:46,167 [25189f65-8263-6d33-f82d-978421cd6200:foreman] INFO c.dremio.exec.catalog.DatasetManager - User Error Occurred [ErrorId: df7bb81a-1149-4ada-bafc-df1f65bfa4aa]
com.dremio.common.exceptions.UserException: Failure while attempting to read metadata for xyz
…
caused by: java.lang.NumberFormatException: For input string: “%{NCES Latitude}”
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[na:1.8.0_172]
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122) ~[na:1.8.0_172]
at java.lang.Float.parseFloat(Float.java:451) ~[na:1.8.0_172]
My ES6 mapping for this index is:
PUT xyz/_mapping/doc
{
"properties" : {
"@timestamp": {"type": "date"},
"@version": {"type": "keyword"},
"Claim Date": {
"type": "date",
"format": "M/d/yy"
},
"NCES Location" : {"type" : "geo_point", "ignore_malformed": true},
"NCES Latitude" : {"type" : "half_float"},
"NCES Longitude" : {"type" : "half_float"}
}
}
It seems like half_float
might not be fully support by Dremio. Is there a workaround for this or I need to wait for ES6 support? Any timeline on this, as I was planning to make Dremio our organization’s main data connector?
Thank you in advance,
George Zoto