Elastic connector not supported date format

1

I have an elastic search indices with date field in epoch_millis format (which is a valid date format in elastic),
while adding this data source I am seeing exception in server.log and data source is not added

2017-11-27 16:24:04,549 [qtp849280097-107] WARN c.d.exec.store.CatalogServiceImpl - Failure while attempting to update metadata for source lab19elastic. Terminating update of this source.
com.dremio.common.exceptions.UserException: Found invalid custom date format, epoch_millis
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:648) ~[dremio-common-1.2.2-201710100151430989-d40e31c-mapr.jar:1.2.2-201710100151430989-d40e31c-mapr]
at com.dremio.plugins.elastic.DateFormats.getFormatterAndType(DateFormats.java:346) ~[dremio-elasticsearch-plugin-1.2.2-201710100151430989-d40e31c-mapr.jar:1.2.2-201710100151430989-d40e31c-mapr]
at com.dremio.plugins.elastic.DateFormats$2.apply(DateFormats.java:158) ~[dremio-elasticsearch-plugin-1.2.2-201710100151430989-d40e31c-mapr.jar:1.2.2-201710100151430989-d40e31c-mapr]
at com.dremio.plugins.elastic.DateFormats$2.apply(DateFormats.java:155) ~[dremio-elasticsearch-plugin-1.2.2-201710100151430989-d40e31c-mapr.jar:1.2.2-201710100151430989-d40e31c-mapr]
at com.google.common.collect.Iterators$8.transform(Iterators.java:799) ~[guava-18.0.jar:na]
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-18.0.jar:na]
at com.google.common.collect.Iterators.addAll(Iterators.java:362) ~[guava-18.0.jar:na]
at com.google.common.collect.Lists.newArrayList(Lists.java:160) ~[guava-18.0.jar:na]
at com.google.common.collect.Iterables.toCollection(Iterables.java:337) ~[guava-18.0.jar:na]
at com.google.common.collect.Iterables.toArray(Iterables.java:315) ~[guava-18.0.jar:na]
at com.google.common.collect.FluentIterable.toArray(FluentIterable.java:474) ~[guava-18.0.jar:na]
at com.dremio.plugins.elastic.DateFormats.getFormatterAndType(DateFormats.java:155) ~[dremio-elasticsearch-plugin-1.2.2-201710100151430989-d40e31c-mapr.jar:1.2.2-201710100151430989-d40e31c-mapr]
at com.dremio.plugins.elastic.mapping.ElasticMappingSet$ElasticField.(ElasticMappingSet.java:259) ~[dremio-elasticsearch-plugin-1.2.2-201710100151430989-d40e31c-mapr.jar:1.2.2-201710100151430989-d40e31c-mapr]

2

Hey @dpolonsky,

Would you be able to share the index/type mapping that you have as well as which Elasticsearch version you are on? I just did some quick testing with ES6 and we should support epoch_millis so hopefully we can get you up and running quickly!

Cheers,
Rohan

3

Hey, we are facing a similar issue. Upon inspection, we have found that our format field in the mapping in ES cluster was something like this

"format": "MM/dd/yyyy || epoch_millis || strict_date_optional_time"

And, line 144 or 158 of DateFormats.java in dremio elastic plugin doesn’t seem to have any trim whitespaces being done. Hence, the format is taken as " epoch_millis " with the whitespaces included, resulting in skipping cases to default, to raise the exception “Found invalid custom date format, epoch_millis”(with a whitespace) at line 346.

Is there a way to fix this? Since, there are cases where we don’t own the elastic clusters and the schema can potentially be locked from modifying by a third party. A possible fix would be to use trim before calling the getFormatterAndType method

4

@kelly If you guys are fine, we can put a patch for this fix and contribute to your open source version. This is a bug that will help lot of people. Let us know how we can put a patch.

5

@sambitdixit appreciate it!

Here are details on contributing: https://docs.dremio.com/contributing-to-dremio.html