ElasticSearch support

[ravenr]

ElasticSearch driver doesn’t support many features.
I believe those features are presented as unexplained errors.
It’s not really documented, and there are no advanced options for this, but that is what I found out so far -

1. No support for ip type - causing a NumberFormatError exception
2. No support for nested, array types - causing all king of unexplained and undetailed errors, about the structure of the index.

Most trials so far to deal with the conversion of elasticsearch structure use flattening for the json hierarchy which is also not supported in Dremio’s driver.
(Exposing format of field1.nested_field2.nested_field3)
Also, you’ll find out soon enough, that array flattening is needed.

I can show references to such implementations if needed.
Without that most of the indexes out there will not be processed right within Dremio.

Also control of some parameters like scrolling timewait, etc, should be there is order to be able to tune performance when needed.

[rohan]

Hi @ravenr,

I’d be glad to help you resolve your issues with Dremio and Elasticsearch.

Dremio should support the IP data type that Elastic provides. We expose Elastic IPs as strings in the dashboard as shown below:

Where the mapping of “my_index” is:

Let me know if you have a different mapping or if you use an older version of Elasticsearch (I tested this on Elasticsearch 5.5.2).

We also support nested objects and arrays, both through the UI as well as the SQL interface.
Working with an example array field “categories”, you can flatten this field through the UI steps below, or through the flatten command which is highlighted in the last picture.

You can also employ a similar process on nested objects and arrays:

Once again, let me know if you use different mappings, object structures, or Elasticsearch versions and I’d be happy to help you get up and running with Dremio!

Thanks,
Rohan