Is Elasticsearch 6.2 supported?

1

I am using the AWS Elasticsearch service. I am using Elasticsearch 6.2 without authentication. I am also using the latest version of dremio from dockerhub. When I try to visualize an index with 3 docs I get the following error in dremio:

2018-07-11 05:53:25,995 [24ba64a9-d71d-ff88-d796-9e0810861100:foreman] INFO  c.dremio.exec.catalog.DatasetManager - User Error Occurred [ErrorId: a564f0f9-acea-456c-852d-c2ea203c9692]
com.dremio.common.exceptions.UserException: Failure while attempting to read metadata for ESDev.twitter."_doc".
	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:315) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.DatasetManager.getTable(DatasetManager.java:190) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.CatalogImpl.getTable(CatalogImpl.java:134) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.DelegatingCatalog.getTable(DelegatingCatalog.java:57) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.CachingCatalog.getTable(CachingCatalog.java:66) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.DremioCatalogReader.getTable(DremioCatalogReader.java:79) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.DremioCatalogReader.getTable(DremioCatalogReader.java:65) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:71) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:189) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2859) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2844) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3077) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:208) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:866) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:577) [calcite-core-1.12.0-201805292230260062-7cc8f8e.jar:1.12.0-201805292230260062-7cc8f8e]
	at com.dremio.exec.planner.sql.SqlConverter.validate(SqlConverter.java:168) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode(PrelTransformer.java:176) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:165) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:161) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:293) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: java.lang.NullPointerException: null
	at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:239) ~[dremio-elasticsearch-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:161) ~[dremio-elasticsearch-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:142) ~[dremio-elasticsearch-plugin-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:297) [dremio-sabot-kernel-2.0.5-201806021755080191-767cfb5.jar:2.0.5-201806021755080191-767cfb5]
	... 32 common frames omitted
2018-07-11 05:53:25,996 [24ba64a9-d71d-ff88-d796-9e0810861100:foreman] INFO  query.logger - {"queryId":"24ba64a9-d71d-ff88-d796-9e0810861100","schema":"ESDev.twitter","queryText":"SELECT *\nFROM ESDev.twitter.\"_doc\"","start":1531288405182,"finish":1531288405995,"outcome":"FAILED","username":"admin","commandDescription":"<unknown>"}

Is ES 6.2 supported by dremio?

Thanks!

c99f1767-2acb-482f-8f0c-5d2a7c5dda9f.zip (2.7 KB)

2

Elastic Search 6.2 should be supported in our next major release out soon. Stay posted :slight_smile:

3

Elasticsearch 6 is now available in the latest Dremio release, more details here

4

Hi @xrayrey

We have enhanced Dremio to support elasticsearch on AWS. If this is still a requirement, kindly download our latest version 3.0.5 from https://www.dremio.com/download/, upgrade, test and let us know

Thanks
@balaji.ramaswamy