Error when accessing Elasticsearch data source

1

Hi

i am getting the following error when querying Elasticsearch data source . Weird thing is it works when the number of records < 5000. When the number of records > 5000 then i would get the following error

Anyone experiencing the same issue ?

2018-07-30 05:53:16,676 [24a15832-eecf-c663-0275-3373418c9e00:foreman] INFO c.d.p.elastic.ElasticConnectionPool - User Error Occurred [ErrorId: bbe1824d-1ee1-4272-ae43-15ebd22834f7]
com.dremio.common.exceptions.UserException: Failure executing Elastic request get next search result.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:622) ~[dremio-common-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticConnectionPool.handleException(ElasticConnectionPool.java:468) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticConnectionPool.access$400(ElasticConnectionPool.java:82) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticConnectionPool$ElasticConnection.execute(ElasticConnectionPool.java:533) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchRecordReader.getNextPage(ElasticsearchRecordReader.java:266) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.execution.ElasticsearchRecordReader.next(ElasticsearchRecordReader.java:313) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.getSampledSchema(ElasticTableBuilder.java:317) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:183) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:154) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:135) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.store.SimpleSchema.getTableFromDataset(SimpleSchema.java:302) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.store.SimpleSchema.getTableWithRegistry(SimpleSchema.java:265) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.store.SimpleSchema.getTable(SimpleSchema.java:377) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:67) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.1.0-201708121825170680-436784e]
at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:219) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.1.0-201708121825170680-436784e]
at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:117) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:106) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:73) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:71) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:189) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2859) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2844) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3077) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:208) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:866) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:577) [calcite-core-1.12.0-201707180020390814-16f46c2.jar:1.12.0-201707180020390814-16f46c2]
at com.dremio.exec.planner.sql.SqlConverter.validate(SqlConverter.java:187) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode(PrelTransformer.java:167) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:155) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:285) [dremio-sabot-kernel-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
Caused by: javax.ws.rs.NotAcceptableException: HTTP 406 Not Acceptable
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1026) ~[jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:819) ~[jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92) ~[jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:701) ~[jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:228) ~[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444) ~[jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:697) ~[jersey-client-2.25.1.jar:na]
at com.dremio.plugins.elastic.ElasticConnectionPool$ElasticConnection.execute(ElasticConnectionPool.java:531) [dremio-elasticsearch-plugin-1.1.0-201708121825170680-436784e.jar:1.1.0-201708121825170680-436784e]
… 39 common frames omitted
2018-07-30 05:53:16,678 [24a15832-eecf-c663-0275-3373418c9e00:foreman] INFO query.logger - {“queryId”:“24a15832-eecf-c663-0275-3373418c9e00”,“schema”:"",“queryText”:“SELECT *\nFROM elasticsearch3.“test-payments6”.payment”,“start”:1532929996561,“finish”:1532929996677,“outcome”:“FAILED”,“username”:“admin”,“commandDescription”:""}

2

Can you confirm what version of ES you are using? Also does this happen for both preview & run mode?

3

im using ES 6.3.1 and dremio 1.1

yup it happens to both preview and run mode

thanks

4

Thanks. ES6 is currently not supported but will be real soon, stay tuned!

5

I know that the release to support ES6 was due by the end of July. Has that been made available?

6

The 2.1 release is expected to be out within the next 2 weeks.

7

Great. Thanks for the update! Looking forward to it as it will allow me to keep using this great tool.

8

Any update on the support for ES 6+?

9

Yes, ES6 will be supported with the 2.1 release. Coming within the next week or two.

10

Hello,
support for Elasticsearch 6.0 is now available in our latest release, Dremio 2.1. More details here.