Giving Dremio a shot to connect Elasticsearch to Tableau. I’m very new to Dremio so I’m not sure if this problem is easy to fix.
I have an alias ‘tickets’ that combines indicies ‘tickets-2017.01.01’, …, ‘tickets-2018.01.12’, i.e., every day we bring the day’s ticket data into a daily index.
There are currently a little over 1M documents in my ‘tickets’ alias, and the alias pretty much maps to 365+ different indices (and growing daily!).
I can access any individual index just fine, but the following error appears when I try to access the alias:
Something went wrong, error code 400. Please provide the correct host and credentials.
Is there any known limitation on using Dremio to connecting to an Elasticsearch alias?
Got our IT admin to pull a log of my attempt to access the tickets alias, here’s what the log says…
2018-01-12 08:46:58,208 [jersey-client-async-executor-11] WARN c.d.p.e.e.ElasticsearchRecordReader - Exception while deleting scroll
javax.ws.rs.ProcessingException: java.io.IOException: Error writing to server
at org.glassfish.jersey.client.internal.HttpUrlConnector$3.run(HttpUrlConnector.java:299) [jersey-client-2.25.1.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299) [jersey-guava-2.25.1.jar:na]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) [na:1.8.0_151]
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50) [jersey-guava-2.25.1.jar:na]
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37) [jersey-guava-2.25.1.jar:na]
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:293) [jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:180) [jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:208) [jersey-client-2.25.1.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: java.io.IOException: Error writing to server
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:699) ~[na:1.8.0_151]
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:711) ~[na:1.8.0_151]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1585) ~[na:1.8.0_151]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[na:1.8.0_151]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_151]
at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:399) [jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.client.internal.HttpUrlConnector.access$000(HttpUrlConnector.java:96) [jersey-client-2.25.1.jar:na]
at org.glassfish.jersey.client.internal.HttpUrlConnector$3.run(HttpUrlConnector.java:297) [jersey-client-2.25.1.jar:na]
... 20 common frames omitted
2018-01-12 08:46:58,261 [25a7197e-667c-760d-c560-c5a1c1428000:foreman] WARN c.d.p.elastic.ElasticConnectionPool - Error encountered: HTTP 400 Bad Request
2018-01-12 08:46:58,262 [25a7197e-667c-760d-c560-c5a1c1428000:foreman] INFO c.d.p.elastic.ElasticConnectionPool - User Error Occurred [ErrorId: bd8044cb-8a78-49a4-9552-d7cc522a94ef]
com.dremio.common.exceptions.UserException: Something went wrong, error code 400. Please provide the correct host and credentials.
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:648) ~[dremio-common-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.plugins.elastic.ElasticConnectionPool.addContextAndThrow(ElasticConnectionPool.java:427) [dremio-elasticsearch-plugin-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.plugins.elastic.ElasticConnectionPool.access$500(ElasticConnectionPool.java:82) [dremio-elasticsearch-plugin-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.plugins.elastic.ElasticConnectionPool$ElasticConnection.executeAndHandleResponseCode(ElasticConnectionPool.java:606) [dremio-elasticsearch-plugin-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:276) [dremio-elasticsearch-plugin-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:161) [dremio-elasticsearch-plugin-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:142) [dremio-elasticsearch-plugin-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.store.SimpleSchema.getTableFromSource(SimpleSchema.java:377) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.store.SimpleSchema.getTableWithRegistry(SimpleSchema.java:297) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.store.SimpleSchema.getTable(SimpleSchema.java:415) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:67) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.3.1-201712020438070881-a7af5c8]
at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:219) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.3.1-201712020438070881-a7af5c8]
at org.apache.calcite.prepare.CalciteCatalogReader.getTableFrom(CalciteCatalogReader.java:117) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:106) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.prepare.CalciteCatalogReader.getTable(CalciteCatalogReader.java:73) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.EmptyScope.getTableNamespace(EmptyScope.java:71) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.DelegatingScope.getTableNamespace(DelegatingScope.java:189) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:104) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2859) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2844) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3077) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:84) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:910) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:891) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:208) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:866) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:577) [calcite-core-1.12.0-201711022309440460-bd4149e.jar:1.12.0-201711022309440460-bd4149e]
at com.dremio.exec.planner.sql.SqlConverter.validate(SqlConverter.java:188) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateNode(PrelTransformer.java:167) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.planner.sql.handlers.PrelTransformer.validateAndConvert(PrelTransformer.java:155) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:43) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:66) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:290) [dremio-sabot-kernel-1.3.1-201712020438070881-a7af5c8.jar:1.3.1-201712020438070881-a7af5c8]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-01-12 08:46:58,262 [25a7197e-667c-760d-c560-c5a1c1428000:foreman] INFO query.logger - {"queryId":"25a7197e-667c-760d-c560-c5a1c1428000","schema":"","queryText":"SELECT *\nFROM elasticsearch.\"tickets\".logs","start":1515775617238,"finish":1515775618262,"outcome":"FAILED","username":"admin","commandDescription":"<unknown>"}
Thank you for providing the log. Would it be possible to also look at the query profile, and see if there is any more detail regarding the error message? Follow these instructions to look at the profile:
In this case, it should be enough just to copy and paste the entire error message from the error tab.
Here’s the error message from the error tab (verbose):
CONNECTION ERROR: Something went wrong, error code 400. Please provide the correct host and credentials.
responseCode 400
action com.dremio.plugins.elastic.ElasticActions$Count@461910a4
base url http://10.40.202.67:9200
error message HTTP 400 Bad Request
Sql Query SELECT *
FROM elasticsearch."tickets".logs
I have the full zip file of the query as well if needed.
Any luck on replicating the issue? I have a feeling it’s got to do with the number of indices being pulled in. Haven’t had time to investigate, will hopefully do so today
This does indeed look like an issue where Dremio cannot access too many indices at once. I reindexed my data into to monthly instead of daily views and it works perfectly.
I’ve ran into this exact same issue, although re-indexing to a monthly index did not work for me. Now Dremio won’t load the index at all. The daily index has about 2 million records, rolling it into a monthly index creates upwards of 50 million records.