ES-Dremio: error in retrieving tables

1

Hello :slight_smile:

I have a problem with Dremio + Elasticsearch, which I’m hoping you can help to solve.

I am running Dremio locally, and I have connected to my ES cluster on AWS. I can see the indexes and the doc types, but when I try to access any of the tables I get an error:

Failure while attempting to read metadata for “EU BI”.bi_loginattempts.loginAttempts.

Logs:

 2018-09-18 12:08:47,305 [qtp627722054-216] INFO  c.dremio.exec.catalog.DatasetManager - User Error Occurred [ErrorId: f6d93779-6734-4a04-9194-d2b7a4fa75c1]
    com.dremio.common.exceptions.UserException: Failure while attempting to read metadata for "EU BI".bi_loginattempts.loginAttempts.
    	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:316) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.exec.catalog.DatasetManager.getTable(DatasetManager.java:191) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.exec.catalog.CatalogImpl.getTable(CatalogImpl.java:135) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.exec.catalog.DelegatingCatalog.getTable(DelegatingCatalog.java:58) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.exec.catalog.CachingCatalog.getTable(CachingCatalog.java:66) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at jdk.internal.reflect.GeneratedMethodAccessor356.invoke(Unknown Source) ~[na:na]
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    	at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
    	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287) [hk2-utils-2.5.0-b32.jar:na]
    	at org.jvnet.hk2.internal.MethodInterceptorImpl.internalInvoke(MethodInterceptorImpl.java:109) [hk2-locator-2.5.0-b32.jar:na]
    	at org.jvnet.hk2.internal.MethodInterceptorImpl.invoke(MethodInterceptorImpl.java:125) [hk2-locator-2.5.0-b32.jar:na]
    	at org.jvnet.hk2.internal.MethodInterceptorInvocationHandler.invoke(MethodInterceptorInvocationHandler.java:62) [hk2-locator-2.5.0-b32.jar:na]
    	at com.sun.proxy.$Proxy112.getTable(Unknown Source) [na:na]
    	at com.dremio.dac.explore.DatasetsResource.getDatasetSummary(DatasetsResource.java:264) [dremio-dac-backend-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:137) [dremio-dac-backend-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.dac.explore.DatasetsResource.newUntitledFromParent(DatasetsResource.java:204) [dremio-dac-backend-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at jdk.internal.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na]
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    	at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
    	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
    	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
    	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
    	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
    	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
    	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
    	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
    	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
    	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
    	at java.base/java.lang.Thread.run(Thread.java:844) [na:na]
    Caused by: java.lang.NullPointerException: null
    	at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:239) ~[dremio-elasticsearch-plugin-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:161) ~[dremio-elasticsearch-plugin-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:142) ~[dremio-elasticsearch-plugin-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:298) [dremio-sabot-kernel-2.1.4-201808302048550610-0981242.jar:2.1.4-201808302048550610-0981242]
    	... 60 common frames omitted

Also if I check the source configurations (the “Edit source”), I get an error at the top:

Detected Elastic version null. Full query pushdown in Dremio requires version 2.1.2 or above.

How can I solve this issue?

Thank you!

2

The metadata of the node does not contain the host information, which causes Dremio to complain. Why is the host information needed? Can we get around this?

3

What version of ES are you running on AWS? And which version of Dremio?

4

Dremio version: 2.1.4
ES version on AWS: 6.+

5

Hi @rme

Although Dremio 2.1 has full support for ES6, we are working on supporting ES6 on AWS. We should have the support by the next few releases (Kindly watch for the release notes)

Thanks
@balaji.ramaswamy

1 Like
6

Hi @balaji.ramaswamy,

Any news on the release? I too am running into this same problem. I see that 3.0.0 was released. I am new to dremio and have it running in a docker container locally, but it seems to connect fine to the cluster, then gives this cryptic error whenever I try to see data.

Thanks,
Mark Haley

7

Hi @Mark_Haley

Not yet, we are looking to add elasticsearch 6 on AWS support soon (~6-8 weeks). Should be there in the Dremio release notes

Thanks
@balaji.ramaswamy

8

Good evening @balaji.ramaswamy ,
Firstly I’d like to say that I love Dremio. Top notch product. I did want to point out that I experienced the following error message on Elastic Cloud today:

“Detected Elastic version null. Full query pushdown in Dremio requires version 2.1.2 or above.”

I am hoping this is a cloud issue in general and the Elastic Cloud version will be resolved as a part of that release?

Thanks in advance

9

Hi @Andrew_Greiner

This is a known issue and we are working on fixing the message. Do you see your push down capabilities work as expected?

Thanks
@balaji.ramaswamy

10

+1

We are seeing the same issue on AWS with ES 6.0.1.

11

@balaji.ramaswamy We are seeing the same issue w/ ES 5.6 on AWS

12

Hi @Fima_Leshinsky

Yes, its new feature and currently not supported on both 5.x and 6.x versions of Elasticserch on AWS

Thanks
@balaji.ramaswamy

13

Hi @Fima_Leshinsky

We have enhanced Dremio to support elasticsearch on AWS. If this is still a requirement, kindly download our latest version 3.0.5 from https://www.dremio.com/download/, upgrade, test and let us know

Thanks
@balaji.ramaswamy