Failure while attempting to read metadata for

dkelly · November 8, 2018, 4:49pm

I’m successfully connected my Dremio server to my Elasticsearch Cluster (AWS Managed) and I can see all of my indices. However, when I click on one, and then click on a type to get to the query screen, I always receive the error message, “Failure while attempting to read metadata for <cluster.index.type>”.

I’ve looked at the server logs and they didn’t reveal anything interesting:

2018-11-08 10:45:26,611 [qtp813408648-207] INFO  c.dremio.exec.catalog.DatasetManager - User Error Occurred [ErrorId: 8a977d5c-77c3-4ab2-8b31-8f76919bdf9a]
com.dremio.common.exceptions.UserException: Failure while attempting to read metadata for <cluster.index.type>.
	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:316) [dremio-sabot-kernel-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.exec.catalog.DatasetManager.getTable(DatasetManager.java:191) [dremio-sabot-kernel-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.exec.catalog.CatalogImpl.getTable(CatalogImpl.java:128) [dremio-sabot-kernel-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.exec.catalog.DelegatingCatalog.getTable(DelegatingCatalog.java:59) [dremio-sabot-kernel-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.exec.catalog.CachingCatalog.getTable(CachingCatalog.java:66) [dremio-sabot-kernel-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1287) [hk2-utils-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorImpl.internalInvoke(MethodInterceptorImpl.java:109) [hk2-locator-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorImpl.invoke(MethodInterceptorImpl.java:125) [hk2-locator-2.5.0-b32.jar:na]
	at org.jvnet.hk2.internal.MethodInterceptorInvocationHandler.invoke(MethodInterceptorInvocationHandler.java:62) [hk2-locator-2.5.0-b32.jar:na]
	at com.sun.proxy.$Proxy117.getTable(Unknown Source) [na:na]
	at com.dremio.dac.explore.DatasetsResource.getDatasetSummary(DatasetsResource.java:268) [dremio-dac-backend-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.dac.explore.DatasetsResource.newUntitled(DatasetsResource.java:141) [dremio-dac-backend-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.dac.explore.DatasetsResource.newUntitledFromParent(DatasetsResource.java:208) [dremio-dac-backend-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.lang.NullPointerException: null
	at com.dremio.plugins.elastic.ElasticTableBuilder.populate(ElasticTableBuilder.java:239) ~[dremio-elasticsearch-plugin-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.plugins.elastic.ElasticTableBuilder.buildIfNecessary(ElasticTableBuilder.java:161) ~[dremio-elasticsearch-plugin-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.plugins.elastic.ElasticTableBuilder.getDataset(ElasticTableBuilder.java:142) ~[dremio-elasticsearch-plugin-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.exec.catalog.DatasetManager.getTableFromPlugin(DatasetManager.java:298) [dremio-sabot-kernel-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	... 62 common frames omitted

What’s the next step for troubleshooting this issue?

dfleckinger · November 20, 2018, 3:50pm

I have the same issue here

balaji.ramaswamy · November 20, 2018, 5:00pm

Hi @dfleckinger @dkelly

From the error stack it seems this is ES cluster 6.0 on AWS. If so, we are in the final stages of completing development and should have the support out in the next few weeks.

Thanks
@balaji.ramaswamy

dkelly · November 20, 2018, 6:39pm

Understood. Thanks!

Duncan Kelly

Senior Software Engineer

dfleckinger · November 21, 2018, 11:01am

For me it happens with Elastic Search 5.5.2 hosted on AWS

balaji.ramaswamy · December 7, 2018, 7:40am

@dfleckinger @dkelly

We have enhanced Dremio to support elasticsearch on AWS. If this is still a requirement, kindly download our latest version 3.0.5 from https://www.dremio.com/download/, upgrade, test and let us know

Thanks
@balaji.ramaswamy

dfleckinger · December 7, 2018, 2:44pm

Thanks a log @balaji.ramaswamy
I will have to check that next week.

Thanks

dfleckinger · December 12, 2018, 10:22am

Hi, are there release notes for 3.0.5 ?

I’ve followed the RPM upgrade procedure, but the master takes time to start up,
after 15 minutes, it is
still indicating
KVStore version is 3.0.5-201812021947440442-9a23f45 in the server.out, and CPU activity is low.

dfleckinger · December 12, 2018, 11:59pm

I confirm that the connection to Elastic Search on AWS is working fine now . Querying fields that are not on nested objects work like a charm, thanks !

Topic		Replies	Views
ES-Dremio: error in retrieving tables	12	1370	December 7, 2018
Failure while retrieving metadata when connected to AWS Elasticsearch 5.5	7	1681	December 7, 2018
Failure while attempting to read metadata - AWS ElasticSearch	4	1049	December 7, 2018
Failure while retrieving metadata for table	2	2381	February 15, 2018
Opening elasticsearch table fails (tested on 5.5 and 6.0)	2	1900	January 10, 2018

Failure while attempting to read metadata for

Related Topics