Hi all,
We tried to connect Dremio to Glue Data Catalog on 22.0.0 version but keep receiving this error.
2022-07-13 15:19:56,041 [start-datalake] WARN c.d.e.catalog.ManagedStoragePlugin - Error starting new source: datalake
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf$ConfVars
at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.getConfForGlue(AWSGlueStoragePlugin.java:262)
at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.setup(AWSGlueStoragePlugin.java:230)
at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.start(AWSGlueStoragePlugin.java:600)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:571)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf$ConfVars
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 9 common frames omitted
2022-07-13 15:19:56,054 [start-datalake] INFO c.d.s.s.LocalSchedulerService - Cancelling task metadata-refresh-wakeup-datalake
2022-07-13 15:19:56,833 [qtp1232433403-13] ERROR c.dremio.exec.catalog.PluginsManager - Exception while creating source.
com.dremio.common.exceptions.UserException: org/apache/hadoop/hive/conf/HiveConf$ConfVars
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:890)
at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:436)
at com.dremio.exec.catalog.ManagedStoragePlugin.createSource(ManagedStoragePlugin.java:328)
at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:174)
at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:424)
at com.dremio.exec.catalog.CatalogServiceImpl.access$500(CatalogServiceImpl.java:115)
at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:773)
at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:1658)
at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:319)
at com.dremio.exec.catalog.SourceAccessChecker.createSource(SourceAccessChecker.java:363)
at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:319)
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:169)
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:160)
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:156)
at com.dremio.dac.resource.EnterprisePutSourceResource.putSource(EnterprisePutSourceResource.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at com.dremio.dac.server.GenericResponseHeadersFilter.doFilter(GenericResponseHeadersFilter.java:46)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:69)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf$ConfVars
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1046)
at com.dremio.exec.catalog.ManagedStoragePlugin.createOrUpdateSource(ManagedStoragePlugin.java:369)
... 78 common frames omitted
Suppressed: java.lang.NullPointerException: null
at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.getState(AWSGlueStoragePlugin.java:585)
at com.dremio.exec.catalog.ManagedStoragePlugin.setLocals(ManagedStoragePlugin.java:949)
at com.dremio.exec.catalog.ManagedStoragePlugin.replacePlugin(ManagedStoragePlugin.java:1061)
... 79 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf$ConfVars
at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.getConfForGlue(AWSGlueStoragePlugin.java:262)
at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.setup(AWSGlueStoragePlugin.java:230)
at com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.start(AWSGlueStoragePlugin.java:600)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:571)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:639)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf$ConfVars
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 9 common frames omitted
We decided to go through the exact same configuration but using the previous version 21.2.2 and connected fine to glue catalog. Is there any known issue with 22.0.0 version? Would appreciate any suggestions.