Dremio 2.0.1rc - unable to connect to Mongo 3.4.9-ent

1

Hello,

We’re trying to connect dremio 2.0.1rc to our UAT instance of mongo 3.4.9-ent which is setup as a primary/secondary with communication between the mongo servers set to use SSL, but allow clients to connect without SSL.

On the dremio web admin page to add a source, it just says “Failure while configuring source [MongoUAT]”

Mongo Compass connects fine with the identical connection parameters and SSL turned OFF, so it’s definitely allowing non-secure clients.

In the server log I see that it has connected to the primary but it seems to failing when accessing the secondary (i only specify the primary in the connection details). Apologies for the spam, but here are the logs if it helps (and thanks in advance for any help!):

2018-04-19 10:45:40,302 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - Created connection to [address=<our_primary>:27000, user=<our_user>].
2018-04-19 10:45:40,302 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - Number of open connections 1.
2018-04-19 10:45:40,302 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - MongoClientOptions:
MongoClientOptions{description='null', readPreference=primary, writeConcern=WriteConcern{w=1, wtimeout=0, fsync=false, j=false, codecRegistry=org.bson.codecs.configuration.ProvidersCodecRegistry@e95a9714, minConnectionsPerHost=0, maxConnectionsPerHost=100, threadsAllowedToBlockForConnectionMultiplier=5, serverSelectionTimeout=2000, maxWaitTime=120000, maxConnectionIdleTime=0, maxConnectionLifeTime=0, connectTimeout=10000, socketTimeout=0, socketKeepAlive=false, sslEnabled=false, sslInvalidHostNamesAllowed=false, alwaysUseMBeans=false, heartbeatFrequency=10000, minHeartbeatFrequency=500, heartbeatConnectTimeout=20000, heartbeatSocketTimeout=20000, localThreshold=15, requiredReplicaSetName='null', dbDecoderFactory=com.mongodb.DefaultDBDecoder$1@2f0a66a, dbEncoderFactory=com.mongodb.DefaultDBEncoder$1@4deb4741, socketFactory=javax.net.DefaultSocketFactory@27cb862b, cursorFinalizerEnabled=true, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitQueueSize=500, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, keepAlive=false, receiveBufferSize=0, sendBufferSize=0}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=20000, readTimeoutMS=20000, keepAlive=false, receiveBufferSize=0, sendBufferSize=0}}
2018-04-19 10:45:40,302 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - MongoCredential:MongoCredential{mechanism=null, userName='<our_user>', source='admin', password=<hidden>, mechanismProperties={}}
2018-04-19 10:45:40,339 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - Created connection to [address=<our_secondary>:27001, user=<our_user>].
2018-04-19 10:45:40,339 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - Number of open connections 2.
2018-04-19 10:45:40,339 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - MongoClientOptions:
MongoClientOptions{description='null', readPreference=primary, writeConcern=WriteConcern{w=1, wtimeout=0, fsync=false, j=false, codecRegistry=org.bson.codecs.configuration.ProvidersCodecRegistry@e95a9714, minConnectionsPerHost=0, maxConnectionsPerHost=100, threadsAllowedToBlockForConnectionMultiplier=5, serverSelectionTimeout=2000, maxWaitTime=120000, maxConnectionIdleTime=0, maxConnectionLifeTime=0, connectTimeout=10000:
, socketTimeout=0, socketKeepAlive=false, sslEnabled=false, sslInvalidHostNamesAllowed=false, alwaysUseMBeans=false, heartbeatFrequency=10000, minHeartbeatFrequency=500, heartbeatConnectTimeout=20000, heartbeatSocketTimeout=20000, localThreshold=15, requiredReplicaSetName='null', dbDecoderFactory=com.mongodb.DefaultDBDecoder$1@2f0a66a, dbEncoderFactory=com.mongodb.DefaultDBEncoder$1@4deb4741, socketFactory=javax.net.DefaultSocketFactory@27cb862b, cursorFinalizerEnabled=true, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitQueueSize=500, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, keepAlive=false, receiveBufferSize=0, sendBufferSize=0}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=20000, readTimeoutMS=20000, keepAlive=false, receiveBufferSize=0, sendBufferSize=0}}
2018-04-19 10:45:40,339 [Plugin Startup: MongoUAT] INFO  c.d.p.m.c.MongoConnectionManager - MongoCredential:MongoCredential{mechanism=null, userName='<our_user>', source='admin', password=<hidden>, mechanismProperties={}}
2018-04-19 10:45:42,342 [qtp714321161-290] WARN  c.dremio.exec.catalog.PluginsManager - Exception while shutting down source.
java.lang.NullPointerException: null
        at com.dremio.exec.catalog.ManagedStoragePlugin.close(ManagedStoragePlugin.java:552) ~[dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.PluginsManager.deleteSource(PluginsManager.java:287) ~[dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:627) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:376) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:100) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:946) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:529) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:182) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:149) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:140) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:136) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at sun.reflect.GeneratedMethodAccessor220.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-04-19 10:45:42,344 [qtp714321161-290] INFO  c.d.exec.catalog.CatalogServiceImpl - User Error Occurred [ErrorId: 870dd5f2-0f63-4d9c-b90c-32b66931eff8]
com.dremio.common.exceptions.UserException: Failure while configuring source [MongoUAT]
        at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:629) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:376) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:100) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:946) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:529) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:182) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:149) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:140) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:136) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at sun.reflect.GeneratedMethodAccessor220.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: java.util.concurrent.ExecutionException: com.mongodb.MongoTimeoutException: Timed out after 2000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=<our_secondary>:27001, type=REPLICA_SET_SECONDARY, roundTripTime=0.7 ms, state=CONNECTED}]
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69) ~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:107) ~[guava-18.0.jar:na]
        at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:617) [dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        ... 54 common frames omitted
Caused by: com.mongodb.MongoTimeoutException: Timed out after 2000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=<our_secondary>:27001, type=REPLICA_SET_SECONDARY, roundTripTime=0.7 ms, state=CONNECTED}]
        at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:370) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:104) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:97) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:85) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.operation.CommandWriteOperation.execute(CommandWriteOperation.java:55) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.Mongo.execute(Mongo.java:747) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.Mongo$2.execute(Mongo.java:730) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:111) ~[mongo-java-driver-3.0.2.jar:na]
        at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:101) ~[mongo-java-driver-3.0.2.jar:na]
        at com.dremio.plugins.mongo.metadata.MongoVersion.getVersionForConnection(MongoVersion.java:51) ~[dremio-extra-plugin-mongo-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.plugins.mongo.metadata.MongoTopology.getMinVersion(MongoTopology.java:220) ~[dremio-extra-plugin-mongo-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.plugins.mongo.metadata.MongoTopology.getClusterTopology(MongoTopology.java:168) ~[dremio-extra-plugin-mongo-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.plugins.mongo.connection.MongoConnectionManager.getTopology(MongoConnectionManager.java:174) ~[dremio-extra-plugin-mongo-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.plugins.mongo.connection.MongoConnectionManager.validateConnection(MongoConnectionManager.java:310) ~[dremio-extra-plugin-mongo-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.plugins.mongo.connection.MongoConnectionManager.connect(MongoConnectionManager.java:145) ~[dremio-extra-plugin-mongo-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.plugins.mongo.MongoStoragePlugin.start(MongoStoragePlugin.java:126) ~[dremio-extra-plugin-mongo-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.exec.catalog.ManagedStoragePlugin$2.run(ManagedStoragePlugin.java:238) ~[dremio-sabot-kernel-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) ~[dremio-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) ~[dremio-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) ~[dremio-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
        at com.dremio.concurrent.Runnables$1.run(Runnables.java:45) ~[dremio-common-2.0.1-201804132205050000-10b1de0.jar:2.0.1-201804132205050000-10b1de0]
2

Hi Ben,

We’re looking into your issue. Hopefully we’ll have a response back to you soon.

Christy

3

Thanks Christy. If there’s any more infomation i can provide to help, let me know.

4

Hi Christy,

Just to let you know, but the connection is now magically working. We’re not entirely sure what has changed since last week, but I’d suspect the disabling of SSL we made in Mongo prior to trying dremio hadn’t fully kicked into action until the weekend restarts of Mongo.

So for now, we’re good. I’ll post back if it happens again.

Cheers,
Ben

1 Like
5

Thanks for the update Ben!

6

Hey Ben,

Thanks a lot. I will close the internal ticket I raised to track this.

Christy

7

No problem. I may have been a bit too quick though. Now when we try to access the saved connection we see:

The source ["Mongo UAT"] is currently unavailable. Info: [[Message{level=ERROR, msg=Timed out after 2000 ms while waiting for a server that matches PrimaryServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=<other mongo server>, type=REPLICA_SET_SECONDARY, roundTripTime=0.8 ms, state=CONNECTED}]}]].

I’m guessing our setup switched primary and secondary for whatever reason and that has confused dremio. We have both primary and secondary specified in the connection details, but that refers to sharded servers, which these aren’t. However, if i specify only one server (which may or not be the primary at the time) it fails to connect if that isn’t the primary - so it smells like there’s still something odd going on.

What is the recommended way to connect non-sharded servers in a primary/secondary setup?

Cheers,
Ben

8

Hey Ben,

Dremio relies on the mongo jdbc driver to handle failover events properly. Is this something you can reproduce, and if so, can you share steps so we can try to do the same here?

9

Hi Kelly,

We’ve replaced the default 3.0.2 Mongo JDBC driver with the 3.4.2 one and it seems to be reliably connecting. I’ll post back with more information if I see problems with this version.

Cheers
Ben

1 Like
10

Thanks Ben - we’ll take a look on our end as well.