Dremio integration with Mongodb

1

Hi , I am using dremio 3.0 standalone and trying it to coonect to Mongodb 4.0. Below is my server.log file. Kindly help me to resolve this issue.

2019-04-04 09:07:03,162 [qtp1272438966-306] INFO c.d.exec.catalog.CatalogServiceImpl - User Error Occurred [ErrorId: ddf7cb11-589a-4188-bc59-dc1e5445a5bf]
com.dremio.common.exceptions.UserException: Failure while configuring source [M]
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:639) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:378) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.exec.catalog.CatalogServiceImpl.access$600(CatalogServiceImpl.java:102) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:972) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:519) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:189) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:157) [dremio-dac-backend-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:148) [dremio-dac-backend-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:144) [dremio-dac-backend-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80) [dremio-dac-backend-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_201]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_201]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
Caused by: java.util.concurrent.ExecutionException: com.mongodb.MongoTimeoutException: Timed out after 2000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=10.26.37.151:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:401) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:104) ~[guava-20.0.jar:na]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:627) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
… 55 common frames omitted
Caused by: com.mongodb.MongoTimeoutException: Timed out after 2000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=10.26.37.151:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.mongodb.connection.BaseCluster.getDescription(BaseCluster.java:167) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo.getConnectedClusterDescription(Mongo.java:885) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo.createClientSession(Mongo.java:877) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo$3.getClientSession(Mongo.java:866) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo$3.execute(Mongo.java:823) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.MongoIterableImpl.execute(MongoIterableImpl.java:130) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.MongoIterableImpl.iterator(MongoIterableImpl.java:77) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.MappingIterable.iterator(MappingIterable.java:40) ~[mongo-java-driver-3.6.4.jar:na]
at com.dremio.plugins.mongo.metadata.MongoCollections.getDatabasesFromListDatabases(MongoCollections.java:228) ~[dremio-extra-plugin-mongo-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.plugins.mongo.metadata.MongoCollections.fetchDatabaseAndCollectionNames(MongoCollections.java:102) ~[dremio-extra-plugin-mongo-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.plugins.mongo.connection.MongoConnection.getCollections(MongoConnection.java:37) ~[dremio-extra-plugin-mongo-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.plugins.mongo.connection.MongoConnectionManager.validateConnection(MongoConnectionManager.java:301) ~[dremio-extra-plugin-mongo-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.plugins.mongo.connection.MongoConnectionManager.connect(MongoConnectionManager.java:145) ~[dremio-extra-plugin-mongo-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.plugins.mongo.MongoStoragePlugin.start(MongoStoragePlugin.java:129) ~[dremio-extra-plugin-mongo-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.exec.catalog.ManagedStoragePlugin$1.run(ManagedStoragePlugin.java:255) ~[dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.concurrent.RenamingRunnable.run(RenamingRunnable.java:36) ~[dremio-common-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.concurrent.SingletonRunnable.run(SingletonRunnable.java:41) ~[dremio-common-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.concurrent.SafeRunnable.run(SafeRunnable.java:40) ~[dremio-common-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
at com.dremio.concurrent.Runnables$1.run(Runnables.java:45) ~[dremio-common-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
[root@devupgnew4 log]# vi server.log
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
Caused by: java.util.concurrent.ExecutionException: com.mongodb.MongoTimeoutException: Timed out after 2000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=10.26.37.151:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:401) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:68) ~[guava-20.0.jar:na]
at com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:104) ~[guava-20.0.jar:na]
at com.dremio.exec.catalog.CatalogServiceImpl.createOrUpdateSource(CatalogServiceImpl.java:627) [dremio-sabot-kernel-3.1.7-201903210426240688-c843b51.jar:3.1.7-201903210426240688-c843b51]
… 55 common frames omitted
Caused by: com.mongodb.MongoTimeoutException: Timed out after 2000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=10.26.37.151:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
at com.mongodb.connection.BaseCluster.getDescription(BaseCluster.java:167) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo.getConnectedClusterDescription(Mongo.java:885) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo.createClientSession(Mongo.java:877) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo$3.getClientSession(Mongo.java:866) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.Mongo$3.execute(Mongo.java:823) ~[mongo-java-driver-3.6.4.jar:na]
at com.mongodb.MongoIterableImpl.execute(MongoIterableImpl.java:130) ~[mongo-java-driver-3.6.4.jar:na]

2

Hi, have you tried to manually connect to your Mongo instance? Using telnet or netcat?
The stacktrace quite clearly says that 10.26.37.151:27017 refuses connections…
We’re using MongoDB with Dremio quite heavily and it works fine…

Best, Tim

3

Yes I tried using telnet .So What is your point ??

4

Do we need to add any Jar file to integrate with Mongo ??

5

My point is that the error message clearly says that there is a network-related problem.
You don’t need any additional drivers, everything included in Dremio.

6

Telnet is not working for me as well .Connection refused is the status

7

Aha, you could try to log-in to the machine where Mongo runs and then telnet localhost 27017. If that does not work, your Mongo server isn’t running. If it does work, you should check firewall configurations and routes between your hosts.

8

I tried with the same and when I giving the telnet localhost 27017 And displayed this status

telnet localhost 27017
Trying ::1…
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.

9

But when I giving the IP with Telnet is shows this
telnet 10.26.37.151 27017
Trying 10.26.37.151…
telnet: connect to address 10.26.37.151: Connection refused

10

And Tim I am using the admin database with No authentication enabled there in Dremio

11

You should check to which interfaces MongoDB binds! Could be that it only listens to the loopback device (which would explain why telnet 127.0.0.1 27017 works).

See bind_ip in /etc/mongod.conf. You can set it to bind_ip=0.0.0.0 to make it listen to all interfaces (for testing – might lead to security problems)

Best, Tim

12

Thanks Tim now issue is being resolved .

13

Hi Abhba,

Can you please let me know what did you do to make the issue resolved? We are not able to connect mongoDB 4.0