Cannot connect to MongoDB from Dremio

We cannot connect to MongoDB (version 4.0.6-3) which is running on a DigitalOcean Droplet from Dremio (version 3.1.9) anymore. It used to work, but https was added to the Droplet, which might have caused the connection issue now? When we use RoboMongo, it connects without problems to this MongoDB on the Digital Ocean Droplet. Please advise.

Thanks!

Regards
Willem

Hello @willemderu,

Can you share your configuration for the Mongo source?
https://docs.dremio.com/data-sources/mongo.html

What is the specific error that you’re seeing?

Hi @ben , thanks for the assistance, it is appreciated very much. Apologies, I should have added more details to the original topic question above, thanks for you patience :slight_smile:

We have previously used the instructions in the document that you have linked to as follows:

We are using the Dremio MongoDB Source and have set the Host to:
188.226.181.157 and Port 27017

For Authentication, we use ‘Master Credentials’
Username: StratSmart and Password: stratpwd (I will change this password after getting it going. The db now only contains test data.)

For the Authentication Database we have the following: formioapp

Other than the above, we have just kept the default settings in the Dremio MongoDB Source.

If the ‘Encrypt connection’ setting is checked, it gives the following error:
Failure creating/updating source [StratSmart MongoDB]

If the ‘Encrypt connection’ setting is not checked, it then gives the following error:
Could not find/access authentication database formioapp

(The above works when linking to the db with RoboMongo (we specify the Auth Mechanism as SCRAM-SHA-1))

I can provide the log output as well, if required.

Thanks!

Regards
Willem

@willemderu, can you try the connection and follow the logs as you attempt this? I believe they should give a more verbose error message.

@ben, below are the log output for reference. Maybe I should try to move to a previous version of MongoDB. (it is strange though that robomongo and ‘NoSQLBooster for MongoDB’ both connect to MongoDB without any issues at all)

2019-04-24 22:18:13,856 [qtp1249000159-215] INFO c.d.e.catalog.ManagedStoragePlugin - User Error Occurred [ErrorId: 2967ab76-2e7a-48fb-be1e-d565912fcc5f]
com.dremio.common.exceptions.UserException: The source [“StratSmart MongoDB”] is currently unavailable. Info: [[Message{level=ERROR, msg=For input string: “6-3”}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:356) [dremio-sabot-kernel-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:600) [dremio-sabot-kernel-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867) [dremio-sabot-kernel-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
at com.dremio.dac.service.source.SourceService.listSource(SourceService.java:374) [dremio-dac-backend-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
at com.dremio.dac.resource.SourceResource.getSource(SourceResource.java:159) [dremio-dac-backend-3.1.10-201904162146020182-adf690d.jar:3.1.10-201904162146020182-adf690d]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_201]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_201]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_201]
at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_201]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:365) [jetty-servlets-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.26.v20180806.jar:9.2.26.v20180806]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_201]

Thanks @willemderu, unfortunately the error is message is frustratingly generic. A couple of questions:

  1. Is the mongo instance a replica set and, if so, is one of the members of that replica set down?
  2. Are there any additional client connection options you added to RoboMongo to get it to connect to Mongo with SSL?

Thanks for the assistance @ben It is much appreciated. I have done a lot of experimentation the past while, (without success, though). To answer your questions: (1) The mongo instance is not a replica set. (2) It seems that it might not be related to moving to https. After removing the SSL Certificate to test, Dremio still did not want to connect to MongoDB. (although client apps such as MongoChef (Robo 3T) and noSQLBooster for MongoDB, connects without problems from the same environment)

I see in this thread in the Dremio Community Forum: Percona Mongodb 4.0.9-4 null pointer that Mongodb 4 is not supported yet by Dremio? Our issue might be related to this? (We cannot move back to MongoDB 3 though) What is the approx roadmap for Mongodb 4 support and what is the approx time-frames for moving the MongoDB connector out of Beta?

Thanks!

Just for some more feedback, we spun up an experimental Digital Ocean Droplet and installed MongoDB 3.6.12. Dremio connected successfully to this MongoDB instance.

However, it is not viable to use MongoDB 3 because MongoDB 4 (as well the the Percona MongoDB 4 builds) contains functionality that is not available in MongoDB 3.

Thanks!
Willem

Just for feedback, I have managed to connect to MongoDB 4 with Dremio. I am not sure how it started working though. (I have not upgraded my version of Dremio yet when it started working again.) However, just providing feedback here that Dremio can connect to MongoDB 4 as well. Thanks!

Thanks for reporting your testing efforts @willemderu,

What version of Dremio did you connect to Mongo?

Hi @ben , I am currently using Dremio Build 3.2.3, which connects to MongoDB 4 without a problem.
(This is probably the wrong place to say this, but I just want to say that Dremio is awesome. I have tried many other alternatives to connect to and query MongoDB, but the features provided by Dremio is spot-on for our requirements. Thanks also for the Community Edition, which provides a very accessible and realistic way of getting up to speed with Dremio and getting real value from Dremio.)