Metabase is an open source business intelligence tool. I am writing a Metabase driver plugin to Dremio.
I have already installed the Dremio JDBC driver with my project.
Now I am trying to make a connection with the following attributes:
classname: “com.dremio.jdbc.Driver”
connection uri: “jdbc:dremio:direct=127.0.0.1:31010”
I am running Dremio locally on a Docker container, with the Dremio services at port 31010.
However, when I do the Metabase setup, entering connection attributes in the “Add your data” section, and hit the “Next” button, I would get the following “Connection Refused” error message from the Metabase logs:
ERROR rpc.BasicClient :: Failed to establish connection
java.util.concurrent.ExecutionException: cdjd.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:31010
at cdjd.io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:284)
at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:272)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
at cdjd.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
at cdjd.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
at cdjd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
at cdjd.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at cdjd.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: cdjd.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:31010
Caused by: java.net.ConnectException: Connection refused
From the Network console, I get the following error message:
Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.ConnectionFailedException: cdjd.com.dremio.exec.rpc.RpcException: CONNECTION : cdjd.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:31010"
I suspect that my connection string is correct, but its requests are just not reaching the Dremio service.
The Dremio logs do not exhibit any reaction when connection requests are made.
What could be going on? Please help!
@rdavid48
Is your JDBC tool installed on the same machine running the Dremio coordinator?
Yes, Balaji. Both the Dremio coordinator and the JDBC client are running on the same machine, on separate Docker containers.
@rdavid48, From the JDBC docker container are you able to do “telnet dremio-coordinator-docker 31010”?
Sorry for the late reply, Balaji. I was on vacation.
When I run the following:
docker exec -it metabase telnet 52.123.456.789 31010
I get:
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:262: starting container process caused “exec: "telnet": executable file not found in $PATH”
When I run:
telnet 52.123.456.789 31010
I get:
Trying 52.123.456.789…
Connected to 52.123.456.789.
Escape character is ‘^]’.
Connection closed by foreign host.
Would I need to install telnet in my container? What do these messages mean?
@rdavid48
Did you run the second telnet from one of the docker’s? That mean the port is open
What happens if you try “telnet localhost 31010” and “telnet 127.0.0.1 31010” from the JDBC docker container
Hi @balaji.ramaswamy,
Let me clarify my current configuration. My Dremio instance is now running remotely at, say, ip address 52.123.456.789
My Metabase client which has the Dremio JDBC driver installed is running in a docker container.
The connection string I have been using (as well as trying many other permutations) in the container is:
jdbc:dremio:direct=127.0.0.1:31010
When you speak of “dremio-coordinator-docker” I am assuming you are referring to the Dremio instance running at 52.123.456.789
Is this assumption correct? I am not too knowledgeable about what a Dremio Coordinator is, or how it is different from the Dremio Executor service, or if they are on separate ip addresses.
So when you say “telnet dremio-coordinator-docker 31010” I would run:
docker exec -it metabase telnet 52.123.456.789 31010
and then I get:
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:262: starting container process caused “exec: “telnet”: executable file not found in $PATH”
When I ran “telnet 52.123.456.789 31010” it was ran from my local host machine, not the container.
And that’s where I got:
Trying 52.123.456.789…
Connected to 52.123.456.789.
Escape character is ‘^]’.
Connection closed by foreign host.
So does that mean that the Dremio port 31010 at 52.123.456.789 is open and my docker container should be able to access it?
Does that mean that the problem is with the connectivity from my container?
When I run either “docker exec -it metabase telnet localhost 31010” or “docker exec -it metabase telnet 127.0.0.1 31010”, I get:
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:262: starting container process caused “exec: “telnet”: executable file not found in $PATH”
Does this mean that the problem is that my container cannot access port 31010 on the local host?
What must I do?
Thanks for the details @rdavid48
Your jdbc container needs to talk to the Dremio coordinator (you got it right) on port 31010. So you should NOT run anything on your local machine. Instead from your container (which I suppose is the JDBC metabase client) you should be able to run “telnet 52.123.456.789 31010” and you should get “Escape character is ‘^]’.” but instead if you get “connection refused” or “Trying …” then the port is not open and you should open inbound on the coordinator on 31010
Thanks, @balaji.ramaswamy
Dremio is not running on my local machine. It is running on 52.123.456.789.
Only my JDBC Metabase client is running as a docker container on my local machine.
However, my container does not have telnet installed. So I simply use ping.
From the container (with the jdbc Metabase client), I can ping 52.123.456.789.
Assuming the container can access the Dremio coordinator at 52.123.456.789, I changed the JDBC connection string to:
jdbc:dremio:direct=52.123.456.789:31010
Now, the connection error I get changed to:
ERROR rpc.RpcExceptionHandler :: Exception in RPC communication. Connection: null <–> null (user client). Closing connection.
java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available
ERROR driver.util :: Database connection error
java.sql.SQLException: Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.RpcException: HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <–> null (user client)
DEBUG middleware.log :: POST /api/setup/validate 400 2.7 s (0 DB calls)
{:errors {:dbname “Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.RpcException: HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <–> null (user client)”}}
In case this meant that there was a problem with authentication and SSL, I toggled between “Encrypt SSL: true” and “Encrypt SSL: false”, but the error message was the same.
What does this mean?
Thanks.
@balaji.ramaswamy
I managed to install telnet inside my docker container.
Here are the results:
telnet localhost 31010
telnet: can’t connect to remote host (127.0.0.1): Connection refused
telnet 127.0.0.1 31010
telnet: can’t connect to remote host (127.0.0.1): Connection refused
But when I run:
telnet 52.123.456.789 31010
telnet hangs or does nothing, and I have to hit Ctrl^C and “e” to escape telnet.
What does this all mean?
@rdavid48
Can you please send me a screenshot of what happens after you give “telnet 52.123.456.789 31010” and hit enter?
@balaji.ramaswamy
Here’s the screenshot:
@rdavid48,
As a quick test, is it too difficult to install the JDBC client on the Dremio coordinator and see if it connects? This will atleast narrow down the problem
@balaji.ramaswamy
I’m afraid I have no access to the remote server running the Dremio instance and coordinator.
I cannot test the JDBC client on the same remote machine.
What do the Dremio connection error messages mean?
And what does the telnet unresponsiveness and hanging mean?
@rdavid48
I am not sure if you are getting a blank screen which is +ve and the port is open versus if it hangs its a firewall issue. I think it is the latter. Can you spin up a Dremio instance on the docker container so its local to the JDBC and try?
Hi @balaji.ramaswamy
I am using netstat inside my Metabase docker container to check for any traffic to the remote Dremio instance. I use the command:
netstat -a
I try to connect dozens of times, sometimes going through minor configuration permutations. I almost never get any indication from netstat that any request is going out from the container. But in the container logs, I usually get the error message:
Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.RpcException: HANDSHAKE_COMMUNICATION : [USER]: Channel closed null <--> null (user client)
Once every few dozen times, I would catch this message from netstat:
Here in the 2nd row I see that a connection to 52.170.198.158:31010 is ESTABLISHED !
But a split-second later, it disappears from netstat.
Was that a true connection where the container reached the remote server and the remote serve replied back? Or was that a fluke? Why does that line almost instantly disappear? Why doesn’t it stay? What does it mean?
@rdavid48
Is it possible your connection is getting dropped? For the same reason “Can you spin up a Dremio instance on the docker container so its local to the JDBC and try?”
Yes, @balaji.ramaswamy . The clues do appear to suggest that it’s possible my connection is getting dropped. What could be the possible reasons for that?
I don’t know how to spin up a Dremio instance on the docker container, so I’m looking for alternative solutions.
If I understand you correctly, @balaji.ramaswamy you are suggesting that I run a Dremio docker container inside my Metabase JDBC docker container?
Early on, I had separate containers for Dremio and Metabase JDBC running on my local machine, but that didn’t work. So I instead started pointing to a remote instance of Dremio but kept the JDBC container on my local host.
