Cannot create Dremio connection from Superset: Exception in RPC communication

1

Hi all!
I have a Dremio running docker instance that successfully access a Postgres db. I have also a running Superset docker instance created out of alexmerced/dremio-superset image that is not able to connect to Dremio due to this error:

20:43:25.168 [UserServer-1] ERROR com.dremio.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /172.27.0.8:31010 <--> /172.27.0.7:39396 (user client).  Closing connection.
io.netty.handler.codec.CorruptedFrameException: Expected to read a tag of 10 but actually received a value of 3.  Happened after reading 0 message.

SQLALCHEMY URI is set as:

dremio+flight://my-user:my-password.@172.27.0.8:31010/my-dremio-space?UseEncryption=false

Superset docker instance is able to see dremio docker instance as long as both instances are part of the same network. Superset docker instance is derived from alexmerced/dremio-superset so it has the SQL alchemy drivers.

dremio.conf looks like:

paths: {
  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true,
  flight.use_session_service: true
}

I’ve added ‘coordinator.flight.enabled: true’ as I googled that it could be the reason for which I cannot establish a connection to Dremio from Superset.

Dremio docker instance has been created with:

docker run -d --name=dremio --network=db-network -p 9047:9047 -p 31010:31010 -p 45678:45678 -p 32010:32010 -e DREMIO_JAVA_SERVER_EXTRA_OPTS=-Dpaths.dist=file:///opt/dremio/data/dist -v /my-path/conf:/opt/dremio/conf dremio/dremio-oss

Any suggestion? Other similar topics didn’t help me.

2

@relio There seems to be a similar issue reported internally, I would like to verify if the stack is the same. Any chance you can send the profile?

3

Mmh, which profile? :slight_smile:

4

@relio My bad, Are you able to send the logfile so we can see the full stack

5

This is the relevant part of Superset log when I’m trying to connect to Dremio:

2025-01-08 08:12:36,121:DEBUG:superset.stats_logger:[stats_logger] (incr) test_connection_attempt
2025-01-08 08:12:36,129:DEBUG:superset.models.core:Database._get_sqla_engine(). Masked URL: dremio+flight://my-user:XXXXXXXXXX@172.27.0.8:31010/my-dremio-space?UseEncryption=false
2025-01-08 08:12:36,160:DEBUG:superset.stats_logger:[stats_logger] (incr) test_connection_error.DBAPIError
2025-01-08 08:12:36,163:DEBUG:superset.stats_logger:[stats_logger] (incr) DatabaseRestApi.test_connection.error
2025-01-08 08:12:36,163:WARNING:superset.views.base:SupersetErrorsException
Traceback (most recent call last):
  File "/app/superset/databases/commands/test_connection.py", line 163, in run
    raise DBAPIError(ex_str or None, None, None)
sqlalchemy.exc.DBAPIError: (builtins.NoneType) None
[SQL: Flight returned unavailable error, with message: failed to connect to all addresses; last error: UNAVAILABLE: ipv4:172.27.0.8:31010: Socket closed]
(Background on this error at: https://sqlalche.me/e/14/dbapi)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py", line 95, in wraps
    return f(self, *args, **kwargs)
  File "/app/superset/views/base_api.py", line 127, in wraps
    raise ex
  File "/app/superset/views/base_api.py", line 121, in wraps
    duration, response = time_function(f, self, *args, **kwargs)
  File "/app/superset/utils/core.py", line 1518, in time_function
    response = func(*args, **kwargs)
  File "/app/superset/utils/log.py", line 255, in wrapper
    value = f(*args, **kwargs)
  File "/app/superset/views/base_api.py", line 93, in wraps
    return f(self, *args, **kwargs)
  File "/app/superset/databases/api.py", line 905, in test_connection
    TestConnectionDatabaseCommand(item).run()
  File "/app/superset/databases/commands/test_connection.py", line 192, in run
    raise SupersetErrorsException(errors) from ex
superset.exceptions.SupersetErrorsException: [SupersetError(message='(builtins.NoneType) None\n[SQL: Flight returned unavailable error, with message: failed to connect to all addresses; last error: UNAVAILABLE: ipv4:172.27.0.8:31010: Socket closed]\n(Background on this error at: https://sqlalche.me/e/14/dbapi)', error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: 'GENERIC_DB_ENGINE_ERROR'>, level=<ErrorLevel.ERROR: 'error'>, extra={'engine_name': 'Dremio', 'issue_codes': [{'code': 1002, 'message': 'Issue 1002 - The database returned an unexpected error.'}]})]
172.27.0.1 - - [08/Jan/2025:08:12:36 +0000] "POST /api/v1/database/test_connection/ HTTP/1.1" 500 469 "http://localhost:8088/databaseview/list/?pageIndex=0&sortColumn=changed_on_delta_humanized&sortOrder=desc" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
2025-01-08 08:12:46,877:DEBUG:superset.stats_logger:[stats_logger] (incr) test_connection_attempt
2025-01-08 08:12:46,889:DEBUG:superset.models.core:Database._get_sqla_engine(). Masked URL: dremio+flight://my-user:XXXXXXXXXX@172.27.0.8:31010/my-dremio-space?UseEncryption=false
2025-01-08 08:12:46,892:DEBUG:superset.stats_logger:[stats_logger] (incr) test_connection_error.DBAPIError
2025-01-08 08:12:46,897:DEBUG:superset.stats_logger:[stats_logger] (incr) db_creation_failed.SupersetErrorsException
2025-01-08 08:12:46,903:DEBUG:superset.stats_logger:[stats_logger] (incr) DatabaseRestApi.post.error
2025-01-08 08:12:46,903:DEBUG:superset.stats_logger:[stats_logger] (timing) DatabaseRestApi.post.time | 31.010183000034885 
172.27.0.1 - - [08/Jan/2025:08:12:46 +0000] "POST /api/v1/database/ HTTP/1.1" 500 469 "http://localhost:8088/databaseview/list/?pageIndex=0&sortColumn=changed_on_delta_humanized&sortOrder=desc" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
2025-01-08 08:12:52,777:DEBUG:superset.stats_logger:[stats_logger] (incr) health
127.0.0.1 - - [08/Jan/2025:08:12:52 +0000] "GET /health HTTP/1.1" 200 2 "-" "curl/7.88.1"

And this is the content of dremio.conf inside the docker instance:

dremio@52cbec2a46d8:/opt/dremio/conf$ cat dremio.conf
#
# Copyright (C) 2017-2019 Dremio Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

paths: {
  # the local path for dremio to store data.
  local: ${DREMIO_HOME}"/data"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true,
  flight.use_session_service: true
  # coordinator.flight.enabled: true
}

Inside Dremio logs:

14:34:57.542 [dremio-general-1] INFO com.dremio.service.reflection.MaterializationCache - Materialization cache updated. Materializations: reused=0 expanded=0 errors=0. External: reused=0 expanded=0 errors=0
14:34:57.543 [dremio-general-1] DEBUG com.dremio.exec.planner.plancache.PlanCacheSynchronizer - Reconciling plan cache with any changes
14:34:57.546 [dremio-general-1] INFO com.dremio.exec.planner.plancache.PlanCacheSynchronizer - Cleaning up 0 datasets based on entries, 0 datasets based on goals. Total (union) processed datasets: 0
14:34:57.546 [dremio-general-1] INFO com.dremio.exec.planner.plancache.PlanCacheSynchronizer - Completed plan cache sync.  Cache entries before 2.  Cache entries after 2.
14:34:57.546 [dremio-general-1] INFO com.dremio.exec.planner.plancache.CacheRefresher - Materialization cache sync took 1 ms.  Plan cache sync took 3 ms.
14:34:57.694 [profile-sender1] DEBUG com.dremio.exec.work.protector.ForemenWorkManager - About to send all planning profiles
14:34:57.792 [UserServer-1] ERROR com.dremio.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /172.27.0.8:31010 <--> /172.27.0.4:46096 (user client).  Closing connection.
io.netty.handler.codec.CorruptedFrameException: Expected to read a tag of 10 but actually received a value of 82.  Happened after reading 0 message.
        at com.dremio.exec.rpc.MessageDecoder.checkTag(MessageDecoder.java:256)
        at com.dremio.exec.rpc.MessageDecoder.decodeMessage(MessageDecoder.java:150)
        at com.dremio.exec.rpc.MessageDecoder.decode(MessageDecoder.java:68)
        at com.dremio.sabot.rpc.user.UserProtobufLengthDecoder.decode(UserProtobufLengthDecoder.java:32)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:829)

Hope it helps!

6

Thank you Balaji, I’ve update the post with three logs.

Hope it helps!

Bye AR

7

Thanks for the logs, will review and get back

8

@relio
Have you tried this?

I had to place this in requirements-local.txt in superset.
git+https://github.com/lenoyjacob/sqlalchemy_dremio.git and then rebuild the image…

and for the port number of dremio, I used 32010.

This is in case of open source configuration that is… Hope this helps.

9

Sorry, where is located requirements-local.txt in superset project? I cannot find it.

TIA!