Unable to connect to Dremio from Superset . Both are running as Docker containers

Greetings !

I am trying to have a sandbox environment through Docker.

  1. Dremio running on a container – based on AlexMerced’s tutorial docker-compose
  2. Superset based on the non-dev docker-compose. I have included sqlalchemy_dremio in the ./docker/requirements-local.txt as per the Superset documentation. I checked the logs of superset_init container and it looks like the driver was successfully installed.
    ==
    2024-02-14 15:28:08 Installing local overrides at /app/docker/requirements-local.txt
    2024-02-14 15:28:08 Collecting sqlalchemy_dremio
    2024-02-14 15:28:08 Downloading sqlalchemy_dremio-3.0.3-py2.py3-none-any.whl (18 kB)
    2024-02-14 15:28:08 Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.9/site-packages (from sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (14.0.1)
    2024-02-14 15:28:08 Requirement already satisfied: sqlalchemy>=1.3.24 in /usr/local/lib/python3.9/site-packages (from sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (1.4.36)
    2024-02-14 15:28:08 Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.9/site-packages (from pyarrow>=5.0.0->sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (1.23.5)
    2024-02-14 15:28:08 Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.9/site-packages (from sqlalchemy>=1.3.24->sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (2.0.2)
    2024-02-14 15:28:09 Installing collected packages: sqlalchemy_dremio
    2024-02-14 15:28:09 Successfully installed sqlalchemy_dremio-3.0.3
    2024-02-14 16:03:34 superset.commands.database.exceptions.DatabaseTestConnectionDriverError: Could not load database driver: DremioEngineSpec
    ==
    However, when I tried to add a new datasource through superset UI and hit TestConnection i get teh following error.

2024-02-14 16:03:34 superset.commands.database.exceptions.DatabaseTestConnectionDriverError: Could not load database driver: DremioEngineSpec

=== log from superset_app Docker container
2024-02-14 16:03:34 2024-02-14 21:03:34,583:WARNING:superset.views.base:CommandException
2024-02-14 16:03:34 Traceback (most recent call last):
2024-02-14 16:03:34 File “/app/superset/commands/database/test_connection.py”, line 131, in run
2024-02-14 16:03:34 with database.get_sqla_engine_with_context(
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/contextlib.py”, line 119, in enter
2024-02-14 16:03:34 return next(self.gen)
2024-02-14 16:03:34 File “/app/superset/models/core.py”, line 423, in get_sqla_engine_with_context
2024-02-14 16:03:34 yield self._get_sqla_engine(
2024-02-14 16:03:34 File “/app/superset/models/core.py”, line 440, in _get_sqla_engine
2024-02-14 16:03:34 self.db_engine_spec.validate_database_uri(sqlalchemy_url)
2024-02-14 16:03:34 File “/app/superset/db_engine_specs/base.py”, line 1963, in validate_database_uri
2024-02-14 16:03:34 sqlalchemy_uri.get_driver_name(), set()
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py”, line 624, in get_driver_name
2024-02-14 16:03:34 return self.get_dialect().driver
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py”, line 679, in get_dialect
2024-02-14 16:03:34 entrypoint = self._get_entrypoint()
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py”, line 661, in _get_entrypoint
2024-02-14 16:03:34 cls = registry.load(name)
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py”, line 343, in load
2024-02-14 16:03:34 raise exc.NoSuchModuleError(
2024-02-14 16:03:34 sqlalchemy.exc.NoSuchModuleError: Can’t load plugin: sqlalchemy.dialects:dremio
2024-02-14 16:03:34
2024-02-14 16:03:34 The above exception was the direct cause of the following exception:
2024-02-14 16:03:34

I appreciate any guidance/pointers on resolving this.

Sri

@Sri

Have you tried suggestions on thi post by @naren

Thanks Balaji. I did go through that but not sure that is relevant in my case, using docker and a docker-compose file from tutorials… May be i am missing something.

However, I switched the connection string to use flight

dremio+flight://user:pwd@localhost:31010/

The error is different.
SQL: Could not finish writing before closing

I will post the log separately.

Appreciate any pointers on how to resolve this.

Thanks,
Sri

the log:
here is the log:

2024-02-15 13:50:02 SupersetErrorsException
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 161, in run
2024-02-15 13:50:02 raise DBAPIError(ex_str or None, None, None)
2024-02-15 13:50:02 sqlalchemy.exc.DBAPIError: (builtins.NoneType) None
2024-02-15 13:50:02 [SQL: Could not finish writing before closing]
2024-02-15 13:50:02 (Background on this error at: hypertextprotocols://sqlalche.me/e/14/dbapi)
2024-02-15 13:50:02
2024-02-15 13:50:02 The above exception was the direct cause of the following exception:
2024-02-15 13:50:02
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1823, in full_dispatch_request
2024-02-15 13:50:02 rv = self.dispatch_request()
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1799, in dispatch_request
2024-02-15 13:50:02 return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py”, line 95, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 127, in wraps
2024-02-15 13:50:02 raise ex
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 121, in wraps
2024-02-15 13:50:02 duration, response = time_function(f, self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/core.py”, line 1454, in time_function
2024-02-15 13:50:02 response = func(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/log.py”, line 255, in wrapper
2024-02-15 13:50:02 value = f(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 93, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/databases/api.py”, line 918, in test_connection
2024-02-15 13:50:02 TestConnectionDatabaseCommand(item).run()
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 190, in run
2024-02-15 13:50:02 raise SupersetErrorsException(errors) from ex
2024-02-15 13:50:02 superset.exceptions.SupersetErrorsException: [SupersetError(message=‘(builtins.NoneType) None\n[SQL: Could not finish writing before closing]\n(Background on this error at: //sqlalche.me/e/14/dbapi)’, error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: ‘GENERIC_DB_ENGINE_ERROR’>, level=<ErrorLevel.ERROR: ‘error’>, extra={‘engine_name’: ‘Dremio’, ‘issue_codes’: [{‘code’: 1002, ‘message’: ‘Issue 1002 - The database returned an unexpected error.’}]})]
2024-02-15 13:50:02 2024-02-15 18:50:02,170:WARNING:superset.views.base:SupersetErrorsException
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 161, in run
2024-02-15 13:50:02 raise DBAPIError(ex_str or None, None, None)
2024-02-15 13:50:02 sqlalchemy.exc.DBAPIError: (builtins.NoneType) None
2024-02-15 13:50:02 [SQL: Could not finish writing before closing]
2024-02-15 13:50:02 (Background on this error at: hypertextprotocols://sqlalche.me/e/14/dbapi)
2024-02-15 13:50:02
2024-02-15 13:50:02 The above exception was the direct cause of the following exception:
2024-02-15 13:50:02
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1823, in full_dispatch_request
2024-02-15 13:50:02 rv = self.dispatch_request()
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1799, in dispatch_request
2024-02-15 13:50:02 return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py”, line 95, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 127, in wraps
2024-02-15 13:50:02 raise ex
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 121, in wraps
2024-02-15 13:50:02 duration, response = time_function(f, self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/core.py”, line 1454, in time_function
2024-02-15 13:50:02 response = func(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/log.py”, line 255, in wrapper
2024-02-15 13:50:02 value = f(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 93, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/databases/api.py”, line 918, in test_connection
2024-02-15 13:50:02 TestConnectionDatabaseCommand(item).run()
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 190, in run
2024-02-15 13:50:02 raise SupersetErrorsException(errors) from ex
2024-02-15 13:50:02 superset.exceptions.SupersetErrorsException: [SupersetError(message=‘(builtins.NoneType) None\n[SQL: Could not finish writing before closing]\n(Background on this error at: //sqlalche.me/e/14/dbapi)’, error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: ‘GENERIC_DB_ENGINE_ERROR’>, level=<ErrorLevel.ERROR: ‘error’>, extra={‘engine_name’: ‘Dremio’, ‘issue_codes’: [{‘code’: 1002, ‘message’: ‘Issue 1002 - The database returned an unexpected error.’}]})]
2024-02-15 13:50:02 192.168.65.1 - - [15/Feb/2024:18:50:02 +0000] “POST /api/v1/database/test_connection/ hypertextprotocol/1.1” 500 362 “/localhost:8088/superset/welcome/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”

===

Hello Dremio experts, I am still waiting on any pointers to resolve this. Much appreciate it. – regards, Sri

Flight port is by default is 32010, try that instead?

Thank you Lenoy. I tried both ports (31010 and 32010) but issue still persists.

@rsrs Could you try again but with SQLAlchemy from this fork? GitHub - lenoyjacob/sqlalchemy_dremio: SQLAlchemy for Dremio via Flight interface.

Lenoy, Happy to try it. Can you give me step by step instructions on how to swap the driver to this ?

I followed the instructions on Superset docker non-dev version which uses requirements_local.txt earlier to get the SuperSet on Docker.

Thank you !

Lenoy, Never mind my request for detailed instructions. I got it.

Also, good news this (version of the ) driver seems to work. Test Connection was successful Much appreciate your guidance.

regards,
Sri

1 Like

I am facing the same issue since last week. Can you please let me know how did you resolve this?

I have setup Dremio and Superset in Docker Desktop.
Dremio: http://locahost:9047
Superset: http://localhost:8088

I have also installed database driver for Dremio in Superset.

While connecting Superset to Dremio, I am getting below error-

I am using below SQLALCHEMY URI for Dremio-
dremio+flight://pinku:password@localhost:32010/Pinku

On the above URL, Pinku in the last is dataset from next to the home icon from Dremio UI. I have also tried port 31010 but getting same error.