Unable to connect to Dremio from Superset . Both are running as Docker containers

rsrs · February 14, 2024, 9:34pm

Greetings !

I am trying to have a sandbox environment through Docker.

Dremio running on a container – based on AlexMerced’s tutorial docker-compose
Superset based on the non-dev docker-compose. I have included sqlalchemy_dremio in the ./docker/requirements-local.txt as per the Superset documentation. I checked the logs of superset_init container and it looks like the driver was successfully installed.
==
2024-02-14 15:28:08 Installing local overrides at /app/docker/requirements-local.txt
2024-02-14 15:28:08 Collecting sqlalchemy_dremio
2024-02-14 15:28:08 Downloading sqlalchemy_dremio-3.0.3-py2.py3-none-any.whl (18 kB)
2024-02-14 15:28:08 Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.9/site-packages (from sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (14.0.1)
2024-02-14 15:28:08 Requirement already satisfied: sqlalchemy>=1.3.24 in /usr/local/lib/python3.9/site-packages (from sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (1.4.36)
2024-02-14 15:28:08 Requirement already satisfied: numpy>=1.16.6 in /usr/local/lib/python3.9/site-packages (from pyarrow>=5.0.0->sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (1.23.5)
2024-02-14 15:28:08 Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.9/site-packages (from sqlalchemy>=1.3.24->sqlalchemy_dremio->-r /app/docker/requirements-local.txt (line 1)) (2.0.2)
2024-02-14 15:28:09 Installing collected packages: sqlalchemy_dremio
2024-02-14 15:28:09 Successfully installed sqlalchemy_dremio-3.0.3
2024-02-14 16:03:34 superset.commands.database.exceptions.DatabaseTestConnectionDriverError: Could not load database driver: DremioEngineSpec
==
However, when I tried to add a new datasource through superset UI and hit TestConnection i get teh following error.

2024-02-14 16:03:34 superset.commands.database.exceptions.DatabaseTestConnectionDriverError: Could not load database driver: DremioEngineSpec

=== log from superset_app Docker container
2024-02-14 16:03:34 2024-02-14 21:03:34,583:WARNING:superset.views.base:CommandException
2024-02-14 16:03:34 Traceback (most recent call last):
2024-02-14 16:03:34 File “/app/superset/commands/database/test_connection.py”, line 131, in run
2024-02-14 16:03:34 with database.get_sqla_engine_with_context(
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/contextlib.py”, line 119, in enter
2024-02-14 16:03:34 return next(self.gen)
2024-02-14 16:03:34 File “/app/superset/models/core.py”, line 423, in get_sqla_engine_with_context
2024-02-14 16:03:34 yield self._get_sqla_engine(
2024-02-14 16:03:34 File “/app/superset/models/core.py”, line 440, in _get_sqla_engine
2024-02-14 16:03:34 self.db_engine_spec.validate_database_uri(sqlalchemy_url)
2024-02-14 16:03:34 File “/app/superset/db_engine_specs/base.py”, line 1963, in validate_database_uri
2024-02-14 16:03:34 sqlalchemy_uri.get_driver_name(), set()
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py”, line 624, in get_driver_name
2024-02-14 16:03:34 return self.get_dialect().driver
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py”, line 679, in get_dialect
2024-02-14 16:03:34 entrypoint = self._get_entrypoint()
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/url.py”, line 661, in _get_entrypoint
2024-02-14 16:03:34 cls = registry.load(name)
2024-02-14 16:03:34 File “/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py”, line 343, in load
2024-02-14 16:03:34 raise exc.NoSuchModuleError(
2024-02-14 16:03:34 sqlalchemy.exc.NoSuchModuleError: Can’t load plugin: sqlalchemy.dialects:dremio
2024-02-14 16:03:34
2024-02-14 16:03:34 The above exception was the direct cause of the following exception:
2024-02-14 16:03:34

I appreciate any guidance/pointers on resolving this.

Sri

balaji.ramaswamy · February 15, 2024, 12:27am

@Sri

Have you tried suggestions on thi post by @naren

rsrs · February 15, 2024, 7:10pm

Thanks Balaji. I did go through that but not sure that is relevant in my case, using docker and a docker-compose file from tutorials… May be i am missing something.

However, I switched the connection string to use flight

dremio+flight://user:pwd@localhost:31010/

The error is different.
SQL: Could not finish writing before closing

I will post the log separately.

Appreciate any pointers on how to resolve this.

Thanks,
Sri

rsrs · February 15, 2024, 7:13pm

the log:
here is the log:

2024-02-15 13:50:02 SupersetErrorsException
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 161, in run
2024-02-15 13:50:02 raise DBAPIError(ex_str or None, None, None)
2024-02-15 13:50:02 sqlalchemy.exc.DBAPIError: (builtins.NoneType) None
2024-02-15 13:50:02 [SQL: Could not finish writing before closing]
2024-02-15 13:50:02 (Background on this error at: hypertextprotocols://sqlalche.me/e/14/dbapi)
2024-02-15 13:50:02
2024-02-15 13:50:02 The above exception was the direct cause of the following exception:
2024-02-15 13:50:02
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1823, in full_dispatch_request
2024-02-15 13:50:02 rv = self.dispatch_request()
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1799, in dispatch_request
2024-02-15 13:50:02 return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py”, line 95, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 127, in wraps
2024-02-15 13:50:02 raise ex
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 121, in wraps
2024-02-15 13:50:02 duration, response = time_function(f, self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/core.py”, line 1454, in time_function
2024-02-15 13:50:02 response = func(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/log.py”, line 255, in wrapper
2024-02-15 13:50:02 value = f(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 93, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/databases/api.py”, line 918, in test_connection
2024-02-15 13:50:02 TestConnectionDatabaseCommand(item).run()
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 190, in run
2024-02-15 13:50:02 raise SupersetErrorsException(errors) from ex
2024-02-15 13:50:02 superset.exceptions.SupersetErrorsException: [SupersetError(message=‘(builtins.NoneType) None\n[SQL: Could not finish writing before closing]\n(Background on this error at: //sqlalche.me/e/14/dbapi)’, error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: ‘GENERIC_DB_ENGINE_ERROR’>, level=<ErrorLevel.ERROR: ‘error’>, extra={‘engine_name’: ‘Dremio’, ‘issue_codes’: [{‘code’: 1002, ‘message’: ‘Issue 1002 - The database returned an unexpected error.’}]})]
2024-02-15 13:50:02 2024-02-15 18:50:02,170:WARNING:superset.views.base:SupersetErrorsException
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 161, in run
2024-02-15 13:50:02 raise DBAPIError(ex_str or None, None, None)
2024-02-15 13:50:02 sqlalchemy.exc.DBAPIError: (builtins.NoneType) None
2024-02-15 13:50:02 [SQL: Could not finish writing before closing]
2024-02-15 13:50:02 (Background on this error at: hypertextprotocols://sqlalche.me/e/14/dbapi)
2024-02-15 13:50:02
2024-02-15 13:50:02 The above exception was the direct cause of the following exception:
2024-02-15 13:50:02
2024-02-15 13:50:02 Traceback (most recent call last):
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1823, in full_dispatch_request
2024-02-15 13:50:02 rv = self.dispatch_request()
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask/app.py”, line 1799, in dispatch_request
2024-02-15 13:50:02 return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-02-15 13:50:02 File “/usr/local/lib/python3.9/site-packages/flask_appbuilder/security/decorators.py”, line 95, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 127, in wraps
2024-02-15 13:50:02 raise ex
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 121, in wraps
2024-02-15 13:50:02 duration, response = time_function(f, self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/core.py”, line 1454, in time_function
2024-02-15 13:50:02 response = func(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/utils/log.py”, line 255, in wrapper
2024-02-15 13:50:02 value = f(*args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/views/base_api.py”, line 93, in wraps
2024-02-15 13:50:02 return f(self, *args, **kwargs)
2024-02-15 13:50:02 File “/app/superset/databases/api.py”, line 918, in test_connection
2024-02-15 13:50:02 TestConnectionDatabaseCommand(item).run()
2024-02-15 13:50:02 File “/app/superset/commands/database/test_connection.py”, line 190, in run
2024-02-15 13:50:02 raise SupersetErrorsException(errors) from ex
2024-02-15 13:50:02 superset.exceptions.SupersetErrorsException: [SupersetError(message=‘(builtins.NoneType) None\n[SQL: Could not finish writing before closing]\n(Background on this error at: //sqlalche.me/e/14/dbapi)’, error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: ‘GENERIC_DB_ENGINE_ERROR’>, level=<ErrorLevel.ERROR: ‘error’>, extra={‘engine_name’: ‘Dremio’, ‘issue_codes’: [{‘code’: 1002, ‘message’: ‘Issue 1002 - The database returned an unexpected error.’}]})]
2024-02-15 13:50:02 192.168.65.1 - - [15/Feb/2024:18:50:02 +0000] “POST /api/v1/database/test_connection/ hypertextprotocol/1.1” 500 362 “/localhost:8088/superset/welcome/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36”

===

rsrs · February 20, 2024, 8:34pm

Hello Dremio experts, I am still waiting on any pointers to resolve this. Much appreciate it. – regards, Sri

lenoyjacob · February 20, 2024, 9:38pm

Flight port is by default is 32010, try that instead?

rsrs · March 9, 2024, 11:03pm

Thank you Lenoy. I tried both ports (31010 and 32010) but issue still persists.

lenoyjacob · March 13, 2024, 2:08am

@rsrs Could you try again but with SQLAlchemy from this fork? GitHub - lenoyjacob/sqlalchemy_dremio: SQLAlchemy for Dremio via Flight interface.

rsrs · March 15, 2024, 3:16pm

Lenoy, Happy to try it. Can you give me step by step instructions on how to swap the driver to this ?

I followed the instructions on Superset docker non-dev version which uses requirements_local.txt earlier to get the SuperSet on Docker.

Thank you !

Lenoy, Never mind my request for detailed instructions. I got it.

Also, good news this (version of the ) driver seems to work. Test Connection was successful Much appreciate your guidance.

regards,
Sri

Pinku · March 27, 2024, 11:43am

I am facing the same issue since last week. Can you please let me know how did you resolve this?

I have setup Dremio and Superset in Docker Desktop.
Dremio: http://locahost:9047
Superset: http://localhost:8088

I have also installed database driver for Dremio in Superset.

While connecting Superset to Dremio, I am getting below error-

I am using below SQLALCHEMY URI for Dremio-
dremio+flight://pinku:password@localhost:32010/Pinku

On the above URL, Pinku in the last is dataset from next to the home icon from Dremio UI. I have also tried port 31010 but getting same error.

Luke · May 2, 2024, 9:42am

@Pinku try to connect with URI below:
dremio+flight://pinku:password@localhost:32010/Pinku?UseEncryption=false

rosi1899 · May 7, 2024, 3:16am

Hi, I already follow step by step for install all requirement_local.txt and already restart service superset but have same error.
SQLAlchemy 2.0.30
sqlalchemy-dremio 3.0.3
pytest 8.2.0
bumpversion 0.6.0
pyarrow 16.0.0

our environment VM with ubuntu os, superset version 3.1.2 Dremio version 25.x.x

cfrancois · July 3, 2024, 3:29pm

@rosi1899
Have you found a solution to your issue?

I met similar issue with my config.

I install dremio by using the docker-compose from dremio on laptop
I install superset by following the Superset quickstart
I add sqlalchemy-dremio in the requirements-local.txt for superset, it is installed.

Then when I tried to connect with this config:
dremio+flight://dremio:password@localhost:32010/dremio?UseEncryption=false
or
dremio+flight://dremio:password@dremio:32010/dremio?UseEncryption=false
I met this error:

ERROR: (builtins.NoneType) None
[SQL: Could not finish writing before closing]
(Background on this error at: https://sqlalche.me/e/14/dbapi)

I’ve read many blog, stackoverflow but none of them provide answers. They stopped writing without sharing their solutions.

Is there a missing driver to install?

lenoyjacob · July 3, 2024, 4:06pm

@cfrancois Instead of sqlalchemy-dremio in the requirements-local.txt, add the following instead and try again:

git+https://github.com/lenoyjacob/sqlalchemy_dremio.git

c.delivorias · October 9, 2024, 8:06am

I’m seeing the same issue. I’ve tried using the git repo that @lenoyjacob suggested but didn’t seem to fix this.
Did it work for you @rosi1899 ?
What is the underlying cause for this error?
Thnx

Topic		Replies	Views
Superset cannot connect to dremio Dremio University	1	238	May 13, 2024
Unable to connect to Dremio from Superset	7	2368	April 7, 2021
Issue with Superset with Dremio Dremio University	1	1579	July 10, 2020
Superset to Dremio connectivity	3	1713	October 31, 2022
Connectivity issue from Superset to Dremio	11	406	April 2, 2024

Unable to connect to Dremio from Superset . Both are running as Docker containers

Related topics