Superset - Dremio Error 'datetime64[ms]' for timestamp field

1

I connected Superset to Dremio and tried to visualise some data.

image
image1370×884 68.7 KB

I want to use a timestamp field (called started_at) as the time column, but got ** Dremio error datetime64[ms] ** error

I confirm that this behaviour is specific only to timestamp data type. Reason is, when I switch to another column (as_of_date) with the data type date, it is fine without error.

image
image1370×884 85.3 KB

============

I also verified that the SQL which superset sent to Dremio had no problem getting run on Dremio

image
image1919×918 220 KB

Attached is the log file from superset pod on kubernetes.

log.zip (1.8 KB)

2

Given that Dremio’s timesatmp field is basically date PLUS time, does it mean that I would not be able to plot timeseries data in superset where x-axis is a datetime series?

3

@Ken Could you try again but with SQLAlchemy from this fork? GitHub - lenoyjacob/sqlalchemy_dremio: SQLAlchemy for Dremio via Flight interface.

4

Thank you for the suggestion.

Looks like it might work because of the updated mapping for datetime64[ms] → types.datetime

5

@lenoyjacob

Thank you so much for your suggestion.

image
image1920×999 149 KB

I am using the version in your fork and it is working.

Do we know when your fork could be merged back into the main repository?

6

I believe this PR would have a similar effect like your fork.

I added a comment to chase the maintainer to see if the PR could be merged.

THank you!

7

It is a bit sad that there is no sign the PR is getting merged :cry:

8

I think that repo is abandoned, my PR is not merged yet.
Any posibility that @naren transfer repo to another mantainer?
or Dremio make a official repo? @balaji.ramaswamy or @lenoyjacob can you help here?

1 Like
9

Getting the sqlalchemy_dremio pip package updated would really nice. It’s been hard to find the correct one to use. There are a number of packages that seem to aim to do the same thing, but they all seem to be out of date, even the Preset one.

This is the pip package that would be ideal to update:

To clarify, when I install this package in Superset (any version), when I try to create a database connection to Dremio I get

Error: 'datetime64[ms]'

This user is one of the maintainers. Any chance they’re at Dremio? :smiley:

If I install the package here, everything works: GitHub - lenoyjacob/sqlalchemy_dremio: SQLAlchemy for Dremio via Flight interface.

I have an Ansible role that attempts to leverage the Superset github repo as much as possible, and I am making sure git is installed on all the docker containers that start up via the docker-compose-non-dev.yml file (by adding “apt update; apt install git -y” near the top of docker/docker-bootstrap.sh); I have also added “git+https://github.com/lenoyjacob/sqlalchemy_dremio.git” to docker/requirements-local.yml file.

Figured I’d leave a description for someone else looking to solve this.

2 Likes
10

For the record, I am also doing the same in my requirements.txt file.


# For connecting to dremio
# sqlalchemy_dremio==3.0.3
## datetime64[ms] issue here: https://community.dremio.com/t/superset-dremio-error-datetime64-ms-for-timestamp-field/11827
git+https://github.com/lenoyjacob/sqlalchemy_dremio.git

It is working for me.

1 Like