INFORMATION_SCHEMA.COLUMNS query fails in web ui, succeeds in datagrip

1

Hello!

On our helm chart based setup (image sha256:95bb1841335c41ae32b1e9fb9a4c779ec047903e94cc66e0a6b7ed3657feddfe) if I try to query information_schema.columns I get

IllegalArgumentException: No enum constant com.dremio.common.types.TypeProtos.MinorType.TIMEMICRO.Show less

Screenshot from 2021-10-11 20-16-57
Screenshot from 2021-10-11 20-16-571133×600 40.4 KB

However, the same query from DataGrip and through the JDBC driver returns correctly.

Please let me know whether there is a way to query this information also from the Web UI.

Thank you,
Matteo

2

@ma82 Are you able to send us the profile from the query that failed via UI and the successful query from JDBC?

3

Hi @balaji.ramaswamy,

thank you for your reply.

I reproduced the problem and checked the generated jobs: both the query from DataGrip and the one from the Web UI are actually listed as “failed”, with the same error.

However, DataGrip rather strangely manages to show valid-looking output and doesn’t emit any error.

For what concerns sending you the profile: are you suggesting to upload the zip files that one can download from the UI?

In that case, is it OK if I avoid sending the header.json file that appears in them?

There seems to be a lot of private information in it.

4

@ma82 Yes, send me other files, you can leave header.json

5

Thank you @balaji.ramaswamy

I could reproduce once more using the simple query

select * from information_schema.columns

  • web ui in preview mode: the query succeeded and the corresponding dremio job is in status completed
  • web ui in run mode: the query failed with the previously reported error and the corresponding dremio job is in status failed
  • datagrip: 500 rows appeared in the client, no error was shown but the corresponding dremio job is in status failed

Here are the profiles:

community-dremio-com-8116.zip (29.4 KB)

It would be very valuable for us to be able to query this table.

Thank you for any help on this!

6

@ma82 It seems like one of the datatypes is causing this, do you have any TIMESTAMP(6) like MICRO seconds datatypes?

1 Like
7

@balaji.ramaswamy

I think this problem was generated by a mysql source, which we removed yesterday for other reasons [*]: we don’t currently have this problem in production anymore.

We’re planning to simplify the deployment of dremio instances in our qa environments: when we have them we’ll try and find out what mysql types were generating this issue on the dremio side and send you an update.

Thank you for your help!

[*] We’re in the process of moving away from a PoC federation setup to more consistently use s3 as our main data source.

8

@ma82 Thanks a lot for the update, wish you good luck on your simplified deployment process

10

Reposting because i accidentaly deleted the last one:

I went through the same issue and found a cause that might affect others as well. In my case, when generating an Iceberg table, Dremio incorrectly configured it as Parquet. Simply going to the folder/table settings and setting the type to Iceberg resolved the issue

11

@patrick.silva Which catalog was the Iceberg table created on?

12

I just have the main catalog here in my Dremio. I use DBT to perform full updates on certain tables daily. These tables are stored in an S3 data source.

Occasionally (although I haven’t identified a consistent pattern yet), some S3 folders fail to automatically convert into tables. To address this issue, I enabled the option to ‘Automatically format files into physical datasets when users issue queries.’

However, after making this change, I encountered the enum errors. It turned out that the Iceberg S3 folder had been mistakenly interpreted as a generic Parquet folder."

13

@patrick.silva

“mistakenly interpreted as a generic Parquet folder” by Dremio?