Spark Array Support - Flight JDBC Connector

Dremio University
1

We use on-prem installation of 23.2.2-202307280452300417-9276e770, Enterprise Edition.
Recently we tried to connect to Dremio via flight-sql-jdbc-driver-10.0.0.jar from Dremio JDBC Drivers, using iceberg-spark-runtime-3.4_2.12-1.4.3.

In generally it works (for primitive types including timestamps), but we get an exception if we try to access ARRAY columns.
The error:
org.apache.spark.SparkSQLException: [UNRECOGNIZED_SQL_TYPE] Unrecognized SQL type - name: ARRAY, id: ARRAY.

Anyone has any idea how this problem could be fixed ?

2

Hi @Alexarl

I don’t believe that connector was written by Dremio. However, I see in the code of that connector that the corresponding for array in Arrow is list: spark-flight-connector/README.md at 46ce7b6559705d0828f7e5b863495908563718f4 · qwshen/spark-flight-connector · GitHub

Hopefully that helps, otherwise I suggest you get in touch with qwshen.

Thanks, Bogdan