I was trying to use dremio to read from hive tables with nested fields.
Unfortunately I can’t. I’m getting error Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column ‘xxx’ not found in any table, but the column exists.
Hey @maver1ck, sorry for the delay. You should be able to directly point Dremio to the Parquet/JSON files backing these Hive tables and query nested fields. We currently don’t support accessing nested structures when used over Hive.
Thanks fot the advise. It worked.
Are there any plans to support such a fields in near future ?
Hi @can ,
We met the same issue now on Dremio 4.9.1. We want to query Dremio Hive tables for Array<struct<…>>. Is this supported now by any settings?
@balaji.ramaswamy , Would you help suggest on this?
Dremio fully supports structs, arrays (LISTS) but not MAP data types for Hive tables, what help would you exactly need?
In one of the use case, we were working with the nested struct values from Hive. below schema has “Parent__c” column as a nested struct. I have searched for data type compatibility of Dremio and hive it shows struct Compatability with single level struct i.e. key-value pair structkey1:value1,key2:value2 in Dremio documentation. Just wanted to understand does the Dremio support the nested struct.
so the schema that I am using is shown below
root
|-- Id: string (nullable = true)
|-- Name: string (nullable = true)
|-- Parent__c: struct (nullable = true)
| |-- Id: string (nullable = true)
| |-- Name: string (nullable = true)
| |-- Parent__c: struct (nullable = true)
| | |-- Id: string (nullable = true)
| | |-- Name: string (nullable = true)
| | |-- Parent__c: struct (nullable = true)
| | | |-- Id: string (nullable = true)
| | | |-- Name: string (nullable = true)
| | | |-- Parent__c: struct (nullable = true)
| | | | |-- Id: string (nullable = true)
| | | | |-- Name: string (nullable = true)
| | | | |-- Parent__c: struct (nullable = true)
| | | | | |-- Id: string (nullable = true)
| | | | | |-- Name: string (nullable = true)
| | | | | |-- Parent__c: struct (nullable = true)
| | | | | | |-- Id: string (nullable = true)
| | | | | | |-- Name: string (nullable = true)
| | | | | | |-- Parent__c: struct (nullable = true)
| | | | | | | |-- Id: string (nullable = true)
| | | | | | | |-- Name: string (nullable = true)
| | | | | | | |-- Parent__c: struct (nullable = true)
| | | | | | | | |-- Id: string (nullable = true)
| | | | | | | | |-- Name: string (nullable = true)
| | | | | | | | |-- Parent__c: struct (nullable = true)
| | | | | | | | | |-- Id: string (nullable = true)
| | | | | | | | | |-- Name: string (nullable = true)
| | | | | | | | | |-- Type__c: string (nullable = true)
| | | | | | | | | |-- SubType__c: string (nullable = true)
| | | | | | | | | |-- IsKit__c: boolean (nullable = true)
| | | | | | | | |-- Type__c: string (nullable = true)
| | | | | | | | |-- SubType__c: string (nullable = true)
| | | | | | | | |-- IsKit__c: boolean (nullable = true)
| | | | | | | |-- Type__c: string (nullable = true)
| | | | | | | |-- SubType__c: string (nullable = true)
| | | | | | | |-- IsKit__c: boolean (nullable = true)
| | | | | | |-- Type__c: string (nullable = true)
| | | | | | |-- SubType__c: string (nullable = true)
| | | | | | |-- IsKit__c: boolean (nullable = true)
| | | | | |-- Type__c: string (nullable = true)
| | | | | |-- SubType__c: string (nullable = true)
| | | | | |-- IsKit__c: boolean (nullable = true)
| | | | |-- Type__c: string (nullable = true)
| | | | |-- SubType__c: string (nullable = true)
| | | | |-- IsKit__c: boolean (nullable = true)
| | | |-- Type__c: string (nullable = true)
| | | |-- SubType__c: string (nullable = true)
| | | |-- IsKit__c: boolean (nullable = true)
| | |-- Type__c: string (nullable = true)
| | |-- SubType__c: string (nullable = true)
| | |-- IsKit__c: boolean (nullable = true)
| |-- Type__c: string (nullable = true)
| |-- SubType__c: string (nullable = true)
| |-- IsKit__c: boolean (nullable = true)
|-- Type__c: string (nullable = true)
|-- SubType__c: string (nullable = true)
|-- IsKit__c: boolean (nullable = true)
|-- Parent__c_ancestors: array (nullable = true)
| |-- element: string (containsNull = true)
|-- children: array (nullable = true)
| |-- element: string (containsNull = true)
@rohankumardubey Dremio has started support for MAP types too, yes there is support for nested structs
thanks @balaji.ramaswamy will look at the documentation