Hi,
in Dremio 17.0.0 release notes (Dremio) it is indicated that
“Dremio has removed support for columns with mixed data types. A standard schema is now enforced.”
I have a parquet dataset containing many years of log having some mixed type columns, those mixed type columns are curated in a VDS. I’m unsure this will continue to work or not.
Do I need to fix the parquet dataset format before migration to Dremio 17 or will my VDS continue to work ?
Thanks for your advise
David
@dfleckinger If the data types are compatible then Dremio should coerce to the higher data type, for example integer and double should coerce to double or integer and string should coerce to string, some types that cannot be coerced will fail the query
thanks @balaji.ramaswamy .
In 17.0, in a JSON dataset, a column with FLOAT8 and BIGINT types was identified as mixed type, not sure if it is expected ?
I could cast it to float without any issue.
@dfleckinger In 17.0 the automatic coercion is behind a support key, you have to enable store.disable.mixed_types, when 18.0 is released you can reset the key
That makes sense. Thank you @balaji.ramaswamy !
mixed type functionality can be disabled ?
Upgrading to version 18.1, I lost entire mongoDB database sources, I can no longer consume the collections.
Unable to coerce from the file's data type "varbinary" to the column's data type "varchar" in table "ops_dao.tarificador.machines", column "implementations._id".
The apl key is null, is that the problem?
@jlcornejo Unfortunately this coercion is not supported so best is to work with your ETL team to make the data types consistent
