We ran into this issue too. I would expect any valid parquet file to be readable by Dremio. As it stands, the user runs into issues unexpectant upon the first occurrence. Maybe you send a parquet file with 1k cols, fail… send a parquet file using the legacy list structure, fail, send a parquet file with list col over 128 entries, fail… this list of limitations on top of the parquet format leaves the end user questioning when the next generated parquet file will break things.
1 Like