FLATTEN with empty rows fails

#1

ac64b3f0-2f0e-4d8d-9eb9-9c7ce85f4103.zip (9.1 KB)

If I try and return a simple query with FLATTEN and the array is empty, I get a query error

Unable to find the referenced field: [expr000].

#2

@jhaynie, what type of data source is devdata? What are the formats of the source datasetsdevdata.jira.sprint and devdata.pinpoint.sprint_detail" (Parquet, JSON, Mongo docs, etc)

#3

it’s a parquet file. the contents of the array are string (defined as a LIST in parquet with string value type). it appears that if there’s at least one row in the data set that has a value it works fine. when none of the rows have a value (such as []) it fails. to work around, we had to create a dummy row in the DB (which is horrible).

#4

@jhaynie, any chance you can share a sample file that produces the flatten error?

#5

yes, i’ll upload here shortly.

#6

Hello @jhaynie,

Any chance you can supply the file here with the empty list?

Is this the logical parquet type to which you refer: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists?