I have S3 folders set up as “table”/date1, “table”/date2,… containing parquet files.
I can read all the data, but not all folders are getting dir0 set to the name of the folder. What could cause this to occur? (I’m on 2.1.6)
select count(*) from X where dir0 is not null
select count(*) from X where dir0 is null
The first 21 folders have dir0 populated, after that they are null.
Removing the first folder of the 21 did not change any of the null values.
Removing the 21st folder of the 21 did not change any of the null values.
Removing the first 21 folders caused the remaining dir0 values to be populated.
Could this have been caused by a schema change?