S3 data dir0 is null in 2/3rds of the records

1

I have S3 folders set up as “table”/date1, “table”/date2,… containing parquet files.

I can read all the data, but not all folders are getting dir0 set to the name of the folder. What could cause this to occur? (I’m on 2.1.6)

select count(*) from X where dir0 is not null

“72226”

select count(*) from X where dir0 is null

“128485”

The first 21 folders have dir0 populated, after that they are null.
Removing the first folder of the 21 did not change any of the null values.
Removing the 21st folder of the 21 did not change any of the null values.
Removing the first 21 folders caused the remaining dir0 values to be populated.

Could this have been caused by a schema change?

2

Any ideas on what could cause this? I figure it must be a defect right?