2020-02-08x19.parquet is 53121961. Max supported footer size is 16777216

i got such error when dremio tried to read the parquet file generated by pyarrow. can you please give some hints on how to resolve it?


I used for loop to give 20 files. only the last and the 2nd last one have this issue. when i removed these 2 files, alter pds refresh metadata can subsequently give the successful info


Currently we have a Parquet Footer Size limit of 16 MB