2020-02-08x19.parquet is 53121961. Max supported footer size is 16777216

1

hi,
i got such error when dremio tried to read the parquet file generated by pyarrow. can you please give some hints on how to resolve it?

thanks!

2

I used for loop to give 20 files. only the last and the 2nd last one have this issue. when i removed these 2 files, alter pds refresh metadata can subsequently give the successful info

3

@gypsysunny

Currently we have a Parquet Footer Size limit of 16 MB

Thanks
@balaji.ramaswamy