Too many splits encountered when processing parquet metadata

1

java.io.IOException: com.dremio.exec.store.parquet.Metadata$TooManySplitsException: Too many splits encountered when processing parquet metadata at file /file/0176_part_00.parquet, maximum is 60000 but encountered 60008 splits thus far.

Is this limit can be changed ?

2

@v-shaal

Our current limit for maximum number of splits for file system sources is 60,000. What is the row group size defined in the ETL job? Can it be altered to a bigger size so the total number of splits come down?

Thanks
Bali

3

thats whole lot of data that needs to be reprocessed and altered to bigger size in that case…:frowning:

4

I have the same problem, not other solutions? some support key?

thanks

5

@LucGth

What is the average size of each Parquet file?