While reading Hive table (partitioned table) getting exceeds dataset split limit error

krajkumar14 · June 16, 2020, 12:07pm

Number of splits (265389) in dataset exceeds dataset split limit of 200000

balaji.ramaswamy · June 22, 2020, 6:56am

What version Dremio is this? Currently we support up to 300,000 splits on Hive

Thanks
Bali

krajkumar14 · June 29, 2020, 5:40am

Hi, I am using 4.2.1…

balaji.ramaswamy · July 1, 2020, 8:10am

Can you please upgrade and test. We should allow upto 300,000 splits

Topic		Replies	Views
Dataset split limits errors	6	2641	April 22, 2020
Query exceeds the query split limit of 300000	1	1905	December 29, 2020
"Number of splits (432658) in dataset XXX.2019 exceeds dataset split limit of 300000" when trying to use the "Format Folder" feature on aws load balancer access logs in s3 at the load balancer level and the year level s3 folder	4	769	March 29, 2023
Parquet file too many splits	7	1243	January 15, 2023
Too many splits encountered when processing parquet metadata	4	1752	April 4, 2021