While reading Hive table (partitioned table) getting exceeds dataset split limit error

krajkumar14 · June 16, 2020, 12:07pm

Number of splits (265389) in dataset exceeds dataset split limit of 200000

balaji.ramaswamy · June 22, 2020, 6:56am

What version Dremio is this? Currently we support up to 300,000 splits on Hive

Thanks
Bali

krajkumar14 · June 29, 2020, 5:40am

Hi, I am using 4.2.1…

balaji.ramaswamy · July 1, 2020, 8:10am

Can you please upgrade and test. We should allow upto 300,000 splits

Topic		Replies	Views
Dataset split limits errors	6	2787	April 22, 2020
Query exceeds the query split limit of 300000	1	2098	December 29, 2020
"Number of splits" error	1	1236	June 20, 2022
"Number of splits (432658) in dataset XXX.2019 exceeds dataset split limit of 300000" when trying to use the "Format Folder" feature on aws load balancer access logs in s3 at the load balancer level and the year level s3 folder	4	1201	March 29, 2023
Parquet file too many splits	7	1512	January 15, 2023