# Dataset split limits errors

**URL:** <https://community.dremio.com/t/dataset-split-limits-errors/5415>\
**Category:** Uncategorized\
**Created:** [April 19, 2020, 6:04am UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415 "2020-04-19T06:04:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![noah](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@noah](https://community.dremio.com/u/noah)\
**Post date:** [April 19, 2020, 6:04am UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415/1 "2020-04-19T06:04:54Z")

</div>

`Number of splits (345476) in dataset <dataset.path> exceeds dataset split limit of 300000`

Anyone have an explanation of what dataset splitting is in the above context? I get this error when trying to format a large number of directories that all contain a large number of json data structures.

---

<div class="post-metadata">

**Author:** ![noah](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@noah](https://community.dremio.com/u/noah)\
**Post date:** [April 19, 2020, 7:01pm UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415/2 "2020-04-19T19:01:48Z")

</div>

Dremio Build 4.0.5-201911202046080257-19b10938

---

<div class="post-metadata">

**Author:** ![ben](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ben/32/1367_2.png) [@ben](https://community.dremio.com/u/ben)\
**Post date:** [April 20, 2020, 2:33am UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415/3 "2020-04-20T02:33:33Z")

</div>

@noah

If the dataset is Parquet files, this refers to the number of Row Groups across all files in the dataset

If the dataset is some other kind of file, then is just the number of files.

---

<div class="post-metadata">

**Author:** ![noah](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@noah](https://community.dremio.com/u/noah)\
**Post date:** [April 20, 2020, 4:53am UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415/5 "2020-04-20T04:53:25Z")

</div>

Can the dataset split limit be increased and if so, what resources need to be considered when increasing them? (e.g. RAM, CPU, disk, OS limits?)

---

<div class="post-metadata">

**Author:** ![ben](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ben/32/1367_2.png) [@ben](https://community.dremio.com/u/ben)\
**Post date:** [April 20, 2020, 5:52pm UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415/6 "2020-04-20T17:52:18Z")

</div>

In newer versions of Dremio (4.1.x or higher) this value has been increased to 60 000:  
[https://docs.dremio.com/advanced-administration/limits.html](https://docs.dremio.com/advanced-administration/limits.html)

Mainly this effects RAM on the Dremio coordinator, as on-heap objects are created for each split while Dremio plans queries.

---

<div class="post-metadata">

**Author:** ![noah](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@noah](https://community.dremio.com/u/noah)\
**Post date:** [April 20, 2020, 9:02pm UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415/7 "2020-04-20T21:02:12Z")

</div>

I am hitting the 300k limit in my case. Is that because the files in my case are on an S3-like object store source?

Is the 60k/300k limit a fixed limit or can it be changed in dremio.conf?

---

<div class="post-metadata">

**Author:** ![ben](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ben/32/1367_2.png) [@ben](https://community.dremio.com/u/ben)\
**Post date:** [April 22, 2020, 1:57pm UTC](https://community.dremio.com/t/dataset-split-limits-errors/5415/8 "2020-04-22T13:57:27Z")

</div>

It cannot be changed with `dremio.conf`. It would be better to alter the query so as to avoid the split error in the first place.