Spill directory space usage

1

My spill directory size is increasing (now 49 GB) and i’m running out of disk space.

Is it safe to delete its content ?

Thanks

2

What version of Dremio are you on? If you’re on version 2, Dremio supports spilling for sorts. If you’re getting a lot of spilling it might suggest your queries aren’t tuned especially regarding partitioning. In general spilling prevents Out Of Memory errors, but will affect performance.

More generally, however, you could try setting the job cleanup to happen for frequently: https://docs.dremio.com/advanced-administration/job-results-cleanup.html

Christy

3

Thanks @christy
I’m running version 2.0.1.
However a lot of directories in spill directories have been created before (november, march, january).
Regarding the job cleanup, “More generally, however, you could try setting the job cleanup to happen for frequently” > do you mean that the job cleanup is supposed to clean the directories within “spill” ?
My setting was already set to 5 days, but it seems the job cleanup does not clean some of the directories created before this delay.

What would be the impact deleting these directories that Dremio has put in this “spill” directory ?