Hi Team,
what is the default spill directory used by dremio?
is it along with master node local data directory?
or executor nodes local data directory?
or distributed storage used for reflections? or can be anywhere?
is there a way to specify a custom spill directory?
Thanks
Spill is usually on the executors and local disks, If it is yarn based you can do it via admin-provisioning-edit and specify under “Spill Directories”, see attached screenshot
If your Dremio executors are on non-yarn then you have to specify via dremio.conf
spilling: [${paths.local}/spill]
