Failed to create directory for spilling

Dremio Cloud
1

Problem ‘Failed to create directory for spilling. Please check that the spill location is accessible and confirm read, write & execute permissions.’ occured in Dremio like AWS Service. Why occured this and how I can fixed?

image
image1059×628 38.9 KB

Job Profile Error:
DATA_WRITE ERROR: Failed to create directory for spilling. Please check that the spill location is accessible and confirm read, write & execute permissions.

Spill directory path /mnt/c1/spilling
Spill directory path /mnt/c1/spilling
Caller agg spilling
SqlOperatorImpl HASH_AGGREGATE
Location 13:2:2
SqlOperatorImpl HASH_AGGREGATE
Location 13:2:2
Fragment 13:0

[Error Id: fc79cbca-249f-4cd7-89a4-9374bcbb0d1a on 192.168.5.134:0]

Failure node: 192.168.21.228:31010

Error ID: 2b9b4b8b-5b8b-4bde-bc51-b8e8aebf78ea

Verbose:

DATA_WRITE ERROR: Failed to create directory for spilling. Please check that the spill location is accessible and confirm read, write & execute permissions.

Spill directory path /mnt/c1/spilling
Spill directory path /mnt/c1/spilling
Caller agg spilling
SqlOperatorImpl HASH_AGGREGATE
Location 13:2:2
SqlOperatorImpl HASH_AGGREGATE
Location 13:2:2
Fragment 13:0

[Error Id: fc79cbca-249f-4cd7-89a4-9374bcbb0d1a on 192.168.5.134:0]

Spill directory path /mnt/c1/spilling
Spill directory path /mnt/c1/spilling
Caller agg spilling
SqlOperatorImpl HASH_AGGREGATE
Location 13:2:2
SqlOperatorImpl HASH_AGGREGATE
Location 13:2:2
Fragment 13:0

2

@lucasft Replied on post Failed to create directory for spilling in Dremio like AWS Service - #2 by balaji.ramaswamy

3

Hi @balaji.ramaswamy
I am facing the same issue/error. In my set up I have spilling defined to multiple directories like /mnt/c1/spilling /mnt/c2/spilling /mnt/c3/spilling. However, when I am running my queries (Using Group BY and Order BY), it is failing with the error message “Failed to create directory for spilling. Please check that the spill location…”.
When checked the error message in Job Profile under the Error tab, it hints the Spill Directory to " /mnt/c2/spilling".
I checked the directory /mnt/c2/spilling and it has got the permissions to write, read and execute.
All my queries where Aggregate/Sort are being used are failing.
Can you please let me know what can be the issue and how to get it fixed.
Any help is really appreciated.

thanks,
Abhi

4

@abhi.mishra29 IS your C3 and spill are on the same disk?

5

Hi @balaji.ramaswamy
My Dremio set up is on single node. So, we have single mount /mnt and different disk.
My fstab have the entries like below: (Just a few portion of stab file)
/dev/nvme1n1 /mnt/c1 ext4
/dev/nvme2n1 /mnt/c2 ext4
/dev/nvme3n1 /mnt/c3 ext4

And output of df -Th is as below :
FIlesystem Type Size Used. Avail Use% Mount On
…
…
/dev/nvme1n1. ext4 1.5T … … 59%. /mnt/c1

6

@abhi.mishra29

Send us the job profile of the failed job

Thanks
Bali

7

Hi @balaji.ramaswamy
As said earlier, all the aggregate and order by queries are spilling to /mnt/c2. So, we decided to remove this folder and will have just /mnt/c1.
We made the changes in dremio.conf file under preview folder and in remaining script. However, whenever we are executing the query, the folder /mnt/c2 is getting created and the queries fail. We also restarted the dremio services by omitting the /mnt/c2 folder but in vain.
It looks like some start up script is creating this folder whenever there are Aggregate or Order BY queries. Can you help us in troubleshooting this cause.

8

@abhi.mishra29 Send us your dremio.conf with passwords masked, I would like to see paths to spill and C3