# IOException: No space left on device on dremio **URL:** \ **Category:** Uncategorized\ **Created:** [June 10, 2020, 4:18pm UTC](https://community.dremio.com/t/ioexception-no-space-left-on-device-on-dremio/5706 "2020-06-10T16:18:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![rajupillai](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@rajupillai](https://community.dremio.com/u/rajupillai)\ **Post date:** [June 10, 2020, 4:18pm UTC](https://community.dremio.com/t/ioexception-no-space-left-on-device-on-dremio/5706/1 "2020-06-10T16:18:41Z")
I am getting the below error when I ran a query. I did a df-m on the master server and I don’t see anything full. Can someone please let me what its running out of and how I can solve it ? I have pasted the output of df -m at the end of the error message ``` SYSTEM ERROR: IOException: No space left on device ``` SqlOperatorImpl ARROW\_WRITER Location 1:4:1 Fragment 1:0 [Error Id: 1dcedbde-360f-4d24-98f9-2f2d72f2432d on ip-10-0-0-22.ec2.internal:-1] (java.lang.RuntimeException) java.io.IOException: No space left on device com.dremio.exec.cache.VectorAccessibleSerializable.writeToStream():333 com.dremio.exec.store.easy.arrow.ArrowRecordWriter.writeBatch():131 com.dremio.sabot.op.writer.WriterOperator.consumeData():131 com.dremio.sabot.driver.SmartOp$SmartSingleInput.consumeData():240 com.dremio.sabot.driver.StraightPipe.pump():59 com.dremio.sabot.driver.Pipeline.doPump():109 com.dremio.sabot.driver.Pipeline.pumpOnce():99 com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():320 com.dremio.sabot.exec.fragment.FragmentExecutor.run():273 com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():87 com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():658 com.dremio.sabot.task.AsyncTaskWrapper.run():104 com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226 com.dremio.sabot.task.slicing.SlicingThread.run():156 Caused By (java.io.IOException) No space left on device java.io.FileOutputStream.writeBytes():-2 java.io.FileOutputStream.write():326 org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write():273 java.io.BufferedOutputStream.flushBuffer():82 java.io.BufferedOutputStream.write():121 org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write():57 java.io.DataOutputStream.write():107 com.dremio.exec.hadoop.FSDataOutputStreamWrapper.write():58 com.dremio.io.FilterFSOutputStream.write():65 com.dremio.exec.hadoop.FSDataOutputStreamWithStatsWrapper.write():53 java.io.DataOutputStream.write():107 com.google.protobuf.CodedOutputStream.refreshBuffer():833 com.google.protobuf.CodedOutputStream.writeRawByte():892 com.google.protobuf.CodedOutputStream.writeRawByte():900 com.google.protobuf.CodedOutputStream.writeRawVarint32():1009 com.google.protobuf.CodedOutputStream.writeTag():994 com.google.protobuf.CodedOutputStream.writeMessage():221 com.dremio.exec.proto.UserBitShared$SerializedField.writeTo():11880 com.google.protobuf.CodedOutputStream.writeMessageNoTag():380 com.google.protobuf.CodedOutputStream.writeMessage():222 com.dremio.exec.proto.UserBitShared$SerializedField.writeTo():11886 com.google.protobuf.CodedOutputStream.writeMessageNoTag():380 com.google.protobuf.CodedOutputStream.writeMessage():222 com.dremio.exec.proto.UserBitShared$RecordBatchDef.writeTo():10013 com.google.protobuf.AbstractMessageLite.writeDelimitedTo():90 com.dremio.exec.cache.VectorAccessibleSerializable.writeToStream():308 com.dremio.exec.store.easy.arrow.ArrowRecordWriter.writeBatch():131 com.dremio.sabot.op.writer.WriterOperator.consumeData():131 com.dremio.sabot.driver.SmartOp$SmartSingleInput.consumeData():240 com.dremio.sabot.driver.StraightPipe.pump():59 com.dremio.sabot.driver.Pipeline.doPump():109 com.dremio.sabot.driver.Pipeline.pumpOnce():99 com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():320 com.dremio.sabot.exec.fragment.FragmentExecutor.run():273 com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():87 com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():658 com.dremio.sabot.task.AsyncTaskWrapper.run():104 com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226 com.dremio.sabot.task.slicing.SlicingThread.run():156 Output of df -m [ec2-user@ip-10-0-0-23 dremio]$ df -m Filesystem 1M-blocks Used Available Use% Mounted on devtmpfs 15752 0 15752 0% /dev tmpfs 15769 0 15769 0% /dev/shm tmpfs 15769 1 15769 1% /run tmpfs 15769 0 15769 0% /sys/fs/cgroup /dev/nvme0n1p1 8180 3276 4905 41% / tmpfs 3154 0 3154 0% /run/user/1000 ---
**Author:** ![Rakesh\_Malugu](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rakesh_malugu/32/2384_2.png) [@Rakesh\_Malugu](https://community.dremio.com/u/Rakesh_Malugu)\ **Post date:** [June 11, 2020, 9:45am UTC](https://community.dremio.com/t/ioexception-no-space-left-on-device-on-dremio/5706/2 "2020-06-11T09:45:51Z")
Hello @rajupillai Can you share the query profile and dremio.conf file? Thanks, @Rakesh_Malugu ---
**Author:** ![rajupillai](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@rajupillai](https://community.dremio.com/u/rajupillai)\ **Post date:** [June 12, 2020, 7:10pm UTC](https://community.dremio.com/t/ioexception-no-space-left-on-device-on-dremio/5706/3 "2020-06-12T19:10:38Z")
here is the content of the dremio.conf and attaches I the profile ``` paths: { # the local path for dremio to store data. local: "/var/lib/dremio" # the distributed path Dremio data including job results, downloads, uploads, etc dist: "pdfs://"${paths.local}"/pdfs" } services: { coordinator.enabled: true, coordinator.master.enabled: true, executor.enabled: false } ``` [cd527a44-d962-4122-8fea-23680f2eb4ca.zip](https://community.dremio.com/uploads/short-url/rU27HCBd2M81UlmZqvO0R3zWvo5.zip) (73.0 KB) ---
**Author:** ![rajupillai](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@rajupillai](https://community.dremio.com/u/rajupillai)\ **Post date:** [June 16, 2020, 5:18pm UTC](https://community.dremio.com/t/ioexception-no-space-left-on-device-on-dremio/5706/4 "2020-06-16T17:18:01Z")
did you get a chance to look into my logs. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [June 22, 2020, 6:40am UTC](https://community.dremio.com/t/ioexception-no-space-left-on-device-on-dremio/5706/5 "2020-06-22T06:40:30Z")
@rajupillai, Try this and let us know if this fixes the issue - Shutdown executors - Shutdown coordinator - vi dremio.conf on the coordinator, and add below line (above services but outside “}” block) {noformat} paths.results: “pdfs://”${paths.local}"/data/results" {noformat} - Restart coordinator - Launch executors Note: We have a patch where we push paths.results from coordinator to all the executors, also the results should map to the larger ephemeral disk automatically so you do not run out