IOException: No space left on device on dremio

1

I am getting the below error when I ran a query. I did a df-m on the master server and I don’t see anything full. Can someone please let me what its running out of and how I can solve it ?
I have pasted the output of df -m at the end of the error message

  SYSTEM ERROR: IOException: No space left on device

SqlOperatorImpl ARROW_WRITER
Location 1:4:1
Fragment 1:0

[Error Id: 1dcedbde-360f-4d24-98f9-2f2d72f2432d on ip-10-0-0-22.ec2.internal:-1]

(java.lang.RuntimeException) java.io.IOException: No space left on device
com.dremio.exec.cache.VectorAccessibleSerializable.writeToStream():333
com.dremio.exec.store.easy.arrow.ArrowRecordWriter.writeBatch():131
com.dremio.sabot.op.writer.WriterOperator.consumeData():131
com.dremio.sabot.driver.SmartOp$SmartSingleInput.consumeData():240
com.dremio.sabot.driver.StraightPipe.pump():59
com.dremio.sabot.driver.Pipeline.doPump():109
com.dremio.sabot.driver.Pipeline.pumpOnce():99
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():320
com.dremio.sabot.exec.fragment.FragmentExecutor.run():273
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():87
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():658
com.dremio.sabot.task.AsyncTaskWrapper.run():104
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226
com.dremio.sabot.task.slicing.SlicingThread.run():156
Caused By (java.io.IOException) No space left on device
java.io.FileOutputStream.writeBytes():-2
java.io.FileOutputStream.write():326
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write():273
java.io.BufferedOutputStream.flushBuffer():82
java.io.BufferedOutputStream.write():121
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write():57
java.io.DataOutputStream.write():107
com.dremio.exec.hadoop.FSDataOutputStreamWrapper.write():58
com.dremio.io.FilterFSOutputStream.write():65
com.dremio.exec.hadoop.FSDataOutputStreamWithStatsWrapper.write():53
java.io.DataOutputStream.write():107
com.google.protobuf.CodedOutputStream.refreshBuffer():833
com.google.protobuf.CodedOutputStream.writeRawByte():892
com.google.protobuf.CodedOutputStream.writeRawByte():900
com.google.protobuf.CodedOutputStream.writeRawVarint32():1009
com.google.protobuf.CodedOutputStream.writeTag():994
com.google.protobuf.CodedOutputStream.writeMessage():221
com.dremio.exec.proto.UserBitShared$SerializedField.writeTo():11880
com.google.protobuf.CodedOutputStream.writeMessageNoTag():380
com.google.protobuf.CodedOutputStream.writeMessage():222
com.dremio.exec.proto.UserBitShared$SerializedField.writeTo():11886
com.google.protobuf.CodedOutputStream.writeMessageNoTag():380
com.google.protobuf.CodedOutputStream.writeMessage():222
com.dremio.exec.proto.UserBitShared$RecordBatchDef.writeTo():10013
com.google.protobuf.AbstractMessageLite.writeDelimitedTo():90
com.dremio.exec.cache.VectorAccessibleSerializable.writeToStream():308
com.dremio.exec.store.easy.arrow.ArrowRecordWriter.writeBatch():131
com.dremio.sabot.op.writer.WriterOperator.consumeData():131
com.dremio.sabot.driver.SmartOp$SmartSingleInput.consumeData():240
com.dremio.sabot.driver.StraightPipe.pump():59
com.dremio.sabot.driver.Pipeline.doPump():109
com.dremio.sabot.driver.Pipeline.pumpOnce():99
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():320
com.dremio.sabot.exec.fragment.FragmentExecutor.run():273
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():87
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():658
com.dremio.sabot.task.AsyncTaskWrapper.run():104
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226
com.dremio.sabot.task.slicing.SlicingThread.run():156

Output of df -m

[ec2-user@ip-10-0-0-23 dremio]$ df -m
Filesystem 1M-blocks Used Available Use% Mounted on
devtmpfs 15752 0 15752 0% /dev
tmpfs 15769 0 15769 0% /dev/shm
tmpfs 15769 1 15769 1% /run
tmpfs 15769 0 15769 0% /sys/fs/cgroup
/dev/nvme0n1p1 8180 3276 4905 41% /
tmpfs 3154 0 3154 0% /run/user/1000

2

Hello @rajupillai

Can you share the query profile and dremio.conf file?

Thanks,
@Rakesh_Malugu

3

here is the content of the dremio.conf and attaches I the profile

paths: {
  # the local path for dremio to store data.
  local: "/var/lib/dremio"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  dist: "pdfs://"${paths.local}"/pdfs"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: false
}

cd527a44-d962-4122-8fea-23680f2eb4ca.zip (73.0 KB)

4

did you get a chance to look into my logs.

5

@rajupillai,

Try this and let us know if this fixes the issue

  • Shutdown executors
  • Shutdown coordinator
  • vi dremio.conf on the coordinator, and add below line (above services but outside “}” block)
    {noformat}
    paths.results: “pdfs://”${paths.local}"/data/results"
    {noformat}
  • Restart coordinator
  • Launch executors

Note: We have a patch where we push paths.results from coordinator to all the executors, also the results should map to the larger ephemeral disk automatically so you do not run out