# java.io.IOException: org.rocksdb.RocksDBException:No space left on device **URL:** \ **Category:** Uncategorized\ **Created:** [October 11, 2018, 9:24pm UTC](https://community.dremio.com/t/java-io-ioexception-org-rocksdb-rocksdbexception-no-space-left-on-device/2007 "2018-10-11T21:24:18Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![George\_Zoto](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/george_zoto/32/542_2.png) [@George\_Zoto](https://community.dremio.com/u/George_Zoto)\ **Post date:** [October 11, 2018, 9:24pm UTC](https://community.dremio.com/t/java-io-ioexception-org-rocksdb-rocksdbexception-no-space-left-on-device/2007/1 "2018-10-11T21:24:18Z")
Hello Dremio team, I am running Dremio on a single EC2 m5.large instance with 8GB of RAM. I was trying to read a 150MB xlsx file from a S3 bucket and upon reading it I got the following errors and exceptions. Currently I cannot even start Dremio and all my data is inaccessible ☹ . I tried rebooting the EC2 instance and even stop and starting it from the AWS console but every time I try to start Dremio, I get the same error. Any idea how I might at least start Dremio 🐬 ? Thank you, George Errors/Exceptions from /var/log/dremio/server.out ```auto hu Oct 11 21:15:22 UTC 2018 Starting dremio on ip-x-y-z-w core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 30658 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 30658 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited KVStore version is 2.1.6-201809161906440178-edb5b4d Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting. java.io.IOException: org.rocksdb.RocksDBException: While appending to file: /var/lib/dremio/db/catalog/000178.sst: No space left on device at com.dremio.datastore.RocksDBStore.exclusively(RocksDBStore.java:373) at com.dremio.datastore.RocksDBStore.close(RocksDBStore.java:335) at com.dremio.datastore.ByteStoreManager$2.onRemoval(ByteStoreManager.java:83) at com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:1963) at com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3562) at com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3538) at com.google.common.cache.LocalCache$Segment.clear(LocalCache.java:3309) at com.google.common.cache.LocalCache.clear(LocalCache.java:4322) at com.google.common.cache.LocalCache$LocalManualCache.invalidateAll(LocalCache.java:4937) at com.dremio.datastore.ByteStoreManager.close(ByteStoreManager.java:250) at com.dremio.common.AutoCloseables.close(AutoCloseables.java:92) at com.dremio.common.AutoCloseables.close(AutoCloseables.java:71) at com.dremio.datastore.CoreStoreProviderImpl.close(CoreStoreProviderImpl.java:197) at com.dremio.datastore.LocalKVStoreProvider.close(LocalKVStoreProvider.java:152) at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:172) at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:103) Caused by: org.rocksdb.RocksDBException: While appending to file: /var/lib/dremio/db/catalog/000178.sst: No space left on device at org.rocksdb.RocksDB.flush(Native Method) at org.rocksdb.RocksDB.flush(RocksDB.java:1760) at com.dremio.datastore.RocksDBStore.lambda$close$2(RocksDBStore.java:339) at com.dremio.datastore.RocksDBStore.exclusively(RocksDBStore.java:365) ... 15 more ``` ---
**Author:** ![anthony](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/anthony/32/670_2.png) [@anthony](https://community.dremio.com/u/anthony)\ **Post date:** [October 14, 2018, 12:56am UTC](https://community.dremio.com/t/java-io-ioexception-org-rocksdb-rocksdbexception-no-space-left-on-device/2007/2 "2018-10-14T00:56:38Z")
You can try our metadata cleanup to free some space - [https://docs.dremio.com/advanced-administration/metadata-cleanup.html](https://docs.dremio.com/advanced-administration/metadata-cleanup.html) ---
**Author:** ![kelly](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kelly/32/11_2.png) [@kelly](https://community.dremio.com/u/kelly)\ **Post date:** [October 15, 2018, 1:27pm UTC](https://community.dremio.com/t/java-io-ioexception-org-rocksdb-rocksdbexception-no-space-left-on-device/2007/3 "2018-10-15T13:27:08Z")
Looks like you are out of space on the local file system where /var/lib/dremio/db/catalog/ is mounted. Can you allocate some more space there? ---
**Author:** ![George\_Zoto](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/george_zoto/32/542_2.png) [@George\_Zoto](https://community.dremio.com/u/George_Zoto)\ **Post date:** [October 22, 2018, 8:41pm UTC](https://community.dremio.com/t/java-io-ioexception-org-rocksdb-rocksdbexception-no-space-left-on-device/2007/4 "2018-10-22T20:41:19Z")
Hi @Anthony, Thank you for this information. I tried all these options but none of them worked. I think me also deleting a file as I am sharing below might have something to do with it. I ended up reinstalling Dremio and loosing all my data. I am sure I can replicate this issue by reading a large file (~150MB) so I can try the admin commands once more. ```auto sudo /opt/dremio/bin/dremio-admin clean -j Failed to complete cleanup. com.beust.jcommander.ParameterException: Expected a value after parameter -j ``` ```auto sudo /opt/dremio/bin/dremio-admin clean --max-job-days Failed to complete cleanup. com.beust.jcommander.ParameterException: Expected a value after parameter --max-job-days ``` ```auto sudo /opt/dremio/bin/dremio-admin clean -I 018-10-12 14:21:31,117 [main] INFO c.d.common.scanner.BuildTimeScan - Loaded prescanned packages [com.dremio.storage ... Failed to complete cleanup. org.rocksdb.RocksDBException: Sst file size mismatch: /var/lib/dremio/db/catalog/000134.sst. Size recorded in manifest 2527531, actual size 0 ``` At this point I think I deleted file /var/lib/dremio/db/catalog/000134.sst thinking that it was the culprit. From the message below, I was wrong: ```auto sudo /opt/dremio/bin/dremio-admin clean --reindex-data Failed to complete cleanup. org.rocksdb.RocksDBException: Sst file size mismatch: /var/lib/dremio/db/catalog/000134.sst. Size recorded in manifest 2527531, actual size 0 ``` Thank you for your feedback, George ---
**Author:** ![George\_Zoto](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/george_zoto/32/542_2.png) [@George\_Zoto](https://community.dremio.com/u/George_Zoto)\ **Post date:** [October 22, 2018, 8:46pm UTC](https://community.dremio.com/t/java-io-ioexception-org-rocksdb-rocksdbexception-no-space-left-on-device/2007/5 "2018-10-22T20:46:55Z")
Hi @kelly, That’s what I thought as well but given that my input file was ~150MB and I had 8GB of space left I could not see how adding more space would help. Is there a way to confirm on which partition is `/var/lib/dremio/db/catalog/` mounted on? I would imagine one of the 2 main operations below: ```auto df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 3.8G 44K 3.8G 1% /dev tmpfs 3.8G 0 3.8G 0% /dev/shm /dev/nvme0n1p1 7.8G 7.2G 518M 94% / ``` I ended up reinstalling Dremio and loosing all my data. I am sure I can replicate this issue by reading a large file (~150MB) but I want to confirm where is the`catalog` folder mounted on first. Thank you again, George