Error on run dremio-admin compact

1

Hi to All!

When I use the compact administrative tool (dremio-admin -c) at the end of process get the following message:

Compacting profiles…

Failed to complete cleanup.
java.io.IOException: org.rocksdb.RocksDBException: While appending to file: /var /lib/dremio/db/catalog/044815.sst: No space left on device
Suppressed: java.lang.RuntimeException: org.rocksdb.RocksDBException: Wh ile appending to file: /var/lib/dremio/db/catalog/044815.sst: No space left on d evice
Suppressed: java.lang.RuntimeException: org.rocksdb.RocksDBExcep tion: While appending to file: /var/lib/dremio/db/catalog/044815.sst: No space l eft on device
Caused by: org.rocksdb.RocksDBException: While appending to file : /var/lib/dremio/db/catalog/044815.sst: No space left on device
Suppressed: java.lang.RuntimeException: org.rocksdb.RocksDBExcep tion: While appending to file: /var/lib/dremio/db/catalog/044815.sst: No space l eft on device
Caused by: org.rocksdb.RocksDBException: While appending to file : /var/lib/dremio/db/catalog/044815.sst: No space left on device
Suppressed: java.lang.RuntimeException: org.rocksdb.RocksDBExcep tion: While appending to file: /var/lib/dremio/db/catalog/044815.sst: No space l eft on device
Caused by: org.rocksdb.RocksDBException: While appending to file : /var/lib/dremio/db/catalog/044815.sst: No space left on device
Suppressed: java.lang.RuntimeException: org.rocksdb.RocksDBExcep tion: While appending to file: /var/lib/dremio/db/catalog/044815.sst: No space l eft on device
Caused by: org.rocksdb.RocksDBException: While appending to file : /var/lib/dremio/db/catalog/044815.sst: No space left on device
Suppressed: java.io.IOException: org.rocksdb.RocksDBException: W hile appending to file: /var/lib/dremio/db/catalog/044815.sst: No space left on device
Caused by: org.rocksdb.RocksDBException: While appending to file : /var/lib/dremio/db/catalog/044815.sst: No space left on device
Caused by: org.rocksdb.RocksDBException: While appending to file: /var/l ib/dremio/db/catalog/044815.sst: No space left on device
Caused by: org.rocksdb.RocksDBException: While appending to file: /var/lib/dremi o/db/catalog/044815.sst: No space left on device

In my environment, on master node, I have this total amount of space

dremio-space

How much more space would I need for the task to complete successfully?

We’re in this version on AWS Cloud

dremio_version
dremio_version811×645 43 KB

2

@Diego When you run the compact command, Dremio would actually need extra free storage temporarily when the command is run as it is trying to move files around. Just like when you do gzip and then it will release the space. The free space available is only 24 GB and hence running out of disk space

Can you run the below API and send the output zip file which should tell us what is using up all the space in rocksdb?

curl --location --request GET ‘localhost:9047/apiv2/kvstore/report?store=none’
–header ‘Authorization: _dremioirr8hj6qnpc3tfr3omiqvev51c’ > kvstore_summary.zip

  • The authorization code needs to be replaced with yours
3

@balaji.ramaswamy the environment same of these

4

@Diego Just answered in Dremio server disk management - #8 by balaji.ramaswamy

5

@Diego By the way the -o is not having any effect, the restart which is needed for -o is what is clearing the files. I looked at your report and the only major chunk is jobs/profiles and not orphan splits (which is why -o is needed)

Can you query any dataset and attach the profile, I want to review your non-default parameters

6

@balaji.ramaswamy sure, it’s here in your hands. Thanks in advance Bali

2d0ddd13-bf94-4665-b4ed-ee4f09b7809f.zip (862,8,KB)