Dremio server disk management

1

Hi!

I’ve been learning Dremio disk management and admnistrative tools. In my production environment, the coordinator node full very quicly. I already grow the vm’s sdd disk but the problem persists until I searched about “dremio-admin clean -o” (to purge orphaned metadata). My disk space dows from 100GB to 61GB after run command. Now on every two day I have to run this command to retrieve disk space, this is not seems fine.

Analysing the dremio installation directories, I discover db folder that equivalent to all space used on my disk

dremio db folder 60GB
dremio db folder 60GB660×535 11.9 KB

Catalog is the largest folder in db folder, what we can about it? Can we move to another place, like AWS S3?

dremio catalog folder 59Gb

UPDATE:
I see a large amount of .sst, .log and LOG.old what are these files? Are they really necessary?
How can I make safe delete of LOG.old.* files?

Thanks in advance

2

@Diego

.sst and .log files are transaction log files so please do not remove them

With the current version, most of the metadata has been moved off the local disk to the distributed store like S3, what version of Dremio are you on?

3

@balaji.ramaswamy we’re in this version on AWS Cloud

dremio_version
dremio_version811×645 43 KB

4

@Diego I would like to see what is using up all the space in your Catalog, can you please run the below command vi REST, need to change your authorization ID

curl --location --request GET 'localhost:9047/apiv2/kvstore/report?store=none' \
--header 'Authorization: _dremioirr8hj6qnpc3tfr3omiqvev51c' > kvstore_summary.zip

Also, can you please query any Parquet dataset and send me the job profile

5

@balaji.ramaswamy take these files:
kvstore_summary.zip (2,0,KB)
8a5b7981-fe6e-4602-bca2-191c63a1f38a.zip (20,0,KB)

6

@Diego The profiles+jobs are using up 60 GB of space. What is the total volume size where the rocksdb is stoed?

7

@balaji.ramaswamy the disk have up to 100 GB

8

@Diego You are using 60% of the total disk space for jobs and profiles, you have 2 options

  • Decrease jobs.max.age_in_days to something like 14 (default is 30 days)

  • Increase disk volume to account for the recover log files needed for rocksdb (say make it 150 GB)

  • Can you also make sure planner.verbose_profile is false

Thanks
Bali

1 Like
9

@balaji.ramaswamy

  • Decrease jobs.max.age_in_days to something like 14 (default is 30 days)
    sure, I will to reduce to 7

  • Increase disk volume to account for the recover log files needed for rocksdb (say make it 150 GB)
    This will be the last resource

  • Can you also make sure planner.verbose_profile is false
    This is already set to false

Thanks Bali

10

@balaji.ramaswamy when I decrease jobs.max.age_in_days to 7 worked fine. Thanks again!

11

@Diego Glad it works, welcome