Hi,
I have migrated Dremio from 3.1 to 4.0.2 last week .All looks good but only worried if i see disk Usage here is the details disk
what Causing issue to Full 250GB Disk to 96%?
on that folder, can you please do the below and send us the outputs?
df -h .
du -sh *
cd db
du -sh *
Shutdown coordinator
cd /bin
./dremio-admin clean
Send us the above output also
Thanks
@balaji.ramaswamy
Hello @balaji.ramaswamy,
Here is screen_shots for above command
Here is Output for ./dremio-admin clean command
dremio@dremio-admin:/opt/dremio/bin$ ./dremio-admin clean
No operation requested, printing store Stats.
refresh_requests
basic rocks store stats
* Estimated Number of Keys: 96
* Estimated Live Data Size: 5959
* Total SST files size: 8996
* Pending Compaction Bytes: 0
reflection_goals
basic rocks store stats
* Estimated Number of Keys: 323
* Estimated Live Data Size: 123983
* Total SST files size: 130207
* Pending Compaction Bytes: 0
Index Stats
* live records: 151
* deleted records: 23
userGroup
basic rocks store stats
* Estimated Number of Keys: 3
* Estimated Live Data Size: 2343
* Total SST files size: 2343
* Pending Compaction Bytes: 0
Index Stats
* live records: 3
* deleted records: 0
catalog-source-data
basic rocks store stats
* Estimated Number of Keys: 52
* Estimated Live Data Size: 1588
* Total SST files size: 5756
* Pending Compaction Bytes: 0
datasetVersions
basic rocks store stats
* Estimated Number of Keys: 23150
* Estimated Live Data Size: 130524326
* Total SST files size: 130901031
* Pending Compaction Bytes: 130901031
tokens
basic rocks store stats
* Estimated Number of Keys: 7
* Estimated Live Data Size: 2279
* Total SST files size: 4654
* Pending Compaction Bytes: 0
profiles
basic rocks store stats
* Estimated Number of Keys: 4190074
* Estimated Live Data Size: 59519882390
* Total SST files size: 70486552876
* Pending Compaction Bytes: 481271359
collaboration_wiki
basic rocks store stats
* Estimated Number of Keys: 3
* Estimated Live Data Size: 5221
* Total SST files size: 5221
* Pending Compaction Bytes: 0
Index Stats
* live records: 3
* deleted records: 0
identity
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
materialization_store
basic rocks store stats
* Estimated Number of Keys: 5991
* Estimated Live Data Size: 32647196
* Total SST files size: 32647196
* Pending Compaction Bytes: 0
Index Stats
* live records: 5991
* deleted records: 75
metadata-dataset-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 302162
* Total SST files size: 1167472
* Pending Compaction Bytes: 1167472
Index Stats
* live records: 16291
* deleted records: 0
external_reflection_store
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
configuration
basic rocks store stats
* Estimated Number of Keys: 6
* Estimated Live Data Size: 3277
* Total SST files size: 5269
* Pending Compaction Bytes: 5269
dependencies
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 1710
* Total SST files size: 3357
* Pending Compaction Bytes: 0
jobs
basic rocks store stats
* Estimated Number of Keys: 4190938
* Estimated Live Data Size: 2529555454
* Total SST files size: 2739722501
* Pending Compaction Bytes: 0
Index Stats
* live records: 4190607
* deleted records: 1390857
sys.options
basic rocks store stats
* Estimated Number of Keys: 1
* Estimated Live Data Size: 7259
* Total SST files size: 7259
* Pending Compaction Bytes: 0
dac-namespace
basic rocks store stats
* Estimated Number of Keys: 3934
* Estimated Live Data Size: 1809040
* Total SST files size: 2658260
* Pending Compaction Bytes: 0
* Estimated Blob Count: 0
* Estimated Blob Bytes: 0
Index Stats
* live records: 2975
* deleted records: 311
collaboration_tags
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
refresh_store
basic rocks store stats
* Estimated Number of Keys: 5830
* Estimated Live Data Size: 1464632
* Total SST files size: 1473245
* Pending Compaction Bytes: 0
Index Stats
* live records: 6047
* deleted records: 77
provisioning
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
reflection_entries
basic rocks store stats
* Estimated Number of Keys: 261
* Estimated Live Data Size: 43595
* Total SST files size: 43595
* Pending Compaction Bytes: 0
reflection_settings
basic rocks store stats
* Estimated Number of Keys: 77
* Estimated Live Data Size: 4186
* Total SST files size: 6173
* Pending Compaction Bytes: 0
upgrade
basic rocks store stats
* Estimated Number of Keys: 18
* Estimated Live Data Size: 2401
* Total SST files size: 2401
* Pending Compaction Bytes: 0
metadata-multi-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Here are the main contributors
Profiles: 55 GB
Jobs: 2.35 GB
For this if you are on the latest version, set the below via support key
jobs.max.age_in_days - defaults to 30 days in later versions and unlimited in old versions. It looks like you are on an older version (this parameter might not be there) then you may have to shutdown Dremio and do a "dremio-admin clean -j "
I also see your pdfs is 35 GB, the only folder you can remove files under is “results”
In later versions you can set the below parameter “results.max.age_in_days” defaults to 1 day. on Older version, just remove folders under the results folder (keep the results folder). By removing all the folders under results, you cannot use the “open results” feature (on the top right corner) when you click on a job
Kindly let me know if you have any other questions
Thanks
@balaji.ramaswamy
Thanks @balaji.ramaswamy it helps
@balaji.ramaswamy
again i am facing Disk full issue alomst 135 GB RockDB size ,i also cant able to clean jobs and profile here is clean job coomand result
reflection_goals
basic rocks store stats
* Estimated Number of Keys: 476
* Estimated Live Data Size: 161506
* Total SST files size: 161506
* Pending Compaction Bytes: 0
Index Stats
* live records: 286
* deleted records: 3
refresh_requests
basic rocks store stats
* Estimated Number of Keys: 140
* Estimated Live Data Size: 8442
* Total SST files size: 8442
* Pending Compaction Bytes: 0
userGroup
basic rocks store stats
* Estimated Number of Keys: 3
* Estimated Live Data Size: 2343
* Total SST files size: 2343
* Pending Compaction Bytes: 0
Index Stats
* live records: 3
* deleted records: 0
catalog-source-data
basic rocks store stats
* Estimated Number of Keys: 21
* Estimated Live Data Size: 1681
* Total SST files size: 1681
* Pending Compaction Bytes: 0
datasetVersions
basic rocks store stats
* Estimated Number of Keys: 27772
* Estimated Live Data Size: 173295413
* Total SST files size: 173295413
* Pending Compaction Bytes: 0
tokens
basic rocks store stats
* Estimated Number of Keys: 13
* Estimated Live Data Size: 1546
* Total SST files size: 1546
* Pending Compaction Bytes: 0
profiles
basic rocks store stats
* Estimated Number of Keys: 6186065
* Estimated Live Data Size: 117701833616
* Total SST files size: 128619854764
* Pending Compaction Bytes: 1962966276
collaboration_wiki
basic rocks store stats
* Estimated Number of Keys: 3
* Estimated Live Data Size: 5221
* Total SST files size: 5221
* Pending Compaction Bytes: 0
Index Stats
* live records: 3
* deleted records: 0
materialization_store
basic rocks store stats
* Estimated Number of Keys: 12026
* Estimated Live Data Size: 34062859
* Total SST files size: 34563806
* Pending Compaction Bytes: 0
Index Stats
* live records: 6099
* deleted records: 10
identity
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
metadata-dataset-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 540492
* Total SST files size: 14960133
* Pending Compaction Bytes: 0
Index Stats
* live records: 2335
* deleted records: 16006
external_reflection_store
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
configuration
basic rocks store stats
* Estimated Number of Keys: 3
* Estimated Live Data Size: 1124
* Total SST files size: 2120
* Pending Compaction Bytes: 0
dependencies
basic rocks store stats
* Estimated Number of Keys: 48
* Estimated Live Data Size: 7143
* Total SST files size: 7143
* Pending Compaction Bytes: 0
jobs
basic rocks store stats
* Estimated Number of Keys: 6187740
* Estimated Live Data Size: 9979333944
* Total SST files size: 10634308526
* Pending Compaction Bytes: 0
Index Stats
* live records: 6186519
* deleted records: 1442531
dac-namespace
basic rocks store stats
* Estimated Number of Keys: 5599
* Estimated Live Data Size: 2807183
* Total SST files size: 5317669
* Pending Compaction Bytes: 0
* Estimated Blob Count: 0
* Estimated Blob Bytes: 0
Index Stats
* live records: 3996
* deleted records: 289
sys.options
basic rocks store stats
* Estimated Number of Keys: 1
* Estimated Live Data Size: 7259
* Total SST files size: 7259
* Pending Compaction Bytes: 0
collaboration_tags
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
refresh_store
basic rocks store stats
* Estimated Number of Keys: 5947
* Estimated Live Data Size: 1479059
* Total SST files size: 1501356
* Pending Compaction Bytes: 0
Index Stats
* live records: 6131
* deleted records: 0
provisioning
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
reflection_settings
basic rocks store stats
* Estimated Number of Keys: 161
* Estimated Live Data Size: 3355
* Total SST files size: 6728
* Pending Compaction Bytes: 0
reflection_entries
basic rocks store stats
* Estimated Number of Keys: 279
* Estimated Live Data Size: 46719
* Total SST files size: 46719
* Pending Compaction Bytes: 0
upgrade
basic rocks store stats
* Estimated Number of Keys: 18
* Estimated Live Data Size: 2401
* Total SST files size: 2401
* Pending Compaction Bytes: 0
metadata-multi-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
i have also given clean jobs from coomand line but it hangs from past 2 hour,please guide me how to cleanup my Rocke DB its always incresing
i also see clean log generated with empty records
dremio@dremio-admin:/var/log/dremio$ ls -g
total 0
-rw-r–r-- 1 dremio 0 Jan 6 09:25 admin_clean_2020-01-06_09:25:37.log
-rw-r–r-- 1 dremio 0 Jan 6 09:29 admin_clean_2020-01-06_09:29:36.log
-rw-r–r-- 1 dremio 0 Jan 6 09:29 admin_clean_2020-01-06_09:29:59.log
-rw-r–r-- 1 dremio 0 Jan 6 09:29 server.out
here is output for dremio@dremio-admin:/opt/dremio/data/db$ du -sh *
8.0K blob
132G catalog
220K metadata
1.5G search
All the space is being consumed by the below 2. So Shutdown Dremio and run
dremio-admin clean -j
jobs - 10GB
Profiles - 120 GB
This is going to take time as you have lots of profiles and jobs. Maybe you can do it in batches of 10 days and try? Is your rocksDB on premier SSD’s? See table in below link for IOPS per disk. Which one do you have?
hello @balaji.ramaswamy i have given clean in 2 days interval and i able to clean Profile now i also have search folder which contain 2 gb how to clean this folder ,i dont want search data should i simply delete those folder
Do not delete “search” as Dremio will not start as it will break “Lucene Search”. Once you complete cleaning jobs in batches, you can do the below 3 to make it more compact. More like a DB reorg
dremio-admin clean -i (reindex)
dremio-admin clean -c (compact)
dremio-admin clean -o (clean orphans, happens in background too)
Thanks @balaji.ramaswamy yes thats works!