Hello,
Im trying to backup my dremio server with the dremio-admin command but the volume of the backup is extremely heavy. Mainly because of profiles:
-rw-r–r-- 1 root root 47185 May 4 08:18 acceleration_materialization_backup.json
-rw-r–r-- 1 root root 512 May 4 08:18 acceleration_materialization_info.json
-rw-r–r-- 1 root root 133928484864 May 4 09:13 profiles_backup.json
-rw-r–r-- 1 root root 650 May 4 08:18 provisioning_backup.json
-rw-r–r-- 1 root root 376 May 4 08:18 provisioning_info.json
-rw-r–r-- 1 root root 2379 May 4 08:18 roles_backup.json
-rw-r–r-- 1 root root 340 May 4 08:18 roles_info.json
Before doing this backup I stopped my server and did a clean with all options and only 8 days retention for jobs and profiles :
profiles basic rocks store stats
* Estimated Number of Keys: 1157818
* Estimated Live Data Size: 19751883693
* Total SST files size: 19751883693
* Pending Compaction Bytes: 0
jobs basic rocks store stats
* Estimated Number of Keys: 1157823
* Estimated Live Data Size: 556538976
* Total SST files size: 1220500846
* Pending Compaction Bytes: 0
Any idea how can I make this work ? In the current state, I can’t do backup because the process fill a terabyte of filesystem and ultimately fail.
@kelly,
I also have tried to backup but its take too much time due to huge size for Catalog ,in mine cased coomand itself hangs i have triyed all coomand
./dremio-admin clean -j
./dremio-admin clean -c
./dremio-admin clean ```
-i
Dremio 4.0 is creating huge backup. I ran the cleanup commands the db folder size comes to about 20GB. But the backup folder size is huge, the profiles_backup.json in the backup folder is more than 50GB in size. Attached the outputs of some of the clean commands.
How do we debug this issue?
/opt/dremio/bin/dremio-admin clean -j 1
/opt/dremio/bin/dremio-admin clean -i
/opt/dremio/bin/dremio-admin clean -c
/opt/dremio/bin/dremio-admin clean -o
Initial Store Status.
reflection_goals
basic rocks store stats
* Estimated Number of Keys: 195
* Estimated Live Data Size: 57982
* Total SST files size: 59656
* Pending Compaction Bytes: 0
Index Stats
* live records: 194
* deleted records: 0
refresh_requests
basic rocks store stats
* Estimated Number of Keys: 82
* Estimated Live Data Size: 5257
* Total SST files size: 6396
* Pending Compaction Bytes: 0
userGroup
basic rocks store stats
* Estimated Number of Keys: 9
* Estimated Live Data Size: 2879
* Total SST files size: 4125
* Pending Compaction Bytes: 0
Index Stats
* live records: 9
* deleted records: 0
catalog-source-data
basic rocks store stats
* Estimated Number of Keys: 95
* Estimated Live Data Size: 1999
* Total SST files size: 6914
* Pending Compaction Bytes: 0
datasetVersions
basic rocks store stats
* Estimated Number of Keys: 3362
* Estimated Live Data Size: 7174688
* Total SST files size: 7211878
* Pending Compaction Bytes: 0
tokens
basic rocks store stats
* Estimated Number of Keys: 15
* Estimated Live Data Size: 2604
* Total SST files size: 11073
* Pending Compaction Bytes: 0
profiles
basic rocks store stats
* Estimated Number of Keys: 448037
* Estimated Live Data Size: 15729323964
* Total SST files size: 17794129611
* Pending Compaction Bytes: 0
collaboration_wiki
basic rocks store stats
* Estimated Number of Keys: 7
* Estimated Live Data Size: 7303
* Total SST files size: 7303
* Pending Compaction Bytes: 0
Index Stats
* live records: 7
* deleted records: 0
identity
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
materialization_store
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 1568509
* Total SST files size: 4047576
* Pending Compaction Bytes: 0
Index Stats
* live records: 176
* deleted records: 0
metadata-dataset-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 5655740
* Total SST files size: 15474884
* Pending Compaction Bytes: 0
Index Stats
* live records: 3382
* deleted records: 0
external_reflection_store
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
configuration
basic rocks store stats
* Estimated Number of Keys: 3
* Estimated Live Data Size: 1118
* Total SST files size: 2106
* Pending Compaction Bytes: 0
dependencies
basic rocks store stats
* Estimated Number of Keys: 324
* Estimated Live Data Size: 16937
* Total SST files size: 33055
* Pending Compaction Bytes: 0
jobs
basic rocks store stats
* Estimated Number of Keys: 448199
* Estimated Live Data Size: 1580896807
* Total SST files size: 1872649941
* Pending Compaction Bytes: 0
Index Stats
* live records: 448197
* deleted records: 0
sys.options
basic rocks store stats
* Estimated Number of Keys: 2
* Estimated Live Data Size: 2094
* Total SST files size: 2094
* Pending Compaction Bytes: 0
dac-namespace
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 654843
* Total SST files size: 1369428
* Pending Compaction Bytes: 0
* Estimated Blob Count: 0
* Estimated Blob Bytes: 0
Index Stats
* live records: 1344
* deleted records: 0
collaboration_tags
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
refresh_store
basic rocks store stats
* Estimated Number of Keys: 170
* Estimated Live Data Size: 87521
* Total SST files size: 101201
* Pending Compaction Bytes: 0
Index Stats
* live records: 170
* deleted records: 0
provisioning
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
reflection_settings
basic rocks store stats
* Estimated Number of Keys: 81
* Estimated Live Data Size: 3352
* Total SST files size: 5509
* Pending Compaction Bytes: 0
reflection_entries
basic rocks store stats
* Estimated Number of Keys: 198
* Estimated Live Data Size: 27449
* Total SST files size: 33446
* Pending Compaction Bytes: 0
upgrade
basic rocks store stats
* Estimated Number of Keys: 10
* Estimated Live Data Size: 1776
* Total SST files size: 1776
* Pending Compaction Bytes: 0
metadata-multi-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Final Store Status.
reflection_goals
basic rocks store stats
* Estimated Number of Keys: 194
* Estimated Live Data Size: 58542
* Total SST files size: 58542
* Pending Compaction Bytes: 0
Index Stats
* live records: 194
* deleted records: 0
refresh_requests
basic rocks store stats
* Estimated Number of Keys: 79
* Estimated Live Data Size: 5260
* Total SST files size: 5260
* Pending Compaction Bytes: 0
userGroup
basic rocks store stats
* Estimated Number of Keys: 9
* Estimated Live Data Size: 2185
* Total SST files size: 2185
* Pending Compaction Bytes: 0
Index Stats
* live records: 9
* deleted records: 0
catalog-source-data
basic rocks store stats
* Estimated Number of Keys: 36
* Estimated Live Data Size: 1946
* Total SST files size: 1946
* Pending Compaction Bytes: 0
datasetVersions
basic rocks store stats
* Estimated Number of Keys: 3362
* Estimated Live Data Size: 7211098
* Total SST files size: 7211098
* Pending Compaction Bytes: 0
tokens
basic rocks store stats
* Estimated Number of Keys: 17
* Estimated Live Data Size: 1762
* Total SST files size: 1762
* Pending Compaction Bytes: 0
profiles
basic rocks store stats
* Estimated Number of Keys: 448058
* Estimated Live Data Size: 11327301174
* Total SST files size: 11327301174
* Pending Compaction Bytes: 0
collaboration_wiki
basic rocks store stats
* Estimated Number of Keys: 7
* Estimated Live Data Size: 7303
* Total SST files size: 7303
* Pending Compaction Bytes: 0
Index Stats
* live records: 7
* deleted records: 0
identity
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
materialization_store
basic rocks store stats
* Estimated Number of Keys: 176
* Estimated Live Data Size: 806582
* Total SST files size: 806582
* Pending Compaction Bytes: 0
Index Stats
* live records: 176
* deleted records: 0
metadata-dataset-splits
basic rocks store stats
* Estimated Number of Keys: 3382
* Estimated Live Data Size: 1439681
* Total SST files size: 1439681
* Pending Compaction Bytes: 0
Index Stats
* live records: 3382
* deleted records: 0
external_reflection_store
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
configuration
basic rocks store stats
* Estimated Number of Keys: 2
* Estimated Live Data Size: 1118
* Total SST files size: 1118
* Pending Compaction Bytes: 0
dependencies
basic rocks store stats
* Estimated Number of Keys: 162
* Estimated Live Data Size: 16952
* Total SST files size: 16952
* Pending Compaction Bytes: 0
jobs
basic rocks store stats
* Estimated Number of Keys: 448197
* Estimated Live Data Size: 912453993
* Total SST files size: 912453993
* Pending Compaction Bytes: 0
Index Stats
* live records: 448197
* deleted records: 0
sys.options
basic rocks store stats
* Estimated Number of Keys: 2
* Estimated Live Data Size: 2094
* Total SST files size: 2094
* Pending Compaction Bytes: 0
dac-namespace
basic rocks store stats
* Estimated Number of Keys: 1344
* Estimated Live Data Size: 579419
* Total SST files size: 579419
* Pending Compaction Bytes: 0
* Estimated Blob Count: 0
* Estimated Blob Bytes: 0
Index Stats
* live records: 1344
* deleted records: 0
collaboration_tags
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
refresh_store
basic rocks store stats
* Estimated Number of Keys: 170
* Estimated Live Data Size: 42776
* Total SST files size: 42776
* Pending Compaction Bytes: 0
Index Stats
* live records: 170
* deleted records: 0
provisioning
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
reflection_settings
basic rocks store stats
* Estimated Number of Keys: 80
* Estimated Live Data Size: 2460
* Total SST files size: 2460
* Pending Compaction Bytes: 0
reflection_entries
basic rocks store stats
* Estimated Number of Keys: 165
* Estimated Live Data Size: 27476
* Total SST files size: 27476
* Pending Compaction Bytes: 0
upgrade
basic rocks store stats
* Estimated Number of Keys: 10
* Estimated Live Data Size: 1776
* Total SST files size: 1776
* Pending Compaction Bytes: 0
metadata-multi-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
indent preformatted text by 4 spaces
@Vikash_Singh Your RocksDB looks clean, in Dremio 4.0 it was a JSON backup and used to take a lot of space. To avoid this we switched to a binary backup (and use parallel thread) which should reduce the backup size by a significant size and also make the backup faster. This was brought in 4.1.4, see 4.1.4 release notes. Kindly upgrade and validate
