I am running the admin metadata cleanup commands. Dremio version 4.7 deployed on Kubernetes
On the master node 340 GB out of 500 GB is being used by the catalog folder.
But the compact command has been running for more than 4 hours. The log files created are all empty.
I can see the process is running, also /proc/<pid>/fd folder is showing which catalog files are being used by the process.
The commands are not reducing the disk usage.
Is it advisable to kill the process?
Output of jobs cleanup
Final Store Status.
reflection_goals
basic rocks store stats
* Estimated Number of Keys: 194
* Estimated Live Data Size: 58542
* Total SST files size: 58542
* Pending Compaction Bytes: 0
Index Stats
* live records: 194
* deleted records: 0
refresh_requests
basic rocks store stats
* Estimated Number of Keys: 79
* Estimated Live Data Size: 5260
* Total SST files size: 5260
* Pending Compaction Bytes: 0
userGroup
basic rocks store stats
* Estimated Number of Keys: 9
* Estimated Live Data Size: 2185
* Total SST files size: 2185
* Pending Compaction Bytes: 0
Index Stats
* live records: 9
* deleted records: 0
catalog-source-data
basic rocks store stats
* Estimated Number of Keys: 36
* Estimated Live Data Size: 1946
* Total SST files size: 1946
* Pending Compaction Bytes: 0
datasetVersions
basic rocks store stats
* Estimated Number of Keys: 3362
* Estimated Live Data Size: 7211098
* Total SST files size: 7211098
* Pending Compaction Bytes: 0
tokens
basic rocks store stats
* Estimated Number of Keys: 17
* Estimated Live Data Size: 1762
* Total SST files size: 1762
* Pending Compaction Bytes: 0
profiles
basic rocks store stats
* Estimated Number of Keys: 448058
* Estimated Live Data Size: 11327301174
* Total SST files size: 11327301174
* Pending Compaction Bytes: 0
collaboration_wiki
basic rocks store stats
* Estimated Number of Keys: 7
* Estimated Live Data Size: 7303
* Total SST files size: 7303
* Pending Compaction Bytes: 0
Index Stats
* live records: 7
* deleted records: 0
identity
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
materialization_store
basic rocks store stats
* Estimated Number of Keys: 176
* Estimated Live Data Size: 806582
* Total SST files size: 806582
* Pending Compaction Bytes: 0
Index Stats
* live records: 176
* deleted records: 0
metadata-dataset-splits
basic rocks store stats
* Estimated Number of Keys: 3382
* Estimated Live Data Size: 1439681
* Total SST files size: 1439681
* Pending Compaction Bytes: 0
Index Stats
* live records: 3382
* deleted records: 0
external_reflection_store
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
configuration
basic rocks store stats
* Estimated Number of Keys: 2
* Estimated Live Data Size: 1118
* Total SST files size: 1118
* Pending Compaction Bytes: 0
dependencies
basic rocks store stats
* Estimated Number of Keys: 162
* Estimated Live Data Size: 16952
* Total SST files size: 16952
* Pending Compaction Bytes: 0
jobs
basic rocks store stats
* Estimated Number of Keys: 448197
* Estimated Live Data Size: 912453993
* Total SST files size: 912453993
* Pending Compaction Bytes: 0
Index Stats
* live records: 448197
* deleted records: 0
sys.options
basic rocks store stats
* Estimated Number of Keys: 2
* Estimated Live Data Size: 2094
* Total SST files size: 2094
* Pending Compaction Bytes: 0
dac-namespace
basic rocks store stats
* Estimated Number of Keys: 1344
* Estimated Live Data Size: 579419
* Total SST files size: 579419
* Pending Compaction Bytes: 0
* Estimated Blob Count: 0
* Estimated Blob Bytes: 0
Index Stats
* live records: 1344
* deleted records: 0
collaboration_tags
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
Index Stats
* live records: 0
* deleted records: 0
refresh_store
basic rocks store stats
* Estimated Number of Keys: 170
* Estimated Live Data Size: 42776
* Total SST files size: 42776
* Pending Compaction Bytes: 0
Index Stats
* live records: 170
* deleted records: 0
provisioning
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
reflection_settings
basic rocks store stats
* Estimated Number of Keys: 80
* Estimated Live Data Size: 2460
* Total SST files size: 2460
* Pending Compaction Bytes: 0
reflection_entries
basic rocks store stats
* Estimated Number of Keys: 165
* Estimated Live Data Size: 27476
* Total SST files size: 27476
* Pending Compaction Bytes: 0
upgrade
basic rocks store stats
* Estimated Number of Keys: 10
* Estimated Live Data Size: 1776
* Total SST files size: 1776
* Pending Compaction Bytes: 0
metadata-multi-splits
basic rocks store stats
* Estimated Number of Keys: 0
* Estimated Live Data Size: 0
* Total SST files size: 0
* Pending Compaction Bytes: 0
@balaji.ramaswamy the logs were all empty. I have shared a screenshot of the files.
I deployed the master again and now I do no have any logs in the /var/log/dremio folder.
@unni When was the last backup taken, if not too old, you can cancel as it should not cause any issues, if it does then you can restore from the most recent backup. If you have not taken a backup in recent times then recommend leaving it
