Housekeeping for the community edition

1

Hi all !
These directories are full of old files and I did not find any documentation on how to do a cleanup :

39G dremio/db/catalog
1,3G dremio/db/search
33G dremio/cm/fs

Could you please tell us what they are used for, and how we could safely remove old data ?

Thank you !

2

dremio/cm/fs is for Dremio so depends on number of queries.

dremio/db/catalog is the rocksDB, What version of Dremio is this? Have you turned on unlimited splits?

3

You are right,
I had dist.caching.enabled set to true in my conf.

Yes unlimited splits is turned on, is it why the directory gets so big ?

Thank you @balaji.ramaswamy !

4

@fetanchaud can you try this?

  • Shutdown Dremio
  • bin/dremio-admin clean -o -c
  • Start Dremio
  • send the log file, it will be generated under the log folder, if not copy/paste from screen
  • See of this reduces the 39G under catalog
  • The 33 GB under fs is all local cache. Can you go inside the folder and see how old the files under it are?
1 Like