Reduce disk usage of /catalog folder of Dremio

1

Hi Team,

is it safe to delete those big .log files in

db/catalog/ ?

Is the retention of these files configurable?

We are using Dremio 2.0.5 on centos 6
Thanks
Sergio

2

Hi @seraus

A big “NO”. Those are the rocksDB catalog transaction log files. Starting 2.1.8 and up we are better maintaining these.

Thanks
@balaji.ramaswamy

3

Thanks,

any chance of doing the cleaning with some Dremio or RockDb tools ?

I observed that they are cleaned automatically when Dremio is restarted

Some of these are VERY old files…
Thanks
Sergio

4

I would not have to do a restart of Dremio only to clean those files.

Thanks
Sergio

5

Hi @seraus

Kindly upgrade to the latest version of Dremio and we clean orphans automatially.

Thanks
@balaji.ramaswamy

6

Thanks for the answer, balaji.ramaswamy; we’ll upgrade as soon as possible.

In the meantime, is there any workaround on this version?

For example: delete .log files older than 7 days… or running a scheduled dremio admin <something>

Thanks

7

@seraus

Not in this version. Once you upgrade, you can do below

./dremio-admin clean -c -o

Thanks
@balaji.ramaswamy

8

Then I think I’ll schedule a restart
Thanks
Sergio