It happened to me as well. I made few fixes as below which are under observation for now. The guess is that dremio cleans up old reflections, scratch, uploads, download, etc. folders periodically. It seems that periodic clean up of reflections is fine, however it seems there is a bug in clean up of other folders like scratch, downloads, etc. It seems that it deletes everything in the immediate parent directory of scratch, download, uploads, etc. folder (which exact it is trying to clean, don’t know). For example, if you have:
foo/reflections
foo/uploads
foo/download
foo/scratch
foo/results
Then, periodically you will find only:
foo/
everything deleted.
If you have db folder under foo:
foo/db
you will wonder why it did not disappear; db has all of dremio metadata thats why you see all of the objects in the UI.
foo/db doesn’t get dropped because dremio has a lock on the db folder by way of having open handles to the underlying files.
So, I have setup so that all of the dremio folders like reflections, downloads, uploads, scratch, etc. are on separate paths folders.
foo1/reflections
foo2/uploads
foo3/downloads
foo4/scratch
foo5/results
fooX/whatever
Try that, observe it per your cleanup schedule - check this: https://docs.dremio.com/advanced-administration/job-results-cleanup.html