we have some problems of executor volumes getting full quickly and found out some strange behaviour regarding the retention of reflection data.
Our observations are the following:
- There is an accumulation of reflection data in the folders /opt/dremio/data/pdfs/accelerator/ for reflections that are deactivated. By deactivated I mean that the reflection is OFF in the UI and there is no materialization_id for this reflection in the sys.materializations table. We can however track the materialisation_id used in the Dremio logs of old reflection runs.
- Large volumes of old data can also be found in the folders /opt/dremio/cloudcache/c0/ . Timestamps are after the reflections run so I guess it is linked…
- Why is old reflection data not purged when the reflection is set to OFF ?
- In the documentation it is mentionned that “the cache will evict unused data when required” (Dremio). When is Dremio considering it “required” ?
- Is there a parameter somewhere in Dremio or an operation to perform in order to avoid such retention of unused reflection data ?