Reflections lost after Dremio redeployment

1

Hi, we are running dremio-oss 25.1 on k8s. i have distributed storage on persistent volume.
we needed to redeploy dremio and reset the dremio persistent volume (different to distributed storage). after redeployment i have lost all reflections.
can you please help me understand what and where is stored when it comes to reflections to avoid this?
also what happens to all those files stored in distributed storage that became orphan? i assume that this does not relate to Iceberg, but is Dremio internal.
thank you for clarification.
Jaro

2

@jaroslav_marko The physical files are stored in the distributed location defined under dremio.conf, so if you changed then you would need to refresh the reflections again so they get recreated physically under the new dist store defined

3

hi @balaji.ramaswamy this is exactly what i thought. we did not change anything in .conf and the distributed location remained untouched (including all files visible after redeployment). but still all (completely) reflection definitions were lost.
any ideas, please?
thanks
Jaro

4

@jaroslav_marko What was exactly done when you say "reset the dremio persistent volume "?

5

@balaji.ramaswamy there is a pvc volume attached to every pod (for executors as well as for master) - so we have reset them (deleted and created new/empty).
we were not expecting that there would be some permanent data stored there.

6

@jaroslav_marko One of two things is going on

  • You have deleted your master PVC which means you have lost your KVstore, but that also means you would have lost not only reflections but also your sources, spaces, folders etc
  • You are storing reflections on local disk (not supported) which got erased, only on executors

Are you able to send your values.yaml deleting any passwords and we can validate

7

hi @balaji.ramaswamy, i think it is the first option :slight_smile:
is it possible to split KV store to a separate PVC?
this happened when we were upgrading from 25.0 to 25.1 and everything stopped working and we couldn’t identify why. after deleting the PVC is has initialized 25.1 ok and worked fine.

attaching the values.yaml
values_forum.zip (1.4 KB)
i would appreciate if you can give me also other recommendations for values.yaml that may be beneficial. thanks.

jaro

8

@jaroslav_marko Bottom line is the KVstore needs to go to a persistent volume and should never be deleted, yes KVstore can go to its own location. You do not need a second PVC on the coordinator

9

@jaroslav_marko You should use Dremio provided Helm charts

10

Hi @balaji.ramaswamy we indeed use helm-chart you’re referring to.
our current understanding is master node stores 2 “sets” of data

  • KVstore
  • other settings or data

Firstly - is our understanding correct and if so, does it make sense to have those two separated on separate PVCs?
we have checked and coordinator and it uses only one PVC, whereas workers use 2 PVCs but those are created automatically.

11

@jaroslav_marko All Dremio catalog, namespace, RBAC objects are within KVtore, the Dremio configuration files go to /opt/dremio

12

Thank you for clarification. Jaro