# Reflections lost after Dremio redeployment **URL:** \ **Category:** Uncategorized\ **Created:** [September 26, 2024, 6:31am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346 "2024-09-26T06:31:09Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![jaroslav\_marko](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@jaroslav\_marko](https://community.dremio.com/u/jaroslav_marko)\ **Post date:** [September 26, 2024, 6:31am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/1 "2024-09-26T06:31:09Z")
Hi, we are running dremio-oss 25.1 on k8s. i have distributed storage on persistent volume. we needed to redeploy dremio and reset the dremio persistent volume (different to distributed storage). after redeployment i have lost all reflections. can you please help me understand what and where is stored when it comes to reflections to avoid this? also what happens to all those files stored in distributed storage that became orphan? i assume that this does not relate to Iceberg, but is Dremio internal. thank you for clarification. Jaro ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [September 30, 2024, 3:09pm UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/2 "2024-09-30T15:09:39Z")
@jaroslav_marko The physical files are stored in the distributed location defined under dremio.conf, so if you changed then you would need to refresh the reflections again so they get recreated physically under the new dist store defined ---
**Author:** ![jaroslav\_marko](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@jaroslav\_marko](https://community.dremio.com/u/jaroslav_marko)\ **Post date:** [September 30, 2024, 6:36pm UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/3 "2024-09-30T18:36:15Z")
hi @balaji.ramaswamy this is exactly what i thought. we did not change anything in .conf and the distributed location remained untouched (including all files visible after redeployment). but still all (completely) reflection definitions were lost. any ideas, please? thanks Jaro ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [September 30, 2024, 7:00pm UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/4 "2024-09-30T19:00:34Z")
@jaroslav_marko What was exactly done when you say "reset the dremio persistent volume "? ---
**Author:** ![jaroslav\_marko](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@jaroslav\_marko](https://community.dremio.com/u/jaroslav_marko)\ **Post date:** [September 30, 2024, 7:02pm UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/5 "2024-09-30T19:02:51Z")
@balaji.ramaswamy there is a pvc volume attached to every pod (for executors as well as for master) - so we have reset them (deleted and created new/empty). we were not expecting that there would be some permanent data stored there. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 2, 2024, 6:57am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/6 "2024-10-02T06:57:44Z")
@jaroslav_marko One of two things is going on - You have deleted your master PVC which means you have lost your KVstore, but that also means you would have lost not only reflections but also your sources, spaces, folders etc - You are storing reflections on local disk (not supported) which got erased, only on executors Are you able to send your values.yaml deleting any passwords and we can validate ---
**Author:** ![jaroslav\_marko](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@jaroslav\_marko](https://community.dremio.com/u/jaroslav_marko)\ **Post date:** [October 2, 2024, 12:23pm UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/7 "2024-10-02T12:23:25Z")
hi @balaji.ramaswamy, i think it is the first option 🙂 is it possible to split KV store to a separate PVC? this happened when we were upgrading from 25.0 to 25.1 and everything stopped working and we couldn’t identify why. after deleting the PVC is has initialized 25.1 ok and worked fine. attaching the values.yaml [values\_forum.zip](https://community.dremio.com/uploads/short-url/dTloVByx0WpLBGGSazllbKS9BOJ.zip) (1.4 KB) i would appreciate if you can give me also other recommendations for values.yaml that may be beneficial. thanks. jaro ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 3, 2024, 5:35am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/8 "2024-10-03T05:35:25Z")
@jaroslav_marko Bottom line is the KVstore needs to go to a persistent volume and should never be deleted, yes KVstore can go to its own location. You do not need a second PVC on the coordinator ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 3, 2024, 5:36am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/9 "2024-10-03T05:36:23Z")
@jaroslav_marko You should use Dremio provided Helm charts > **[GitHub - dremio/dremio-cloud-tools: Dremio Container Tools](https://github.com/dremio/dremio-cloud-tools)** > > Dremio Container Tools ---
**Author:** ![jaroslav\_marko](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@jaroslav\_marko](https://community.dremio.com/u/jaroslav_marko)\ **Post date:** [October 3, 2024, 8:28am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/10 "2024-10-03T08:28:00Z")
Hi @balaji.ramaswamy we indeed use helm-chart you’re referring to. our current understanding is master node stores 2 “sets” of data - KVstore - other settings or data Firstly - is our understanding correct and if so, does it make sense to have those two separated on separate PVCs? we have checked and coordinator and it uses only one PVC, whereas workers use 2 PVCs but those are created automatically. ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [October 8, 2024, 5:20am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/11 "2024-10-08T05:20:29Z")
@jaroslav_marko All Dremio catalog, namespace, RBAC objects are within KVtore, the Dremio configuration files go to `/opt/dremio` ---
**Author:** ![jaroslav\_marko](https://avatars.discourse-cdn.com/v4/letter/j/bc8723/32.png) [@jaroslav\_marko](https://community.dremio.com/u/jaroslav_marko)\ **Post date:** [October 8, 2024, 7:36am UTC](https://community.dremio.com/t/reflections-lost-after-dremio-redeployment/12346/12 "2024-10-08T07:36:18Z")
Thank you for clarification. Jaro