# Reflections stop working **URL:** \ **Category:** Uncategorized\ **Created:** [December 4, 2019, 10:47pm UTC](https://community.dremio.com/t/reflections-stop-working/4691 "2019-12-04T22:47:12Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 4, 2019, 10:47pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/1 "2019-12-04T22:47:12Z")
Today when I signed in all the reflections on my cluster suddenly had a 0B in the advanced settings and an “…” but the history showed that the last activity of a reflection being succesfully built. Also the reflections did not get used in any queries. The only thing I can think to do is rebuild them, but its really bad that sometimes they seem to become unlinked somehow from the queries? What is causing this? ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 4, 2019, 10:50pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/2 "2019-12-04T22:50:23Z")
![00%20PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/d/d3b593fe1fc7f5122892b134fd56b468ed5c57e1.png) ![28%20PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/c/cacca4c1000775e9083a2212bdcb7c74ad43c7a3.png) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 4, 2019, 11:15pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/3 "2019-12-04T23:15:31Z")
@kprifogle Can you filter failed jobs and look at the profile - error If you did not recreate them and just refreshed them then the old reflection id would still exists and you can search for that reflection id in the jobs page and look at them If you dropped and recreated the reflection then you have to filter failed jobs and search for “REFRESH REFLECTION” as the id would have changed ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 5, 2019, 4:55am UTC](https://community.dremio.com/t/reflections-stop-working/4691/4 "2019-12-05T04:55:17Z")
The id for the refleciton hasn’t changed, I looked at the profile and there was no error. As far as I can tell the refleciton was last successfully built with no errors however I see the 0B and the reflections are not used. It just happened again, second time today. This makes reflections practically unusable. ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 5, 2019, 7:01pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/8 "2019-12-05T19:01:28Z")
So I notice if I go into the syslogs the reflection is trying to be refreshed, so its behaving as though its wanting to be refreshed according to a schedule. Then I went into the metadata and checked and its refresh policy was still set to 9 hours and the neverRefresh and neverExpire booleans were not in the source metadata. This makes me think that the issue is a UI bug. I will try manually updating those boolean fields via the rest API and see if that makes a difference. Have you fixed any bugs with refresh settings in the UI since Dremio 3.2? ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 5, 2019, 7:11pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/9 "2019-12-05T19:11:09Z")
I manually updated the reflection using a put request to the rest api and those boolean attributes still will not appear. Is there a bug with updates to those boolean never fields in the rest api for dremio 3.2? ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 5, 2019, 7:20pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/10 "2019-12-05T19:20:15Z")
I’m going to just set refresh time and refresh expiration to 99 million weeks to get around the issue. I think something is buggy in those boolean fields. ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 5, 2019, 10:01pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/11 "2019-12-05T22:01:31Z")
Turns out this didn’t solve the problem. I just signed in and the reflections had become disassociated again. @balaji.ramaswamy do you have any idea whats happening from looking at the above queries? ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 6, 2019, 4:30am UTC](https://community.dremio.com/t/reflections-stop-working/4691/12 "2019-12-06T04:30:14Z")
@kprifogle, reflection\_bad: Is a query that is getting accelerated. Is the wrong reflection getting selected? reflection\_bad2: Is a refresh reflection on “Version-1.transactions.Raw.transactions\_union\_delta”? reflection\_bad3: Is a refresh on “Version-1.transactions.Raw.transactions\_bridge”? delete\_bad: is a drop reflection I also see you are on a very old version 3.2. Both refreshes have a non-zero count. We have some important bug fixes/enhancements from 3.2 to 4.0. I would recommend first upgrading. Rebuild the reflections and then lets draw a baseline and solve issues arising after that ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 6, 2019, 6:22pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/13 "2019-12-06T18:22:39Z")
reflection\_bad: This is a query that is not selecting because reflection\_bad2 is going bad. reflection\_bad2 is the create reflection job and nothing seems to be wrong with it. Same with 3. delete\_bad is dremio deleting the reflection after whatever is happening happens to it to disassociate it from the table. What do you mean by “both refreshes have a non-zero count”, they show as having 0B in the reflection page. ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 6, 2019, 6:22pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/14 "2019-12-06T18:22:47Z")
Is upgrading the only option? ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 6, 2019, 7:25pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/16 "2019-12-06T19:25:25Z")
@balaji.ramaswamy I found that one of the sources that was being joined into the reflection that was expiring incorrectly (or whatever you want to call what it was doing with the 0B and ellipses) had bad reflection refresh settings (3 hours expire and 9 hours refresh). This happened due to another issue I just posted about. Even though it didn’t itself have any reflections I think it was causing things to break possibly because the other reflections may have picked up those settings? I think this may actually be the bug. (fingers crossed) [/quote] ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 7, 2019, 12:56pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/17 "2019-12-07T12:56:16Z")
This appears to have been the issue. The source with bad reflection refresh settings was causing all its dependent downstream reflections to “break” with the ellipses and 0B. I no longer have this issue. ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 9, 2019, 3:13am UTC](https://community.dremio.com/t/reflections-stop-working/4691/18 "2019-12-09T03:13:51Z")
Nope, I was wrong. The issue returned. I think the issue is actually related to recycling the master node. ---
**Author:** ![kprifogle](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/kprifogle/32/1719_2.png) [@kprifogle](https://community.dremio.com/u/kprifogle)\ **Post date:** [December 9, 2019, 4:21pm UTC](https://community.dremio.com/t/reflections-stop-working/4691/19 "2019-12-09T16:21:51Z")
I found this in the log related to the reflection in question: f08e6231-f488-457f-ba8a-847a7750da28/bfabd717-b124-4280-9358-248d2ac0c092",“stack\_trace”:“com.dremio.exec.planner.acceleration.MaterializationExpander$ExpansionException: Materialization bfabd717-b124-4280-9358-248d2ac0c092 have different row types for its table and query rels.\ntable row type REDACTED )\n\tat com.dremio.exec.planner.acceleration.MaterializationExpander.expand(MaterializationExpander.java:109) ~[dremio-sabot-kernel-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.exec.planner.acceleration.MaterializationDescriptor.getMaterializationFor(MaterializationDescriptor.java:147) ~[dremio-sabot-kernel-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionServiceImpl$CacheHelperImpl.expand(ReflectionServiceImpl.java:950) ~[dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionServiceImpl$CacheHelperImpl.expand(ReflectionServiceImpl.java:931) ~[dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.MaterializationCache.updateEntry(MaterializationCache.java:174) ~[dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.MaterializationCache.update(MaterializationCache.java:221) ~[dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionServiceImpl$DescriptorCacheImpl.update(ReflectionServiceImpl.java:996) ~[dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionManager.metadataRefreshJobSucceeded(ReflectionManager.java:752) [dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionManager.handleSuccessfulJob(ReflectionManager.java:574) [dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionManager.handleRefreshingEntry(ReflectionManager.java:344) [dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionManager.handleEntry(ReflectionManager.java:292) [dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionManager.handleEntries(ReflectionManager.java:262) [dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.service.reflection.ReflectionManager.run(ReflectionManager.java:178) [dremio-services-accelerator-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat com.dremio.common.WakeupHandler$1.run(WakeupHandler.java:67) [dremio-common-3.2.0-201905102005330382-0598733.jar:3.2.0-201905102005330382-0598733]\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0\_222]\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0\_222]\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0\_222]\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0\_222]\n\tat java.lang.Thread.run(Thread.java:748) [na:1.8.0\_222]\n”}