# Raw reflection creation never finishes **URL:** \ **Category:** Uncategorized\ **Created:** [February 27, 2024, 2:25pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567 "2024-02-27T14:25:47Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![rrosenzvaig](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rrosenzvaig/32/5502_2.png) [@rrosenzvaig](https://community.dremio.com/u/rrosenzvaig)\ **Post date:** [February 27, 2024, 2:25pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/1 "2024-02-27T14:25:47Z")
Hello, I am using Dremio version 24.3.2 I am trying to create a raw reflection from a view that makes some calculations and joins from some other raw tables. In total, all the tables together use 160 MB of Blob storage space, and the total number of files is 3000. My data source is Azure Blob Storage. Whenever I run the view query, it takes around 30-60 seconds. Now I want to activate a reflection for this view, so that it takes less than 5 seconds to retrieve the data every time I run it. The problem is that, whenever I try to create a reflection it processes the job for many hours, and it never finishes. For example, I have tried running the reflection creation 2 times. One for 66hours and another time for 17 hours. None of them succeeded. I am using a 4 core 32 GM RAM single node. I have tried creating a raw reflection for one of the tables that is used in the view, and it succeeded after 1 hour. Is there a limitation with views that I should be aware of? I have tried every solution that I could find on this forum, but without success. Any help would be greatly appreciated. Thanks ---
**Author:** ![rrosenzvaig](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rrosenzvaig/32/5502_2.png) [@rrosenzvaig](https://community.dremio.com/u/rrosenzvaig)\ **Post date:** [February 27, 2024, 3:00pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/2 "2024-02-27T15:00:04Z")
Here is the last query profile I performed [d1b83bd4-b4f0-4595-aeac-64b790b7d65d.zip](https://community.dremio.com/uploads/short-url/mc0TWgT6Y25DSfAi2FltLmJJgLL.zip) (72.8 KB) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [March 3, 2024, 1:29am UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/4 "2024-03-03T01:29:48Z")
@rrosenzvaig On 24.x this path should have used Iceberg Metadata but instead because the below setting have been disabled (by default true) ```auto { "kind" : "BOOLEAN", "type" : "SYSTEM", "name" : "dremio.execution.support_unlimited_splits", "bool_val" : false }, { "kind" : "BOOLEAN", "type" : "SYSTEM", "name" : "dremio.iceberg.enabled", "bool_val" : false }, ``` In this path, metadata refresh is on the coordinator and every refresh is a Full refresh vs in the new flow it is shared by all the executors just like a query and the metadata part is incremental. I would be interested to see your query profile without the reflection as that would also have used the old code path In this profile, PARQUET\_WRITER is taking all the time butI see no metrics. I need to see the profile with unlimited splits turned on Do you have the steps? It is just not turning the above keys but your metadata will start going to your dist storage and you would have to compute metadata for all PDS by doing a forget and refresh. In fact in the next version these keys will be dropped Thanks Bali ---
**Author:** ![rrosenzvaig](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rrosenzvaig/32/5502_2.png) [@rrosenzvaig](https://community.dremio.com/u/rrosenzvaig)\ **Post date:** [March 4, 2024, 3:02pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/5 "2024-03-04T15:02:13Z")
Thanks for the response. I am testing thew configs on a test server. I did a reset on the entire environment, and I am going to try the reflection again. As soon as I can, I share the query plan as well. Is there anything else I should try? ---
**Author:** ![rrosenzvaig](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rrosenzvaig/32/5502_2.png) [@rrosenzvaig](https://community.dremio.com/u/rrosenzvaig)\ **Post date:** [March 5, 2024, 8:48pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/6 "2024-03-05T20:48:35Z")
@balaji.ramaswamy I have the new query profile after reseting all settings. [940e8fe8-af5a-4901-b83f-8ab08540adc0.zip](https://community.dremio.com/uploads/short-url/dRP7l0Q9lNLz0vR1nYRWzEHDD6f.zip) (55.0 KB) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [March 12, 2024, 6:08am UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/7 "2024-03-12T06:08:15Z")
@rrosenzvaig All the time is spent on PARQUET\_WRITER and the record processing rate is extremely slow. If you see all the time is spent on process so this is not IO, It looks like `2af9eea3408a` is a 4 core box and that could be the issue as the PARQUET\_WRITER is writing 7 million per thread. Still the record processing rate seems slow When you run this reflection and after 30 minutes, is it possible to get a thread dump about 1 every second for say 5 minutes #!/bin/bash for i in `seq -w 3 1 300` do jstack -l dremio-pid \> ThreadDump$i.txt sleep 1 done - chmod +x jstack.sh - `./jstack.sh` - `tar -cvf jstack.tar *.txt` - `gzip jstack.tar` And send the jstack.tar.gz file ---
**Author:** ![rrosenzvaig](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rrosenzvaig/32/5502_2.png) [@rrosenzvaig](https://community.dremio.com/u/rrosenzvaig)\ **Post date:** [March 12, 2024, 12:43pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/8 "2024-03-12T12:43:19Z")
Ok. Thanks a lot for the sugestion. I will run this test and share the results as soon as possible ---
**Author:** ![rrosenzvaig](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rrosenzvaig/32/5502_2.png) [@rrosenzvaig](https://community.dremio.com/u/rrosenzvaig)\ **Post date:** [March 13, 2024, 12:49pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/9 "2024-03-13T12:49:09Z")
@balaji.ramaswamy Here is the file [jstack.tar.zip](https://community.dremio.com/uploads/short-url/pLHorUhCFBDzwdpzi15EA9Gkwq8.zip) (4.2 MB) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [March 15, 2024, 3:31am UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/10 "2024-03-15T03:31:36Z")
@rrosenzvaig Sorry my bad, please also send me the profile of the job that was running when you took the thread dumps as I need to trace the job ID in the thread dump ---
**Author:** ![rrosenzvaig](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/rrosenzvaig/32/5502_2.png) [@rrosenzvaig](https://community.dremio.com/u/rrosenzvaig)\ **Post date:** [March 15, 2024, 6:58pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/11 "2024-03-15T18:58:20Z")
Sure. It is either 1a0f5213-d05d-3d69-c0e5-d4a018bdd800 or 1a0e6326-7327-17ba-72b1-9fc4893f2100 ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [March 17, 2024, 10:00pm UTC](https://community.dremio.com/t/raw-reflection-creation-never-finishes/11567/12 "2024-03-17T22:00:09Z")
@rrosenzvaig Do not see both Job IDs in the ThreadDump files. Can you please send me the profile of the Reflection refresh that never finished when this thread dump was taken?