Dremio CE hangs on reflection refresh

Dremio University
1

I’m using Dremio 22.0.0-202206221430090603-1fa4049f.
In latest days, Dremio reflection refresh jobs are hanging… but It not occurs when I create the reflections on first time.
My data source is Mongodb 4.4.15 and the reflections are of type REFLECTION_SMALL:

image
image1198×324 70.6 KB

The machine CPU also stucks at 100% for hours and hours… No errors are displayed on logs.

I saw a fix on Dremio Cloud changelog Dremio about and issue like mine:
Fixed an issue that was causing REFRESH REFLECTION and REFRESH DATASET jobs to hang when reading Iceberg metadata.

Do you think this issue can happen also on CE edition and what are the plans to apply the same fix on it?

Update: maybe this is happening because all the dependent reflections are recreated simultaneously.

Thanks!

2

@almirb Are you able to send us the job profile?

3

Sorry @balaji.ramaswamy , but the profile has confidential data. Can’t post It here.

4

Can you provide a thread dump? The issue we fixed that you referenced in the release notes won’t have 100% CPU utilization. It was a hang when reading avro Iceberg metadata files.