15 PDS, Raw Reflection, Planning >3mins and Failing

1

Hi everyone,

I’m running into a problem where Dremio query planning takes a very long time and eventually fails.

Setup:

  • ~15 Physical Datasets (PDS)

  • Using a Raw Reflection

  • Queries are relatively straightforward

Issue:

  • Planning stage takes more than 3 minutes

  • Query ends up failing during planning (never executes)

I expected the reflection to speed things up, but it doesn’t seem to be helping. Has anyone experienced this kind of behavior?

Questions:

  1. Is there something wrong with how I’m defining my Raw Reflection?

  2. Could having 15 PDS cause such a huge planning overhead?

  3. Any tips for diagnosing why planning takes so long?

Any suggestions or pointers would be greatly appreciated. Thanks!

2

@andrewfirmansap The job profile should tell us which planner phase the time was spent on, are you able to send it across?

3

Greeting sir Balaji Ramaswamy,
Thank you for your fast response, here I will provide the Profile,
while at it, can you guide me how to read the Job Profile & Also I don’t understand why doing raw reflections on the PDS caused the planning to overplan, Thank you!

b2120fe0-7427-4ab1-9569-a44c94dabaac.zip (157.8 KB)

4

In the planning tab, we can see that the query finished pre-logical planning and got stuck in logical planning which is the step that substitutes reflections/materializations into the plan.

I tried running a similar query with no reflections on the latest version of Dremio and it works fine. I also don’t see anything in the query which would cause it to take a long time to plan.

15 PDS is a very small number and the planner can easily handle hundreds of tables/views.

If you disable reflections with a hint, does the query run successfully?

Can you provide more information on the reflection that you created?

5

Thank you for your responses, I ended up upgrading My Dremio from version 25 to version 26, now everything works fine!

Before, it only works when I turn off all reflections on the PDS, I think it might be bug in older version.

2 Likes