We have been using Dremio 2.0 for a few months and recently copied all our tables/reflections over to using Dremio 3.0. We have a basic query that is able to run in the Dremio 2.0 server environment in about ~30 seconds, but the same query is stuck in “starting” status for over 60 min. before failing and giving us the details of “Invalid Query Exception” and “Failed”.
The queries are the exactly the same.
Our Queue control settings are exactly the same.
Any ideas on what to try to better troubleshoot?
Can you try and disable “planner.experimental.pclean_logical” via the support key under admin-support? and retry the query?
Thank you for the suggestion.
the “planner.experimental.pclean_logical” is already unchecked in both environments. Sorry for not calling that out in my first post.
Can you please compare the differences in convert to rel between the 2.x and 3.x plan? Also do the same for Logical planning as most lilely the 1 minute would have been spent on logical planning. Do you have both profiles? Can you please send it across? the one from 2.x and the one from 3.x?
Dremio 3.x - Query does NOT work
Convert To Rel (139 ms)
Logical Planning (734 ms)
Dremio 2.x - Query does work
Convert To Rel (79 ms)
Logical Planning (959 ms)
What part of the “Job Profile” would be helpful? Query, Planning (what part within planning), Acceleration, or Error?
Tahnks for your help!
I think I found the downloadable profile you were talking about. Please let me know if you meant something different. Profile_2.x.zip (237.1 KB)
619c6c5c-ff76-4133-a3ea-1e36315dde60.zip (241.5 KB)
Thanks for all your responses and help!
Attached is the 2.x prepare statement
prepare_statement_profile_dremio_2.x.zip (78.5 KB)
I turned on the planner.verbose_profile for both Dremio 3.x and 2.x server environments
The prepare statement that successfully completes shows me reflections used, the prepare statements for Dremio 3.x where the prepare statements fails, it does not show me the names of reflections chosen within “Job Profile > Acceleration”. Still looking for detail so I can compare the two.
Can you please send us the r profiles with verbose on, we will look and see what changed. If you are not able to find the missing reflections, that should be ok for now
How do I find the r (reflection) profiles? Do you mean the full download profile for the same query in each environment like I sent before? Attached are the full profiles for each query run, when the planner.verbose_profile switch turned on.
Let me know if you mean something different.
prepare_statement_dremio_2.x-verbose_on.zip (78.5 KB)
prepare_statement_Dremio_3.x-verbose_on.zip (494.3 KB)
The verbose profiles certainly helps. Let us not worry about the same set of reflections getting considered between the 2 profiles for now. Let me look into what has changed and get back to you