Store.max_partitions

1

Heyho, i want to create a raw reflection but i get this error:

image
image1417×46 6.31 KB

I tried to add partitions on the reflection but its not working.

Unfortunately its not documented where i cant find the setting nor where to add “store.max_partitions”.

Here’s the job profile im using dremio version 22.1
f421872e-983a-486d-8695-120dde15e08d.zip (20,7 KB)

EDIT:
I guess i fixed it. I wanted to reflect the view “tBestellPos”.
But i had a view “OrdersWithPosition” which was already reflected and was supposed to use the reflection of “tBestellPos”.

Somehow while creating this reflection dremio also used “OrdersWithPosition” allthough it was not part of my Query in tBestellPos.

I disabled the reflection on “OrdersWithPosition” and now all works fine.

This was the Job that didn’t work

image

This is the version that worked

image

2

The store.max_partitions is a support key that you can set here: Dremio

1 Like
3

Aahhh, thank you, that actually destroys several questionamarks in my brain xD

But store.max_partitions should be added to the documentation or is it deprecated?

Edit:
But the question why Dremio tries to use a reflection thats not part of the query still remains.
This problem increases the amount of partitions “indefinitely”

4

When you say the reflection is “not part of query”, I think you are saying that the user query does not directly reference a view that has either a raw/aggregate reflection turned on, yet the user query is still being accelerated by one of the reflections on the view.

If so, then this is how our algebraic matching works. Your user queries don’t need to directly reference the materialized views. We can detect that the relational operators are equivalent (or can be made equivalent) between a subtree of the user query and the reflection and substitute the reflection into the query. (After that, we still look at costs to determine whether we actually use the matched plan or not).

1 Like