I have created a Refection with partition on the day. When the reflection refreshes happens . The External_sort activity happens on the single thread. why dint it uses hash partitioning and distribute it ?. How can we apply other partitioning schemes on reflection?.
Thanks,
Pavan
@pavankumargundeti By default Dremio tries to minimize files and hence this issue when you partition
Open the reflection layout, click on advanced, click on the tiny cog wheel and choose “Minimize refresh time”
Can you see if this makes a difference? How many rows are there in the table? Is “day” very high cardinality?
Hi Balaji,
I tried out the two combinations. My table has 1M records partition on day basis .
Time taken with :
Minimum of files produced is 1minute 30 sec.
Minimiza Refresh Time: 1 minunte 15 sec.
Not much. Please give me any other way to add reflection strategies.
Current cluster :
One coordinator
One executor
Thanks,
Pavan
@pavankumargundeti I would like to know where the time was spent with Minimize refresh time, can you please send the profile to review
