Hi all,
our dremio cluster didn’t work as our expectation. we had 5 worker and only 1 worker used memory until 77% , other nodes only use under 10% of memory. is there any missing config that i was set. i set dremio-env MAX_HEAP=8GB and MAX_DIRECT=128GB.
Are all nodes configured with the same hep and direct, sometimes due to data skew processing might target to single node, are you able to share with us the Dremio query profile
Thanks
Bali
Hi Bali,
All nodes have same config, is there any strategy to mitigate skew processing since those process are most ly joining 3-6 tables using single key smartphone number ?
Send us the profile and we can see if there is anything that can be done, data skew due to filter may be tough to resolve
Hi Bali,
I need permission from my internal team to send profile. I just curious about dremio internal process. did it work like other distributed system that always shuffle the data using hash to distribute process?
Based on the size of the tables joined, it can be either a shuffle or a broadcast