I noticed Dremio has some ‘vectorized’ implementations for HashJoin and HashAggregate and I am curious to know what is the gain with these new implementations? Any benchmark or analysis to justify these implementations?
Have a look here: https://www.dremio.com/java-vector-enhancements-for-apache-arrow-0-8-0/
Also further improvements coming from Gandiva: https://www.dremio.com/gandiva-performance-improvements-production-query/
More to chat me in Dremio 3.0 later this year.
Hi Kelly, thanks for your response.
The first blog mentioned only Project and Filter. And Gandiva has only two operators as far as I can see. What I asked was about VectorizedHashJoinOperator and VectorizedHashAggOperator. Since they are much more complex and time consuming compared with Project and Filter I would expect implementing them to benefit the performance more. But I could not see any visible improvements when running TPCH queries with “exec.operator.join.vectorize” switched between true and false. So I want to know if you have any internal measurement about these operators?