I am trying to build raw reflection and it takes very long time to execute.
Why thread 04 takes this very long time?
5efaaab7-f60c-472c-913b-ee6bd6993220.zip (159.5 KB)
Problem seems to be with the HASH_JOIN as the two inputs to the HASH JOIN are
01-xx-09 about 28K records
01-xx-08 about 700 Million
Output of HASH_JOIN is 01-xx-06 which is 20 Billion records and PARQUET_WRITER is taking way to long to write these files in 01-xx-01
This looks like an expanding join that is causing the issue. Phase 4 is just waiting on Phase 1 to complete the task