# Dremio parallel job execution

**URL:** <https://community.dremio.com/t/dremio-parallel-job-execution/7393>\
**Category:** Uncategorized\
**Created:** [May 4, 2021, 1:48pm UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393 "2021-05-04T13:48:48Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![JoiceJacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/joicejacob/32/5715_2.png) [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)\
**Post date:** [May 4, 2021, 1:48pm UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/1 "2021-05-04T13:48:48Z")

</div>

Hi all  
I am having some trouble with understanding the execution of minor fragments.  
I am attaching a visualization plan below. There are two leaf branches. Is the execution of minor fragments in leaf branches are parallel? after these processes, from unionAll, is the execution sequential?

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/3/32dee6237fcfdf5237521d7e5f1461d4d326c9fc.png)

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [May 5, 2021, 7:15am UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/2 "2021-05-05T07:15:19Z")

</div>

@JoiceJacob Kindly send us the profile and we can explain, it does seem like your query was planned in a single phase and that means it was all the way single threaded, not sure where you see parallel threads in the scan

> **[How To Share A Query Profile](https://www.dremio.com/tutorials/share-query-profile-dremio/)**
>
> Tutorial explaining how to share a query profile in Dremio.

---

<div class="post-metadata">

**Author:** ![JoiceJacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/joicejacob/32/5715_2.png) [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)\
**Post date:** [May 5, 2021, 10:34am UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/3 "2021-05-05T10:34:53Z")

</div>

@balaji.ramaswamy  
Here is the job profile :  
[8b053f4d-c8e5-4955-8a7e-14eb1e3e3eef.zip](https://community.dremio.com/uploads/short-url/raDFIpFIreKRcbl2z8VPMjegm93.zip) (16.7 KB)

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [May 6, 2021, 6:51am UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/4 "2021-05-06T06:51:35Z")

</div>

@JoiceJacob Your entire query was planned in a single phase so it was all the way single threaded, this is because you had very less rows in JDBC scan and zero rows on your PARQUET\_SCAN

---

<div class="post-metadata">

**Author:** ![JoiceJacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/joicejacob/32/5715_2.png) [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)\
**Post date:** [May 6, 2021, 2:30pm UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/5 "2021-05-06T14:30:22Z")

</div>

@balaji.ramaswamy  
Thank you for your response.  
As you suggested I have executed the query with more data. The visualization plan and the job profile generated are attached below.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/3/36a550909e038276c7e63341f4904113adaef7b3.png)  
Job profile:  
[e9cddd73-440f-462b-96a6-9ed4ffbff934.zip](https://community.dremio.com/uploads/short-url/pdI5y92pxpP1kQA0RgeKqpdyTnr.zip) (35.0 KB)

I am also attaching another visualization plan and the job profile.

 ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/c/c1ad689d76ae7e58ed9a32f22e0435ac2b23d2d3.png)  
Job profile:  
[bfc14e40-7080-45a4-a52d-ed3e27fc3e5e.zip](https://community.dremio.com/uploads/short-url/z2KEk65vaenyJDLHWVsCGIDDwoF.zip) (57.6 KB)

Is the execution of minor fragments in leaf branches are parallel in both cases?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [May 7, 2021, 6:34am UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/6 "2021-05-07T06:34:48Z")

</div>

@JoiceJacob Irrespective of number of rows JDBC\_SCAN is always single threaded. Other than this I see HASH\_AGG phase is multi-threaded, what is the real issue here?

---

<div class="post-metadata">

**Author:** ![JoiceJacob](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/joicejacob/32/5715_2.png) [@JoiceJacob](https://community.dremio.com/u/JoiceJacob)\
**Post date:** [May 9, 2021, 12:20pm UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/7 "2021-05-09T12:20:34Z")

</div>

@balaji.ramaswamy  
I will elaborate my doubt.  
From the above visualization plan there are two JDBC scan.

- Are both JDBC scans running in parallel?  
or

- After the completion of one JDBC scan, the other JDBC scan is executed.

- Same like the JDBC scan, are parquet scan running in parallel or one after another?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [May 10, 2021, 4:43am UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/8 "2021-05-10T04:43:17Z")

</div>

@JoiceJacob

Are both JDBC scans running in parallel? Yes  
Same like the JDBC scan, are parquet scan running in parallel or one after another? Yes

There will be some CPU slicing based on the number of parallel threads in the scan (only for Parquet scan)

---

<div class="post-metadata">

**Author:** ![popejune](https://avatars.discourse-cdn.com/v4/letter/p/8dc957/32.png) [@popejune](https://community.dremio.com/u/popejune)\
**Post date:** [September 5, 2023, 3:57am UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/9 "2023-09-05T03:57:00Z")

</div>

Hi @balaji.ramaswamy ,

We did some testing for HashJoin case, seem the two scans are always running one by one.

I think the pros to run one by one is it will not run another if the first one data is zero.

But is there any config to let them run in parallel?

---

<div class="post-metadata">

**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\
**Post date:** [September 20, 2023, 5:41pm UTC](https://community.dremio.com/t/dremio-parallel-job-execution/7393/10 "2023-09-20T17:41:26Z")

</div>

@popejune Operators in same phase will not run in parallel while all phases start in parallel
