# Dremio Query very slow **URL:** \ **Category:** Uncategorized\ **Created:** [December 23, 2019, 3:48pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816 "2019-12-23T15:48:43Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 23, 2019, 3:48pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/1 "2019-12-23T15:48:43Z")
I have facing issue with dremio query very slow here is logs ``` 2019-12-23 15:33:02,092 [21ff21d0-85fc-c4db-310e-bae3a0735100/0:foreman-planning] WARN c.d.e.p.a.MaterializationList - failed to expand materialization 4b90e379-6477-4a2d-b0b3-e7329bb25dc9 java.lang.IllegalArgumentException: Projected column base size %d is not equal to outcome rowtype %d. [370, 360] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:263) ~[guava-20.0.jar:na] at com.dremio.exec.planner.common.ScanRelBase.setProjectedRowType(ScanRelBase.java:266) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.common.ScanRelBase.(ScanRelBase.java:81) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.calcite.logical.ScanCrel.(ScanCrel.java:53) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.calcite.logical.ScanCrel.copyWith(ScanCrel.java:74) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at org.apache.calcite.plan.CopyWithCluster.visit(CopyWithCluster.java:389) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.rel.core.TableScan.accept(TableScan.java:167) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.plan.CopyWithCluster.visit(CopyWithCluster.java:241) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.rel.logical.LogicalProject.accept(LogicalProject.java:129) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.plan.CopyWithCluster.visit(CopyWithCluster.java:241) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:1.16.0-201909031936280585-3f76472] at org.apache.calcite.rel.logical.LogicalProject.accept(LogicalProject.java:129) ~[calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at com.dremio.exec.planner.acceleration.DremioMaterialization.accept(DremioMaterialization.java:242) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.acceleration.CachedMaterializationDescriptor.getMaterializationFor(CachedMaterializationDescriptor.java:49) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.acceleration.MaterializationList.build(MaterializationList.java:108) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.acceleration.MaterializationList$1.get(MaterializationList.java:50) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.acceleration.MaterializationList$1.get(MaterializationList.java:47) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:120) [guava-20.0.jar:na] at com.dremio.exec.planner.acceleration.MaterializationList.getMaterializations(MaterializationList.java:73) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.acceleration.substitution.AbstractSubstitutionProvider.getMaterializations(AbstractSubstitutionProvider.java:43) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.acceleration.substitution.DremioSubstitutionProvider.findSubstitutions(DremioSubstitutionProvider.java:123) [dremio-ce-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.acceleration.substitution.AccelerationAwareSubstitutionProvider.findSubstitutions(AccelerationAwareSubstitutionProvider.java:59) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.DremioVolcanoPlanner.registerMaterializations(DremioVolcanoPlanner.java:101) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:600) [calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at com.dremio.exec.planner.DremioVolcanoPlanner.findBestExp(DremioVolcanoPlanner.java:83) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:368) [calcite-core-1.16.0-201909031936280585-3f76472.jar:1.16.0-201909031936280585-3f76472] at com.dremio.exec.planner.sql.handlers.PrelTransformer.lambda$transform$2(PrelTransformer.java:498) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.PrelTransformer.doTransform(PrelTransformer.java:512) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.PrelTransformer.transform(PrelTransformer.java:505) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToDrel(PrelTransformer.java:229) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToDrel(PrelTransformer.java:318) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:47) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.commands.HandlerToPreparePlan.plan(HandlerToPreparePlan.java:89) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.work.foreman.AttemptManager.plan(AttemptManager.java:407) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.work.foreman.AttemptManager.lambda$run$0(AttemptManager.java:310) ~[dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.service.commandpool.CommandWrapper.run(CommandWrapper.java:62) ~[dremio-services-commandpool-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_222] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_222] ``` ---
**Author:** ![ben](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/ben/32/1367_2.png) [@ben](https://community.dremio.com/u/ben)\ **Post date:** [December 23, 2019, 7:17pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/2 "2019-12-23T19:17:57Z")
@Vikash_Singh, I’m not sure this is related to the slowness… do you have a job profile for 21ff21d0-85fc-c4db-310e-bae3a0735100? ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 24, 2019, 5:12am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/3 "2019-12-24T05:12:36Z")
> [@ben](#): > > 21ff21d0-85fc-c4db-310e-bae3a0735100 Hello Ben, i dont have same job profile but same error i have getting in other jobs too i am attaching that job profile.Also that error i am getting in loop every 2 sec its calling because of this also mine Query slow [cab16e6d-9e83-4c47-b481-44a85cc9bace.zip](https://community.dremio.com/uploads/short-url/i3BThcox2USzJV9WxOeyvXNaxd1.zip) (104.0 KB) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 25, 2019, 6:24pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/4 "2019-12-25T18:24:04Z")
@Vikash_Singh Is this the same as this [Community Post](https://community.dremio.com/t/com-microsoft-azure-storage-v10-adlsg2-models-datalakestorageerrorexception-status-code-404-error-code-pathnotfound-message-the-specified-path-does-not-exist/4824) ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 26, 2019, 4:31am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/5 "2019-12-26T04:31:39Z")
No both are diffrent post ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 26, 2019, 1:00pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/6 "2019-12-26T13:00:47Z")
@Vikash_Singh The profile you sent is “cab16e6d-9e83-4c47-b481-44a85cc9bace.zip” is for a failed job. Is there a profile for the slow running query? ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 26, 2019, 1:12pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/7 "2019-12-26T13:12:28Z")
> [@Vikash\_Singh](#): > > java.lang.IllegalArgumentException: Projected column base size %d is not equal to outcome rowtype %d. [370, 360] I cannot able to trace which job cause Query slow mainly i am getting this Error continues in Dremio log java.lang.IllegalArgumentException: Projected column base size %d is not equal to outcome rowtype %d. [370, 360] ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 26, 2019, 1:33pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/8 "2019-12-26T13:33:44Z")
@Vikash_Singh Send us the complete server.log as the log snippet you send does not give us complete details ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 26, 2019, 1:46pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/9 "2019-12-26T13:46:24Z")
where to find all dremio logs in container ? ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 26, 2019, 2:01pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/10 "2019-12-26T14:01:38Z")
@Vikash_Singh Are you on K8’s. Do a “ps -ef | grep dremio” and look for “-Ddremio.log.path” in the output ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 26, 2019, 2:10pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/11 "2019-12-26T14:10:57Z")
I am also facing Long running jobs some time half hour ,i have attached job profile ![Screenshot_11](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/c/c5add0188c040515fbf8d40e18576b08dddb00ac.png) [50fbc10c-c281-48f4-9742-e187427fa147.zip](https://community.dremio.com/uploads/short-url/fyVmJx1jzwk933Sfn7MfH1jDV0s.zip) (97.5 KB) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 26, 2019, 2:53pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/12 "2019-12-26T14:53:22Z")
@Vikash_Singh I see the query is waiting on CPU for the full duration. See attached screenshot. How many executors and cores per executor you have? I also see you are running concurrent queries so might be CPU starved ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 26, 2019, 3:28pm UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/13 "2019-12-26T15:28:52Z")
I have only one executer and I given 8 cpu,44 GB ram Some time I have concurrent query is dremio can’t handle this query ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 27, 2019, 4:33am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/14 "2019-12-27T04:33:33Z")
@balaji.ramaswamy any idea suggestion what i have to do to solve above issue ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 27, 2019, 4:45am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/15 "2019-12-27T04:45:51Z")
still so many jobs are in running state due to this my Bi dashboard is loading either slow or hanged please suggest necessary steps to solve above issue ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 27, 2019, 4:46am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/16 "2019-12-27T04:46:47Z")
Failure while attempting substitution, skipping materialization 67fb5e91-bc34-4492-be64-832ae6d31ed0 ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 27, 2019, 6:04am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/17 "2019-12-27T06:04:54Z")
@Vikash_Singh Have you considered increasing capacity in terms of number of executors? ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 27, 2019, 6:18am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/18 "2019-12-27T06:18:43Z")
do you want me to increae executor count to 2 and distribute cpu what i have earlier like 4 cpu each .i am also doing metadata cleanup ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 27, 2019, 6:38am UTC](https://community.dremio.com/t/dremio-query-very-slow/4816/19 "2019-12-27T06:38:20Z")
@Vikash_Singh, You would need to have a separate coordinator and 2 executor nodes to start with. Start with lower concurrency and then you can increase Thanks @balaji.ramaswamy