I have 300k records in my dataset, they are partitioned into 30 date folder.
when I query any data for one date, below error error.
I think the key words are “RecordPruner - Encountered exception during row count estimation” and “Memory was leaked by query”.
My setting is:
DREMIO_MAX_HEAP_MEMORY_SIZE_MB=16384
DREMIO_MAX_DIRECT_MEMORY_SIZE_MB=204800
Dremio 24.1
2023-08-25 08:32:59,701 [1b1798c6-ce8e-a804-d45e-d16d39d71d00/0:foreman-planning] ERROR o.apache.arrow.memory.BaseAllocator - Memory was leaked by query. Memory leaked: (16384)
Allocator(p-s-r-multicob) 0/16384/32768/9223372036854775807 (res/actual/peak/limit)
2023-08-25 08:32:59,702 [1b1798c6-ce8e-a804-d45e-d16d39d71d00/0:foreman-planning] ERROR c.d.e.p.l.partition.RecordPruner - Encountered exception during row count estimation:
java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (16384)
Allocator(p-s-r-multicob) 0/16384/32768/9223372036854775807 (res/actual/peak/limit)
at org.apache.arrow.memory.BaseAllocator.close(BaseAllocator.java:437)
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:139)
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:76)
at com.dremio.sabot.exec.context.OperatorContextImpl.close(OperatorContextImpl.java:322)
at com.dremio.exec.planner.logical.partition.PartitionStatsBasedPruner.prune(PartitionStatsBasedPruner.java:626)
at com.dremio.exec.ops.QueryContext.getSurvivingRowCountWithPruneFilter(QueryContext.java:583)
at com.dremio.exec.planner.logical.partition.PruneFilter.doRewrite(PruneFilter.java:126)
at com.dremio.exec.planner.logical.partition.PruneFilter.safeRewrite(PruneFilter.java:82)
at com.dremio.exec.planner.logical.partition.CompositeFilterPushdown.rewrite(CompositeFilterPushdown.java:122)
at com.dremio.exec.planner.logical.partition.CompositeFilterPushdown.doOnMatch(CompositeFilterPushdown.java:62)
at com.dremio.exec.planner.logical.partition.CompositeFilterPushdown$FilterOnScanRule.onMatch(CompositeFilterPushdown.java:193)
at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:214)
at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:650)
at com.dremio.exec.planner.DremioVolcanoPlanner.findBestExp(DremioVolcanoPlanner.java:101)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:321)
at com.dremio.exec.planner.sql.handlers.PrelTransformer.lambda$transform$1(PrelTransformer.java:621)
at com.dremio.exec.planner.sql.handlers.PrelTransformer.doTransform(PrelTransformer.java:659)
at com.dremio.exec.planner.sql.handlers.PrelTransformer.transform(PrelTransformer.java:634)
at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToDrel(PrelTransformer.java:296)
at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToDrel(PrelTransformer.java:433)
at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:87)
at com.dremio.exec.planner.sql.handlers.commands.HandlerToPreparePlanBase.plan(HandlerToPreparePlanBase.java:91)
at com.dremio.exec.work.foreman.AttemptManager.plan(AttemptManager.java:571)
at com.dremio.exec.work.foreman.AttemptManager.lambda$run$4(AttemptManager.java:462)
at com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3(ReleasableBoundCommandPool.java:140)
at com.dremio.service.commandpool.CommandWrapper.run(CommandWrapper.java:70)
at com.dremio.context.RequestContext.run(RequestContext.java:96)
at com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4(ContextMigratingExecutorService.java:226)
at com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run(ContextMigratingExecutorService.java:206)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
