Query cancelled due to OutOfMemoryException

1

The problem does not seem to be new, but in dremio 19.5, we receive OutOfMemoryException when we execute certain queries, that will abort then.

Extract from data/log/server.log from one of the executors:

2022-08-09 12:02:44,117 [e5 - 1d0db365-4b9e-c55a-d7a2-f0dd2c86cb00:frag:31:11] INFO  c.d.e.s.parquet.ScanTableFunction - Filter received from Frag 19, OpId 1245188 minor fragment 2 into op 2031618
2022-08-09 12:02:44,117 [e5 - 1d0db365-4b9e-c55a-d7a2-f0dd2c86cb00:frag:31:11] INFO  c.d.e.s.parquet.ScanTableFunction - Filter received from Frag 19, OpId 1245188 minor fragment 0 into op 2031618
2022-08-09 12:03:00,153 [e4 - 1d0db365-4b9e-c55a-d7a2-f0dd2c86cb00:frag:153:0] ERROR c.d.e.s.d.i.AdditionalColumnsRecordReader - Query was cancelled because it exceeded the memory limits set by the administrator.
com.dremio.common.exceptions.UserException: Query was cancelled because it exceeded the memory limits set by the administrator.
	at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:885)
	at com.dremio.exec.store.dfs.implicit.AdditionalColumnsRecordReader.userExceptionWithDiagnosticInfo(AdditionalColumnsRecordReader.java:128)
	at com.dremio.exec.store.dfs.implicit.AdditionalColumnsRecordReader.next(AdditionalColumnsRecordReader.java:101)
	at com.dremio.exec.store.parquet.ScanTableFunction.processRow(ScanTableFunction.java:188)
	at com.dremio.sabot.op.tablefunction.TableFunctionOperator.outputData(TableFunctionOperator.java:101)
	at com.dremio.sabot.driver.SmartOp$SmartSingleInput.outputData(SmartOp.java:188)
	at com.dremio.sabot.driver.StraightPipe.pump(StraightPipe.java:56)
	at com.dremio.sabot.driver.Pipeline.doPump(Pipeline.java:108)
	at com.dremio.sabot.driver.Pipeline.pumpOnce(Pipeline.java:98)
	at com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run(FragmentExecutor.java:345)
	at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:294)
	at com.dremio.sabot.exec.fragment.FragmentExecutor.access$1600(FragmentExecutor.java:94)
	at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:747)
	at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:112)
	at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:243)
	at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:171)
Caused by: org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of size 32768 (rounded from 25092) due to memory limit. Current allocation: 2753536
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:270)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:240)
	at com.dremio.exec.store.dfs.implicit.ConstantColumnPopulators$VarCharPopulator.populate(ConstantColumnPopulators.java:1075)
	at com.dremio.exec.store.dfs.implicit.AdditionalColumnsRecordReader.next(AdditionalColumnsRecordReader.java:98)
	... 13 common frames omitted

How to get this debugged?

2

@h.hansmeier The job profile (error tab), should tell you how many queries were running and total memory foot print, the query page should tell you which phase and operator too k most memory