Hello
We get following message : Node [rel#84883210:Subset#54.PHYSICAL.SINGLETON().] could not be implemented.
We are running DREMIOEE 20.4.0.
Regards
@Arnie Wondering if the job failed with a OOM and the second attempt failed with a planner error. Are you able to send the job profile?
Hello
You are right (OOM in Attempt 1 and Physical.Singleton in Attempt2.
In Raw Profile tab we see : Attempt 2 (insufficient memory) and Attempt 1 (OOM).
I cannot upload full Profiles (no way to get zip …) :
- Content of Attempt 1:
OUT_OF_MEMORY ERROR: Query was cancelled because it exceeded the memory limits set by the administrator.
Failure allocating buffer.
Allocator dominators:
Allocator(ROOT) 0/24896816028/28586086242/31138512896 (res/actual/peak/limit) numChildAllocators:25
Allocator(queue-9afd309d-1c04-4185-81e5-b486aa0a1950) 0/24791843740/28197236962/9223372036854775807 (res/actual/peak/limit) numChildAllocators:1
Allocator(query-1cb0643f-548e-e6f3-238f-3dd9a3e3a300) 0/24791819164/24809964444/9223372036854775807 (res/actual/peak/limit) numChildAllocators:5
Allocator(phase-3) 0/20123700608/20125273472/9223372036854775807 (res/actual/peak/limit) numChildAllocators:14
Allocator(frag:3:16) 12000000/3327518336/3327518336/9223372036854775807 (res/actual/peak/limit) numChildAllocators:13
Allocator(op:3:16:6:HashJoinPOP) 1000000/3222872064/3222872064/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(op:3:16:incoming) 0/67633152/109576192/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(frag:3:1) 12000000/2951128704/2951128704/9223372036854775807 (res/actual/peak/limit) numChildAllocators:13
Allocator(op:3:1:6:HashJoinPOP) 1000000/2901008384/2901008384/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(op:3:1:0:HashPartitionSender) 26102496/27013120/27013120/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(phase-6) 0/3371668700/3437233884/9223372036854775807 (res/actual/peak/limit) numChildAllocators:14
Allocator(frag:6:13) 4000000/502756676/503805252/9223372036854775807 (res/actual/peak/limit) numChildAllocators:5
Allocator(op:6:13:3:ParquetSubScan) 1000000/441229508/442278084/9223372036854775807 (res/actual/peak/limit) numChildAllocators:2
Allocator(row-group-reader) 0/247726080/248774656/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(row-group-reader) 0/166985728/166985728/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(op:6:13:0:HashPartitionSender) 26727456/59527168/59543552/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(frag:6:10) 4000000/501708100/503805252/9223372036854775807 (res/actual/peak/limit) numChildAllocators:5
Allocator(op:6:10:3:ParquetSubScan) 1000000/440180932/442278084/9223372036854775807 (res/actual/peak/limit) numChildAllocators:2
Allocator(row-group-reader) 0/166985728/167510016/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(row-group-reader) 0/246677504/248774656/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(op:6:10:0:HashPartitionSender) 26727456/59527168/59543552/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
Allocator(fabric-allocator) 104857600/1441792/160464896/9223372036854775807 (res/actual/peak/limit) numChildAllocators:0
[Error Id: 761f7b98-7257-4bdd-8e03-31142e888071 ]
(org.apache.arrow.memory.OutOfMemoryException) Failure allocating buffer.
io.netty.buffer.PooledByteBufAllocatorL.allocate():67
org.apache.arrow.memory.NettyAllocationManager.():77
org.apache.arrow.memory.NettyAllocationManager.():84
org.apache.arrow.memory.NettyAllocationManager$1.create():34
org.apache.arrow.memory.BaseAllocator.newAllocationManager():315
org.apache.arrow.memory.BaseAllocator.newAllocationManager():310
org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation():298
org.apache.arrow.memory.BaseAllocator.buffer():276
org.apache.arrow.memory.BaseAllocator.buffer():240
com.dremio.exec.rpc.MessageDecoder.decodeMessage():172
com.dremio.exec.rpc.MessageDecoder.decode():69
com.dremio.services.fabric.FabricProtobufLengthDecoder.decode():37
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection():498
io.netty.handler.codec.ByteToMessageDecoder.callDecode():437
io.netty.handler.codec.ByteToMessageDecoder.channelRead():276
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():379
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():365
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead():357
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead():1410
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():379
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():365
io.netty.channel.DefaultChannelPipeline.fireChannelRead():919
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read():163
io.netty.channel.nio.NioEventLoop.processSelectedKey():714
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized():650
io.netty.channel.nio.NioEventLoop.processSelectedKeys():576
io.netty.channel.nio.NioEventLoop.run():493
io.netty.util.concurrent.SingleThreadEventExecutor$4.run():989
io.netty.util.internal.ThreadExecutorMap$2.run():74
java.lang.Thread.run():748
Caused By (io.netty.util.internal.OutOfDirectMemoryError) failed to allocate 16777216 byte(s) of direct memory (used: 31132221719, max: 31138512896)
io.netty.util.internal.PlatformDependent.incrementMemoryCounter():742
io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner():697
io.netty.buffer.PoolArena$DirectArena.allocateDirect():758
io.netty.buffer.PoolArena$DirectArena.newChunk():734
io.netty.buffer.PoolArena.allocateNormal():245
io.netty.buffer.PoolArena.allocate():227
io.netty.buffer.PoolArena.allocate():147
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL():181
io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer():214
io.netty.buffer.PooledByteBufAllocatorL.allocate():58
org.apache.arrow.memory.NettyAllocationManager.():77
org.apache.arrow.memory.NettyAllocationManager.():84
org.apache.arrow.memory.NettyAllocationManager$1.create():34
org.apache.arrow.memory.BaseAllocator.newAllocationManager():315
org.apache.arrow.memory.BaseAllocator.newAllocationManager():310
org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation():298
org.apache.arrow.memory.BaseAllocator.buffer():276
org.apache.arrow.memory.BaseAllocator.buffer():240
com.dremio.exec.rpc.MessageDecoder.decodeMessage():172
com.dremio.exec.rpc.MessageDecoder.decode():69
com.dremio.services.fabric.FabricProtobufLengthDecoder.decode():37
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection():498
io.netty.handler.codec.ByteToMessageDecoder.callDecode():437
io.netty.handler.codec.ByteToMessageDecoder.channelRead():276
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():379
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():365
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead():357
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead():1410
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():379
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():365
io.netty.channel.DefaultChannelPipeline.fireChannelRead():919
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read():163
io.netty.channel.nio.NioEventLoop.processSelectedKey():714
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized():650
io.netty.channel.nio.NioEventLoop.processSelectedKeys():576
io.netty.channel.nio.NioEventLoop.run():493
io.netty.util.concurrent.SingleThreadEventExecutor$4.run():989
io.netty.util.internal.ThreadExecutorMap$2.run():74
java.lang.Thread.run():748
Content of Attempt 2
Node [rel#87896989:Subset#54.PHYSICAL.SINGLETON().] could not be implemented; planner state:
@Arnie Looks like this query used most of the 29 GB direct memory you have per node