NessieNotFoundException: Requested contents do not exist for specified reference

Hello,

we ran some performance tests on our dremio deployment in EKS, which included connecting S3 source and continuously adding new PDS and VDS from this source.

After this test we keep seeing this error

org.projectnessie.error.NessieNotFoundException: Requested contents do not exist for specified reference.
	at org.projectnessie.services.rest.ContentsResource.getContents(ContentsResource.java:67)
	at com.dremio.service.nessie.ContentsApiService.getContents(ContentsApiService.java:50)
	at com.dremio.service.nessieapi.ContentsApiGrpc$MethodHandlers.invoke(ContentsApiGrpc.java:301)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:180)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.opentracing.contrib.grpc.TracingServerInterceptor$2.onHalfClose(TracingServerInterceptor.java:231)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$1.onHalfClose(TransmitStatusRuntimeExceptionInterceptor.java:74)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:814)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Which happens when we try to force refresh metadata of a pds using ALTER PDS ... FORGET METADATA query. There are also logs like

2021-11-30 15:14:08,187 [grpc-default-executor-19171] WARN  com.dremio.common.util.Retryer - Retry attempt 1 for the failure at com.dremio.service.nessie.NessieKVVersionStore:commit:230, Error - A new change was committed concurrently on branch/tag BranchName{name=main}

2021-11-30 15:14:07,586 [grpc-default-executor-19182] INFO  c.d.s.nessie.NessieKVVersionStore - NessieKVVersionStore commit() took longer than 1s: 4344ms total

I am trying to understand the context of these logs as they dont seem to be connected to any particular dataset (at least it is not obvious to me). Any help will be appreciated.

Thank you, Igor

@igorsechyn The message is harmless and have filed an internal ticket to investigate