SYSTEM ERROR: NativeIOException: Operation not permitted

#1

Hi, getting the following error with a new setup when I try to run a query

Setup

AKS, 3 nodes, Azure Storage Gen 2
Cloud Cache enabled
all running under dremio user via ps -ef | grep dremo

Storage Container

image
image913×124 24.2 KB

ERROR

SYSTEM ERROR: NativeIOException: Operation not permitted

SqlOperatorImpl ARROW_WRITER
Location 0:0:3
SqlOperatorImpl ARROW_WRITER
Location 0:0:3
Fragment 0:0

[Error Id: a2b66576-8fca-407f-9df4-89aebbd816b5 on dremio-executor-0.dremio-cluster-pod.bd-dremio.svc.cluster.local:0]

  (org.apache.hadoop.io.nativeio.NativeIOException) Operation not permitted
    org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl():-2
    org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod():234
    org.apache.hadoop.fs.RawLocalFileSystem.setPermission():861
    org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode():547
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission():587
    org.apache.hadoop.fs.RawLocalFileSystem.mkdirs():559
    org.apache.hadoop.fs.RawLocalFileSystem.create():316
    org.apache.hadoop.fs.RawLocalFileSystem.create():351
    com.dremio.exec.store.dfs.PseudoDistributedFileSystem.create():369
    org.apache.hadoop.fs.FileSystem.create():1118
    org.apache.hadoop.fs.FileSystem.create():1098
    org.apache.hadoop.fs.FileSystem.create():987
    org.apache.hadoop.fs.FileSystem.create():975
    com.dremio.exec.hadoop.HadoopFileSystem.create():212
    com.dremio.exec.store.easy.arrow.ArrowRecordWriter.setup():98
    com.dremio.sabot.op.writer.WriterOperator.setup():107
    com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup():255
    com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():73
    com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():63
    com.dremio.sabot.driver.SmartOp$SmartSingleInput.accept():200
    com.dremio.sabot.driver.StraightPipe.setup():103
    com.dremio.sabot.driver.StraightPipe.setup():102
    com.dremio.sabot.driver.StraightPipe.setup():102
    com.dremio.sabot.driver.StraightPipe.setup():102
    com.dremio.sabot.driver.Pipeline.setup():68
    com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():391
    com.dremio.sabot.exec.fragment.FragmentExecutor.run():273
    com.dremio.sabot.exec.fragment.FragmentExecutor.access$1400():94
    com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():711
    com.dremio.sabot.task.AsyncTaskWrapper.run():112
    com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():228
    com.dremio.sabot.task.slicing.SlicingThread.run():159
#2

@rwetzeler

Is this AWSE? Results go to local. Is there permission to write results to local?

Thanks
Bali

#3

This is running in Azure Kubernetes (AKS). Permissions appear to work when I upload a file as it works. But when I query that file, I get the error above.

#4

@rwetzeler

That is right, Dremio is unable to write results to that folder, see stack below

org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmodImpl():-2
org.apache.hadoop.io.nativeio.NativeIO$POSIX.chmod():234
org.apache.hadoop.fs.RawLocalFileSystem.setPermission():861