Hello,
I am working with the latest Dremio Docker image, dremio/dremio-oss:latest, which looks to be version 14. I am using Dremio to connect to an Azure Blob Storage v2, which has a folder with Parquet files. I can build out a connection to ADLS and set up a dataset with the Parquet files. This works without any issue. However, I am not able to use any window functions in my queries. The two window functions I used were row_number() and count(), and both resulted in a UnsupportedOperationException error.
The exact error was as follows:
SYSTEM ERROR: UnsupportedOperationException
SqlOperatorImpl HASH_PARTITION_SENDER
Location 2:0:0
SqlOperatorImpl HASH_PARTITION_SENDER
Location 2:0:0
Fragment 2:0
[Error Id: d8f0553c-8b83-45dd-a2be-dc7176c4485a on 15da02e03768:0]
(java.lang.UnsupportedOperationException) null
org.apache.arrow.vector.complex.UnionVector.getValidityBufferAddress():241
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.():63
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.():41
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier$GenericCopier.():372
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.addValueCopier():449
com.dremio.sabot.op.sender.partition.vectorized.MultiDestCopier.getCopiers():477
com.dremio.sabot.op.sender.partition.vectorized.VectorizedPartitionSenderOperator.setup():185
com.dremio.sabot.driver.SmartOp$SmartTerminal.setup():346
com.dremio.sabot.driver.Pipe$SetupVisitor.visitTerminalOperator():85
com.dremio.sabot.driver.Pipe$SetupVisitor.visitTerminalOperator():63
com.dremio.sabot.driver.SmartOp$SmartTerminal.accept():290
com.dremio.sabot.driver.StraightPipe.setup():103
com.dremio.sabot.driver.Pipeline.setup():68
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():392
com.dremio.sabot.exec.fragment.FragmentExecutor.run():273
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1400():94
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():712
com.dremio.sabot.task.AsyncTaskWrapper.run():112
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():228
com.dremio.sabot.task.slicing.SlicingThread.run():159
I also tried version 13.1 and encountered the same error.
Is it possible to issue window functions to Parquet files on ADLS?