Error during execution of creation of a manuel aggregation Reflection

1

I get the following error IllegalReferenceCountException, that i can reproduce each time I create the aggregated reflection.
When i use the simple aggregation reflection wizard with the same dimensions/measures, the reflection is created and the execution works.
But when I create the reflection using the following script or if i use the advanced wizard,

ALTER TABLE “swn-sframe”.raw_wtb_actions ADD AGGREGATE REFLECTION
DIMENSIONS (ts BY DAY, other_dimensions
)
MEASURES (prc)
PARTITION BY (wtbid)
LOCALSORT BY (ts)
AS my_agg_reflection
,

i have the following error during the underlying execution of

CREATE TABLE “__accelerator”.“d1e1ca8a-4445-4c17-857f-851a62df6cc9/666221e9-0dcd-4b24-8295-5b311edb7ecc” HASH PARTITION BY (wtbid) LOCALSORT BY (ts) AS SELECT * FROM “__materialization”.“d1e1ca8a-4445-4c17-857f-851a62df6cc9”

:

SYSTEM ERROR: IllegalReferenceCountException: refCnt: 0

SqlOperatorImpl PROJECT
Location 1:4:2
Fragment 1:4

[Error Id: 4aac10fd-093e-4126-ab79-bc3d824dbee3 on dremio-coordinator-1:31010]

(io.netty.util.IllegalReferenceCountException) refCnt: 0
io.netty.buffer.AbstractByteBuf.ensureAccessible():1180
io.netty.buffer.ArrowBuf.checkIndexD():130
io.netty.buffer.ArrowBuf.checkBytes():155
org.apache.arrow.vector.util.ByteFunctionHelpers.compare():99
com.dremio.s.ProjectorGen261.doEval():82
com.dremio.sabot.op.project.ProjectorTemplate.projectRecords():53
com.dremio.sabot.op.project.ProjectOperator.outputData():192
com.dremio.sabot.driver.SmartOp$SmartSingleInput.outputData():177
com.dremio.sabot.driver.StraightPipe.pump():56
com.dremio.sabot.driver.Pipeline.doPump():82
com.dremio.sabot.driver.Pipeline.pumpOnce():72
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():288
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run():284
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1807
com.dremio.sabot.exec.fragment.FragmentExecutor.run():243
com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
com.dremio.sabot.task.AsyncTaskWrapper.run():92
com.dremio.sabot.task.slicing.SlicingThread.run():71

2

Hey @dfleckinger we’ve taken a quick look at the code causing the error. To better understand what going on, could you share:

  • Exact steps that causes the issue you are seeing.
  • Definition and datatypes for the “swn-sframe”.raw_wtb_actions dataset.
3

Hey @can,

  • exact step : I simply run the following query in Dremio UI to create the aggregate reflection :

ALTER TABLE “swn-sframe”.raw_wtb_actions ADD AGGREGATE REFLECTION
DIMENSIONS (
ts BY DAY, wtbid,action)
MEASURES (prc)
PARTITION BY (wtbid) LOCALSORT BY (ts)
AS my_agg

It seems the issue occurs when the clause PARTITION BY is used.

  • Definition and datatypes for the “swn-sframe”.raw_wtb_actions:
    “swn-sframe”.raw_wtb_actions source is a Parquet Dataset composed of multiple columns.
    for instance, for the columns used for the acceleration,
    ts is a TIMESTAMP,
    wtbid, action are VARCHAR,
    and prc is DOUBLE.
    Do you need all the columns of the dataset ie

SELECT * FROM INFORMATION_SCHEMA.COLUMNS
where TABLE_SCHEMA like ‘swn-sframe’ and TABLE_NAME = ‘raw_wtb_actions’

?
Thanks

4

Hey @can, with the version 1.4.4, I executed the same query and it worked. So it might be closed.

5

@dfleckinger great! Thanks for the update.