IllegalArgumentException: Comparison method violates its general contract! regarding HiveScanBatchCreator

1

Yellow,

Context:

  • Dremio 2.10 OSS edition;

  • Hive metastore (2.3.3);

  • Hadoop 3.0.1;

  • Parquet files (which can be read directly by Dremio but when going to the same files via Hive, we get the below weirdness);

  • everything containerized;

        SYSTEM ERROR: IllegalArgumentException: Comparison method violates its general contract!

Fragment 0:0

[Error Id: 0de46680-2612-4918-b76f-7751bc0a08c3 on i10-tracking-dremio-p.data.internaldomain.org:-1]

(java.lang.IllegalArgumentException) Comparison method violates its general contract!
java.util.ComparableTimSort.mergeLo():744
java.util.ComparableTimSort.mergeAt():481
java.util.ComparableTimSort.mergeCollapse():404
java.util.ComparableTimSort.sort():213
java.util.Arrays.sort():1312
java.util.Arrays.sort():1506
java.util.ArrayList.sort():1462
java.util.Collections.sort():141
com.dremio.exec.store.hive.exec.HiveScanBatchCreator.createNativeParquet():111
com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():257
com.dremio.exec.store.hive.exec.HiveScanBatchCreator.create():65
com.dremio.sabot.driver.OperatorCreatorRegistry.getProducerOperator():94
com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():89
com.dremio.sabot.driver.UserDelegatingOperatorCreator$4.run():86
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():422
org.apache.hadoop.security.UserGroupInformation.doAs():1807
com.dremio.sabot.driver.UserDelegatingOperatorCreator.getProducerOperator():86
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():210
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitSubScan():115
com.dremio.exec.physical.base.AbstractSubScan.accept():77
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
com.dremio.exec.physical.config.Limit.accept():55
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitLimit():109
com.dremio.exec.physical.config.Limit.accept():55
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():247
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp():115
com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject():84
com.dremio.exec.physical.config.Project.accept():53
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():234
com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen():115
com.dremio.exec.physical.config.Screen.accept():43
com.dremio.sabot.driver.PipelineCreator.get():107
com.dremio.sabot.driver.PipelineCreator.get():101
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():320
com.dremio.sabot.exec.fragment.FragmentExecutor.run():227
com.dremio.sabot.exec.fragment.FragmentExecutor.access$800():83
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():577
com.dremio.sabot.task.AsyncTaskWrapper.run():92
com.dremio.sabot.task.slicing.SlicingThread.run():71

2

Hello,

For anyone interested, fixed with -Djava.util.Arrays.useLegacyMergeSort=true
Original article: https://dertompson.com/2012/11/23/sort-algorithm-changes-in-java-7/

Either the Dremio team finds a fix for this or use the above define.
Now Hive works for us in Dremio using this (the code path was with Parquet native reader).

3

Hi @Antauri

This is a known issue in out current release. To throw more info on this. It happens when your physical Parquet files of Hive > 2GB and the parquet file is a multi row group.

We should be fixing this in the upcoming release (pretty soon)

Thanks,
@balaji.ramaswamy

4

Hello @balaji.ramaswamy

Thanks for the feedback. Anxiously awaiting the new version. The above work-around (of using the legacy merge sort in Java) has fixed this issue for now and we are able to query Hive. But an upstream fix is much awaited :slight_smile:

Thank you!