Dremio 4 failed to read parquet

1

Hi Dremio 4.0.2 running in container and failed reading parquet file:

Error in parquet reader (complex). Message: Failure in setting up reader Parquet Metadata: ParquetMetaData{FileMetaData{schema: message spark_schema { optional int64 dsid; optional binary consumer_id (STRING); optional binary person_id (STRING); optional binary derived_key; } , metadata: {org.apache.spark.sql.parquet.row.metadata={"type":"struct","fields":[{"name":"dsid","type":"long","nullable":true,"metadata":{}},{"name":"consumer_id","type":"string","nullable":true,"metadata":{}},{"name":"person_id","type":"string","nullable":true,"metadata":{}},{"name":"derived_key","type":"binary","nullable":true,"metadata":{}}]}}}, blocks: [BlockMetaData{567053, 60113622 [ColumnMetaData{SNAPPY [dsid] optional int64 dsid [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData{SNAPPY [consumer_id] optional binary consumer_id (STRING) [RLE, BIT_PACKED, PLAIN], 2939903}, ColumnMetaData{SNAPPY [person_id] optional binary person_id (STRING) [RLE, BIT_PACKED, PLAIN], 21515693}, ColumnMetaData{SNAPPY [derived_key] optional binary derived_key [RLE, BIT_PACKED, PLAIN], 40091600}]}]}

Note: if I run the same distr with the same config from edge node on the cluster all is fine. Some libs missing?

Update: Logs show java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

2

Turned out that on container it was jdk11 and su.misc.Unsafe was moved out of there. Switching back to jdk8 resolved the issue

3

Thanks for the update @comphead

4

I have installed dremio in ubuntu, I am receiving below error while loading sample csv file in Dremio

sun.misc.Unsafe or java.nio.DirectByteBuffer.(long, int) not available

I also removed jdk 11 and my jdk version is JDK 8 now, Still i do receive same error.Please help ASAP

java -version
openjdk version “1.8.0_232”
OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
@balaji.ramaswamy

5

I would recommend double check that your java env when you starting dremio is exactly 1.8, not just java -version

6

@nitil.dwivedi

best is to the below line in dremio-env and restart Dremio
JAVA_HOME=path
(or)
export JAVA_HOME=path

7

1- JAVA_HOME is set executing below command for same session.
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export JAVA_HOME
export PATH

2- echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

3-service --status-all
Output

[ - ] dremio

4-
sudo systemctl restart dremio

Output

Job for dremio.service failed because the control process exited with error code.
See “systemctl status dremio.service” and “journalctl -xe” for details.

4- systemctl status dremio.service

Output

● dremio.service - LSB: This service manages the Dremio server
Loaded: loaded (/etc/init.d/dremio; generated)
Active: failed (Result: exit-code) since Thu 2019-12-26 16:02:35 IST; 14s ago
Docs: man:systemd-sysv-generator(8)
Process: 31446 ExecStart=/etc/init.d/dremio start (code=exited, status=1/FAILU
Tasks: 120 (limit: 4915)
CGroup: /system.slice/dremio.service
└─1016 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -Djava.util.loggin

pam_unix(runuser:session): session opened
+=========================================
| Error: JAVA_HOME is not set and Jav
±----------------------------------------
| Dremio requires Java 1.8 or later.
+=========================================
pam_unix(runuser:session): session closed
dremio.service: Control process exited, code=
dremio.service: Failed with result 'exit-code
Failed to start LSB: This service manages the

From the above commands you could see even setting up JAVA_HOME for particular session, dremio services are not starting up and tries to refer JDK 11, which was uninstalled

@balaji.ramaswamy

8

@nitil.dwivedi

Did you say you set JAVA_HOME, inside dremio-env and still running into issues?

If not please set it in dremio-env
Also can you try to start dremio from “bin/dremio start” instead of systemctl

9

The issue is resolved, after removing all the links to jdk 11, as dreimo services were referring jdk 11.

this’s how it looks now after referringto jdk 8

CGroup: /system.slice/dremio.service
└─1621 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djava.util.logging.config.class=org.slf4j.bridge.SLF4JBridgeHandler -Djava.library.path=/opt/dremio/lib -X