Fatal error Java runtime after Dremio upgrade from 4.1.0 to 4.8.0

[vmois]

I have an EC2 instance with Dremio installed. Instance acts as master and coordinator node at the same time, no more Dremio instances exist. Dremio runs inside the Docker container with some volumes mapped.

Steps that were taken to updated Dremio:

1. Stop Dremio container
2. Update Docker image (from 4.1.0 to 4.8.0) by installing a new version from tar
3. Run image, log in to Docker image, run dremio-admin upgrade
4. After, start Dremio in the running container

After starting Dremio again error occurs. After spending some time searching on the Internet, I could find anything that could help solve the issue. You can find server.out and error report in logs.zip.

It doesn’t look like an error with Dremio, more like with Java. I do not know a lot about Java Runtime. Would really appreciate help with that. Thank you.

OS: Debian 9 (container)
OpenJDK 8

logs.zip (28.4 KB)

[balaji.ramaswamy]

@vmois

What version of JDK is this? Also is this openJDK?

[vmois]

My image is based on OpenJDK 8 Docker container. Should I try a newer version?

[laurent]

By any chance, have you simply uncompressed Dremio 4.8.0 on top of Dremio 4.1.0 in your current image, or did you start from the openjdk8 docker image?

[vmois]

My base image is openjdk:8-jdk. Example of my Dockerfile:

FROM openjdk:8-jdk

ARG DOWNLOAD_URL=https://download.dremio.com/community-server/4.8.0-202009180238090394-b0a98b73/dremio-community-4.8.0-202009180238090394-b0a98b73.tar.gz
RUN \
  mkdir -p /opt/dremio \
  && wget -q "${DOWNLOAD_URL}" -O dremio.tar.gz \
  && tar vxfz dremio.tar.gz -C /opt/dremio --strip-components=1 \
  && rm -rf dremio.tar.gz

WORKDIR /opt/dremio

ADD libs jars
ADD conf/dremio.conf conf
ADD conf/dremio-env conf

EXPOSE 9047 31010 45678 47470

ENV DREMIO_HOME /opt/dremio
ENTRYPOINT ["bin/dremio", "start-fg"]

What I did for the Dremio update is replaced the DOWNLOAD_URL from version 4.1.0 to 4.8.0 and re-run the build.

[vmois]

Also, will add extra JAVA runtime params from dremio conf:

DREMIO_JAVA_SERVER_EXTRA_OPTS="
-XX:+UseG1GC 
-XX:G1HeapRegionSize=8M 
-XX:InitiatingHeapOccupancyPercent=25 
-XX:+AggressiveOpts 
-XX:+OptimizeStringConcat 
-XX:+UseFastAccessorMethods"

Maybe, they affect the runtime and cause an error. @balaji.ramaswamy @laurent

[laurent]

I would try to disable the following options and see if the problem disappear or not:

-XX:+AggressiveOpts 
-XX:+OptimizeStringConcat 
-XX:+UseFastAccessorMethods

Note that Dremio also publishes a Docker image available at https://hub.docker.com/r/dremio/dremio-oss

[vmois]

Removing Java options doesn’t help. I will try to use the official Docker image.

[vmois]

My Docker image is essentially the same as the official Docker image.

I checked the layers of the newest image (docker history -no-trunc dremio/dremio-oss:4.8.0). Also, checked the old Dockerfile (https://github.com/dremio/dremio-cloud-tools/blob/master/images/dremio-oss/Dockerfile). Generals Dremio install steps are the same, the same openjdk-8.

[vmois]

From logs, ‘aprMajorVersion’ ‘()I’ in ‘io/netty/internal/tcnative/Library’. Maybe, something there.

[vmois]

@balaji.ramaswamy @laurent any ideas regarding the issue?

[laurent]

I tried to reproduce with no luck. I noticed that your Docker includes external jars. Without the list of those, I cannot reproduce exactly your situation, but this is very likely that one of them might include a copy of netty native library in some form which might conflict and cause the crash you’re experiencing.

[vmois]

You can be right. The only external jar is dremio-flight-connector-0.11.0-SNAPSHOT-shaded.jar. This is quite an old version. The newest is 0.26.0. I will try to run dremio with this version.

[vmois]

Using dremio-flight-connector version 0.26.0 solved the problem. Essentially, problem with external jars. Dremio is not failing shortly after the start and continues to work. Thank you very much for pointing in the right direction @laurent!

[laurent]

Check out Dremio 4.9.1 recently released: flight connector is now built in.