Dremio crash during Reflection preview

1

Hello,
I’ve installed Dremio on various servers and one of them has systematic crashes during reflection refresh, the coredump and hs_err file are not really self explaining.
It looks like this issue Dump thread java when execution Gandiva Execution (fixed in 4.7.3 but I’m running 4.8 and it’s not fixed).

Here is the hs_err file is you can investigate.
hs_err.zip (52.0 KB)

Setup:

  • RHEL7 Virtualized with OpenJDK-1.8

Merci,
Dimitri BAELI

2

@dbaeli

Can you please provide the proile?

1 Like
3

Hello,
Sorry for the delay, I missed the answer notification.
I reproduced this crash on a fresh RHEL / Dremio 4.8 install with the sample sources.

This is the only job request played, not related to the crash:
3a5aba60-f359-45d7-819f-62a7acbed900.zip (9.3 KB)

The Error Log zip file is here:
hs_err_pid86628.log.zip (54.1 KB)

To reproduce the crash:

  • Create a Sample Source (from the Dremio sample)
  • Format the source : SF weather 2018-2019.csv (headers, and \n as end of line)
  • Go to Reflections, enable raw reflections and Save
  • The dremio server crashes at the save

Thank you,
Dimitri

4

@dbaeli

Thanks for the profile and Error file, we will check on this and get back to you

Thanks
Bali

5

Update :

  • Same crash on Ubuntu Dremio 4.8 (on an Hyper-V VM)
  • No Crash on Ubuntu or RHEL 7 on Dremio 4.1.8 (on an Hyper-V VM)
  • No Crash on a Physical server with RHEL 7
6

We’ve tried to fix the Allocated memory at Hyper-V level, same error.

Last lines of logs are the following:

Creating gandiva cache with capacity: 500

A fatal error has been detected by the Java Runtime Environment:

SIGILL (0x4) at pc=0x00007f08786240c0, pid=11190, tid=0x00007f085ffff700

I’ll do a test with each RPM version to find the first failing (between 4.1.8 and 4.8.0)

7

After testing quite all, 4.1.8 is the last one working correctly. Since 4.2.0 it’s broken.

Any idea of when you can tell us what causes this issue and if you’re able to fix this issue (like the gandiva one).

Thank you in advance

8

@dbaeli

We have this as a To Do for this week and will update on any findings

9

Any update on this one ? Any user on Dremio RHEL + Hyper-V to better understand if a specific setting on their side was required ?

10

@dbaeli

Just went through your reproduction steps and everything went fine, I see you have JRE, wondering if you can start Dremio using JDK 8 and retry?

11

Hi,

Yes it’s the JDK (OpenJDK)

Did you try with Hyper-V ? It works fine for us on a physical hardware, and it fails only with RHEL + Hyper-V.

12

How to determine wether JDK or JRE is used?
“java -version” returns:

openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)
13

@Muffex

output says JDK,

Do you see tools like jstack, jmap under “bin”? Then is it is JDK

14

@balaji.ramaswamy

I can’t find these tools in /bin/

I ran
sudo yum install java-1.8.0-openjdk
to install the JDK.

But these files are still missing in /bin/

regards

15

@Muffex

Can you please do the below
which java -->> This will give a symlink, follow the symlink and see if you can go to bin and find the tools

What is the path you did not find the java tools?

16

@balaji.ramaswamy

The symlink points to “/etc/alternatives/java” and thus to a file.
The folder “/etc/alternatives” does also not contain the mentioned tools.

regards

17

@Muffex

Can you please try this?

sudo apt-get install openjdk-8*

You probably need devel

18

@balaji.ramaswamy

Your answer pointed me in the right direction.

I found the installation command in here:
https://www.digitalocean.com/community/tutorials/how-to-install-java-on-centos-and-fedora

The installation command for Fedora and CentOS is:
sudo yum install java-1.8.0-openjdk-devel

regards