Dremio crash during Reflection preview

Hello,
I’ve installed Dremio on various servers and one of them has systematic crashes during reflection refresh, the coredump and hs_err file are not really self explaining.
It looks like this issue Dump thread java when execution Gandiva Execution (fixed in 4.7.3 but I’m running 4.8 and it’s not fixed).

Here is the hs_err file is you can investigate.
hs_err.zip (52.0 KB)

Setup:

  • RHEL7 Virtualized with OpenJDK-1.8

Merci,
Dimitri BAELI

@dbaeli

Can you please provide the proile?

1 Like

Hello,
Sorry for the delay, I missed the answer notification.
I reproduced this crash on a fresh RHEL / Dremio 4.8 install with the sample sources.

This is the only job request played, not related to the crash:
3a5aba60-f359-45d7-819f-62a7acbed900.zip (9.3 KB)

The Error Log zip file is here:
hs_err_pid86628.log.zip (54.1 KB)

To reproduce the crash:

  • Create a Sample Source (from the Dremio sample)
  • Format the source : SF weather 2018-2019.csv (headers, and \n as end of line)
  • Go to Reflections, enable raw reflections and Save
  • The dremio server crashes at the save

Thank you,
Dimitri

@dbaeli

Thanks for the profile and Error file, we will check on this and get back to you

Thanks
Bali

Update :

  • Same crash on Ubuntu Dremio 4.8 (on an Hyper-V VM)
  • No Crash on Ubuntu or RHEL 7 on Dremio 4.1.8 (on an Hyper-V VM)
  • No Crash on a Physical server with RHEL 7

We’ve tried to fix the Allocated memory at Hyper-V level, same error.

Last lines of logs are the following:

Creating gandiva cache with capacity: 500

A fatal error has been detected by the Java Runtime Environment:

SIGILL (0x4) at pc=0x00007f08786240c0, pid=11190, tid=0x00007f085ffff700

I’ll do a test with each RPM version to find the first failing (between 4.1.8 and 4.8.0)

After testing quite all, 4.1.8 is the last one working correctly. Since 4.2.0 it’s broken.

Any idea of when you can tell us what causes this issue and if you’re able to fix this issue (like the gandiva one).

Thank you in advance

@dbaeli

We have this as a To Do for this week and will update on any findings

Any update on this one ? Any user on Dremio RHEL + Hyper-V to better understand if a specific setting on their side was required ?

@dbaeli

Just went through your reproduction steps and everything went fine, I see you have JRE, wondering if you can start Dremio using JDK 8 and retry?