Dremio fault down and indexing si too much solw

#1

Hi all,

dremio version --> dremio-community-2.1.6
VM Centos 7 JVM java-1.8.0-openjdk-1.8.0.131-2.b11.el7_3.x86_64

config: Xmx 4096 MB direct memory 8192MB

dremio is often killed by OOM KILLER; the VM has 64GB of RAM.

The rockDB is 260GB but indexing lasts for more than 16 hours, is this normal?
At the moment it is using only 2 of the 16 cores for the indexing: can we set the number of threads for indexing?
Is it normal to have such a big rockDB?

Thanks in advance

#2

Hi @disoardi

Are you saying that Dremio crashes because of OOM killer and everytime on startup reindexing takes forever due to your rocksDB size of 260 GB

We handle this better in 3.0, can you please try to upgrade and retry?

More importantly we need to find out why you are running into OOM KILLER? Are their any memory consuming queries that were running when OOM KILLER happened?

Thanks
@balaji.ramaswamy

#3

Hi @balaji.ramaswamy

thank you for your fast reply.

This is the situation:
dremio 5 -15 46.839g 0.041t 3.004g S 21.9 67.1 1742:02

In this moments dremio have 4GB for Xmx and 8GB for direct memory (default configuration), but it consume (VIRT) over than 45G. it’s not normal even for this version.

We are waiting for the end of indexing for upgrade to v3 (wish me good luck for preserve all reflection in the upgrade).

In the meantime could you please suggest any improvements to accelerate the indexing process?

Best regrards

#4

Hi @disoardi

When you say VIRT over 45GB. What is consuming all that memory? Dremio jobs?

Thanks
@balaji.ramaswamy

#5

Yes, it’s all memory used by dremio jobs re-indexing.

#6

Hi @disoardi

Were you able to upgrade to 3.0.x?

Thanks
@balaji.ramaswamy