Dremio using too much /tmp disk space

1

Hi Team I have recently Installed Dremio on CentOS 7 like below and when I start the dremio service when it keep running Dremio using too much /tmp disk space some time my server is hanging due to that

#Install script
sudo groupadd -r dremio
sudo useradd -r -g dremio -d /var/lib/dremio -s /sbin/nologin dremio
sudo mkdir /opt/dremio
sudo mkdir /opt/dremio/log && sudo chown dremio : dremio /opt/dremio/log
sudo mkdir /opt/dremio/run && sudo chown dremio : dremio /opt/dremio/run
sudo mkdir /opt/dremio/data && sudo chown dremio : dremio /opt/dremio/data
# Latest version link can be obtained on https://www.dremio.com/download/ under 'TAR'
cd /opt
wget https : //download.dremio.com/community-server/dremio-community-LATEST.tar.gz
sudo tar -xvf dremio-community-LATEST.tar.gz -C /opt/dremio/ --strip-components=1
sudo ln -s /opt/dremio/conf /etc/dremio
sudo cp /opt/dremio/share/dremio.service /etc/systemd/system/dremio.service
# Add service
sudo systemctl daemon-reload
sudo systemctl start dremio
# Optionally, set Dremio service to start on boot:
sudo systemctl enable dremio

It’s created files like below inside the /tmp appreciate your support regarding this issue

-rw-r–r--. 1 dremio dremio 8212269 Oct 22 02:06 librocksdbjni2977380202217409908.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 22 02:05 librocksdbjni408510301532648189.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 22 02:06 librocksdbjni470331870951617463.so

Thank you,

Nuwan

2

@nuwan1

If you are having limited disk space on /tmp, then you can reconfigure your dremio-env so we put our temporary files in a different filesystem/folder, use below in dremio-env and restart coordinator

DREMIO_JAVA_EXTRA_OPTS=“Djava.io.tmpdir=/lot-of-free-space/dremio/tmp”

Thanks
@balaji.ramaswamy

3

Hello Balaji,

First, thank you very much for this valuable tip. A few more questions in case you know:
How often OR what are the criteria for a cronjob to remove these temporary files?
How and why are they generated (their purpose… we would like to control the creations of those files)?

Thank you very much
Best regards

4

Thank you very much. Balaji and Ajaspart I will give a try and get back you.

Nuwan.

5

Hi Balaji,

I have change the dremio-env file like this DREMIO_JAVA_EXTRA_OPTS="Djava.io.tmpdir=/mnt/snas/tmp” but still when the run the dremio it’s created file underneath /tmp like below.

-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:29 librocksdbjni5871504129770964558.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:31 librocksdbjni5970982237250503656.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:28 librocksdbjni6337065271324306705.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:26 librocksdbjni6376093629950495191.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:25 librocksdbjni6582883062349048020.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:23 librocksdbjni6586184864391685338.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:28 librocksdbjni6631347181007443091.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:25 librocksdbjni7435513677609370009.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:23 librocksdbjni7636409376564898532.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:25 librocksdbjni7848977354632933848.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:31 librocksdbjni8118851021089532136.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:32 librocksdbjni8132299498643753025.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:32 librocksdbjni8134137027359354614.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:30 librocksdbjni8172593704575134570.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:27 librocksdbjni8366804736417088661.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:29 librocksdbjni8430616115556576606.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:30 librocksdbjni8592705944121728228.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:31 librocksdbjni8744128431585245950.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:26 librocksdbjni9001427895243196722.so
-rw-r–r--. 1 dremio dremio 8212269 Oct 25 06:27 librocksdbjni9003492478213267895.so

6

Here is the server.log file outout
[root@915468-ElasticDemo log]# tail server.log
Suppressed: java.lang.IllegalStateException: #start was not invoked, so metadataManager is not available
at com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[guava-20.0.jar:na]
at com.dremio.datastore.ByteStoreManager.getMetadataManager(ByteStoreManager.java:422) ~[dremio-services-datastore-4.0.4-201910212053580380-773b665.jar:4.0.4-201910212053580380-773b665]
at com.dremio.datastore.ByteStoreManager.close(ByteStoreManager.java:429) ~[dremio-services-datastore-4.0.4-201910212053580380-773b665.jar:4.0.4-201910212053580380-773b665]
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:92) ~[dremio-common-4.0.4-201910212053580380-773b665.jar:4.0.4-201910212053580380-773b665]
at com.dremio.common.AutoCloseables.close(AutoCloseables.java:71) ~[dremio-common-4.0.4-201910212053580380-773b665.jar:4.0.4-201910212053580380-773b665]
at com.dremio.datastore.CoreStoreProviderImpl.close(CoreStoreProviderImpl.java:286) ~[dremio-services-datastore-4.0.4-201910212053580380-773b665.jar:4.0.4-201910212053580380-773b665]
at com.dremio.datastore.LocalKVStoreProvider.close(LocalKVStoreProvider.java:198) ~[dremio-services-datastore-4.0.4-201910212053580380-773b665.jar:4.0.4-201910212053580380-773b665]
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:163) ~[dremio-dac-daemon-4.0.4-201910212053580380-773b665.jar:4.0.4-201910212053580380-773b665]
… 1 common frames omitted

7

@nuwan1

Was the coordinator restarted after the change?

8

How can I do it ? I mean coordinator restarted

systemctl restart dremio or any other method ?

9

Investigation 1

I saw:

Starting dremio, logging to /opt/dremio/log/server.out

Check the log file to make sure dremio started successfully and did not run into any issues.

[root@915468-ElasticDemo ~]# netstat -pluton | grep 9047

[root@915468-ElasticDemo ~]# netstat -pluton | grep 9047

The port was not opened. And so I looked in /opt/dremio/log/server.out:

17:35:10,937 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[null] - Active log file name: /opt/dremio/log/hive.deprecated.function.warning.log

17:35:10,937 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[null] - File property is set to [/opt/dremio/log/hive.deprecated.function.warning.log]

17:35:10,938 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@2d710f1a - Propagating ERROR level on Logger[ROOT] onto the JUL framework

17:35:10,938 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT level set to ERROR

17:35:10,940 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [text] to Logger[ROOT]

17:35:10,940 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [json] to Logger[ROOT]

17:35:10,940 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.

17:35:10,941 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@13fd2ccd - Registering current configuration as safe fallback point

Dremio is exiting. Failure while starting services.

java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni6790346469476059877.so: /tmp/librocksdbjni6790346469476059877.so: failed to map segment from shared object: Operation not permitted

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)

    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)

    at java.lang.Runtime.load0(Runtime.java:809)

    at java.lang.System.load(System.java:1086)

    at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:78)

    at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:56)

    at org.rocksdb.RocksDB.loadLibrary(RocksDB.java:64)

    at org.rocksdb.RocksDB.<clinit>(RocksDB.java:35)

    at org.rocksdb.Options.<clinit>(Options.java:25)

    at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:201)

    at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:192)

    at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:157)

    at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160)

    at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:99)

    Suppressed: java.lang.IllegalStateException: #start was not invoked, so metadataManager is not available

So, I looked “https://stackoverflow.com/questions/13502156/what-are-possible-causes-of-failed-to-map-segment-from-shared-object-operation” and saw:

Permission issue. Need to remount /tmp. The following command works for me (Centos 7):
sudo mount /tmp -o remount,exec

So, I applied:

[root@915468-ElasticDemo tmp]# sudo mount /tmp -o remount,exec

[root@915468-ElasticDemo tmp]# netstat -pluton | grep 9047

[root@915468-ElasticDemo tmp]#

Investigation 2

And restarted:

[root@915468-ElasticDemo tmp]# service dremio restart

stopping dremio

.

#######################################################################

WARNING:

Please increase the maximum number of file descriptors to at least 65536.

Dremio utilizes a large number of file descriptors.

The maximum number of file descriptors, currently set at 8192, is not sufficient.

#######################################################################

Starting dremio, logging to /opt/dremio/log/server.out

Check the log file to make sure dremio started successfully and did not run into any issues.

I updated the live maximum number of descriptors:

echo “65536” > /proc/sys/fs/file-max

The, updated for reboot:

vi /etc/sysctl.conf

fs.file-max = 65536

And looked at the logs

[root@915468-ElasticDemo tmp]# vi /opt/dremio/log/server.out

17:42:52,783 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@2d710f1a - Propagating ERROR level on Logger[ROOT] onto the JUL framework

17:42:52,783 |-INFO in ch.qos.logback.classic.joran.action.LevelAction - ROOT level set to ERROR

17:42:52,785 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [text] to Logger[ROOT]

17:42:52,785 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [json] to Logger[ROOT]

17:42:52,785 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.

17:42:52,786 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@13fd2ccd - Registering current configuration as safe fallback point

Dremio is exiting. Failure while starting services.

com.dremio.common.exceptions.UserException: No Cluster Identity found

    at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776)

    at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:172)

    at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:162)

    at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:99)

According to “https://community.dremio.com/t/dremio-dockers-stop-as-soon-as-i-started-them/2523” we need to “remove” the data/db directory:

[root@915468-ElasticDemo tmp]# cd /opt/dremio/data

[root@915468-ElasticDemo data]# mkdir ~/dremio-backup

[root@915468-ElasticDemo data]# mv db ~/dremio-backup/

[root@915468-ElasticDemo data]# service dremio restart

[root@915468-ElasticDemo data]# vi /opt/dremio/log/server.out

And then SUCCESS: