3.1 fails upon machine restart in local mode

1

I’m experimenting with 3.1 in local, single-node on CentOS 7.6. I can install 3.1, start it, and use Dremio happily. But when I restart my machine, Dremio no longer works without intervention. It appears that permissions are being reset on /var/run/dremio.

[user1@localhost ~]$ sudo yum localinstall dremio-community-3.1.0-201901172111160703_dc6f6e5_1.noarch.rpm
# permissions are good
[user1@localhost ~]$ ls -ltrh /var/run/
drwxr-xr-x.  2 dremio         dremio           40 Jan 17 16:24 dremio
[user1@localhost ~]$ sudo service dremio start
starting dremio, logging to /var/log/dremio/server.out
# permissions are good
[user1@localhost ~]$ ls -ltrh /var/run/
drwxr-xr-x.  2 dremio         dremio           40 Jan 17 16:24 dremio
# open Dremio in browser, everything is good

[user1@localhost ~]$ reboot
# permissions are changed and Dremio doesn't work
[user1@localhost ~]$ ls -ltrh /var/run/
drw-r--r--.  2 dremio         dremio           40 Jan 29 10:31 dremio
[user1@localhost ~]$ sudo service dremio start
starting dremio, logging to /var/log/dremio/server.out
[user1@localhost ~]$ tail /var/log/dremio/server.out 
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
/opt/dremio/bin/dremio: line 117: /var/run/dremio/dremio.pid: Permission denied
/var/run/dremio not found or dremio does not have access to it.
Please create /var/run/dremio and/or give the user dremio ownership to the directory.
# now I fix, and we are happy
[user1@localhost ~]$ sudo chmod u+x /var/run/dremio/
[user1@localhost ~]$ ls -ltrh /var/run/
drwxr--r--.  2 dremio         dremio           40 Jan 29 10:31 dremio
[user1@localhost ~]$ sudo service dremio start
# can use Dremio normally again
# but, when I reboot, need to reset permissions all over again

When I install 3.0.6 and then reboot, Dremio keeps working. I tried 2-3 different 3.1 installs and all had this problem.

2

I’m facing the same problem after update to 3.1

3

@can1 and @Paulo_Vasconcellos,

We’re you using a Tarball or RPM installs?

4

We’re using the RPM version, @ben

5

@ben rpm version for me too

6

@can1 and @Paulo_Vasconcellos,

On your systems do you know if /var/run is recreated on reboot?

In /etc/sysconfig/dremio Can you check which user runs the Dremio service?
https://docs.dremio.com/deployment/rpm-tarball-install.html#changing-the-dremio-user-and-group

7

@ben I do not know if it is recreated. I do know that the permissions on /var/run/dremio seem to be reset in the same way upon reboot with 3.0.6, but somehow that does not stop 3.0.6. It is run in both cases as the dremio user.

8

@can1,

Did you do any extra configuration besides yum install for 3.1.0 or 3.0.6? For instance, any extra system service configuration?

9

@ben what I provided above was the exact steps I took to produce the problem. As you see, completely default installation.

10

Hey @ben, @balaji.ramaswamy, this is still an issue. Including with 3.1.1

11

Hi @can1

I think I found out what is going on. On the later Linux releases, it look like /run is a tmpfs file system that is recreated on machine reboot

[root@scp-support-target var]# df -h /run
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 174M 1.7G 10% /run

and /run is symlinked to /var/run

ls -l /var/run

lrwxrwxrwx. 1 root root 6 Apr 26 2017 /var/run -> …/run

What you may want to edit dremio-env and change the below to a non tmpfs location so it is not removed on startup

DREMIO_PID_DIR=${DREMIO_HOME}/run

Kindly let me know if this resolves the issue

Thanks
@balaji.ramaswamy