Error - Too many open files

1

Hi everyone,

currently we have to restart dremio everyday, because too many files are opened.

“ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing GET http://XXX.net:9047/apiv2/job/24420c3a-4490-9178-d5cb-7a74b0c10400/details : java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program “ls”: error=24, Too many open files”

We have 2 reflections with partitioning running every hour. Using lsof -p in can see a lot of files beeing opened when the reflection is refreshed and when the number of open files reach around 2000, dremio dies.

I tried to set up file open limits with ulimit but it has no effect.

2

Please edit your /etc/security/limits.conf and add

* soft nofile 65536
* hard nofile 65536

Note this will require a machine reboot to take effect.

3

I already did this and if i do “ulimit -Hn” and “ulimit -Hs” as dremio user i get “900000” and “unlimited”.

4

I even tried downgrading dremio. Still the same result. Can you guys confirm this must be an OS issue?

5

Yes this is likely an OS issue. Where are you storing your reflections? I assume NAS/PDFS? If yes, you can try using another dist storage to possibly prevent this issue.

6

I solved it.
It turns out modern linuxes dont listen to ulimit anymore and you have to use systemd to set the maximum files opened.

You have to set DefaultLimitNOFILE in /etc/systemd/system.conf and /etc/systemd/user.conf
You can query limits with prlimit --pid=…

3 Likes
7

Thanks for sharing. Can you share what OS your are running?

8

No problem. Debian 9 (stretch).