currently we have to restart dremio everyday, because too many files are opened.
“ERROR c.d.d.server.GenericExceptionMapper - Unexpected exception when processing GET http://XXX.net:9047/apiv2/job/24420c3a-4490-9178-d5cb-7a74b0c10400/details : java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: Cannot run program “ls”: error=24, Too many open files”
We have 2 reflections with partitioning running every hour. Using lsof -p in can see a lot of files beeing opened when the reflection is refreshed and when the number of open files reach around 2000, dremio dies.
I tried to set up file open limits with ulimit but it has no effect.
Please edit your /etc/security/limits.conf and add
* soft nofile 65536
* hard nofile 65536
Note this will require a machine reboot to take effect.
I already did this and if i do “ulimit -Hn” and “ulimit -Hs” as dremio user i get “900000” and “unlimited”.
I even tried downgrading dremio. Still the same result. Can you guys confirm this must be an OS issue?
Yes this is likely an OS issue. Where are you storing your reflections? I assume NAS/PDFS? If yes, you can try using another dist storage to possibly prevent this issue.
I solved it.
It turns out modern linuxes dont listen to ulimit anymore and you have to use systemd to set the maximum files opened.
You have to set DefaultLimitNOFILE in /etc/systemd/system.conf and /etc/systemd/user.conf
You can query limits with prlimit --pid=…
Thanks for sharing. Can you share what OS your are running?
No problem. Debian 9 (stretch).