# Dremio auto shutdown **URL:** \ **Category:** Uncategorized\ **Created:** [November 20, 2019, 5:41am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595 "2019-11-20T05:41:00Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [November 20, 2019, 5:41am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/1 "2019-11-20T05:41:00Z")
Hi , I am still struglling with dremio stablity its auto shutdown not able to access gui 2019-11-20 05:31:01,148 [222b3083-e945-9936-0b2d-c8ae5cddf000:foreman] INFO c.d.e.p.s.h.commands.FragmentStarter - User Error Occurred [ErrorId: a52310dd-4d25-4362-adc9-c647ce78d216] com.dremio.common.exceptions.UserException: Exceeded timeout (5000) while waiting after sending work fragments to remote nodes. Sent 1 and only heard response back from 0 nodes. at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776) ~[dremio-common-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.commands.FragmentStarter.startFragments(FragmentStarter.java:157) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.commands.FragmentStarter.start(FragmentStarter.java:80) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.commands.AsyncCommand.startFragments(AsyncCommand.java:98) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:332) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0\_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0\_222] at java.lang.Thread.run(Thread.java:748) [na:1.8.0\_222] 2019-11-20 05:31:01,148 [222b3083-b764-ad0e-427f-5ad6b89c1200:foreman] INFO c.d.e.p.s.h.commands.FragmentStarter - User Error Occurred [ErrorId: 00d35199-3b9c-402d-a2ce-76c1931481bf] com.dremio.common.exceptions.UserException: Exceeded timeout (5000) while waiting after sending work fragments to remote nodes. Sent 1 and only heard response back from 0 nodes. at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:776) ~[dremio-common-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.commands.FragmentStarter.startFragments(FragmentStarter.java:157) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.commands.FragmentStarter.start(FragmentStarter.java:80) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.planner.sql.handlers.commands.AsyncCommand.startFragments(AsyncCommand.java:98) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:332) [dremio-sabot-kernel-4.0.2-201910020123580864-a98a0b9.jar:4.0.2-201910020123580864-a98a0b9] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0\_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0\_222] at java.lang.Thread.run(Thread.java:748) [na:1.8.0\_222] ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/b/bc836057a517e79152403fc246634afd91ba5fa3.png) But in kubernate pods all are in running state ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/e/eb097082aa3946c31b224977dfcffe24e2572936.png) ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [November 20, 2019, 5:42am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/2 "2019-11-20T05:42:50Z")
after restart dremio master node it works ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [November 20, 2019, 2:23pm UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/3 "2019-11-20T14:23:32Z")
i need solution i cant always restart Master pods as its a production ready system ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [November 20, 2019, 4:27pm UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/4 "2019-11-20T16:27:55Z")
@Vikash_Singh Send us the complete server.log from the master coordinator, few questions #1 Do you have a second coordinator other than the master? If yes, we do not recommend that #2 Have you by any chance provisioned all the OS RAM to Dremio, if yes , it could be the oom-killer that is killing Dremio. You need to at least leave 4 GB to the OS #3 Have set heap on the coordinator explicitly? if you have set MAX then everything minus 2 GB goes to heap and may end up in full GC cycles. Check the Dremio master pod log to see if you see Full GC’s ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [November 21, 2019, 5:45am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/5 "2019-11-21T05:45:13Z")
Hello @balaji.ramaswamy, 1. Do you have a second coordinator other than the master? --\> No 2. Have you by any chance provisioned all the OS RAM i have 128 Gb Ram where only 116 GB Ram Assigned to dremio rest all free for OS 3. Have set heap on the coordinator explicitly? yes this is value in dremio env DREMIO\_MAX\_HEAP\_MEMORY\_SIZE\_MB=10240 DREMIO\_MAX\_DIRECT\_MEMORY\_SIZE\_MB=51200 please find atatche d logs for above error [dremio-master-0.zip](https://community.dremio.com/uploads/short-url/3K0TBGEJ6TFbL9rk0mkdE2IhxvB.zip) (1.1 MB) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [November 21, 2019, 6:14am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/6 "2019-11-21T06:14:42Z")
@Vikash_Singh The log file you sent has no Dremio startup or shutdown. Do these logs rollover when you restart Dremio? ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [November 21, 2019, 6:27am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/7 "2019-11-21T06:27:07Z")
yes might be since its a production system i cant stop server and debub ,next time i will provide when i get same issue again i guess every week i faced this type of issue. can you also guide how to copy full dremio log to local filesystem from kubernate pods as kubectl logs -f dremio-master-0 \> dremio-master-0.log looks hanged for me ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 13, 2019, 11:13am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/8 "2019-12-13T11:13:07Z")
@balaji.ramaswamy i am still facing Dremio ui auto shutdown automatically and also see master pods status 0/1 i am attaching master log of issue [dremio-master-0.zip](https://community.dremio.com/uploads/short-url/tw2nHeqTLo5uKRt55OKgBrvMWRl.zip) (531.4 KB) ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/e/e4dd63a10414a9fc8fa9e856210fa1602b154815.png) ---
**Author:** ![Vikash\_Singh](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/vikash_singh/32/1069_2.png) [@Vikash\_Singh](https://community.dremio.com/u/Vikash_Singh)\ **Post date:** [December 13, 2019, 11:14am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/9 "2019-12-13T11:14:12Z")
Its quite urgent for me i cant able to run dremio to production every day i have to manually restart dremio master pods plese suggest permanent solution ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [December 16, 2019, 4:26am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/10 "2019-12-16T04:26:06Z")
@Vikash_Singh The attached logfile has nor entries on Dremio shutting down or starting up, Would you be able to send the logfile that has keyword “KVstore” or “localhost” Thanks @balaji.ramaswamy ---
**Author:** ![Diego](https://avatars.discourse-cdn.com/v4/letter/d/f04885/32.png) [@Diego](https://community.dremio.com/u/Diego)\ **Post date:** [March 23, 2023, 9:58pm UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/11 "2023-03-23T21:58:34Z")
Hello, @balaji.ramaswamy There is some way to force stop/start worker nodes programmatically? ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [March 29, 2023, 12:48am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/12 "2023-03-29T00:48:26Z")
@Diego Depends on the deployment - K8’s and Yarn should restart automatically - Standalone VM’s would require a custom script to start on failure Thanks Bali ---
**Author:** ![Diego](https://avatars.discourse-cdn.com/v4/letter/d/f04885/32.png) [@Diego](https://community.dremio.com/u/Diego)\ **Post date:** [March 29, 2023, 5:43am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/13 "2023-03-29T05:43:06Z")
@balaji.ramaswamy I use Dremio AWS version. I’m interested on schedule an hour to stop nodes and hour to start these nodes ---
**Author:** ![Diego](https://avatars.discourse-cdn.com/v4/letter/d/f04885/32.png) [@Diego](https://community.dremio.com/u/Diego)\ **Post date:** [April 5, 2023, 10:18am UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/14 "2023-04-05T10:18:29Z")
@balaji.ramaswamy, I noticed that when I stop the Dremio service, after a few minutes, the executor nodes are also stopped. I set the nodes to auto start so that when the service is turned on they go up together. But, the second engine does not go up ![image](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/1/1c81b10995d9b00e169dcac165cc2468bff75e66.png) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [April 8, 2023, 10:21pm UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/15 "2023-04-08T22:21:47Z")
@Diego The second engine is a preview engine that will be used only for queries run in the preview mode, try hitting preview instead of run and see of that helps? ![Screen Shot 2023-04-08 at 3.21.31 PM](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/5/576aecd3d26a6ace1b4cc80e4e7169a44801c775.png) ---
**Author:** ![Diego](https://avatars.discourse-cdn.com/v4/letter/d/f04885/32.png) [@Diego](https://community.dremio.com/u/Diego)\ **Post date:** [April 11, 2023, 1:33pm UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/16 "2023-04-11T13:33:29Z")
@balaji.ramaswamy When I click on preview the nodes automatically startup? I guess no ![MicrosoftTeams-image (14)](https://us1.discourse-cdn.com/flex020/uploads/dremio/original/2X/a/a567fb22c4650f2cdae4fd27e97ca1db975ea866.png) ---
**Author:** ![balaji.ramaswamy](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/balaji.ramaswamy/32/843_2.png) [@balaji.ramaswamy](https://community.dremio.com/u/balaji.ramaswamy)\ **Post date:** [April 12, 2023, 10:22pm UTC](https://community.dremio.com/t/dremio-auto-shutdown/4595/17 "2023-04-12T22:22:27Z")
@Diego If you hit preview on a query, the engine should automatically startup