Reconnected, suspended, reconnected, suspended

Randalism · October 16, 2018, 2:17pm

Dremio web UI won’t display and I checked log and see this for about every second. Any ideas why?

2018-10-16 09:01:07,444 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to RECONNECTED
2018-10-16 09:01:07,544 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to SUSPENDED
2018-10-16 09:01:09,252 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to RECONNECTED
2018-10-16 09:01:09,353 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to SUSPENDED

anthony · October 16, 2018, 4:16pm

Can you talk a little more about your deployment? There seems to an issue with your ZK connection - https://docs.dremio.com/advanced-administration/zookeeper.html

Randalism · October 16, 2018, 6:31pm

we are using external cluster

anthony · October 16, 2018, 6:36pm

Do you mean external ZK? Is this part of a Hadoop cluster? Can you try adding a ZK path as seen here and try again?

Randalism · October 16, 2018, 7:00pm

we have ZK path like that.

anthony · October 16, 2018, 7:05pm

Can you share your dremio.conf, server.out, and server.log?

Randalism · October 16, 2018, 7:55pm

Below are both. For log, it just keeps repeating “reconnected” and “suspended” at end, i just included a few of the repeats

server log.
2018-10-16 13:48:29,767 [main] INFO c.d.common.scanner.BuildTimeScan - Loaded prescanned packages [com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.options, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.exec.store, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.service.jobs, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.options, com.dremio.resource, com.dremio.resource.basic, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.options, com.dremio.service.namespace, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.storage, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.options, com.dremio.service.namespace, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.storage, org.apache.hadoop.hive, com.dremio.exec.fn.hive, com.dremio.exec.store.hive, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.options, com.dremio.service.namespace, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, org.apache.hadoop.hive, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.provision, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.store, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.options, com.dremio.resource, com.dremio.resource.basic] from locations [jar:file:/opt/dremio/jars/dremio-services-accelerator-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-common-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-logical-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-jobs-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-datastore-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-common-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-extra-sabot-scheduler-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-kernel-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-hive-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-backend-2.1.6-201809161911340177-edb5b4d-mapr.jar!/META-INF/dremio-module-scan/registry.json]
2018-10-16 13:48:30,767 [main] INFO c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.plugins.s3.store, com.dremio.plugins.adl, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.resource, com.dremio.resource.basic, com.dremio.exec.store.jdbc, com.dremio.extras.plugins.elastic, com.dremio.exec.store.jdbc, com.dremio.plugins.elastic, com.dremio.service.jobs, com.dremio.extra.exec.store.dfs, com.dremio.exec.planner.acceleration.substitution, com.dremio.plugins.mongo, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.service.users, com.dremio.plugins.mongo, com.dremio.provision.yarn.service, com.dremio.exec.store.hbase, com.dremio.exec.expr.fn.impl.conv, com.dremio.options, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical.PlannerSettings, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.exec.store, org.apache.hadoop.hive, com.dremio.exec.fn.hive, com.dremio.exec.store.hive, com.dremio.provision, com.dremio.service.namespace, com.dremio.dac, com.dremio.dac.support.SupportService, org.apache.hadoop.hive] in locations [jar:file:/opt/dremio/jars/dremio-s3-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-jdbc-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-elasticsearch-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-coordinator-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-jdbc-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-hive-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-sabot-kernel-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-mongo-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-users-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-extra-plugin-mongo-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-yarn-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-hbase-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-options-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-pdfs-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-provision-common-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/dremio-services-namespace-2.1.6-201809161911340177-edb5b4d-mapr.jar!/, jar:file:/opt/dremio/jars/3rdparty/dremio-hive-exec-shaded-2.1.6-201809161911340177-edb5b4d-mapr.jar!/] took 995ms
2018-10-16 13:48:30,877 [main] INFO com.dremio.dac.daemon.DACDaemon - Dremio daemon write path: /app_2/dremio/dremio
2018-10-16 13:48:30,883 [main] INFO com.dremio.dac.daemon.DACDaemon - This node is the master node, xxx111.zzz.com. This node acts as a coordinator.
2018-10-16 13:48:30,907 [main] INFO com.dremio.common.config.SabotConfig - Configuration and plugin file(s) identified in 19ms.
Base Configuration:
- jar:file:/opt/dremio/jars/dremio-common-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-default.conf

Intermediate Configuration and Plugin files, in order of precedence:
- jar:file:/opt/dremio/jars/dremio-s3-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-accelerator-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-common-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-jdbc-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-elasticsearch-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-coordinator-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-jdbc-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-logical-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-jobs-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-hive-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-sabot-kernel-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-datastore-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-mongo-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-common-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-sabot-scheduler-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-users-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-extra-plugin-mongo-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-yarn-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hbase-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-options-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-kernel-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-pdfs-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hive-plugin-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-provision-common-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-namespace-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-backend-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/3rdparty/dremio-hive-exec-shaded-2.1.6-201809161911340177-edb5b4d-mapr.jar!/sabot-module.conf

2018-10-16 13:48:31,421 [main] INFO c.d.s.fabric.FabricServiceImpl - fabric service has 104857600 bytes reserved
2018-10-16 13:48:31,473 [main] INFO c.dremio.dac.daemon.DACDaemonModule - Internal user/group service is configured.
2018-10-16 13:48:32,044 [main] INFO c.d.s.coordinator.zk.ZKClusterClient - Starting ZKClusterClient
2018-10-16 13:48:32,177 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to CONNECTED
2018-10-16 13:48:32,280 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to SUSPENDED
2018-10-16 13:48:33,712 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to RECONNECTED
2018-10-16 13:48:33,812 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to SUSPENDED
2018-10-16 13:48:35,106 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to RECONNECTED
2018-10-16 13:48:35,207 [Curator-ConnectionStateManager-0] INFO c.d.s.coordinator.zk.ZKClusterClient - ZK connection state changed to SUSPENDED

server.out
Tue Oct 16 13:48:27 CDT 2018 Starting dremio on xxxx2222
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 257578
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 63536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
No database found. Skipping upgrade

balaji.ramaswamy · October 17, 2018, 7:28am

Hi @Randalism

To narrow down the issue, do this

edit dremio.conf and comment out the external zoo keeper quorum would be something like below

zookeeper: “<ZOOKEEPER_HOST_1>:2181,<ZOOKEEPER_HOST_2>:2181”

Make the below line true

services.coordinator.master.embedded-zookeeper.enabled: false

Now restart the executors and then the coordinators

Does it start successfully?

Thanks
@balaji.ramaswamy

Randalism · October 17, 2018, 1:54pm

I had done something like this but didn’t work.

zookeeper: “host1:2181/dremio_1,host2:2181/dremio_2,host3:2181/dremio_3”

anthony · October 19, 2018, 1:16am

FYI it should be zookeeper: “host1:2181/dremio_1,host2:2181/dremio_1,host3:2181/dremio_1”
Try the same ZK paths, not different from your example.

Randalism · October 29, 2018, 3:50pm

tried that too, but still occurring

Topic		Replies	Views
Unexpected Coordinator service shutting down	2	1723	January 3, 2022
Replacing old master node and dremio UI will not start	3	1302	December 17, 2018
Query failing because of ZK connection SUSPENDED, RECONNECT results in master killing the job	16	3818	November 11, 2020
Unable to start Dremio on Windows	27	5501	June 19, 2019
Some time dremio webui not working contact support	7	1736	December 30, 2019

Reconnected, suspended, reconnected, suspended

Related topics