Master Coordinator is Down. Unable to connect with JDBC

1
  • Docker Desktop: 4.10.1 (82475)
  • Windows 10 Enterprise
  • Apache JMeter 5.5

Running Dremio as a docker instance wherein the single instance acts as

  • Zookeeper
  • Master Coordinator
  • Executor

I see messages in logs that says “Master coordinator is down”

2022-07-27 05:57:08,937 [scheduler-3] WARN  c.d.e.catalog.MetadataSynchronizer - Source 'sys' sync failed unexpectedly. Will try again later
java.lang.NullPointerException: Master coordinator is down
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
        at com.dremio.service.conduit.client.ConduitProviderImpl.getOrCreateChannelToMaster(ConduitProviderImpl.java:175)
        at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightClient(SysFlightStoragePlugin.java:128)
        at com.dremio.plugins.sysflight.SysFlightStoragePlugin.getFlightTableList(SysFlightStoragePlugin.java:258)
        at com.dremio.plugins.sysflight.SysFlightStoragePlugin.listDatasetHandles(SysFlightStoragePlugin.java:210)
        at com.dremio.exec.catalog.MetadataSynchronizer.getDatasetHandleListing(MetadataSynchronizer.java:172)
        at com.dremio.exec.catalog.MetadataSynchronizer.synchronizeDatasets(MetadataSynchronizer.java:185)
        at com.dremio.exec.catalog.MetadataSynchronizer.go(MetadataSynchronizer.java:133)
        at com.dremio.exec.catalog.SourceMetadataManager$RefreshRunner.refreshFull(SourceMetadataManager.java:441)
        at com.dremio.exec.catalog.SourceMetadataManager$AdhocRefresh.run(SourceMetadataManager.java:507)
        at com.dremio.exec.catalog.SourceMetadataManager.refresh(SourceMetadataManager.java:188)
        at com.dremio.exec.catalog.ManagedStoragePlugin.refresh(ManagedStoragePlugin.java:1119)
        at com.dremio.exec.catalog.CatalogServiceImpl.refreshSource(CatalogServiceImpl.java:377)
        at com.dremio.exec.catalog.CatalogServiceImpl.lambda$start$0(CatalogServiceImpl.java:267)
        at com.dremio.service.scheduler.LocalSchedulerService$CancellableTask.run(LocalSchedulerService.java:226)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

While connecting JMeter to Dremio via. JDBC, the request times out. Telnet also responds with Timeout. Since, the coordinator is down, I expect the request to be timed out. Can anyone guide me what’s wrong?

Here is the JMeter JDBC Configuration. The IP is the docker ip

image
image1448×768 15.7 KB

Tried running Dremio as a container mounting the configuration under volumes. Used the default dremio.conf provided in dremio docs and marked web to false, to test if it works or not, but it doesn’t

- ./dremio/dremio.conf:/opt/dremio/dremio.conf

Log File:
dremio.zip (6.2 KB)

2

@hemant.gupta

Is the question that master coordinator is up but still JDBC is still complaining that master is down?

Have you made sure the IP address and port are matching to what has been configured?

3

As the title mentions, Coordinator is down and so JDBC is unable to connect.
I tried different commands to run Dremio in Docker and every time coordinator is down as can be seen in the error logs attached

4

Hi @hemant.gupta Sorry for the late reply, if this is still an issue can you please send us the server.log and server.out from the coordinator

Apologies for the late reply again