Dremio on kubernate not able to start

Vikash_Singh · February 7, 2019, 12:00pm

I have configured dremio with adl Distributed file system ,i have upgraded Kubernate version from Azure .after that when i am running helm command its never getting up,
this was my dist: “adl://.azuredatalakestore.net/dremio”,

com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Source not yet started.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:347) [dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:588) [dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867) [dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:409) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:404) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:273) [dremio-services-jobs-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) [dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) [dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) [dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102) [dremio-dac-daemon-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
2019-02-07 11:35:24,967 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Source not yet started.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:347) ~[dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:588) ~[dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867) ~[dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:409) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:404) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:273) ~[dremio-services-jobs-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102) ~[dremio-dac-daemon-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
Dremio is exiting. Failure while starting services.
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Source not yet started.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746)
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:347)
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:588)
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867)
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:409)
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:404)
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:273)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)3

If i change dist to other Adl folder it is woking now,but in that case I have lost all file and Acceleration,how to solve above issue…As i have Large Datasource connected to dremio

balaji.ramaswamy · February 7, 2019, 2:05pm

Hi @Vikash_Singh

I see you have a “.” as the first character in your ADLS account name. Is that correct?

“adl://.azuredatalakestore.net/dremio”

Thanks
@balaji.ramaswamy

Vikash_Singh · February 7, 2019, 2:32pm

No, i have correct url adl://althing.azuredatalakestore.net/dremio,reflection getting store in adl

balaji.ramaswamy · February 7, 2019, 8:17pm

Hi @Vikash_Singh

Somehow results is not able to write back to ADLS. Is there a results folder and the permissions right?

You can also try to move results to local?

# stores data associated with the job results cache.
  results: ${paths.dist}/results

Thanks
@balaji.ramaswamy

Vikash_Singh · February 28, 2020, 7:22am

@balaji.ramaswamy i am still facing issue in job result not able to start drremio master here is the log file for that and this is config

paths: {

Local path for dremio to store data.

local: ${DREMIO_HOME}"/data"
dist: “dremioAzureStorage://:///dremio”
}

dremio-master-0.zip (4.8 KB)

Topic		Replies	Views
Connecting Dremio to Storage Grid - s3	1	1184	April 24, 2019
Failure while starting services. com.dremio.datastore.DatastoreException: Process user (dremio) doesn't match local catalog db owner (root). Please run process as root Dremio University	20	4512	November 25, 2020
Dremio-master-coordinator unable to access volume cause CrashLoopBackOff	9	1302	January 15, 2023
Error Starting Dremio	2	2517	July 30, 2018
Dremio community edition in aks	8	1238	October 7, 2021

Dremio on kubernate not able to start

Local path for dremio to store data.

Related topics