Dremio on kubernate not able to start


#1

I have configured dremio with adl Distributed file system ,i have upgraded Kubernate version from Azure .after that when i am running helm command its never getting up,
this was my dist: “adl://.azuredatalakestore.net/dremio”,

com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Source not yet started.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:347) [dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:588) [dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867) [dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:409) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:404) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:273) [dremio-services-jobs-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) [dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) [dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) [dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189) [dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102) [dremio-dac-daemon-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
2019-02-07 11:35:24,967 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Source not yet started.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:347) ~[dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:588) ~[dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867) ~[dremio-sabot-kernel-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:409) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:404) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:273) ~[dremio-services-jobs-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33) ~[dremio-common-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189) ~[dremio-dac-backend-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102) ~[dremio-dac-daemon-3.1.1-201901281837360699-30c9d74.jar:3.1.1-201901281837360699-30c9d74]
Dremio is exiting. Failure while starting services.
com.dremio.common.exceptions.UserException: The source ["__jobResultsStore"] is currently unavailable. Info: [[Message{level=ERROR, msg=Source not yet started.}]].
at com.dremio.common.exceptions.UserException$Builder.build(UserException.java:746)
at com.dremio.exec.catalog.ManagedStoragePlugin.checkState(ManagedStoragePlugin.java:347)
at com.dremio.exec.catalog.ManagedStoragePlugin.unwrap(ManagedStoragePlugin.java:588)
at com.dremio.exec.catalog.CatalogServiceImpl.getSource(CatalogServiceImpl.java:867)
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:409)
at com.dremio.dac.daemon.DACDaemonModule$3.get(DACDaemonModule.java:404)
at com.dremio.service.jobs.LocalJobsService.start(LocalJobsService.java:273)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)3

If i change dist to other Adl folder it is woking now,but in that case I have lost all file and Acceleration,how to solve above issue…As i have Large Datasource connected to dremio


#2

Hi @Vikash_Singh

I see you have a “.” as the first character in your ADLS account name. Is that correct?

“adl://.azuredatalakestore.net/dremio”

Thanks
@balaji.ramaswamy


#3

No, i have correct url adl://althing.azuredatalakestore.net/dremio,reflection getting store in adl


#4

Hi @Vikash_Singh

Somehow results is not able to write back to ADLS. Is there a results folder and the permissions right?

You can also try to move results to local?

# stores data associated with the job results cache.
  results: ${paths.dist}/results

Thanks
@balaji.ramaswamy