Dremio OS provisioning: DREMIO_HOME not set (Windows)

1

On Windows when trying to set-up YARN provisioning, I get the following error:

I set the environment variable in the additional properties (environment), in my Windows system variables and in dremio-env in the conf dir.

How should DREMIO_HOME environment be configured in Windows for this error to be resolved?

java.lang.NullPointerException: Environment variable DREMIO_HOME is not set
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:787) ~[guava-20.0.jar:na]
	at com.dremio.provision.yarn.DacDaemonYarnApplication.<init>(DacDaemonYarnApplication.java:76) ~[dremio-yarn-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.provision.yarn.YarnController.createPreparer(YarnController.java:109) ~[dremio-yarn-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.provision.yarn.YarnController.startCluster(YarnController.java:99) ~[dremio-yarn-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.provision.yarn.service.YarnService.startClusterAsync(YarnService.java:131) ~[dremio-yarn-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.provision.yarn.service.YarnService.startCluster(YarnService.java:111) ~[dremio-yarn-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.provision.service.ProvisioningServiceImpl.startCluster(ProvisioningServiceImpl.java:285) ~[dremio-provision-common-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.provision.service.ProvisioningServiceImpl.createCluster(ProvisioningServiceImpl.java:150) ~[dremio-provision-common-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at com.dremio.provision.resource.ProvisioningResource.createCluster(ProvisioningResource.java:106) ~[dremio-provision-common-3.0.0-201810262305460004-5c90d75.jar:3.0.0-201810262305460004-5c90d75]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [jersey-common-2.25.1.jar:na]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [jersey-server-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228) [jersey-container-servlet-core-2.25.1.jar:na]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:301) [jetty-servlets-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-server-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-io-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.22.v20170606.jar:9.2.22.v20170606]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
1 Like
2

I would imagine that you can set DREMIO_HOME as any other env. variable on Windows. E.g.: http://www.dowdandassociates.com/blog/content/howto-set-an-environment-variable-in-windows-command-line-and-registry/

Regarding YARN deployment - do you have just coordinator running on Windows and it can submit YARN applications to your YARN/Hadoop cluster? Or it is a different setup?

1 Like
3

@yufeldman, thanks for the instructions,

I did

PS ~> echo $Env:DREMIO_HOME
C:/Temp/dremio

Dremio is installed in C:\Program Files\Dremio
and Dremio application data is located in ~\AppData\Local\Dremio

Coordinator config is still set so that the local coordinator node can still function as an executor as well.
yarn-site.conf, core-site.conf and hdfs-site.conf included and adapted according to configuration guidelines.

I am testing if the coordinator on Windows can submit YARN applications to the YARN/hadoop cluster. But it seems to already block at the missing environment variable DREMIO_HOME.

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true,
...
1 Like