Error while starting dremio on executor: "Unknown store creator com.dremio.service.users.SimpleUserService"

1

Hi guys.
This morning I’ve noticed that my dremio coordinator node wasn’t working. It seems that during the weekend my coordinator node ran out of memory, so I restarted the service, and it’s working now. However, when I tried to start the server on the executor node, I got this error:

Mon Feb 25 14:43:27 UTC 2019 Starting dremio on ip-11-10-05-100.ec2.internal
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 61726
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
Dremio is exiting. Failure while starting services.
java.lang.RuntimeException: Failure while attempting to create com.dremio.service.users.SimpleUserService.
        at com.dremio.service.BinderImpl$InjectableReference.get(BinderImpl.java:427)
        at com.dremio.service.BinderImpl.lookup(BinderImpl.java:109)
        at com.dremio.service.BinderImpl$DeferredProvider.get(BinderImpl.java:83)
        at com.dremio.exec.server.ContextService.newSabotContext(ContextService.java:225)
        at com.dremio.exec.server.ContextService.start(ContextService.java:168)
        at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
        at com.dremio.dac.daemon.NonMasterSingletonRegistry.start(NonMasterSingletonRegistry.java:54)
        at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:183)
        at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:189)
        at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:102)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.dremio.service.BinderImpl$InjectableReference.get(BinderImpl.java:421)
        ... 9 more
Caused by: java.lang.NullPointerException: Unknown store creator com.dremio.service.users.SimpleUserService$UserGroupStoreBuilder
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:864)
        at com.dremio.datastore.RemoteKVStoreProvider.getStore(RemoteKVStoreProvider.java:76)
        at com.dremio.service.users.SimpleUserService.<init>(SimpleUserService.java:96)
        ... 14 more

My Coordinator/Master node is working like a charm, but my executor isn’t. Also, I’ve noticed that the db folder is missing from /var/lib/dremio directory. The pdfs and zk folder is also missing.

I’m using Dremio 3.1.0-201901172111160703-dc6f6e5 and using a S3 as my dist storage.

Does anybody know what is going on?

2

Hi @Paulo_Vasconcellos,

How are you deploying your Dremio cluster?

If you are configuring each node separately with it’s own dremio.conf. Can you share that file for
the executor?

If you are using the default “out-of-box” configuration, the executor will not have a db folder or a zk dir, but if you start the executor successfully, it will create a pdfs directory.

3

Hi @ben, this is my dremio.conf file on executor node:


paths: {
  # the local path for dremio to store data.
  #local: "/var/lib/dremio"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  dist: "s3a://data.hotmt.com/dremio"
}

services: {
  coordinator.enabled: false,
  coordinator.master.enabled: false,
  executor.enabled: true
}
zookeeper: "COORDINATOR_IP:2181"
4

P.S.: I configured each node separately

5

@Paulo_Vasconcellos, can you try a couple things?

  1. Comment back in local: "/var/lib/dremio"
  2. Make sure you do not have an orphaned executor:
    bash$ ps -ef | grep Dremio
    If there’s a dremio process still running, kill it and try and start the executor again.
6

I did these thing but the same error still happening, @ben

7

@Paulo_Vasconcellos, can you attach the server.log and server.out from the executor?

8

Hi Ben! I’ve solved the problem by reinstalling Dremio.

9

@Paulo_Vasconcellos, are you using RPM or tarball? How are you starting Dremio ($ sudo service dremio start?). It seems that Dremio was not reading some part of your configuration (or picking up the wrong config perhaps?).

10

Same problem here. I’ve installed using RPM. Here my executor conf file:

paths: {
  # the local path for dremio to store data.
  local: "/var/lib/dremio"
  spilling: ["/data/spill"]
  dist: "s3a://data-sprints/dremio-storage/"
}


services: {
  coordinator.enabled: false,
  coordinator.master.enabled: false,
  executor.enabled: true
}

zookeeper: "<COORD_IP>:2181"

Coordinator conf:

paths: {
  # the local path for dremio to store data.
  local: "/var/lib/dremio"
  spilling: ["/data/spill"]
  dist: "s3a://data-sprints/dremio-storage/"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: false 
  coordinator.web.port: 80
}

zookeeper: "<COORD_IP>:2181"