after upgrading to 1.3 we are seeing this when trying to start the coordinator. Any ideas?
Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting.
java.lang.NullPointerException: Unknown store creator com.dremio.exec.store.CatalogServiceImpl$CatalogSourceDataCreator
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:250)
at com.dremio.datastore.LocalKVStoreProvider.getStore(LocalKVStoreProvider.java:80)
at com.dremio.exec.store.CatalogServiceImpl.start(CatalogServiceImpl.java:139)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:171)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:177)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:172)
Could you provide details on how you upgraded to 1.3? as well as list <install_dir>/jars directory?
I followed the steps outlined here. https://docs.dremio.com/advanced-administration/upgrade/rpm.html
we have just the one coordinator so I ran steps 5 and 6 on that.
here are the jars
3rdparty
bundled
dremio-common-1.3.0-201711211846350824-31f8d91.jar
dremio-dac-backend-1.3.0-201711211846350824-31f8d91.jar
dremio-dac-common-1.3.0-201711211846350824-31f8d91.jar
dremio-dac-daemon-1.3.0-201711211846350824-31f8d91.jar
dremio-dac-ui-1.3.0-201711211846350824-31f8d91.jar
dremio-elasticsearch-plugin-1.3.0-201711211846350824-31f8d91.jar
dremio-extra-plugin-elasticsearch-1.3.0-201711211846350824-31f8d91.jar
dremio-extra-plugin-hive-1.3.0-201711211846350824-31f8d91.jar
dremio-extra-plugin-jdbc-1.3.0-201711211846350824-31f8d91.jar
dremio-extra-plugin-mongo-1.3.0-201711211846350824-31f8d91.jar
dremio-extra-plugin-parquet-1.3.0-201711211846350824-31f8d91.jar
dremio-extra-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar
dremio-extra-sabot-scheduler-1.3.0-201711211846350824-31f8d91.jar
dremio-hbase-plugin-1.3.0-201711211846350824-31f8d91.jar
dremio-hive-plugin-1.3.0-201711211846350824-31f8d91.jar
dremio-jdbc-plugin-1.3.0-201711211846350824-31f8d91.jar
dremio-mongo-plugin-1.3.0-201711211846350824-31f8d91.jar
dremio-pdfs-plugin-1.3.0-201711211846350824-31f8d91.jar
dremio-protocol-1.3.0-201711211846350824-31f8d91.jar
dremio-provision-common-1.3.0-201711211846350824-31f8d91.jar
dremio-s3-plugin-1.3.0-201711211846350824-31f8d91.jar
dremio-sabot-kernel-1.3.0-201711211846350824-31f8d91.jar
dremio-sabot-logical-1.3.0-201711211846350824-31f8d91.jar
dremio-services-accelerator-1.3.0-201711211846350824-31f8d91.jar
dremio-services-base-rpc-1.3.0-201711211846350824-31f8d91.jar
dremio-services-coordinator-1.3.0-201711211846350824-31f8d91.jar
dremio-services-datastore-1.3.0-201711211846350824-31f8d91.jar
dremio-services-fabric-rpc-1.3.0-201711211846350824-31f8d91.jar
dremio-services-jobs-1.3.0-201711211846350824-31f8d91.jar
dremio-services-metrics-1.3.0-201711211846350824-31f8d91.jar
dremio-services-namespace-1.3.0-201711211846350824-31f8d91.jar
dremio-services-scheduler-1.3.0-201711211846350824-31f8d91.jar
dremio-services-users-1.3.0-201711211846350824-31f8d91.jar
dremio-tpch-sample-data-1.3.0-201711211846350824-31f8d91.jar
dremio-yarn-1.3.0-201711211846350824-31f8d91.jar
ext
hadoop-aws-2.6.0.jar.old
jdbc-driver
jets3t-0.9.4.jar
jets3t-0.9.4.zip
Is the error you posted the only output in server.out/log ? Anything else is there? Could you upload full server.log/out ?
this is the entire output
Thu Nov 30 02:48:12 UTC 2017 Starting dremio on ip-XX-XX-X-XX
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 120513
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 120513
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting.
java.lang.NullPointerException: Unknown store creator com.dremio.exec.store.CatalogServiceImpl$CatalogSourceDataCreator
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:250)
at com.dremio.datastore.LocalKVStoreProvider.getStore(LocalKVStoreProvider.java:80)
at com.dremio.exec.store.CatalogServiceImpl.start(CatalogServiceImpl.java:139)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:74)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:171)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:177)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:172)
what about server.log - anything there?
Please check if user you start dremio with has write permissions the following directory: /var/lib/dremio/db (or one where metadata is located) and all subdirs
I tried to start the process as root but that did not help. we get the same error.
Thanks
@Russ_Wilson
Thanks for the update. Couple of things
Can you please paste out out of below
ls -ltrh <dremio-data-directory>
Is this a rpm or tar ball install?
If tar ball install, did you run the upgrade utility as “root” or user owning dremio?
Thanks
@balaji.ramaswamy
Hi all,
I had the same problem after upgrading to 1.3.0. rpm installation, RHEL 7. I ran an upgrade script as a root user, but Dremio is running under the dremio user.
After checking server.log, the following fix works for me:
sudo chown dremio:dremio /var/lib/dremio/db/metadata/catalog-source-data_metadata.json
sudo chown dremio:dremio /var/lib/dremio/db/metadata/.catalog-source-data_metadata.json.crc
After this dremio service is started.
Hi @Dmitry_R, @Russ_Wilson
Please see upgrade documentation below, the upgrade script needs to be run as the user running Dremio process
Dremio Upgrade - RPM
Kindly let me now if you have any questions,
Thanks,
@balaji.ramaswamy
@shashi take a look at the upgrade script