We had some trouble upgrading our 3.1.110 version to the new 3.2 version. Even after the upgrade script dremio was complaining that the KVStore belonged to 3.1.110. Hence, we did a re-install, moving /var/lib/dremio to dremio_bak and /opt/dremio to /opt/dremio_bak.
The new install starts up, but now we want to move our users, tables and reflections back to the new instance.
Hi,
You cannot really just copy files over without running the upgrade command - can you run the upgrade command and post the output here?
The server.out log of a failed startup ends with this:
Dremio is exiting. Failure while starting services.
java.lang.IllegalStateException: KVStore has an older version (3.1.10-201904162146020182-adf690d) than the server (3.2.1-201905191350330803-1a33f83), please run the upgrade tool first
at com.google.common.base.Preconditions.checkState(Preconditions.java:721)
at com.dremio.dac.daemon.DremioDaemon$AutoUpgrade.ensureUpgradeSupported(DremioDaemon.java:67)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:178)
at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:160)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:97)
The output from the dremio-admin upgrade command is:
2019-05-23 07:47:41,599 [main] INFO com.dremio.common.config.SabotConfig - Configuration and plugin file(s) identified in 96ms.
Base Configuration:
- jar:file:/opt/dremio/jars/dremio-common-3.2.1-201905191350330803-1a33f83.jar!/sabot-default.conf
Intermediate Configuration and Plugin files, in order of precedence:
- jar:file:/opt/dremio/jars/dremio-services-users-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-common-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-common-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hive-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-pdfs-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-adls-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-provision-common-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-azure-storage-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-s3-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-elasticsearch-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-hive-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-configuration-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-jdbc-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-coordinator-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-mongo-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-kernel-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-datastore-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-sabot-kernel-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-ce-sabot-scheduler-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-daemon-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-mongo-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-jobs-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-sabot-logical-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-namespace-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-hbase-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-accelerator-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-dac-backend-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-options-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-jdbc-plugin-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/dremio-yarn-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
- jar:file:/opt/dremio/jars/3rdparty/dremio-hive-exec-shaded-3.2.1-201905191350330803-1a33f83.jar!/sabot-module.conf
2019-05-23 07:47:42,220 [main] INFO c.d.common.scanner.BuildTimeScan - Loaded prescanned packages [com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.storage, org.apache.hadoop.hive, com.dremio.exec.fn.hive, com.dremio.exec.store.hive, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.options, com.dremio.service.namespace, com.dremio.datastore, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, org.apache.hadoop.hive, com.dremio.storage, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.options, com.dremio.service.namespace, com.dremio.datastore, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.storage, com.dremio.datastore, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.options, com.dremio.service.namespace, com.dremio.datastore, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.service.jobs, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.datastore, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.options, com.dremio.resource, com.dremio.resource.basic, com.dremio.storage, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.datastore, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.options, com.dremio.service.users, com.dremio.resource, com.dremio.resource.basic, com.dremio.exec.store, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, com.dremio.provision, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.datastore, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.service.users, com.dremio.exec.store, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.options, com.dremio.resource, com.dremio.resource.basic, com.dremio.services.configuration, com.dremio.services.configuration.ConfigurationStore] from locations [jar:file:/opt/dremio/jars/dremio-common-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-common-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-hive-plugin-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-kernel-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-datastore-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-ce-sabot-scheduler-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-jobs-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-sabot-logical-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-services-accelerator-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json, jar:file:/opt/dremio/jars/dremio-dac-backend-3.2.1-201905191350330803-1a33f83.jar!/META-INF/dremio-module-scan/registry.json]
2019-05-23 07:47:43,239 [main] INFO c.d.common.scanner.ClassPathScanner - Scanning packages [com.dremio.service.users, com.dremio.exec.store.mock, com.dremio.common.logical, com.dremio.exec.store.dfs, com.dremio.exec.server.options, org.apache.hadoop.hive, com.dremio.exec.fn.hive, com.dremio.exec.store.hive, com.dremio.exec.store, com.dremio.plugins.adl.store, com.dremio.provision, com.dremio.plugins.azure, com.dremio.plugins.s3.store, com.dremio.extras.plugins.elastic, com.dremio.services.configuration, com.dremio.services.configuration.ConfigurationStore, com.dremio.exec.store.jdbc, com.dremio.plugins.mongo, com.dremio.exec.ExecConstants, com.dremio.exec.catalog, com.dremio.exec.compile, com.dremio.exec.expr, com.dremio.exec.physical, com.dremio.exec.planner.physical, com.dremio.exec.server.options, com.dremio.exec.store, com.dremio.exec.store.dfs.implicit.ImplicitFilesystemColumnFinder, com.dremio.exec.rpc.user.security, com.dremio.sabot, com.dremio.sabot.op.aggregate.vectorized, com.dremio.datastore, com.dremio.extra.exec.store.dfs, com.dremio.exec.planner.acceleration.substitution, com.dremio.sabot.task.slicing.SlicingTaskPool, com.dremio.plugins.elastic, com.dremio.plugins.mongo, com.dremio.service.jobs, com.dremio.service.namespace, com.dremio.exec.store.hbase, com.dremio.exec.expr.fn.impl.conv, com.dremio.service.accelerator, com.dremio.service.reflection, com.dremio.service.voting, com.dremio.dac, com.dremio.dac.support.SupportService, com.dremio.options, com.dremio.exec.store.jdbc, com.dremio.resource, com.dremio.resource.basic, com.dremio.provision.yarn.service, org.apache.hadoop.hive] in locations [jar:file:/opt/dremio/jars/dremio-services-users-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-pdfs-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-adls-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-provision-common-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-azure-storage-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-s3-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-ce-elasticsearch-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-ce-hive-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-services-configuration-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-ce-jdbc-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-services-coordinator-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-ce-mongo-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-ce-sabot-kernel-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-dac-daemon-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-elasticsearch-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-mongo-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-services-namespace-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-hbase-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-services-options-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-jdbc-plugin-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-services-resourcescheduler-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/dremio-yarn-3.2.1-201905191350330803-1a33f83.jar!/, jar:file:/opt/dremio/jars/3rdparty/dremio-hive-exec-shaded-3.2.1-201905191350330803-1a33f83.jar!/] took 1015ms
2019-05-23 07:47:43,309 [main] INFO c.d.datastore.LocalKVStoreProvider - Starting LocalKVStoreProvider
2019-05-23 07:47:43,316 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopping LocalKVStoreProvider
2019-05-23 07:47:43,317 [main] INFO c.d.datastore.LocalKVStoreProvider - Stopped LocalKVStoreProviderThe bigger question here is: How do we get our Dremio environment up and running again without having to redefine all our tables from scratch?
I can attempt to re-install the 3.1 version, but will that work with partially upgraded files? I don’t know what got upgraded and what didn’t (or what needs upgrading). Am I just going to break things even more that way?
It’s been a couple of days and I haven’t seen any progress on this case. Thankfully, this isn’t a production environment yet, but it’s supposed to be soon (we’re already past our deadline). We’re starting to have some doubts about the reliability of Dremio…
We managed to fix the problem by removing 3.2 and reinstalling 3.1.10. Apparently, dremio-admin upgrade had not upgraded any relevant files.
We found that, while the (rpm-)upgrade instructions tell to create a copy of /opt/dremio, the really important files appear to be in /var/lib/dremio instead. Creating a copy of that directory as well seems like a good idea for the upgrade instructions.
In hindsight, skipping over the “Create a backup” part of the instructions for what seems to be a minor version upgrade of an environment that’s still much POC in our case, wasn’t exactly a brilliant move. Turns out even a POC can be important enough to warrant creating backups… We were lucky this time.