Hi, I’m trying to restore backup from kubernetes cluster dremio version 4.1.8 to aws ec2 Dremio version 4.2.1 and it’s fails with this error.
2020-10-23 10:04:15,866 [main] INFO admin - Restore failed
io.protostuff.UninitializedMessageException: null
at io.protostuff.JsonInput.mergeObject(JsonInput.java:437)
at com.dremio.dac.proto.model.backup.BackupFileInfo.mergeFrom(BackupFileInfo.java:206)
at com.dremio.dac.proto.model.backup.BackupFileInfo.mergeFrom(BackupFileInfo.java:20)
at io.protostuff.JsonIOUtils.mergeFrom(JsonIOUtils.java:277)
at com.dremio.common.utils.ProtostuffUtil.fromJSON(ProtostuffUtil.java:101)
at com.dremio.dac.util.BackupRestoreUtil.scanInfoFiles(BackupRestoreUtil.java:243)
at com.dremio.dac.util.BackupRestoreUtil.restore(BackupRestoreUtil.java:443)
at com.dremio.dac.cmd.Restore.main(Restore.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.dremio.dac.cmd.AdminCommandRunner.runCommand(AdminCommandRunner.java:80)
at com.dremio.dac.cmd.AdminCommandRunner.main(AdminCommandRunner.java:68)
How can I resolve this error?
@VahagnBleyan
Currently, Dremio only supports backup and restore on the same version. So you can restore to 4.1.8 and then upgrade. Alternatively, shutdown Dremio in 4.1.8, copy the db folder to the new machine on 4.2 and run a “dremio-admin upgrade” and then start Dremio on 4.2. In both method uploads would to be uploaded again. During backup/restore to a different machine, the copying of the db folder would work better
Thanks
Bali
Hi @balaji.ramaswamy, thanks for quick replay.
I tried the other way, I have upgraded my kubernetes dremio cluster to latest version (4.9.0 ) and restored backup from it on dremio enterprise 4.8.1 on aws instance. Restore was succesfulll, but dremio didn’t start after that. It’s continuously throwing error related to elasticsearch plugin when trying connect to my aws elasticsearch resources.
ERROR c.dremio.exec.catalog.PluginsManager - Failure while starting plugin “search-logs-recallmasters” after 0ms.
java.util.concurrent.ExecutionException: java.lang.Exception: Unavailable
Suggested User Action: Unable to find handler for source of type [AMAZONELASTIC].
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at com.dremio.exec.catalog.PluginsManager.start(PluginsManager.java:224)
at com.dremio.exec.catalog.CatalogServiceImpl.start(CatalogServiceImpl.java:191)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:137)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:88)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:33)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:195)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:201)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:146)
I found elasticsearch-5.5.3.jar file in my 4.9.0 version dremio cluster in jars/3rdparty but putting it in dremio 4.8.1 ee edition throws another error:
ERROR ROOT - Dremio is exiting. Failure while starting services.
com.google.common.util.concurrent.ExecutionError: java.util.ServiceConfigurationError: An SPI class of type org.apache.lucene.codecs.PostingsFormat with classname org.apache.lucene.search.suggest.document.Completion50PostingsFormat does not exist, please fix the file ‘META-INF/services/org.apache.lucene.codecs.PostingsFormat’ in your classpath.
also now I can’t downgrade my Dremio cluster on kubernetes to 4.8.1 or 4.8.0 even with uninstalling helm release and installing again.
Thanks
One more thing, I noticed that I can’t add elasticsearch source in Dremio EE 4.8.1 edition. Should I install any plugin or something to have that ability?And how should I do that if yes.
On Dremio CE I have access to aws elastic and cloud elastic.
Well, after removing elasticsearch sources and creating new backup on 4.9.0 Dremio kubernetes Cluster, restoring on Dremio 4.8.1 EE throws this error
sudo -u dremio ./dremio-admin restore -d /opt/dremio_backup_2020-10-26_10.06
Restore failed
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for s3ablock-0001-
Exception in thread “files-delete-on-exit” java.lang.NullPointerException
at org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1194)
at org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1184)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1294)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:285)
at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1292)
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2254)
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2209)
at org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:1757)
at com.dremio.plugins.util.ContainerFileSystem.delete(ContainerFileSystem.java:363)
at com.dremio.exec.hadoop.HadoopFileSystem.delete(HadoopFileSystem.java:343)
at com.dremio.io.file.FileSystemUtils$1.run(FileSystemUtils.java:54)
@VahagnBleyan
Are you trying to restore from Community to Enterprise edition?
Hi everyone!
I’m facing the same problem.
I’m migrating from version 4.3.1 to 4.3.1.
From an AWS EC2 instance to an GCP instance.
Please any solution for this problem or any ideas?
@joell.garcia The restore seems to be successful, the stack below is fixed in version 4.6 and above
