Error restore backup

1

Hi,
I am trying to move Dremio docker container to another host.

First, I created backup:
bin/dremio-admin backup -d /local_files/ -i -j

Backup created at /local_files/dremio_backup_2020-08-18_14.55, dremio tables 24, uploaded files 0

Then I created tar and move to another docker host.
I run docker-compose and then restore backup:
m -rf data/db/
bin/dremio-admin restore -d /local_files/dremio_backup_2020-08-18_14.55/ -v -r

Restored from backup at /local_files/dremio_backup_2020-08-18_14.55/, dremio tables 24, uploaded files 0

Then I say
sudo docker-compose down
sudo docker-compose up

and I got next errors:

2020-08-18 15:27:30,642 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
dremio_1 | org.rocksdb.RocksDBException: Can’t access /000149.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000149.sst: No such file or directory
dremio_1 | Can’t access /000141.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000141.sst: No such file or directory
dremio_1 | Can’t access /000133.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000133.sst: No such file or directory
dremio_1 | Can’t access /000131.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000131.sst: No such file or directory
dremio_1 | Can’t access /000139.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000139.sst: No such file or directory
dremio_1 | Can’t access /000096.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000096.sst: No such file or directory
dremio_1 | Can’t access /000064.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000064.sst: No such file or directory
dremio_1 | Sst file size mismatch: /opt/dremio/data/db/catalog/000058.sst. Size recorded in manifest 17879590, actual size 1376
dremio_1 | Sst file size mismatch: /opt/dremio/data/db/catalog/000062.sst. Size recorded in manifest 21799013, actual size 1949
dremio_1 | Can’t access /000068.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000068.sst: No such file or directory
dremio_1 | Can’t access /000070.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000070.sst: No such file or directory
dremio_1 | Can’t access /000074.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000074.sst: No such file or directory
dremio_1 | Can’t access /000076.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000076.sst: No such file or directory
dremio_1 | Can’t access /000082.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000082.sst: No such file or directory
dremio_1 | Can’t access /000086.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000086.sst: No such file or directory
dremio_1 | Can’t access /000088.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000088.sst: No such file or directory
dremio_1 | Can’t access /000092.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000092.sst: No such file or directory
dremio_1 | Can’t access /000134.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000134.sst: No such file or directory
dremio_1 | Can’t access /000137.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000137.sst: No such file or directory
dremio_1 | Can’t access /000130.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000130.sst: No such file or directory
dremio_1 | Can’t access /000078.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000078.sst: No such file or directory
dremio_1 | Can’t access /000072.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000072.sst: No such file or directory
…
Dremio is exiting. Failure while starting services.
dremio_1 | org.rocksdb.RocksDBException: Can’t access /000149.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000149.sst: No such file or directory
dremio_1 | Can’t access /000141.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000141.sst: No such file or directory
dremio_1 | Can’t access /000133.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000133.sst: No such file or directory
dremio_1 | Can’t access /000131.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000131.sst: No such file or directory
dremio_1 | Can’t access /000139.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000139.sst: No such file or directory
dremio_1 | Can’t access /000096.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000096.sst: No such file or directory
dremio_1 | Can’t access /000064.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000064.sst: No such file or directory
Can’t access /000120.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000120.sst: No such file or directory
dremio_1 | Can’t access /000122.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000122.sst: No such file or directory
dremio_1 | Can’t access /000124.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000124.sst: No such file or directory
dremio_1 | Can’t access /000126.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000126.sst: No such file or directory
dremio_1 | Can’t access /000136.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000136.sst: No such file or directory
dremio_1 |
dremio_1 | at org.rocksdb.RocksDB.open(Native Method)
dremio_1 | at org.rocksdb.RocksDB.open(RocksDB.java:286)
dremio_1 | at com.dremio.datastore.ByteStoreManager.openDB(ByteStoreManager.java:281)
dremio_1 | at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:229)
dremio_1 | at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:148)
dremio_1 | at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:152)
dremio_1 | at com.dremio.datastore.adapter.LegacyKVStoreProviderAdapter.start(LegacyKVStoreProviderAdapter.java:69)
dremio_1 | at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:182)
dremio_1 | at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:141)
dremio_1 | Suppressed: java.lang.IllegalStateException: #start was not invoked, so metadataManager is not available
dremio_1 | at com.google.common.base.Preconditions.checkState(Preconditions.java:508)
dremio_1 | at com.dremio.datastore.ByteStoreManager.getMetadataManager(ByteStoreManager.java:424)
dremio_1 | at com.dremio.datastore.ByteStoreManager.close(ByteStoreManager.java:431)
dremio_1 | at com.dremio.common.AutoCloseables.close(AutoCloseables.java:126)
dremio_1 | at com.dremio.common.AutoCloseables.close(AutoCloseables.java:76)
dremio_1 | at com.dremio.datastore.CoreStoreProviderImpl.close(CoreStoreProviderImpl.java:235)
dremio_1 | at com.dremio.datastore.LocalKVStoreProvider.close(LocalKVStoreProvider.java:196)
dremio_1 | at com.dremio.datastore.adapter.LegacyKVStoreProviderAdapter.close(LegacyKVStoreProviderAdapter.java:110)
dremio_1 | at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:185)
dremio_1 | … 1 more
dremio_1 | Heap
dremio_1 | def new generation total 7744K, used 2957K [0x00000006c0000000, 0x00000006c0860000, 0x0000000715550000)
dremio_1 | eden space 6912K, 37% used [0x00000006c0000000, 0x00000006c02804c0, 0x00000006c06c0000)
dremio_1 | from space 832K, 47% used [0x00000006c0790000, 0x00000006c07f3008, 0x00000006c0860000)
dremio_1 | to space 832K, 0% used [0x00000006c06c0000, 0x00000006c06c0000, 0x00000006c0790000)
dremio_1 | tenured generation total 17108K, used 14356K [0x0000000715550000, 0x0000000716605000, 0x00000007c0000000)
dremio_1 | the space 17108K, 83% used [0x0000000715550000, 0x0000000716355218, 0x0000000716355400, 0x0000000716605000)
dremio_1 | Metaspace used 19933K, capacity 20178K, committed 20352K, reserved 1067008K
dremio_1 | class space used 2258K, capacity 2339K, committed 2432K, reserved 1048576K

I got error with .sst files.
What I did wrong?
May be I should not to docker-compose down | up on new server?
But how to apply the backup?

My docker-compose file:

version: ‘3’
services:
dremio:
restart: always
image: dremio/dremio-oss:4.5.0
volumes:
- /data/dremio-test/local_files:/local_files
- /data/dremio-test/db:/opt/dremio/data/db
ports:
- 9047:9047
- 31010:31010

2

After running this did you create a blank db directory mkdir db?

Restoring dremio step-by-step
On the master node, delete contents of <DREMIO_LOCAL_DATA_PATH> and then create an empty directory called db readable and writable by the user running restore tool and Dremio daemon under <DREMIO_LOCAL_DATA_PATH> .

3

Yes

dremio@d192945c74cb:/opt/dremio$ ls -la data/db/
total 8
drwxr-xr-x 2 dremio dremio 4096 Aug 19 19:57 .
drwxr-xrwx 6   1000   1000 4096 Aug 19 19:57 ..

I did restore on manual (first “-v” and second “-r”) and got error:

bin/dremio-admin restore -d /local_files/dremio_backup_2020-08-18_14.55/ -v
Restored from backup at /local_files/dremio_backup_2020-08-18_14.55/, dremio tables 24, uploaded files 0

dremio@2c7da2f03369:/opt/dremio$ bin/dremio-admin restore -d /local_files/dremio_backup_2020-08-18_14.55/ -r
Restore failed
java.lang.IllegalArgumentException: Path /opt/dremio/data/db must be an empty directory.

then I did “restore - rv”

Restored from backup at /local_files/dremio_backup_2020-08-18_14.55/, dremio tables 24, uploaded files 0

And nothing happens after that, web page “login” don’t work, only signup.
And if I do “docker-compose down and docker-compose up” I got error, which I wrote above:

2020-08-19 21:12:36,003 [main] ERROR ROOT - Dremio is exiting. Failure while starting services.
dremio_1 | org.rocksdb.RocksDBException: Can’t access /000149.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000149.sst: No such file or directory
dremio_1 | Can’t access /000141.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000141.sst: No such file or directory
dremio_1 | Can’t access /000133.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000133.sst: No such file or directory
dremio_1 | Can’t access /000131.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000131.sst: No such file or directory
dremio_1 | Can’t access /000139.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000139.sst: No such file or directory
dremio_1 | Can’t access /000094.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000094.sst: No such file or directory
dremio_1 | Can’t access /000064.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000064.sst: No such file or directory
dremio_1 | Sst file size mismatch: /opt/dremio/data/db/catalog/000058.sst. Size recorded in manifest 17883749, actual size 1373
dremio_1 | Sst file size mismatch: /opt/dremio/data/db/catalog/000062.sst. Size recorded in manifest 21794843, actual size 2140
dremio_1 | Can’t access /000068.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000068.sst: No such file or directory
dremio_1 | Can’t access /000070.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000070.sst: No such file or directory
…
at org.rocksdb.RocksDB.open(Native Method)
dremio_1 | at org.rocksdb.RocksDB.open(RocksDB.java:286)
dremio_1 | at com.dremio.datastore.ByteStoreManager.openDB(ByteStoreManager.java:281)
dremio_1 | at com.dremio.datastore.ByteStoreManager.start(ByteStoreManager.java:229)
dremio_1 | at com.dremio.datastore.CoreStoreProviderImpl.start(CoreStoreProviderImpl.java:148)
dremio_1 | at com.dremio.datastore.LocalKVStoreProvider.start(LocalKVStoreProvider.java:152)
dremio_1 | at com.dremio.datastore.adapter.LegacyKVStoreProviderAdapter.start(LegacyKVStoreProviderAdapter.java:69)
dremio_1 | at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:182)
dremio_1 | at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:141)
dremio_1 | Suppressed: java.lang.IllegalStateException: #start was not invoked, so metadataManager is not available
dremio_1 | at com.google.common.base.Preconditions.checkState(Preconditions.java:508)
dremio_1 | at com.dremio.datastore.ByteStoreManager.getMetadataManager(ByteStoreManager.java:424)
dremio_1 | at com.dremio.datastore.ByteStoreManager.close(ByteStoreManager.java:431)
dremio_1 | at com.dremio.common.AutoCloseables.close(AutoCloseables.java:126)
dremio_1 | at com.dremio.common.AutoCloseables.close(AutoCloseables.java:76)
dremio_1 | at com.dremio.datastore.CoreStoreProviderImpl.close(CoreStoreProviderImpl.java:235)
dremio_1 | at com.dremio.datastore.LocalKVStoreProvider.close(LocalKVStoreProvider.java:196)
dremio_1 | at com.dremio.datastore.adapter.LegacyKVStoreProviderAdapter.close(LegacyKVStoreProviderAdapter.java:110)
dremio_1 | at com.dremio.dac.cmd.upgrade.Upgrade.run(Upgrade.java:185)
dremio_1 | … 1 common frames omitted
dremio_1 | Dremio is exiting. Failure while starting services.
dremio_1 | org.rocksdb.RocksDBException: Can’t access /000149.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000149.sst: No such file or directory
dremio_1 | Can’t access /000141.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000141.sst: No such file or directory
dremio_1 | Can’t access /000133.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000133.sst: No such file or direct

4

@alexdi

Couple of things

Thanks
Bali

5

Good afternoon,

Anyone got some solution for this?

I’m facing the same issue of @alexdi.

Thank you in advance.

6

@alexdi @FulvioMascara

After restore is complete, do the file sizes and number of files match with the source?

8

Answering your question, the sst files wasn’t on backup file created by dremio-admin backup, but the file sizes of json files are ok.

9

Hi @balaji.ramaswamy

I’m trying to restore a backup made in a Docker version (ubuntu), using Dremio-admin backup. The destination host is a Mac with Docker too, with same version of Dremio (4.6.0).
First, I tried to follow these procedures: https://docs.dremio.com/advanced-administration/restore.html

To do this, I accessed docker container with docker exec -it Dremio /bin/bash, removed contents of /opt/dremio/data and recreated db folder inside /opt/dremio/data. After this, I ran ./dremio-admin restore -d folder_backup_files -r -v and got success message.

After this, I left container shell, restarted the container and I got those messages where sst files are missing, avoiding to Dremio start.

Did I forget something?

Thank you.

10

@FulvioMascara

It seems the restore may not have worked as expected. Can you go to the folder you restored and validate the same SST files exists as source. Are you able to provide the backup log and restore log?

11

@balaji.ramaswamy,

The thing is that backup procedure doesn’t copy sst files in output backup folder.
I tried to copy manually data folder from my source server and copy (changing owner and group) to new server. After restart, Dremio didn’t recognise files on folder, but the files are there.

I’m gonna run backup again and save logs to send for you, as well restore logs.

Thank you.

12

@FulvioMascara

I understand that the backup does not expose SST files, my question was that after the restore is done, have we matched source and target “db” folders for same number of sst and log files?

Will review your backup and restore logs, once I receive them

13

Hi @balaji.ramaswamy,

Here is a ZIP file with backup logs, restore logs, screenshots and service start log in Docker after restore, with messages about missing sst files.

Thank you a lot.dremio.zip (2.3 MB)

14

@FulvioMascara

Can you please check if this file actually exists and just that Dremio is not able to read it? Maybe the disk is not readable inside the docker? Can you please validate if you are able to do a

ls -ltrh /opt/dremio/data/db/catalog/000114.sst
ls -ltrh /opt/dremio/data/db/search/collaboration_wiki/core/write.lock

Also I see a size mismatch reported, can you please do “md5sum” or “shasum” on the below file and compare with the same on the source where you took a backup?

/opt/dremio/data/db/catalog/000058.sst

15

@balaji.ramaswamy

I check that 000114.sst doesn’t exists, but write.lock exists and it is empty.

About 000058.sst file, it doesn’t exists on the source.

Maybe I’m doing backup in a wrong way, using Docker.

Can you share some advice about it, so I can try to backup again?

Thank you again.

16

@FulvioMascara

Try an alternate method and then we can do backup/restore

  • Shutdown Dremio on source

  • Copy the db folder to the target

  • Start target (assume source and target are the same version)

  • Reflections need to be recreated

17

@balaji.ramaswamy

Unfortunately, I couldn’t copy files to target docker container.

I started a new instance with docker run, entered in container, removed and recreated data folder (with correct ownership and group) and stopped the container …
When I tried to copy db folder (source) with docker cp -a to keep ownership and group, it doesn’t work.

So I started container again, removed and recreated data folder again and with container up I ran docker cp without -a option and copy was ok (but with wrong ownership. So I entered in container and ran chown in data folder to change for dremio:dremio

So I restarted container, but Dremio doesn’t recognize the files in catalog folder.

Do you have some other idea?

I’m stuck on this for 2 weeks :frowning:

18

@FulvioMascara

When you say Dremio does not recognize the files, are you

#1 Getting an error, if so what is it
#2 When you logon to the UI, are you getting the page to signup? This means your dremio.conf is not pointing to the “db” folder location

Thanks
Bali

19

I have the same problem, generate the backup according to the guide.

 ./dremio-admin backup -u dremio -p dremio123 -d /tmp/backup

Then I started dremio from scratch and uploaded the bkp to the tmp / directory

./dremio-admin restore -d /tmp/dremio_backup_2021-09-02_19.29

generated error for db / directory not empty.

Then I deleted the directory and rebuilt it with all permissions …

Run the restore again.

./dremio-admin restore -d /tmp/dremio_backup_2021-09-02_19.29

I finish correctly.

Restart pod and it won’t lift, indicates missing .sst

Can't access /000082.sst: IO error: while stat a file for size: /opt/dremio/data/db/catalog/000082.sst: No such file or directory

any option to restore? T_T

20

I just had the same issue. For me this was caused by doing the ‘restore’ action in a docker container which was also running dremio-fg. Once I setup a sidecar container to do the restore action while the dremio service was down the restore worked like a charm.