Basic guidelines for using dremio docker

1

are there any basic guidelines for using the dremio docker image?

even like how to make configuration and user data persist from one run to the next?

i suppose it is a docker volume configuration, but not sure what the dremio path should be, etc.

2 Likes
2

You will find some information in https://github.com/dremio/containers

For a Dremio cluster in a single container, yes you can use docker volume configuration to persist the data locally. E.g.

docker run --rm -v /tmp/dremio:/opt/dremio/data dremio/dremio-oss

to have the local /tmp/dremio persist the data from dremio container.

Now, for a clustered deployment, the recommendation would be to look at the helm chart to deploy Dremio a kubernetes managed environment. These are examples that you can take and adjust for your environment. For example, the memory and cpu in values.yaml should be adjusted to account for the resource availability in your kubernetes environment. And set path.dist to point to a distributed file store (note that requires additional configuration in Dremio).

3

hmm…that didn’t seem to work.

i used the options you suggested (-v /tmp/dremio:/opt/dremio/data), making sure /tmp/dremio existed locally.

  • brought up home page in browser

  • registered an admin account

  • logged in as that account

  • logged out of that account

  • stopped container

  • nothing written to /tmp/dremio

  • restarted container

  • forced to create an admin account again

again, how do i persist the configuration and the user accounts?

4

hmm…just to make sure things haven’t changed, I tried it out and it worked. After the container is started and when stopped, the local /tmp/dremio should be like:

$ ls -l /tmp/dremio
total 0
drwxr-xr-x  5 nirmalyasen  wheel  160 Sep 27 09:09 db
drwxr-xr-x  7 nirmalyasen  wheel  224 Sep 27 09:04 pdfs
drwxr-xr-x  3 nirmalyasen  wheel   96 Sep 27 09:09 zk

What is your OS? What is your Docker version? What is the image used by your container? Do you mind sharing your full docker run command?

5

argh - i had a typo…"/opt/fremio/data".
fixed that and it works as you specified. sorry for that oversight.

thanks - this approach will really help me take some baby steps in looking at a dremio-class tool for experimentation and discovery.

2 Likes
6

Hi @nsen,

Do you know if persistence is possible for Windows 7?

I’ve tried following the previous posts and end up with a:

Dremio is exiting. Failure while starting services.
org.rocksdb.RocksDBException: While fsync: a directory: Invalid argument

docker@default:/dremio$ ls -l
total 0
drwxrwxrwx 1 docker staff 0 Nov 18 23:59 db
drwxrwxrwx 1 docker staff 0 Nov 18 23:58 zk

OS: Windows 7
Docker Version: Docker version 18.03.0-ce, build 0520e24302
Image: dremio/dremio-oss
run command: docker run --name dremio2 -p 9047:9047 -p 31010:31010 -p 45678:45678 -v /dremio:/opt/dremio/data dremio/dremio-oss

The folders zk and db make it to the folder on my laptops C:User/… drive but don’t make it to pdfs.

Any insight is greatly appreciated!

Thanks,
Mathew

7

Hi @mathew.lee

What type of mount are you writing your rocksDB to? The local or db parameter in your dremio.conf?

Thanks
@balaji.ramaswamy

8

Hi @balaji.ramaswamy,

How would I check on docker which type of mount I’m writing to? I apologize as I’m very new to docker and containers!

In addition, let me know if I should start another thread or post regarding this as well!

Thanks,
Mathew

9

HI @mathew.lee

Look at your /etc/fstab or /etc/mtab and the options that go with it

Thanks
@balaji.ramaswamy

10

@balaji.ramaswamy,

Hmm when I use:

docker exec -it -u root dremio1 bash

and try to cd into /etc/fstab or /etc/mtab, neither can be found.

Thanks for all the help!
Mathew

11

Sorry to ask again @mathew.lee, can you please provide the complete docker command you are using to start the Dremio container?