Connecting Dremio to S3 compatible store (Minio) with docker

1

Hi,

it is unclear to me why this configuration does not work. I followed the documentation. Minio bucket credentials are setup and the tool is accessible. Both tools are running on the same VM to which I connect via SSH.

Encrypt connection is off.
Access keys are correct.
Compatibility mode is on.
Added properties:
fs.s3a.path.style.access: true
fs.s3a.endpoint: http://localhost:9000

Dremio was started with:
docker run -p 9047:9047 -p 31010:31010 -p 45678:45678 -d dremio/dremio-oss

and Minio with:

docker run \
   -d \
   -p 9000:9000 \
   -p 9001:9001 \
   --name minio \
   -v ~/minio/data:/data \
   -e "MINIO_ROOT_USER=admin" \
   -e "MINIO_ROOT_PASSWORD=..." \
   quay.io/minio/minio server /data --console-address ":9001"

Error message is: Could not connect to S3 source. Check your S3 data source settings and credentials.

2

if you are running both in docker containers, in Dremio config you should use “host ip” you cannot use localhost because this address refers to localhost of container running dremio, and you have running minio in “another host” because is in another container.

i recommend you use docker compose wich by default connect containers using internal network so you can use “container name” in the dremio config to reference minio container.

also by security do not expose ports no all networks, minio for example must be expose only inside host machine, and if you connect them with docker network no port expose is needed

3

The fs.s3a.endpoint endpoint value cannot contain the http(s):// prefix. For example, if the endpoint is http://123.1.2.3:9000 , the value should be 123.1.2.3:9000. See here.