I’m running Dremio on a Docker container on remote server. In the same server I have an another Docker Container with MinIO. But I can’t connect Dremio and MinIO.
This is my dremio.conf file:
# the local path for dremio to store data.
local: ${DREMIO_HOME}"/data"
# the distributed path Dremio data including job results, downloads, uploads, etc
#dist: "pdfs://"${paths.local}"/pdfs"
dist: "dremioS3:///dtlake/"
}
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true
}
And this is my core-site.xml file:
<configuration>
<property>
<name>fs.dremioS3.impl</name>
<description>The FileSystem implementation. Must be set to com.dremio.plugins.s3.store.S3FileSystem</description>
<value>com.dremio.plugins.s3.store.S3FileSystem</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<description>Minio server access key ID.</description>
<value>xxxx</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<description>Minio server secret key.</description>
<value>xxxx</value>
</property>
<property>
<name>fs.s3a.aws.credentials.provider</name>
<description>The credential provider type.</description>
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>
<property>
<name>fs.s3a.endpoint</name>
<description>Endpoint can either be an IP or a hostname, where Minio server is running . However the endpoint value cannot contain the http(s) prefix. E.g. 175.1.2.3:9000 is a valid endpoint. </description>
<value>127.0.0.1:9000</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
<property>
<name>dremio.s3.compat</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<description>Value can either be true or false, set to true to use SSL with a secure Minio server.</description>
<value>false</value>
</property>
</configuration>
I this is my configuration on Dremio:
I’ll apreciated all your help.
@chuto81
Kindly send us the server.log to investigate
Thanks
Bali
Bali thanks a lot for your time. I was unable to find the server.log file, I was looking and all “~” and “/” files on ubuntu remote server and I can’t find it. But this is the server.out file.
Fri Mar 26 15:45:30 UTC 2021 Starting dremio on 44d9f3b3a9d1
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15689
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Fri Mar 26 15:46:57 UTC 2021 Starting dremio on 44d9f3b3a9d1
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15689
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Bali, this the node activity:
I used this command for run dremio on docker:
docker run -p 9047:9047 -p 31010:31010 -p 45678:45678 dremio/dremio-oss
And those are all the server remote ip’s used for Dremio and MinIO:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-n 449 systemd-network 19u IPv4 959169 0t0 UDP 172.31.44.236:68
systemd-r 464 systemd-resolve 12u IPv4 18247 0t0 UDP 127.0.0.53:53
systemd-r 464 systemd-resolve 13u IPv4 18248 0t0 TCP 127.0.0.53:53 (LISTEN)
sshd 740 root 3u IPv4 21916 0t0 TCP *:22 (LISTEN)
sshd 740 root 4u IPv6 21927 0t0 TCP *:22 (LISTEN)
docker-pr 2035 root 4u IPv4 31725 0t0 TCP *:9000 (LISTEN)
docker-pr 10119 root 4u IPv4 69647 0t0 TCP *:45678 (LISTEN)
docker-pr 10131 root 4u IPv4 69758 0t0 TCP *:31010 (LISTEN)
docker-pr 10142 root 3u IPv4 961426 0t0 TCP 127.0.0.1:9047->127.0.0.1:45078 (ESTABLISHED)
docker-pr 10142 root 4u IPv4 70726 0t0 TCP *:9047 (LISTEN)
docker-pr 10142 root 5u IPv4 961428 0t0 TCP 172.17.0.1:46222->172.17.0.3:9047 (ESTABLISHED)
docker-pr 10142 root 11u IPv4 961309 0t0 TCP 127.0.0.1:9047->127.0.0.1:45038 (ESTABLISHED)
docker-pr 10142 root 12u IPv4 961311 0t0 TCP 172.17.0.1:46182->172.17.0.3:9047 (ESTABLISHED)
sshd 21370 root 4u IPv4 566575 0t0 TCP 172.31.44.236:22->191.156.140.133:49776 (ESTABLISHED)
sshd 21496 ubuntu 4u IPv4 566575 0t0 TCP 172.31.44.236:22->191.156.140.133:49776 (ESTABLISHED)
sshd 21496 ubuntu 11u IPv4 961858 0t0 TCP 127.0.0.1:45078->127.0.0.1:9047 (ESTABLISHED)
sshd 21496 ubuntu 14u IPv4 961789 0t0 TCP 127.0.0.1:45038->127.0.0.1:9047 (ESTABLISHED)
sshd 21506 root 4u IPv4 567876 0t0 TCP 172.31.44.236:22->191.156.140.133:49761 (ESTABLISHED)
sshd 21589 ubuntu 4u IPv4 567876 0t0 TCP 172.31.44.236:22->191.156.140.133:49761 (ESTABLISHED)
sshd 21600 root 4u IPv4 568091 0t0 TCP 172.31.44.236:22->191.156.140.133:49788 (ESTABLISHED)
sshd 21684 ubuntu 4u IPv4 568091 0t0 TCP 172.31.44.236:22->191.156.140.133:49788 (ESTABLISHED)
Thanks again.
@chuto81
Wondering if the server.log is going to the standard out of the Docker? Are you able to use “journalctl” commands to view application logs?
