Issue: No FileSystem for scheme: dremioS3

1

I am using dremio 3.2.4 and using AWS s3 as a distributed storage. But encounter an issue No FileSystem for scheme: dremioS3 . This issue is resolved after providing AWS access key and secret key in core-site.xml
https://docs.dremio.com/deployment/distributed-storage.html#amason-s3

But I don’t want to use IAM user. Because I have given all the access to the machine using iam role.

paths: {

the local path for dremio to store data.

local: “/var/lib/dremio”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “dremioS3:///dremio_bucket/folder1/folder2”
}

services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true
}

While it is working fine with version 3.1.9. And we don’t require any access key and secret key. It is working fine with iam role

paths: {

the local path for dremio to store data.

local: “/var/lib/dremio”

the distributed path Dremio data including job results, downloads, uploads, etc

dist: “s3a://dremio_bucket/folder1/folder2”
}

services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true
}

2

Hi, from your post, it seems like you are trying to use EC2 metadata as an authentication method for distributed storage. I am assuming you are running Dremio on an EC2 machine with IAM role attached. The dist store path you provided looks correct.

In 3.2.4 with EC2 Metadata, we recommend using a core-site.xml with the following properties:

  1. Property Name: fs.dremioS3.impl
    Value: com.dremio.plugins.s3.store.S3FileSystem
  2. Property Name: fs.s3a.aws.credentials.provider
    Value: org.apache.hadoop.fs.s3a.SharedInstanceProfileCredentialsProvider
  3. Property Name: dremio.s3.async
    Value: true

If the above core-site.xml settings do not work, do you mind sharing with me what error are you encountering when trying to use EC2 metadata with 3.2.4? You can enable debug logging by changing settings in conf/logback.xml. Look for logger name “com.dremio”, and please change log level from “-info” to “-debug”. You can obtain the stack trace for the error from log/server.log.

3

Hi, but I tried old distributed path as s3a rather than dremioS3 written in the updated documentation. I am able to run dremio 3.2.4 successfully. This solution works for me. This is very strange.

local: “/var/lib/dremio”
dist: “s3a://dremio_bucket/folder1/folder2”
}

services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true
}

4

Which one is safer between dremio.s3.async is true or not?