Error configuring Storage s3

1

Hello! I’m configuring the storage s3 in Dremio to store Dremio data (Reflections, Uploads, etc …), but I’m having problems. I did the configuration following the doc, but it doesn’t work. Here are the settings:

core-site.xml:

<?xml version="1.0"?>
<configuration>
<property>
    <name>fs.dremioS3.impl</name>
    <description>The FileSystem implementation. Must be set to com.dremio.plugins.s3.store.S3FileSystem</description>
    <value>com.dremio.plugins.s3.store.S3FileSystem</value>
</property>
<property>
    <name>fs.s3a.access.key</name>
    <description>AWS access key ID.</description>
    <value>MY-ACCESS-KEY</value>
</property>
<property>
    <name>fs.s3a.secret.key</name>
    <description>AWS secret key.</description>
    <value>MY-SECRET-KEY</value>
</property>
<property>
    <name>fs.s3a.aws.credentials.provider</name>
    <description>The credential provider type.</description>
    <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
</property>
</configuration>

dremio.conf:

paths: {
  # the local path for dremio to store data.
  local: "/var/lib/dremio"

  # the distributed path Dremio data including job results, downloads, uploads, etc
  #dist: "pdfs://"${paths.local}"/pdfs"
  dist: "s3a://dremio-reflections/reflections"
}

services: {
  coordinator.enabled: true,
  coordinator.master.enabled: true,
  executor.enabled: true
}

Caused by: java.nio.file.AccessDeniedException: dremio-reflections: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: The requested metadata is not found

2

you are missing / in s3a:///dremio-reflections/reflections

according documentation must be "dremioS3:///<bucket_name>/<folder1>/<folder2>"

EDIT
if above not work, please consider this:
if you use InstanceProfileCredentialsProvider you can’t use MY-ACCESS-KEY and MY-SECRET-KEY you must use org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

1 Like
3

Sorry, lack of attention on my part. It worked here, thanks @dacopan!