Configuring S3 as distributed cache

1

I see the example to configure HDFS as the custom distributed cache location. I’m trying to configure my custom cache location to S3 like this…

paths: {
local: “/data/dremio”
dist: “s3a://my-dremio-bucket/dremio-cache”
}

Where do I configure the AWS Credentials? When starting the dremio service, I’m getting this error.

Catastrophic failure occurred. Exiting. Information follows: Failed to start services, daemon exiting.
java.lang.RuntimeException: Failed to create FileSystemWrapper for proxy user: doesBucketExist on my-dremio-bucket: com.amazonaws.AmazonClientException: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/: No AWS Credentials provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider SharedInstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/

2

I believe you need to do it in core-site.xml:

    <property>
      <name>fs.s3a.access.key</name>
      <value>ACCESS-KEY</value>
    </property>

    <property>
      <name>fs.s3a.secret.key</name>
      <value>SECRET-KEY</value>
    </property>

same way you would do it for Hadoop and reference that core-site.xml in dremio classpath (e.g by setting DREMIO_CLASSPATH_USER_FIRST to path to your hadoop conf dir) or copy that file to dremio conf directory

1 Like
3

This worked for me. As stated, I create the core-site.xml file and put it in the dremio conf folder.