Connecting S3 bucket via ec2 metadata

1

Hi Team,
I am having only one S3 bucket and that needs to connect dremio. Due to security concerns i cannot provide access key and secret key. So i have attached one IAM policy without * resource ( Which mentioned below) to Dremio instance and trying to fetch the data. But it is giving the error related to “sts:AssumeRole” and this assumerole policy cannot be given due to security concerns. So Please take this issue and share me the resolution.

Attached IAM Policy:

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “Stmt1554423012000”,
“Effect”: “Allow”,
“Action”: [
“s3:GetBucketLocation”,
“s3:ListAllMyBuckets”,
“s3:ListBucket”
],
“Resource”: [
“arn:aws:s3::: my-bucketname”
]
},
{
“Sid”: “Stmt1554423050000”,
“Effect”: “Allow”,
“Action”: [
“s3:GetObject”
],
“Resource”: [
“arn:aws:s3::: my-bucketname*”
]
}
]
}

Error Mentioned by dremio:
"
No AWS Credentials provided by STSCredentialProviderV1 : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::795770151185:assumed-role/bhged-quad-s3-access-new/i-0c7d6945d08245639 is not authorized to perform: sts:AssumeRole on resource: bhged-quad-s3-access-new (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 31eae4f2-2087-440f-8a75-10c3cca25611) "

Thanks and Regards,
Sarvasiddi Manikanta.

2

@msarvasiddi

If you edit the S3 source and under advanced options, enable compatibility mode, does it help?

3

We have enabled the compatibility mode and attached all the details in the required spaces. But getting the same error related to Sts:AssumeRole

Please find the exact error below

“java.nio.file.AccessDeniedException: gcp-s3-migration: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by STSCredentialProviderV1 : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::795770151185:assumed-role/bhged-quad-s3-access/i-0c7d6945d08245639 is not authorized to perform: sts:AssumeRole on resource: bhged-quad-s3-access (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: 85341c17-89cc-401d-8f64-b9353cc40b79)”

Please check this

4

@msarvasiddi

Can you also add the below to core-site.xml and restart Dremio cluster, retry?

<property>>
<name>dremio.s3.compat</name>
<description>Compatiable mode, not calling AWS STS</description>
<value>true</value>
</property>