stsTagSession error on first query execution - suceeds in second attempt

Dremio Cloud
1

Hello there,
Since last few days we are getting following error for every table in Dremio when we execute the query first time. Strangely it works fine when executed again:
Credential verification failed. arn:aws:sts:::assumed-role/ServerlessDataAccessRole/eks-serverless-dremio-exe-81ee839b-642f-4f49-a653-ba44780c823b is not authorized to perform sts:TagSession on arn:aws:iam:::role/trg-am-dev-dremio-cross-account-role. Verify your AWS credentials and IAM permissions.

We are using our own AWS S3 storage. Trust was established between Dremio and AWS when project was created. We have two projects in Dremio so far and error is seen in both.

Has anyone else faced this and what can be the fix?

Our devOps team has verified policies on our end
The trust policy on the target cross-account role (in the S3 account we control) grants both sts:AssumeRole and sts:TagSession to the Dremio account root, in a single statement under one sts:ExternalId condition. Since both actions are evaluated together in the same STS call and AssumeRole clearly succeeds on retry with the same ExternalId, the condition isn’t gating TagSession differently. The target trust side appears correctly configured.

The failing identity (ServerlessDataAccessRole) is in Dremio’s own serverless account, which we don’t control, so we can’t inspect its identity policy.

2

Hi @DnyaneshwarD ,

Thanks for reaching out on the community forums.

Would you mind sharing your Dremio Cloud organization id and project ID so we could investigate further? You can paste it here directly or DM it to me if you prefer.

Best,

3

Hi @cindy.la
Here is the organization id: ab727564-050e-41da-9203-4602f1508c2f
We have two projects and error occurs in both of them.
Project 1 id: 1d07c28f-b988-4d8a-9cb4-ab12417b0479
Project 2 id: ac53caee-4990-4d5a-abf8-bfedc10c9e44

Thanks so much for looking into it.

4

Happy Monday @cindy.la
Just wanted to follow up on this and see if you got chance to look into this further.

5

Hi @DnyaneshwarD ,

Apologies on the delay on answering this!

We have identified that this is a bug on Dremio Cloud’s side. We are planning to release a fix for this sometime this week.

There is, however, a workaround that can be applied if you do not want to wait for the permanent fix. You can use this policy, which should resolve the intermittent query failures with the stsTagSession error.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "AllowAssumeRoleWithExternalId",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::894535543691:root"
           },
           "Action": "sts:AssumeRole",
           "Condition": {
               "StringEquals": {
                   "sts:ExternalId": "<project ID>"
               }
           }
       },
       {
           "Sid": "AllowAssumeRoleWithRoleSessionName",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::894535543691:root"
           },
           "Action": "sts:AssumeRole",
           "Condition": {
               "StringEquals": {
                   "sts:RoleSessionName": "<project ID>"
               }
           }
       },
       {
           "Sid": "AllowTagSessionFromCallerRole",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::894535543691:root"
           },
           "Action": "sts:TagSession"
       }
   ]
}

We have validated internally that using this policy will work as a mitigation, until the fix is out. Please let us know if using this policy works for you.

Best,

6

@cindy.la
Will wait for the fix. Thank you so much for looking into it and the mitigation as well. Much appreciated! Have a good rest of the week.