stsTagSession error on first query execution - suceeds in second attempt

Dremio Cloud
1

Hello there,
Since last few days we are getting following error for every table in Dremio when we execute the query first time. Strangely it works fine when executed again:
Credential verification failed. arn:aws:sts:::assumed-role/ServerlessDataAccessRole/eks-serverless-dremio-exe-81ee839b-642f-4f49-a653-ba44780c823b is not authorized to perform sts:TagSession on arn:aws:iam:::role/trg-am-dev-dremio-cross-account-role. Verify your AWS credentials and IAM permissions.

We are using our own AWS S3 storage. Trust was established between Dremio and AWS when project was created. We have two projects in Dremio so far and error is seen in both.

Has anyone else faced this and what can be the fix?

Our devOps team has verified policies on our end
The trust policy on the target cross-account role (in the S3 account we control) grants both sts:AssumeRole and sts:TagSession to the Dremio account root, in a single statement under one sts:ExternalId condition. Since both actions are evaluated together in the same STS call and AssumeRole clearly succeeds on retry with the same ExternalId, the condition isn’t gating TagSession differently. The target trust side appears correctly configured.

The failing identity (ServerlessDataAccessRole) is in Dremio’s own serverless account, which we don’t control, so we can’t inspect its identity policy.

2

Hi @DnyaneshwarD ,

Thanks for reaching out on the community forums.

Would you mind sharing your Dremio Cloud organization id and project ID so we could investigate further? You can paste it here directly or DM it to me if you prefer.

Best,

3

Hi @cindy.la
Here is the organization id: ab727564-050e-41da-9203-4602f1508c2f
We have two projects and error occurs in both of them.
Project 1 id: 1d07c28f-b988-4d8a-9cb4-ab12417b0479
Project 2 id: ac53caee-4990-4d5a-abf8-bfedc10c9e44

Thanks so much for looking into it.

4

Happy Monday @cindy.la
Just wanted to follow up on this and see if you got chance to look into this further.

5

Hi @DnyaneshwarD ,

Apologies on the delay on answering this!

We have identified that this is a bug on Dremio Cloud’s side. We are planning to release a fix for this sometime this week.

There is, however, a workaround that can be applied if you do not want to wait for the permanent fix. You can use this policy, which should resolve the intermittent query failures with the stsTagSession error.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "AllowAssumeRoleWithExternalId",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::894535543691:root"
           },
           "Action": "sts:AssumeRole",
           "Condition": {
               "StringEquals": {
                   "sts:ExternalId": "<project ID>"
               }
           }
       },
       {
           "Sid": "AllowAssumeRoleWithRoleSessionName",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::894535543691:root"
           },
           "Action": "sts:AssumeRole",
           "Condition": {
               "StringEquals": {
                   "sts:RoleSessionName": "<project ID>"
               }
           }
       },
       {
           "Sid": "AllowTagSessionFromCallerRole",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::894535543691:root"
           },
           "Action": "sts:TagSession"
       }
   ]
}

We have validated internally that using this policy will work as a mitigation, until the fix is out. Please let us know if using this policy works for you.

Best,

6

@cindy.la
Will wait for the fix. Thank you so much for looking into it and the mitigation as well. Much appreciated! Have a good rest of the week.

7

@cindy.la just wanted to follow up and check if the bug fix is delayed and what is the updated ETA. We are still seeing the error.

8

Hello @DnyaneshwarD,

Would you mind sharing the exact trust relationship of the arn:aws:iam::<aws_account_id>:role/trg-am-dev-dremio-cross-account-role IAM role? To double-check, this is the role used for the project store during project creation, correct?

Additionally,

  • what kind of source are you using for your queries (S3, Azure Data Lake Storage, Iceberg Rest Catalog, Dremio Catalog, …)?
  • can you give an example or two of the type of queries that you are running?
  • what kind of data are you running those queries on (raw parquet tables, iceberg tables, …)?

Best,
Francisco

9

@franciscocastanheira
Hi Francis,
thanks for following up. I’m a bit confused, though: @cindy.la confirmed on Sept 22 that this is a Dremio Cloud bug with a fix on the way. Has something changed in the findings, or is this additional verification? Either way, I’m happy to share the details below.

1. Trust relationship of trg-am-dev-dremio-cross-account-role (our account ID masked)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::894535543691:root" },
      "Action": ["sts:AssumeRole", "sts:TagSession"],
      "Condition": { "StringEquals": { "sts:ExternalId": "<dev project ID>" } }
    }
  ]
}

The principal is the Dremio account root. It was originally narrowed to role/ServerlessDataAccessRole and widened back to :root in February, because the metadata-refresh role (ServerlessCoordinatorAccessRole) also needs to assume this role. sts:TagSession has been in the policy since February and was re-verified in August. The same error occurs in our QA project, which uses trg-am-qa-dremio-cross-account-role with the same structure and its own project ID as the external ID.

2. Is this the project store role? Yes. We supplied it as the role ARN under “Your own storage” when creating the project. It has read/write on the root bucket (ListBucket, GetObject, PutObject, DeleteObject, multipart actions).

3. Source and data Apache Iceberg tables on S3, queried through the Dremio Open Catalog. The tables are written by AWS Glue jobs and organised as bronze, silver and gold namespaces. We don’t query raw Parquet.

4. Example query that fails on first run

select * from "bronze_optimize_spend".employees

This is a plain scan of a bronze Iceberg table. The first execution fails, and an identical retry succeeds. We see it on every bronze table, in both of the projects whose IDs I shared earlier. Our gold views have never shown the error. Metadata-only operations (SHOW CREATE TABLE, catalog brows

Retrying is only a workaround. Our automated runners now have to retry with backoff.

5. What we’ve ruled out on the AWS side

  • Trust policies, permissions boundaries, bucket policies, KMS (plain AES256) and SCPs a
  • CloudTrail over about 30 hours showed 750 AssumeRole events. About 475 came from PolarisAwsCredentialsStorageIntegration and succeeded against both roles.
  • None of the events matched the failing session pattern (eks-serverless-dremio-exe-*), and none carried a Tags parameter. The failing call leaves no trace in our CloudTrail.

This points to the failure happening inside Dremio before the call reaches AWS, which fits the bug @cindy.la confirmed on Sept 22. We’re still seeing the error.

Happy to share job IDs and timestamps of failing runs if that helps.

10

@DnyaneshwarD

Thanks for the details!

Regarding your question, I believe it was a slight misunderstanding as we shipped a fix to a bug that manifested in a very similar way to the one you are experiencing (and that you would have probably experienced if not for the current issue).

Would you mind changing the trust relationship of your trg-am-dev-dremio-cross-account-role IAM Role to the shape suggested in the project creation docs?

Essentially, this would mean removing the sts:ExternalId restriction for the sts:TagSession requests, as Dremio does not include it in such requests. I reproduced the intermittent failures internally with a trust policy shape similar to yours, and they disappear after sts:TagSession clause stops requiring the std:ExternalId.

Example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAssumeRoleWithExternalId",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::894535543691:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "<dev project ID>"
                }
            }
        },
        {
            "Sid": "AllowTagSessionFromCallerRole",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::894535543691:root"
            },
            "Action": "sts:TagSession"
        }
    ]
}

This separation does not by itself introduce a security concern. The project-specific sts:ExternalId still protects sts:AssumeRole, which is the operation that issues temporary credentials. sts:TagSession only controls whether tags may be attached to that session; it cannot independently be used to assume the role.

Please do let us know if this change fixes the errors you’ve been seeing.

Best,
Francisco