How to configure Amazon Glue Catalog Data Source to read from separate AWS Account

1

I have Dremio installed on ec2 in one AWS Account. I would like to connect to a Glue Catalog Data Source in another AWS Account. I’ve gone through the configuration of allowing access to that glue catalog. I don’t see a way to configure the Account ID for the Glue Catalog in this data source connector. Is this possible or is there another suggested approach?

2

Welcome to Dremio Community, @jcliche!

This needs to be done outside of Dremio via AWS IAM policies/roles (i.e. for cross account access). The high level steps are:

  1. Say there are two AWS accounts → acct_abc (where Dremio is hosted) and acct_xyz (where Glue/S3 is)
  2. In acct_abc, get the ARN for the IAM role that is attached to the Dremio EC2 nodes. Let’s say it is arn:aws:iam::acct_abc:role/DremioNodeRole
  3. Head over to acct_xyz and create IAM policies that gives access to Glue and S3. Refer sample policies from Dremio Docs. One policy for Glue, and another policy for S3
  4. In acct_xyz, create an IAM role and attach the two IAM policies created above. Make note of the ARN for this IAM role. Let’s say it is, arn:aws:iam::acct_xyz:role/DremioGlueS3Role
  5. In this role, add a Trust Relationship, which contains the following. Modify to the ARN you obtained in step 2.
			{
			"Version": "2012-10-17",
			"Statement": [
			{
			"Effect": "Allow",
			"Principal": {
			"Service": "ec2.amazonaws.com",
			"AWS": "arn:aws:iam::acct_abc:role/DremioNodeRole"
			},
			"Action": "sts:AssumeRole"
			}
			]
			}
  1. Switch to acct_abc, add the following to the Trust Relationship of IAM Role arn:aws:iam::acct_abc:role/DremioNodeRole. Modify to the ARN you obtained in Step 4.
			{
		    "Version": "2012-10-17",
		    "Statement": {
		        "Effect": "Allow",
		        "Action": "sts:AssumeRole",
		        "Resource": "arn:aws:iam::acct_xyz:role/DremioGlueS3Role"
		    }
			}
  1. In Dremio, add a Glue Source with “EC2 Metadata”, and use arn:aws:iam::acct_xyz:role/DremioGlueS3Role in the “IAM Role to Assume” section.

You should have now Glue access from the cross account.

1 Like