Multiple datasources to S3 share credentials

1

It seems that if you create multiple datasources to S3 dremio then after creation dremio uses only the credentials of the first datasource created (or perhaps the credentials stored in core-site.xml)

  • Update: I have verified I can’t access a particular object through S3 for the credentials I used for the datasource. But I am able to access the object through Dremio.
2

@swarren I’d expect credentials provided (if provided) for individual source configurations to trump the one coming from core-site.This is for scenarios where you use one set of credentials for writing reflections to S3 (read /write) and different sets for accessing various sets of S3 sources/buckets (read only).

We’re investigating internally to see if there is an issue here, thanks for pointing this out. Can you confirm the order of the configuration steps/changes you performed?

3

I used 2 sets of AWS credentials to do the following:

  1. Updated core-site.xml to store reflections, scratch/upload/download to BUCKET:/dremio (those credentials have BUCKET/* permission.
  2. Created an S3 datasource using the same credentials to the same bucket. As expected I can use that datasource to see/read/upload all objects in the bucket.
  3. Created an S3 datasource using different credentials to the same bucket. These credentials had permission to delete/read/upload to BUCKET:/group

The unexpected behavior is that navigating the bucket using datasource created in step 3 selected from the sources menu I was able to see and read all objects.

4

Thanks for the details.

5

Is there update to this issue? I’m using s3 bucket for reflections, but can’t use any other buckets on the same account because of this credentials bug. This is on Dremio 3.0.