Dremio unable to find bucket but reads keys of JSON file anyway

1

We currently have a functioning S3 Connection reading JSON files from three buckets, we’ll call:
BucketA
BucketB
BucketC

My test file test.json:
[{
“dremio”: “yep”,
“test” “hurray”
}]

We added permissions to read from BucketD, which has identical data to that which is successfully being read in BucketA-C, yet Dremio sees the bucket, let’s you navigate into it and select and save the file as a JSON file, but as soon as it takes you into the Data/SQL view it says Failure reading JSON file - Unable to find bucket named BucketD, yet if I dismiss the error message, I see displayed in the data view the keys dremio and test, but no data with them.

Dremio Build: 4.0.2-201910020123580864-a98a0b9

2

@kwcrook

Not exactly following this line “We added permissions to read from BucketD, which has identical data to that which is successfully being read in BucketA-C” but in general, we list all buckets for that account and when you try to select if there is no read permissions we fail

Is this your question?

Thanks
@balaji.ramaswamy

3

Let me include the policy to clarify:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:*"
      ],
      "Effect": "Allow",
      "Resource": [
        "BucketA",
        "BucketB",
        "BucketC",
        "BucketD"
      ]
    },
    {
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:ListBucket",
        "s3:HeadBucket"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

We added BucketD to the policy. When testing the permissions via the aws cli, we are able to connect, decrypt, and read data from BucketA-D but Dremio acts as though it cannot find BucketD even though it can read the json keys (but not the values) from the file within the BucketD.

4

@balaji.ramaswamy Does that clarify the question?

5

@kwcrook

Yes got it, can you please send us the job profile

Share Query Profile

Thanks
@balaji.ramaswamy

6

Balaji,

Had a similar issue that I got through by readding the S3 connection. (Unable to find bucket named error when querying S3)
Hope that helps.

Thanks,
Chip