Connect S3 as a data source

Dremio Cloud
1

Hi,
How can we connect S3 as a data source in dremio cloud and access all the files that are present over bucket. And what are the prerequistics authentication we need to connect S3?

2

Hi @Kom_al

Please refer to this documentation for step by step guidelines on how to connect Amazon S3 as a datasource in Dremio Cloud - Dremio

Let me know if you face any issues. Would be happy to help.

Thanks,
Payal
Dremio

3

Hi @Payal_Gupta , Thanks for the help.
One more thing ,basically I have my AWS free tier account so I have limited resources like 2000 request for s3 storage etc. So, When I was connecting to S3 with dremio, I hardly hit 10-15 request or run the query on S3 storage files, but my request limit exceeded. How?

4

Hi @Kom_al ,

Dremio Cloud uses S3 bucket as a project store also with read/write permissions (other than the data source) which contains all project-specific data, such as metadata, reflections, and file uploads to the homespace [1].

Hence, to calculate the total requests made to S3 from Dremio Cloud, you would have to check the S3 data source + project store S3 bucket.

Further, since this request limit is for the complete S3 storage account, I would suggest you to check the AWS usage report for Amazon S3 in order to understand and check what action/resource caused the limit to exceed [2].

Ref [1] : Dremio
Ref [2] : AWS usage report for Amazon S3 - Amazon Simple Storage Service

Hope it helps.

Best,
Payal