How can we connect S3 as a data source in dremio cloud and access all the files that are present over bucket. And what are the prerequistics authentication we need to connect S3?
Please refer to this documentation for step by step guidelines on how to connect Amazon S3 as a datasource in Dremio Cloud - Dremio
Let me know if you face any issues. Would be happy to help.
Hi @Payal_Gupta , Thanks for the help.
One more thing ,basically I have my AWS free tier account so I have limited resources like 2000 request for s3 storage etc. So, When I was connecting to S3 with dremio, I hardly hit 10-15 request or run the query on S3 storage files, but my request limit exceeded. How?
Hi @Kom_al ,
Dremio Cloud uses S3 bucket as a project store also with read/write permissions (other than the data source) which contains all project-specific data, such as metadata, reflections, and file uploads to the homespace .
Hence, to calculate the total requests made to S3 from Dremio Cloud, you would have to check the S3 data source + project store S3 bucket.
Further, since this request limit is for the complete S3 storage account, I would suggest you to check the AWS usage report for Amazon S3 in order to understand and check what action/resource caused the limit to exceed .
Ref  : Dremio
Ref  : AWS usage report for Amazon S3 - Amazon Simple Storage Service
Hope it helps.