Context: Unable to create reflections since I haven’t set up the distributed storage. In my case, I want S3 to be configured as my distributed storage.
I already provided AWS Access Key and AWS Access Secret. Prior to that, I also created the bucket, role, and policy in the side of AWS. However, when I try to connect it to Dremio as an Object Storage/Source, the following error shows:
“Could not connect to S3 source. Check your S3 data source settings and credentials.”
Is this an issue with Dremio or AWS?
Another question is, would it work even when I’m just using the Community Edition of Dremio? Most of the guides/documentations want me to edit the dremio config file which does not exist in the first place. Is there a workaround here?
Yes, distributed storage and reflections work in Community Edition. Could you let us know what you’ve put in core-site.xml and dremio.conf? Feel free to create them if they doesn’t exist.
Quick update: I already managed to connect S3 as my source after creating a new IAM user with console permissions.
However, when I tried to edit dremio.conf (which was located in opt/dremio/conf, not in etc/dremio directory), dremio failed to start.
I added this line below in the config file:
dist: “dremioS3:///bucket-name/folder-name”
I would also want to know the difference between creating the actual core-site.xml (with properties such as access key and secret key, etc.) in contrast with just adding each property in the Connection Properties under the Advanced Options in the S3 source settings.