I installed the dremio cloud and followed through on using the provided cloud formation template for the project and the cloud. I tried to connect to an AWS RDS source but unable to connect. I suspect that it may be related to the security groups that we have on the RDS.
Is there any documentation related for configuring RDS security group to allow Dremio to access? e.g. which Dremio SG would I use to refer to as the source SG in the rule.
In order to save money overnight while not working on this, I manually stopped the EC2 in the AWS console. I see this morning that it been removed. Any way to recover/re-created the EC2 or do I need to delete the AWS cloud and AWS stack and create again. I suppose shutting down the engine via the Dremio UI is maybe the preferred way to stop the resources.
The recommended way to save on engine costs is to set the min replicas of your engine to zero in the Dremio Cloud UI and make sure that your source metadata refresh settings are set to an interval that makes sense for you.
@hope thank you for the quick response. I reviewed the post quickly, but will have to review more later to fully understand. I used the option you provided to create the cloudformation, so just want to make sure you are saying I have to add other security group changes that were not covered in the cloudformation yaml you provided.
I figured I would just need to add an inbound SG rule on the RDS to allow communication from the engine, I just wasn’t sure which SG or resource to specify for the source.
Any way to recover the EC2 that is now gone, or do I need to delete stacks/cloud/project and start over again. Not a big deal, it doesn’t take too long.
Do you mean “Max Replicas” ? Though, I tried to set this to 0 and it would not save because it said it must be 1 to 100. I re-created the project and cloud stacks. For some reason this time the EC2 did not start, maybe because I did not try to add a source yet. I would like to understand how to shutdown the ec2 to avoid costs. Is turning the “Enabled” off for an engine an acceptable way?
Yes, you can flip the ‘enabled’ switch to ensure the engine does not spin up.
‘Max replicas’ does need to be 1 or greater. If ‘min replicas’ is set to zero, the engine will scale down automatically to zero when not in use. Keep in mind that there are some background tasks that use the engine, such as metadata refreshes. Because of this, we recommend that you set your metadata refreshes to something that makes sense for you. For example, if you know your data is not changing frequently, you can set the refresh to a large interval to ensure the engine does not spin up often.
Regarding your engine not starting, what is the error on the engine events page? It is likely there is a misconfiguration. You can find troubleshooting steps here: Dremio