# Dremio Cloud unable to start engine instances **URL:** \ **Category:** Dremio Cloud\ **Created:** [February 16, 2023, 8:34pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302 "2023-02-16T20:34:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 16, 2023, 8:34pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/1 "2023-02-16T20:34:12Z")
When attempting to run EC2 instances for Engines in Dremio Cloud, the instance starts up and then quickly shuts itself down. The end of the EC2 logs show the following: 2023/02/16 20:02:06Z: Amazon SSM Agent v3.1.1732.0 is running 2023/02/16 20:02:06Z: OsProductName: Amazon Linux 2023/02/16 20:02:06Z: OsVersion: 2 [124.753674] nessie.sh[4747]: /opt/dremio/bin/vendor\_specific.sh: line 25: local: `=': not a valid identifier [124.773346] nessie.sh[4747]: Nessie IP (gw.dremio.cloud) is not a valid IP This exact behavior occurs both when using the CloudFormation “path” in which the CloudFormation stack creates all required resources, as well as when using the “manual path” in which the required resources are created manually and then the ARNs, etc. are configured in the wizard. This appears to be a bug in Dremio as there is no user-config that would affect the vendor\_specific.sh script or the Dremio Cloud IP. ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 16, 2023, 8:35pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/2 "2023-02-16T20:35:46Z")
Hi @chough ! Welcome to the Dremio Community. I can look into this further for you. Could you share your organization ID to me via private message? Cindy ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 16, 2023, 8:45pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/3 "2023-02-16T20:45:40Z")
Hi Cindy, sure I can do that. I don’t see a way to private message you however. Could you send me a message first? ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 16, 2023, 9:04pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/4 "2023-02-16T21:04:31Z")
Hi @chough , I have sent you a message Cindy ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 16, 2023, 9:10pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/5 "2023-02-16T21:10:52Z")
I see that your engines are failing to scale up with the error message “Scaling failed. Replica creation timeout exceeded” in the engine events page (Project Settings → Engines → Engine Events). This typically means that your engines are not able to reach out. [https://docs.dremio.com/cloud/appendix/troubleshooting-cloud-resources/#ec2-instances-are-not-launching](https://docs.dremio.com/cloud/appendix/troubleshooting-cloud-resources/#ec2-instances-are-not-launching) I recommend checking over these resources here, depending on how your VPC network is configured: - DNS hostnames and resolutions are enabled on your VPC - Auto-assign public IPv4 address are enabled on your public subnets - Route table routed to an internet gateway for public subnets or a NAT gateway for private subnets ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 16, 2023, 9:17pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/6 "2023-02-16T21:17:19Z")
Thank you! I will explore that further and report back. ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 16, 2023, 10:19pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/7 "2023-02-16T22:19:47Z")
@cindy.la I’ve gone through the steps in the document you linked and the issue is not resolved. It appears that the /opt/dremio/bin/vendor\_specific.sh script is referencing the dremio forwarder endpoint “gw.dremio.cloud” however, the mentioned script is expecting an IP address rather than a DNS hostname. Is it possible that the user data / script / hostname were updated recently and resulted in this bug? I see that the AMI that is being used was created on 02/07/2023 which seems like a recent enough change that this may not have been detected yet. ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 16, 2023, 10:34pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/8 "2023-02-16T22:34:08Z")
Hi @chough , I was able to successfully launch a new organization with CFT and spin up engines on my end. Are you able to spin up a EC2 instance (not the one Dremio Cloud is launching) in the same VPC network you are trying to use for Dremio Cloud and run this curl command? This will check for outbound connectivity from your subnets. ```auto curl -v https://gw.dremio.cloud ``` We also have a prerequisite page here that you can look over if you haven’t yet. A few things to note: you cannot mix public and private subnets, subnets cannot be in the same availability zone, etc. [https://docs.dremio.com/cloud/getting-started/prerequisites/](https://docs.dremio.com/cloud/getting-started/prerequisites/) ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 16, 2023, 10:40pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/9 "2023-02-16T22:40:54Z")
Thanks for the additional info. Yes, I am able to hit [https://gw.dremio.cloud](https://gw.dremio.cloud) from an EC2 instance inside of the subnet without any issues. I have also confirmed that the only subnets I have specified are private (none mixed with public) and that the two subnets are in different availability zones. ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 16, 2023, 10:56pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/10 "2023-02-16T22:56:41Z")
Thanks for checking! I’d like to look into this further on my end. I’ll get back to you as soon as possible ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 16, 2023, 11:01pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/11 "2023-02-16T23:01:05Z")
@chough , I just wanted to confirm, you created this current project with the CloudFormation template from the Dremio Cloud UI? And did you have a NAT gateway set up (with a routing rule to the (public) subnet where the NAT public IP is pointing it to the Internet Gateway)? ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 17, 2023, 5:54pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/12 "2023-02-17T17:54:42Z")
Yes, the project in that organization was created with the CloudFormation template from the Dremio Cloud UI. I had originally setup a different organization and created the resources manually (instead of using the CloudFormation Template), but after hitting this same issue, I decided to try a clean slate and use the CloudFormation template to see where I went wrong, but then ended up hitting the same issue anyway. And yes, there is a NAT gateway setup which allows the private subnet to communicate to the public internet. I have confirmed that I can reach gw.dremio.cloud from within the private subnet where the Dremio engine nodes are created. ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 17, 2023, 6:46pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/13 "2023-02-17T18:46:48Z")
Hi @chough , Thanks for being patient as we looked into this and bringing this to our attention! It is indeed causing an issue for AWS PrivateLink connections. We are currently working on a fix and I can update you once that is live! Cindy ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 17, 2023, 7:03pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/14 "2023-02-17T19:03:53Z")
Hi Cindy, Thanks for confirming! Is there any workaround that we might be able to use in the meantime while waiting for a fix? ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 17, 2023, 7:56pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/15 "2023-02-17T19:56:50Z")
Hi @chough , If you are OK using public subnets for now, you can use that in the meantime as a workaround. ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 17, 2023, 9:05pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/16 "2023-02-17T21:05:56Z")
I’ll have to keep it in a private subnet for now, but I may attempt configuring it without using PrivateLink. Do you have any documentation on a configuration without PrivateLink? I saw it was noted in the setup wizard as optional, but didn’t see what the alternative configuration would be. ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 21, 2023, 5:14pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/17 "2023-02-21T17:14:55Z")
@cindy.la It looks like my attempt to run it in a private subnet, and omit the PrivateLink configuration was unsuccessful. With that, are you able to provide a targeted timeline for when this PrivateLink bug might be resolved? ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 22, 2023, 8:23pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/18 "2023-02-22T20:23:38Z")
Hi @chough , Apologies for the late response. We rolled out the PrivateLink fix over the weekend and I was able to launch Dremio Cloud in a private subnet on my end. I was viewing your organization and noticed that it is in a bad state, did you modify any IAM permissions or roles recently? Cindy ---
**Author:** ![chough](https://avatars.discourse-cdn.com/v4/letter/c/43a26b/32.png) [@chough](https://community.dremio.com/u/chough)\ **Post date:** [February 22, 2023, 8:49pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/19 "2023-02-22T20:49:03Z")
Hi @cindy.la Thanks for providing the update! I did mess with that organization as I was re-deploying and testing different configs in different orgs, so I will re-deploy and see if it is working as expected and report back. ---
**Author:** ![cindy.la](https://sea2.discourse-cdn.com/flex020/user_avatar/community.dremio.com/cindy.la/32/6730_2.png) [@cindy.la](https://community.dremio.com/u/cindy.la)\ **Post date:** [February 22, 2023, 9:57pm UTC](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302/20 "2023-02-22T21:57:35Z")
Sounds good, when you spin up the engine and see the instance in your AWS Dashboard, could you also double check the Instances’ system logs for a section called **“Dremio Node Diagnostics”** and send it over? That should appear in your system logs (toward the end) and checks for S3 connectivity and Dremio Gateway connectivity. [Next page](https://community.dremio.com/t/dremio-cloud-unable-to-start-engine-instances/10302.md?page=2)