Automate away AWS Auto Scaling pitfalls

By Kevin Teague
Published on Mar 06, 2020

Waterbear Cloud has added a new application to our Paco Products suite of AWS automation applications called EC2 Image Automator. This application allows you to automate the process of building new Amazon Machine Images (AMIs) and deploying them into an AutoScalingGroup.

Common pitfalls of EC2 Auto Scaling

AWS introduced EC2 Auto Scaling over a decade ago. It was the first fire lit to prove the potential of the cloud — being able to automatically add servers on-demand to respond to traffic spikes and then spin them down when not used was a game changer. There was now an alternative to the “old way” of purchasing physical servers and trying to make a cost trade-off between having your application fall over during busy times or filling up a server rack with expensive servers that would sit unused 99% of the time.

The basic way to take advantage of EC2 Auto Scaling is to provision an AutoScalingGroup (ASG) with Launch Configuration for every application you want to run. Then whenever the AutoScalingGroup needs to scale up to handle load, it uses the Launch Configuration to run commands to install the application and its dependencies.

While that is the simplest method to get up and running with AutoScalingGroups, it has some pitfalls:

  • New EC2 Instances may fail to install the application’s dependencies correctly. If the installation commands rely on downloading application dependency packages over the internet, an outage from those services can break your ability to launch new instances. Also thorny is the issue of installing the latest versions of dependency packages — which may not have been tested against your application and leave you with a broken application.
  • The installation process can take a long time. Downloading dependencies, installation packages, compiling your application — all this takes time. Sometimes only a few minutes, but we’ve worked on applications where this process took over twenty minutes. If you have a traffic spike and the AutoScalingGroup responds by automatically adding a new instance to its pool, your customers will have a slow and unresponsive experience until the new instance becomes available.
  • You may not have the latest security updates applied. In addition to having your application installed, you want the base operating system to have the latest security updates applied. You could apply these security updates when a new EC2 Instance is launched, but again this will take time and doesn’t give you a chance to test your application against these new updates.

The solution to avoiding all those pitfalls is to create a base Amazon Machine Image (AMI) with all your application’s dependencies and security updates applied, then use that AMI to launch your AutoScalingGroup. Only minimal configuration needs be applied at launch time — such as deploying an application artifact and applying secrets such as database credentials. This step is nicknamed “last mile” configuration.

That process can be done by manually launching a new EC2 Instance in an “AMI Build” sandbox and performing installation, security updates, testing there — and then finally generating a new AMI Image and applying it to your applications AutoScalingGroup.

At Waterbear Cloud, it’s our mission to deliver automation. To solve that process with automation we’ve built a new addition to our family of Paco Products called EC2 Image Automator.

Introducing EC2 Image Automator

Our EC2 Image Automator, which automates robust management of AutoScalingGroups, is available to all our customers on the Waterbear Cloud Platform. How does it work?

We have built an application using AWS Lambda to automate the process of:

  1. Launching a new EC2 Instance in a “sandbox” AutoScalingGroup.
  2. Security updates and packages updates are applied to the new instance in this sandbox.
  3. A new Amazon Machine Image (AMI) is created.
  4. The production application AutoScalingGroup is updated with the new AMI. Typically ASG Rolling Updates are configured so that this can happen without any downtime or service interruption.
  5. The oldest unused AMI’s are automatically removed to avoid running up your AWS bill needlessly.

The EC2 Image Automator is designed to be configurable to support different workflows. For example, it supports date-based scheduling to allow security updates to be automatically applied on the same day every month. Or it can be triggered using Paco’s DeploymentPipeline from a commit to CodeCommit or GitHub. Custom DevOps hours can be used to customize any step of this automation process — for example, to add a hook to run your application’s tests on the new Image before it’s applied to production.

Waterbear Cloud’s growth-ready automation allows you to reduce the time and cost of automating your AWS environment. Our expertise and focus on automation removes error-prone manual steps from the equation, saving you money and headaches.