Scaling (Down) with AWS
Launching a new webapp is never easy - even one as simple as a URL shortener. Will it catch on? If it does, what does that mean in terms of traffic? 10, 100, 1000 requests per second?
A few weeks ago we did some back of the envelope calculations for how big wi.nr could get in the best case. The calculation went something like: 10,000 active users x 10 shortened URLs per day x 100 people clicking on each of those URLs = 10M shortened URLs clicked a day or ~120 per sec. For a variety of reasons, we use an AJAX request to actually check if the person viewing a link has won a prize. This can be a very intensive process - we need to take our official rules and check them against our database and logs (to prevent spammers and people trying to game the system). So we roughly tripled that estimate to ~400 per second. Add to that serving static content, the more expensive operations of viewing statistics for a link, actually shortening the URL, signing people in, sending emails to prize winners, now we’re at ~500 req/sec. Now, following the rule I use once I’ve come up with by best estimate for a job quote, web traffic or telling my wife how long I’ll be at the bar: double it - 1000 req/sec.
After a little benchmarking on my local machine (Core 2 Duo, 8GB of Memory, 10K RPM drives) and getting ~1500 req/sec - I decided better safe than sorry and launched a new EC2 Large instance running the Ubuntu Karmic Canonical AMI, installed all the necessary python libraries and applications and wi.nr was up and running at ~1200 req/sec - perfect.
The initial plan was to run the web server (Nginx), app server (Tornado) and DB server (MySQL with Percona patches) on the same instance. Easy setup, easy to manage and a clear path to scale up: split the DB off to another instance, serve static files and proxy the app server(s) from another instance and run as many instances as needed for the app server.
After days of reading about RAID arrays across EBS volumes and performance optimizations for running MySQL on EC2, Amazon comes to the rescue and does it all for me - enter Amazon Relational Data Store. I downloaded the RDS CLI tools and a few commands later and I’ve got MySQL up and running and the wi.nr app connecting to a small RDS instance, churning out ~1500 req/sec. Not bad.
We launched wi.nr a couple weeks ago with a good amount of initial adoption and steadily growing traffic. The numbers we came up with in our earlier calculations are still a ways off, but we were seeing some nice traffic spikes coinciding with some popular twitter users mentioning us. Then we got our first statement from AWS, and a few more back-of-the-envelope calculations later and we realized that our grandiose dreams of making millions thousands hundreds dozens of dollars from our URL shortener were going to be hard to achieve. This is not to say that our bill from Amazon was unexpected or large by any means - we’re still a month away from our Amazon bills matching the setup fee at a local COLO I’ve had my eye on.
I looked back at the cloudwatch monitoring for the last few weeks and saw a flatline for CPU utilization at just over 0%. The decision to scale back our EC2 Large instance was a pretty obvious one. Our traffic is growing in a fairly steady manner, and our best case scenario of 1000 req/sec is more than a couple orders of magnitude away.
This morning, I fired up a small instance with the 32 bit version of the Karmic AMI, did a dpkg —get-selections on the large instance, a dpkg —set-selections and apt-get dselect-upgrade on the small instance, fired up a new EBS volume from a snapshot of the existing one, rsync’d some configuration to the new instance and remapped the elastic IP - less than 30 minutes later and we’re saving $185 a month and still able to handle ~400 requests per second.
Before spinning down the Large instance, I created a private AMI of the running instance. If wi.nr goes gangbusters, I can probably get traffic hitting a large instance in less than a couple minutes.
So there’s the story of my first experience with AWS and the birth of wi.nr.
— Tom
