Cykod Blog

Recently we were tasked with the following problem: one of the websites we host was going to be featured prominently on a national television program. They are not a huge-traffic website, averaging probably a couple thousand hits a month and are hosted on a shared server. As we also haven't seen what the tv program was going to be and what sort of mention the website was going get, we essentially had no idea how much traffic to expect.

A couple of previous times, under normal circumstances the game plan would be something along the lines of: throw up a bunch of x-large amazon instances and move their site, which contains a number of static and dynamic elements onto the cloud. Unfortunately as this isn't something that we do very often so it's a somewhat painful manual process that takes a fair number of hours to do (and is not something I look forward to as it's a human-error-prone process) and we don't have stock EC2 images up-to-date with all the required software. So, by the time the sites are up and tested and load balanced there is a significant expense to the client - both in the cost of the instances and the consulting time to get them up and functioning properly. This client both didn't have the budget to do that and as we didn't know how much traffic was going to come in, we didn't want to waste their money if the rampaging horde of internet users never arrived.

So we can up with a solution that would be cheap but, for the most part at least, would be able to handle whatever traffic came their way - put the static pages of the site on Amazon S3 and run the website from there, but bring users back to the DB driven back-end for the pages that needed it (logins, user accounts, contributions, volunteering, etc) - all of which we were able to do with a just a couple of hours of time and a couple of quick and dirty scripts. Here's what we did, assuming a linux/apache2 hosting environment:

First the Preliminary steps:

Get yourself an s3 account and download the s3sync.rb scripts and get them configured properly. See the included README - it's pretty simple and mostly involves configuring ~/.s3conf/s3config.yml

Then, move all the file and image assets on the live site to S3 (how you do this depends on your setup - many CMS's have some S3 module or support, on a static site s3sync.rb and sed should do the trick) This isn't actually necessary for the rest of what's below but as the point is scaling to handle increased traffic it'll do it's part to reduce the load on the live server as well by greatly reducing the number of requests that even hit the server.

Now the meat of the process:

1. Download a static copy of your site. There's an export script built into our CMS Webiva but I believe using wget to mirror the site should get you there too:

$ wget -m http://sample.com

Note: Depending on how your site downloads - if the pages aren't named .html you'll have to modify the commands and scripts that follow - there's no reason S3 can't serve pages ending with a .php extension with a text/html mime type.

Note 2: This is more of a tutorial than a set of drop-in-place scripts - don't expect to follow these steps exactly as your own site may be set up differently. 

2. We need to replace any references to the root homepage with index.html - a couple of quick sed calls searching for '/' and "/" did the trick for us:

$ find . -name "*.html" -exec sed -i s/\'\\/\'/\'\\/index.html\'/g {} \;
$ find . -name "*.html" -exec sed -i s/\"\\/\"/\"\\/index.html\"/g {} \;

This is necessary because amazon s3 doesn't support index.html files. We can get around this issue for other pages on the site but not for the homepage.

3. Create an s3 bucket with s3cmd.rb with the full sub-domain that you will be hosting on (in our case example static.sample.com):

$ s3cmd.rb createbucket static.sample.com

4. Drop the static javascripts and stylesheets onto our s3 bucket - this applies for files that are hosted locally that your static site will need:

$ cd public/
$ s3sync.rb -v -p -r javascripts/ static.sample.com:javascripts/
$ s3sync.rb -v -p -r stylesheets/ static.sample.com:stylesheets/

5. Next I wrote a quick script (url_replace.rb) to replace links to specific pages that have forms or dynamic content to point all the way back to the real site. This will allow any relative links to something like the /contribute page to always go back to the live site. The script just searches for certain single - ' - or double - " - quoted relative urls and replaces them with full http://sitename.com/url urls. This worked well enough for us but depending on your site's content you might need to modify. This script needs to be called once for each page with dynamic content with the domain to redirect to and the relative url of the page.

Here's the script:

#!/usr/bin/env ruby
domain = ARGV[0]; url = ARGV[1]
url = url.gsub("/","\\\\\\/")

`find . -name \"*.html\" -exec sed -i s/\\'#{url}\\'/\\'http:\\\\/\\\\/#{domain}#{url}\\'/g {} \\;`
`find . -name \"*.html\" -exec sed -i s/\\\"#{url}\\\"/\\\"http:\\\\/\\\\/#{domain}#{url}\\\"/g {} \\;`

Now here's a sample usage:

$ ./url_replace.rb static.sample.com /email_signup
$ ./url_replace.rb static.sample.com /create_event
$ ./url_replace.rb static.sample.com /contribute
$ ./url_replace.rb static.sample.com /contact

6. Finally I wrote another quick script (s3put.rb) to copy all the exported html files over to S3. Since S3 doesn't support index files (in fact, s3 doesn't really support directories, everything is just a key) it needs to just give them directory-like names but make sure they get sent to the client as text/html files (this is why I couldn't just use s3sync.rb to do the heavy lifting)::

Here's the script:

#!/usr/bin/env ruby
bucket = ARGV[0]; file = ARGV[1..-1].join(' ')

# Files are coming in from a find command, so they are all prefixed with a ./ (i.e. ./dir/filename.html)

# Special case for index.html files - let them work as /directoryname and /directoryname/
puts("Handling: #{file}")
if file =~ /^\.\/(.+)\/index.html$/
  filebase = $1
  `s3cmd.rb put \"#{bucket}:#{filebase}\" \"#{file}\"  x-amz-acl:public-read content-type:text/html`
  `s3cmd.rb put \"#{bucket}:#{filebase}/\" \"#{file}\"  x-amz-acl:public-read content-type:text/html`
end

# copy the file normally
filepath = file[2..-1]
`s3cmd.rb put \"#{bucket}:#{filepath}\" \"#{file}\"  x-amz-acl:public-read content-type:text/html`

And then a find command piped to xargs should do the trick (note the script above is counting on a "find ." command so all files paths coming in have a leading ./ you can adjust as necessary).

$ find . -name "*.html" -print | xargs -L 1 ./s3put.rb static.sample.com

7.  Now do s3sync for any other static files that need to be hosted (in my case, all static files were in the system/ directory)

$ s3sync.rb -v -p -r system/ static.sample.com:system/

8.  Now to take advantage of the fact that any sub-domain prefix to the s3.amazonaws.com domain is treated as a bucket name, we need to add in a quick CNAME to our site's DNS to get a nice looking static domain - in our case we want static.sample.com would serve the static site. Add something like the following to your BIND setup or configure your DNS with whatever you use:

static.sample.com CNAME static.sample.com.s3.amazonaws.com

Once that propogates, http://static.sample.com/index.html will take you to the homepage of the static site. 

You can test your site out on http://static.example.com/index.html and make sure everything is good to go. Any links to dynamic pages should take you back to the live site.

Once it's time to deploy the static site:

Add some rewrite rules to your the apache2 config of the live site to kick it over to static.sample.com on the appropriate pages. For example to kick the homepage and anything under /about, /news /blog, /partners you can use:

RewriteRule /$ http://static.sample.com/index.html [NC,QSA,L]
RewriteRule ^/(about|news|blog|partners)(.*)$ http://static.sample.com/$1$2 [NC,QSA,L]
...Any other rules you need...

And were done - the majority of users will see only the static site unless they to do something active on the site.

Update 9/20:  As someone pointed out, pushing from S3 to Cloudfront will give you much better latency and is a good idea for public-facing hosted files at the cost of being unable to modify your site instantly (if you discover a typo in some html for example). A good compromise would be to turn cloud front on in your bucket, and then do a sed replacementfor any static media files (leaving your .html files on s3). Since the url for the media files is much less important, leaving them on the deafult cloudfront domain name, as ugly as it is, won't really hurt.

Now what happens to as soon as the rush is over and people have linked to the static site - but the site contains old content? Not a problem, because we used the cname feature of s3 we can kick the s3 bucket to the curb and just add in another virtual host on your live server to redirect people to the real site:

<virtualhost>
  ServerName static.sample.com
  RewriteEngine On
  RewriteRule /(.*)$ http://sample.com/$1 [R=permanent,NC,QSA,L]
  ...
</virtualhost> 

Now point that cname back to you live local server and all the static links will redirect to the live site.

Overall, not a huge deal and we saved the client a bunch of money. Now we just have to hope too many people don't try to contribute all at once on the live server.

=====================

Followup - the TV spot was cut to a quick 5 minute spot without a mention of the website and we didn't even need the extra bandwidth of the static hosted site. Oh well, one more for the bag of tricks, I'm sure the experience will come in handy.

I've always thought that two of the toughest industries to work in would be the wedding or the funeral business - not, however for the reason that most people complain about - which is dealing with people under vast amounts of stress. There are lots of industries that deal with people under stress (I'd say IT has it pretty bad as well, ever tried restoring two years of someone's work from a crashed hard drive?)

No - the problem facing people in the wedding and funeral industries is that their day-to-day task is dealing with people on some of the most important days of their life. Even the slightest slip-up on a couple's wedding day will sit as a stain in their memory for the rest of their lives. Similarly, any sort of mistake during a funeral will feel like it tarnishes the memory of a loved one whom everyone attending wished to honor.

This puts a lot of stress on workers in those industries as they need to be on their game 100% of the time, every day, or things can turn in an instant. Consistently going 4 for 5 would make you a god as a baseball player but will get you fired pretty quickly as a wedding planner.

What does this have to do with Web Development? Nothing really, and no, despite what some clients seem to believe I don't think a little downtime is the same as setting the cake cart on fire. What got me thinking about this issue was an interview I saw with Google's CEO on PBS and how any implementation of cloud computing should probably be tempered with a fallback.

If Google* only loses 10 people's data because of a failure of two of their redundant systems at the same time, out of their total of 10 million users that's a pretty good rate. Except for those ten people. They are going to be pissed and there probably won't be much they can do.

Just like your wedding, your data is a good deal more important to you than it is to anyone else. So while an engineer probably won't get fired because he spilled his coffee on the SAN that was storing your info ( "it only took 2 accounts out, we have 20 million here, it's not that big of a deal right?" ) you are going to be in for a headache.

Now the reason this is different than Bill in the next cubicle accidentally watering your PC along with his desk cactus is that as the name sort of implies - you have very little recourse or ability to try to fix any problems hovering around in the cloud. For google, the amount of lost revenue will be minimal and their EULA's and their stable of lawyers are going to protect them against any bigger issues, but for you, good luck getting that data back. If it was really important, your company would probably be willing to spend a few grand to get the data back from a data forensics team, but Google will most likely not be bothered to try to recover the data - it's just going to be gone and they aren't going to give you the hard drive to try to fix.

Now, even if you have the ultimate faith that Google will never screw up, imagine that a hacker guesses a password your admin used to access a service and replaces all your data with Uv_bn_haXored.txt - good luck getting Google to fish out a backup and restore your data. A couple of backup tapes at the CTO's house would have solved that problem.

So what to do, ignore the cloud? Of course not - the overall benefits far outweigh the downsides - just remember that you need to treat your valuable data with the same amount of care whether it's on the cloud or in your server room. Don't assume that it'll be taken care of.

As with anything where the outcome is orders of magnitude more important to you than to those providing a service for you - tread carefully and choose wisely. There's a reason no one goes with "AAA Wedding Planners" even if they are listed first in the phone book and offer the second wedding free.

Update 9/21:  Just saw this story and thought it illustrated the issues with going to a commodity service on the cloud quite well:  Whoops! Students 'Going Google' Get to Read Each Other's Emails - a 3-days response time while people can read each other's emails is lauded as as  "prompt response." Would that sort of thing be acceptable in the business world? 

* And by Google I mean any large company providing commodity level cloud services for cheap ( Amazon, Sun/Oracle, Microsoft, Yahoo, etc)

To badly paraphrase from memory the joke told during the credits of the classic Eddie Murpy comedy Coming to America (back before Eddie's live acting carreer became pretty darn irrelevant):

Customer : Waiter come taste my soup.
Waiter : Is there there a problem with the soup?
Customer : Just taste the soup.
Waiter : Is it too hot ? Is it too cold ?
Customer : Just taste the soup.
Waiter : Is there a fly in the soup ?
Customer : Just taste the soup.
Waiter : Ok, fine I'll taste the soup! Where's the Spoon?
Customer : Aha!

 This joke illulstrates the sort of rut that it's important to avoid as a Developer when dealing with complaints and Bug reports from clients and users. In an ideal world, any web-related user-reported* bug should always include at least the following info:

What url did happen at and what were you doing? When did it happen? What did you expect to have happen? What actually happened? What browser are you using? What account are you using? [ Also helpful: What "Ad Supported Toolbars" do you have installed? How many different P2P Clients have your children installed on your computer? ... ]

But 90% of the time the bug report consists of:

It didn't work. 

Playing the Jerk Programmer card isn't the best way to build repeat business. Sometimes, the right thing to do is just try the soup. Take what little information you are given and do your best to replicate the problem - provided you're not wasting hours of time -  and then gently prod the submitter for the additional information you need.

I've found that more often that I would like to admit, a quick attempt to replicate the problem myself will turn up the bug without too much back and forth. And everyone will be happier for it. Feel free to send over that detailed "Bug Report" form to the client for next time. They won't actually use it, but it will make you feel better.

*As an aside, I say user-reported because if your system is generating an exception you should already have all that information wrapped up in a nice package yourself - In our case using Rails' rescue_action_in_public lets us drop exception data into the Db and send an email with a stack trace directly to an administrator.