Making Scout feel young again: our 4-part tonic

BY Doug Breaker

September 28, 2011

Scout is no longer a puppy: in dog years, he’s old enough to drink, get drafted, and rent a car. During that time, cruft gathered around the edges of our server infrastructure.

We’ve been using a hodgepodge of server hardware, some performing multiple roles, some not, all individually configured and tuned. Small changes to our stack seemed to involve a lengthy checklist. Our staging environment didn’t mirror production: what happened on staging didn’t always happen on production. Finally, database changes were painful.

We wanted to get lean in the right places: could we make the young adult Scout as easy to manipulate as the baby Scout? 7.weeks.ago we followed a four-part process to get there.

1. Homogeneous Servers

In our old setup, one app server had 8 CPUs and 2 GB of memory. Another had 4 CPUs and 4 GB of memory. Another had 2 CPUs and 2 GB of memory. The varying hardware profiles had a number of disadvantages:

Now, each of our servers perform a single role and each server that performs that role has the same hardware profile. We only need to remember one configuration profile. Testing performance enhancements is easier: we can try something on one server and be reasonably confident the impact will be the same on others with the same role.

2. Configuration Management with Moonshine

Like an old married couple debating the merits of some unfamiliar but oft-lauded item, Andre and I were reluctant to use a configuration management tool to manage our infrastructure. Neither of us was excited about performing the plumbing work involved in documenting our setup in code.

Our host, Rails Machine, said they’d do the configuration for us using Moonshine, their Puppet-based solution for managing server infrastructure. The process went something like this:

We were happy paying Rails Machine to Moonshine Scout: it’s their area of expertise. It’s their passion: they wrote Moonshine. Additionally, it’s much easier for Rails Machine to support us now as well: their support staff can look at Scout’s Moonshine configuration as the single blueprint of our infrastructure. If they need to make adjustments, they can do that by submitting a pull request.

3. A proper staging environment

Our previous staging environment did not resemble our production environment: the entire stack was on one server. Now, it also has a homogeneous setup that mirrors production but with lower-powered servers. This mirrored, lower-powered setup has a number of advantages:

4. Multi-Master Replication Manager for MySQL

When we were young, we could run ALTER TABLE … on a whim: not anymore. Large schema changes are slow and can have a dramatic impact on Scout’s performance while they run. To give us more flexibility to tune MySQL, Rails Machine added Multi-Master Replication Manager (MMM) to our MySQL stack. This dramatically improved our ability to tune MySQL in two key ways:

Summary

Working with an app where (1) the stack and configuration is easy to understand (2) is easy to change (3) is easy to test (4) and tuning isn’t a “Hold on to your butts” moment is a lot of fun. It’s not limited to web apps just out of the womb.