Production Server Sysadmin Essentials

Developers! You can churn out a Rails or Sinatra app in no time. What about putting it out there in production? Occasionally forget the syntax for crontab or logrotate? Yeah, me too.

That's why I wrote up a few essential notes for a serviceable production environment.

This article covers Centos/Red Hat and Ubuntu, which is what I always end up on. My approach is to get some minimal configurations working quickly so I can see some results. From there, I can go back and refine the configurations.

The five things you need:

1. Schedule tasks
2. Ensure services survive a reboot
3. Restart processes if they die
4. Backup your database off-site
5. Rotate your logs

Plus, a bonus tip: easiest possible alias editing.

1. Schedule tasks: cron

Often, the first thing a non-trivial web application needs is the ability to schedule overnight tasks, such as emails to users.

Red Hat and Ubuntu: crontab -e

Examples:

# running every minute:
* * * * * /usr/bin/scout
# running every 30 minutes:
*/30 * * * * /usr/bin/scout
# running every day at 4:10am:
10 4 * * * /usr/bin/scout

Things to remember:

• User vs. system cron. If you use vi /etc/cron instead of crontab -e, you'll get the system cron instead of the user cron. Some people don't like to edit the system cron. If you do decide to use system cron, you'll need one extra field, the user under whom the task will be run: 10 4 * * * deploy /usr/bin/scout (with deploy being the user)
• Something works in console but not in cron. It's probably a path problem -- remember, cron runs with a limited environment. The fastest way to debug to direct output and error streams to files, like this: * * * * * /usr/bin/scout >~/scout.out 2>~/scout.err

2. Ensure services survive a reboot: run levels

Have a nagging feeling everything might not come back up if your machine were rebooted? Check your run levels. This example shows enabling/disabling the monit service. You'll need monit for the next section, so it makes a good example.

Centos

turn on:  /sbin/chkconfig monit on
turn off: /sbin/chkconfig monit off
list which are on: /sbin/chkconfig --list

Things to remember: /sbin/ typically isn't in your path ... I tend to forget that chkconfig exists for this reason.

Ubuntu

turn on: update-rc.d -f monit defaults
turn off: update-rc.d -f monit remove

Unlike chkconfig, update-rc.d can't provide a simple list of which services are configured to restart automatically on startup. Instead, I use the following, which shows the runlevels, if any, invoke the given service.:

find /etc/rc* | grep monit

More information: A basic introduction to run levels is here. Google linux run levels and chkconfig for lots more information.

3. Restart processes if they die: monit

Centos

# install
yum install monit

# configure to run
vi /etc/monit.conf
  # set the value of startup to be 1:
  startup=1

vi /etc/monit/monitrc. 
# uncomment the line:   
  include /etc/monit.d/*

Now put your individual rules in /etc/monit.d/, for example, here's /etc/monit.d/scout_task_server.conf, which exemplifies restarting a process (in this case a task server for Scout):

check process scout-task-server with pidfile /var/run/task_server.rb.pid
  group task-server
  start program = "/usr/bin/ruby /var/www/apps/scout/current/script/task_server_control.rb start -- --environment=task_server" as uid deploy
  stop program  = "/usr/bin/ruby /var/www/apps/scout/current/script/task_server_control.rb stop -- --environment=task_server"

Note that Monit requires a PID to be written to the location you've specified. This doesn't happen magically. See the Monit FAQ if you need a wrapper script to create the PID for you.

Finally, you need to start Monit itself: /etc/init.id/monit start. You'll want to configure Monit to survive a reboot via run levels (#2 above).

Ubuntu

All is the same as Centos above, EXCEPT the config file is /etc/default/monit. And of course, use apt-get install monit to install, instead of yum install monit.

4. Backup your DB: automysqlbackup + s3synch.rb

Your DB usually contains the lifeblood of your business. If you're not backing it up nightly and storing it offsite, you're leaving yourself vulnerable. Fortunately, it's easy to backup your data. The automysqlbackup + s3sync combo works great for small-to-medium databases. Obviously, this is not a substitute for database replication and failover, but this will protect you from catestrophic failure with daily off-site DB backups. These steps are the same for both Centos and Ubuntu.

Get Automysqlbackup from http://sourceforge.net/projects/automysqlbackup/. It's a cinch to deploy:

• put in your home directory, chmod to executable for the user running it
• edit the script; add your DB username and password, and the name of the DBs you want to back up. You can back up multiple databases.
• add a crontab to run nightly. Example running nightly at 1:30am: 30 01 * * * /home/deploy/automysqlbackup.sh.2.5

Set up s3sync:

• http://www.linode.com/wiki/index.php/Backups_with_s3sync

How much does it cost to store your DB backups in s3? Really cheap. I have a couple databases that take up around 150MB when dumped (25MB gzipped). I've been syncing the daily, weekly, and monthly rotating backups to S3 nightly for a year or or so, and my monthly s3 charge is currently $.40 a month. Forty cents a month for peace-of-mind-inducing offsite backups is hard to beat.

5. Rotate your logs: logrotate

Log rotation is one of those things that's easy to ignore because you don't need it immediately. Don't procrastinate. It's easy to set up, and you'll sleep easy knowing that you're not going to come across some 2GB monster log a month from now.

Centos

• yum install logrotate if it's not installed. Note that the binary doesn't seem to be in the path so which logrotate might not show anything even if it is installed.
• logrotate is run via cron. When you 'yum install logrotate', you will find a new daily task in /etc/cron.daily/logrotate. This is what fires up logrotate once a daily.
• create as many config files in /etc/logrotate.d/ as you need -- each one specifies a log or group of logs to rotate.
• FYI, on both Centos and Ubuntu, the script in /etc/cron.daily/logrotate runs with /etc/logrotate.conf, which in turn picks up everything in the /etc/logrotate.d/ dir -- that's why you can just drop a file in there.

Logrotate's settings are very understandable. Here's an example to get you started, rotating all the logs in a Rails application daily:

/var/www/apps/MY_AWESOME_APP/shared/log/*log {
   daily
   rotate 7
   missingok
   compress
   delaycompress
   copytruncate
}

Notice the copytruncate -- this configuration works well for Apache and Rails logs because it leaves the current file alone, copying the contents out and truncating the current file. This is important for processes that continue to write to the same filehandle.

Ubuntu

• apt-get install logrorate
• double-check that /etc/cron.daily/logrotate exists
• drop the same snippet above into a file in /etc/logrotate.d/MY_AWESOME_APP

Just knowing you’ve got a lock on these things will make it more likely you’ll try that cool project you’ve been thinking about!

Bonus tip: easiest possible .bashrc editing

I love setting up aliases. So why not set up an alias for setting up aliases? My technique is one line in my .bashrc:

alias brc='vi ~/.bashrc;. ~/.bashrc'

All this does is fire up vi to edit your .bashrc, and then re-sources .bashrc after you're done editing. Although only saves a few keystrokes, it has saved me many, many hours over time.