December 18, 2009

Backup folders to Amazon S3 automatically

After reading of Jeff Atwood’s data loss issues this past week I started thinking about the web servers I look after.  The servers are all backed up by the hosting provider – SiteHost – and I have a lot of faith in those backups (I work there, but I have also needed to restore some data in the past and it was flawless), but that didnt stop me from thinking it would be a good idea to start doing backups that I control.
I considered getting another VPS hosted in a separate data-centre and copying data over at frequent intervals (and I might do that at some stage so the traffic can be kept within NZ), but I already had an Amazon S3 account and use it for automated backups of the SVN repositories hosted with Storehouse.  So I wrote a little cli app (written in PHP) that tarballs, gzips, and copies the compressed folders up to S3.  The config file allows you to specify as many folders as you want and say whether you want them to be backed up daily, weekly, or monthly (these intervals can easily be changed and added to).

For detailed usage, keep reading.

Download the files to your server (should run on anything that has a recent version of PHP).

Create a config.php file in the ./library folder.

Any option that you see in the ./library/defaults.php file can be extended in the config.php file.  To get started, have a look at the following: (exchange the => for a greater-than sign => – whats the deal WP-Syntax?)

return array(
   'server_name' => 'srv1.example.com',
   'folders'     => array(
      array(
         'name'      => 'all-websites',
         'dir'       => '/home/www',
         'frequency' => 'daily',
      ),
      array(
         'name'      => 'mysql-backups',
         'dir'       => '/home/backup',
         'frequency' => 'daily',
      ),
      array(
         'name'      => 'ssl-certs',
         'dir'       => '/root/ssl',
         'frequency' => 'weekly',
      ),
   ),
   'amazon'      => array(
      'aws_key'      => 'Your-AWS-Key',
      'aws_secret'  => 'Your-Secret-Key',
      's3_bucket'   => 'Bucket-Name',
   ),
);

Then, you can simply setup a cron job to run the ./s3_backup/Run.php file each night when the server is less likely to be busy.

The first iteration of the app choked on large files because it was loading everything into memory, but this version streams the compressed data to S3 and in my tests it has never used more than 2MB memory (even with files of hundreds of MB).
Right now, the app size is pretty hefty (6.3MB compressed) because it includes the whole Zend Framework library.

Leave a Reply