Easy backups with rsnapshot

I recently posted a general overview of my backup system, but didn’t go into much detail about the most important component. I said,

My backup system is based mostly around rsnapshot, which provides multiple “snapshots” of your system at specified intervals. Using rsync and hard links, it approximates the .snapshot backups you get on NetApp file servers. A “snapshot” is a copy of your data as it existed at the time the snapshot was taken. With these snapshots available, deleted files are quickly recovered, and modified files can be easily rolled back to a previous state.

Using rsnapshot, I have six snapshots taken per day, at four hour intervals. Every night the most recent one of those is saved as the daily snapshot. Daily snapshots are kept for seven days, at which time the most recent is copied to a weekly. A similar process takes monthly snapshots from the weeklies, and those monthlies are kept for six months. I’ve only got three weeklies right now because I rearranged things a bit when I installed the new drive and started the process over. I’m keeping snapshots from before this point somewhere else until I get a month or two saved here.

kchrist@cairo:~>$ ls /Volumes/Backup/cairo/
daily.0     daily.3     daily.6     hourly.2    hourly.5    weekly.2
daily.1     daily.4     hourly.0    hourly.3    weekly.0    
daily.2     daily.5     hourly.1    hourly.4    weekly.1

All of this is completely automated via cron and, through the use of links rather than actual copies, the disk space required is only the amount needed to store one full copy of the data plus any incremental changes: each snapshot does not actually use the same amount of space as the original data. A single copy of our backed-up data is around 12 GB in size, yet our individual snapshots are mostly between 10 and 50 MB each, depending on how much has changed in the time elapsed since the last snapshot was taken. The snapshots above only take up around 13 GB combined. These are taken and stored on a single machine, but you can also have one machine take snapshots of any number of remote machines over SSH.

I have rsnapshot configured to backup pretty much everything outside of the OS X system and application directories, so restoring to the current state is easy once I reinstall the OS and applications. I thought about backing up the application directory too, but because some apps require more than a drag-and-drop installation, restoring everything this way may not be possible. Application installation is easy enough that it’s not really a big deal, as long as I still have my original installers and disk images, which I do; they’re archived in one of the directories that is backed up. rsnapshot also provides a way to run a script before making backups, in order to dump databases and the like. I use this functionality to make MySQL and Subversion backups, which are then kept with the rest of the files. The contents of each snapshot directory look like this:

kchrist@cairo:~>$ ls /Volumes/Backup/cairo/daily.0
Library    mysql      private    sw
Users      opt        subversion usr

In addition to using it at home, I’m in the process if implementing it for remote backups of all our servers at work, to replace the home-rolled rsync jobs we’re running now.

Installation

Due to its use of filesystem links, rsnapshot can only be used on Unix or Unix-like systems (eg, Linux, Mac OS X). It’s included in the standard Debian distribution and can be installed on OS X via Darwinports. The application itself consists simply of a Perl script and a configuration file, so it’s trivial to drop in if you don’t want to use a package manager. The config file is well commented and the on-line documentation is clearly written with lots of examples. It’s only dependencies are Perl and rsync, and SSH if you want to make backups remotely, all of which are included in OS X. Linux users may have to install rsync first if their distro doesn’t install it by default.

Configuration

Once rsnapshot is installed, you’ll need to configure it for your system, and set cronjobs to run it. Configuration is simple, the defaults should work for most people. Most people will only need to change a few things:
  • snapshot_root – Set this to the directory you want your snapshots stored in. I’ve set mine to a subdirectory of my dedicated backup drive.
  • interval – There are four interval lines: hourly, daily, weekly, and monthly, each followed by a number specifying how many of each type of snapshot you want to save. The defaults are probably fine for most people. Uncomment the monthly line if you want to keep monthlies as well (I did).
  • exclude – Specify any file/directory name patterns you want excluded from the backups. Wildcards are allowed. For example, you might use this to exclude temp files by using the pattern *.tmp.
  • backup – This is the important part. Each backup line should be a directory you want backed up, followed by the where you would like to put it.
  • backup_script – A script that can be run in order to back up the output along with everything else. I’m using this to call two short shell scripts that dump my MySQL databases and hotcopy my Subversion repositories.

Examples usage for all of the above can found in the comments in the config file or in the documentation.

My Subversion and MySQL backup scripts are also available:

Once this is done. all that’s left is to add them to your crontab. I’ve added the following jobs to /etc/crontab:

0 */4  *  *  *  root    /opt/local/bin/rsnapshot hourly
50 23  *  *  *  root    /opt/local/bin/rsnapshot daily
40 23  *  *  0  root    /opt/local/bin/rsnapshot weekly
30 23  1  *  *  root    /opt/local/bin/rsnapshot monthly

This runs the hourly backups every four hours, beginning at midnight, the nightly backups every night at 11:50, the weeklies every Sunday at 11:40pm, and the monthlies on the first of the month, at 11:30. The times are staggered so that I can be certain that the previous job was finished before the next one runs. One of these days I’ll get around to writing launchd jobs for this instead of using the now-deprecated (in OS X) crontab.

That’s it! Enjoy the peace of mind you get from knowing you can easily recover any thing you lose, whether from hardware failure, data corruption, or accidental deletion.

Comments

John says:

Have you tried backing up a remote database with rsnapshot? I’ve got everything else working, except the database backups. Their example mysql script just looks like it would be dumping a local database, not the remote one. Any ideas?
thanks,
John

Kenn Christ says:

Grab a copy of my MySQL script from the above link (when you can; it looks like the site I posted the scripts on is down at the moment) and add -h [hostname] to the MySQL commands. By default mysql and mysqldump will connect to localhost unless you specify a remote host using -h.

Another option would be to run the above backup script on the target database server and have it dump the databases into a directory that will then be backed up by the remote rsnapshot process. The first method is preferable, I would think.

Seppel says:

Hi,

can you tell me how to make a backup up of a remote Subversion repository?

Thanx Seppel

Kenn Christ says:

Seppel: What I do is dump the database with a local cron job before my remote backups begin. You can do something like this:

svnadmin dump --incremental --deltas --quiet /path/to/repos > /path/to/backup

Just make sure that the backup directory is one of those backed up by your rsnapshot job.

Seppel says:

Hi Kenn,

i found http://www.le-gall.net/pierrick/blog/index.php/2007/04/17/98-subversion-incremental-backup which looks very interesting to me. I lokated this Script on my SVN-Server and invoke it by ssh. The results (the deltas) are backuped by a “backup” line in my rsnapshot.conf

Seppel

vishu says:

Hi,
i tried your subversion backup script, but when i run the script i get ‘/Repo/soarepo/ls/format’: No such file or directory error

pls help i am new to subverion

toasty says:

vishu,

Make sure that you are using back ticks ` as opposed to single quotes ‘

The back tick is located next to the 1 key on a standard keyboard.

Cheers,

~mc