Scheduled Backups with rdiff-backup on Mac OS X

2007-03-11 by Ilari Scheinin <firstname.lastname@helsinki.fi>

Update on 2008-03-26: I changed the behavior of my backup script a while back, but never updated this page since I felt that I did not have the time to upgrade the description. Now I figured that I might as well just post the script as some people can still find it useful. It is no longer making backups to a disk image, but straight to the external drive, and it also has some additional features. If you set up a scheduled wake up in Energy Saver, the script will put the computer back to sleep if these two conditions are met 1) the computer was actually waken up as scheduled (i.e. not left on on purpose to do something), and 2) the computer has been idle for a specified time (i.e. nobody started using it while the backup was running). By the way, I recommend cron for scheduling the script in this scenario, as launchd is a bit buggy (at least was on Tiger). Here is the new script:
/usr/local/bin/newbackup.sh - Download - View in browser

This article describes how I am using rdiff-backup to do nightly backups to a disk image on an external hard drive. Scheduling is done using launchd, and any potential errors are reported via email. This document is not meant to be a general guide on how to do backups, but instead it just describes how I ended up doing it.

NOTICE: Backing up your work is one of the most important things to take care of when dealing with computers. It is necessary to know what you are doing, so if you do not feel comfortable working with the command line, writing shell scripts, or editing configuration files, you might want to look for other alternatives.

Mac OS X 10.5 Leopard is only a couple of months away and will bring us Time Machine, which is a backup solution integrated to the operating system. In the mean time, I thought that I still need some way of backing up my files. When thinking about the solutions, I came up with the following criteria for the system:

runs from the command line
only copies files that have changed
does incremental backups instead of just mirroring

While going through different solutions, I came across rdiff-backup, which seemed to do what I wanted. I decided to set up nightly backups using rdiff-backup and launchd. In this document I am outlining that process and also discussing how to keep external hard disks mounted when no users are logged in, and how to get mail to work.

Installing rdiff-backup

Installing rdiff-backup is easy using MacPorts as it automatically installs all the dependencies as well. In addition to installing the rdiff-backup package, you should also install the xattr python module, which takes care of extended attributes. These two commands will install rdiff-backup and xattr:

sudo port install py-xattr sudo port install rdiff-backup-devel

As you can see, I went for the development version of rdiff-backup. The current stable release is 1.0.5 (MacPorts has 1.0.4), and the development version is at 1.1.9 (1.1.5 on MacPorts). For the xattr module, MacPorts has 0.2 while the latest is 0.4. After a while, I decided that I want to go for the cutting edge releases. So, I deactivated the versions installed through MacPorts (sudo port deactivate rdiff-backup-devel and sudo port deactivate py-xattr), downloaded the source packages, and built them myself. That was not too hard either. The newest xattr can be downloaded here. Even if you want to go for the latest versions, installing first with MacPorts does have the benefit of automatically installing all the dependencies.

My Backup Script

I have an external hard drive that I am using for backups. However, the disk also has some other uses and has a separate Mac OS X installation for testing purposes. I did not want to interfere with this installation, and more importantly, I did not want this test installation to interfere with my backups. So, instead of backing up directly to this drive, I decided to create a disk image to store my backups. This backup script mounts the disk image, performs the backup, and unmounts the image.

Also, the script has an alert level and gives a warning when the disk usage goes over this limit. This was added because rdiff-backup has an option to remove backups older than some specified time limit. Naturally, I would like to keep the backups for as long as possible in terms of the available disk space. So, instead of specifying a time limit right away, I have set the script to alert at 90%. When the time comes, I will be notified and can then make a decision about the appropriate time to store old backups.

/usr/local/bin/mybackup-script.sh - Download - View in browser

#!/bin/sh # backup script # 2007-03-11 # Ilari Scheinin # firstname.lastname@helsinki.fi # MIT License # config section SOURCE=/ # what to back up DESTINATION_VOLUME=MyExternalHD # name of the backup drive BACKUP_NAME=Backup # name of the disk image on the backup drive IMAGE_SIZE=220g # max size how big the disk image can grow KEEP_TIME=3M # how long to keep backups, see man rdiff-backup ALERT=90% # when to alert about disk space running out RDIFF_BACKUP=/opt/local/bin/rdiff-backup # end of config section VOL=/Volumes/$DESTINATION_VOLUME IMAGE=/Volumes/$DESTINATION_VOLUME/$BACKUP_NAME.sparseimage if [ ! -d $VOL ]; then # can we find the backup volume? echo 1>&2 Error: Backup volume not found. exit 1 fi if [ ! -f $IMAGE ]; then # can we find the disk image? echo Backup image not found, creating... FORCE="--force" hdiutil create -quiet -size $IMAGE_SIZE -type SPARSE -fs HFS+J \ -volname $BACKUP_NAME -uid 0 -gid 80 -mode 0775 $IMAGE fi # mount the image and use grep and awk to grab the path to the mounted volume MOUNT=`hdiutil attach -owners on $IMAGE | grep Apple_HFS | awk '{print $3}'` if [ ! -d $MOUNT ]; then # did the image mount successfully? echo 1>&2 Error: Could not mount backup image. exit 1 fi # turn Spotlight indexing off for the mounted disk image mdutil -i off $MOUNT >/dev/null # do the backup $RDIFF_BACKUP --exclude-sockets --exclude-other-filesystems \ --exclude /.Spotlight-V100/ --exclude /automount/ \ --exclude /cores/ --exclude /Network/ \ --exclude /private/tmp/ --exclude /private/var/launchd/ \ --exclude /private/var/run/ --exclude /private/var/spool/postfix/ \ --exclude /private/var/vm/ --exclude /tmp/ \ --exclude "/Users/*/Library/Caches/" --exclude /Volumes/ \ $FORCE $SOURCE $MOUNT # remove files older than KEEP_TIME # $RDIFF_BACKUP --remove-older-than $KEEP_TIME $MOUNT # I have left the above command commented out, because I am waiting until # I receive a warning about disk usage exceeding the ALERT level. Then I # will see what is a suitable KEEP_TIME and will then uncomment the command. # get the disk space usage, and give a warning if necessary USAGE=`df -h $MOUNT | grep -v Capacity | awk '{print $5}'` if [ $USAGE = "100%" ] || [ $USAGE = $ALERT ] || [[ $USAGE > $ALERT ]]; then echo Warning: Disk usage is $USAGE fi # unmount the disk image hdiutil detach -quiet -force $MOUNT exit 0

Scheduling Backups with launchd

The standard way of scheduling tasks on Mac OS X is using launchd. The following file will run the backup script daily at 3:30 AM. You are free to name the file whatever you want, but it should be placed in /Library/LaunchDaemons/ so that it is loaded when the computer boots.

/Library/LaunchDaemons/com.myip.backup.plist - Download - View in browser

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>com.myip.backup</string> <key>ProgramArguments</key> <array> <string>/usr/local/bin/mybackup.sh</string> </array> <key>LowPriorityIO</key> <true/> <key>Nice</key> <integer>1</integer> <key>StartCalendarInterval</key> <dict> <key>Hour</key> <integer>3</integer> <key>Minute</key> <integer>30</integer> </dict> </dict> </plist>

You probably noticed that the plist refers to mybackup.sh and not mybackup-script.sh. The reason will come clear in a moment.

To load the plist manually right now without restarting, use this command:

sudo launchctl load /Library/LaunchDaemons/com.myip.backup.plist

Reporting Errors via Email

Imagine your hard drive does break some day and you need to recover everything from the backups. It would not be very nice to find out that for whatever reason, your last backup is three months old. Maybe you renamed your external hard drive forgetting that the backup script will not be able to find it anymore. For this reason, I wanted to make sure that if something goes wrong, I will notice it. You can always direct the output from the backup script to some log file, but that would require you to check that file on a regular basis. That is something that you are likely to forget, so I figured that it would be better to get these notices by email.

launchd does allow you to specify log files for stdout and stderr, but I was not able to figure out how you could pipe them to mail. So, I wrote another script called mybackup.sh, which is just a small wrapper for mybackup-script.sh. It detects if you are running it interactively or if it is being run without a terminal. In the first case, it just calls the original backup script. In the latter, it pipes the output to mail, so you will get notified if anything goes wrong. Just make sure that the emails do not get stuck in a spam filter.

/usr/local/bin/mybackup.sh - Download - View in browser

#!/bin/sh # backup script # 2007-03-11 # Ilari Scheinin # firstname.lastname@helsinki.fi # MIT License # config section EMAIL=my@email.address # end of config section if [ -t 0 ]; then /usr/local/bin/mybackup-script.sh else /usr/local/bin/mybackup-script.sh 2>&1 | \ mail -E -s "Backup Script Output" $EMAIL fi exit $?

Configuring Postfix

It is quite like that sending email with mail does not work with the default configuration. This is because quite a few ISPs block port 25 to all other addresses, but their own SMTP server. The solution is to configure Postfix (the builtin tool for sending mail in Mac OS X) to relay all emails through your ISPs server. This is done by adding the following line to the /etc/postfix/main.cf configuration file. Naturally, replace smtp.yourisp.com with the name of the SMTP server of your ISP:

relayhost = smtp.yourisp.com

By default, your mails appear to originate from root@name-of-your-computer.local, which is something that the mail server is likely to reject. The solution is to define what should come after the @ sign:

myorigin = yourisp.com

Using the default configuration, Postfix accepts all computers in the same subnet to send mail. If you do not want to allow this, adding the following line narrows it down to your machine only.

mynetworks_style = host

Keeping External Volumes Mounted

Normally, Mac OS X does not mount external hard drives at boot time, but instead only when a user logs in. Also, the drives are unmounted when the user logs out. For this reason, the backup script fails if there are no users logged in at the time of its execution, as it cannot find the backup volume. If you want to be able to run the backup script at all times, this command will make Mac OS X keep the external volumes mounted even when no one is logged in:

sudo defaults write /Library/Preferences/SystemConfiguration/autodiskmount \ AutomountDisksWithoutUserLogin -bool true

Getting Backup Statistics

There is also a separate tool called rdiff-backup-statistics for getting more information about the backup runs. It tells you how long it took to do the backup, how many new/deleted/changed files there were, etc. As my backup is on a disk image, I cannot run this tool directly without first mounting the image, so I wrote a small script for that purpose as well. It just mounts the image, runs rdiff-backup-statistics, and unmounts.

Without any arguments, you will get an average of all the backup runs. If you are only interested about the last run, run it with argument last.

/usr/local/bin/mybackup-statistics.sh - Download - View in browser

#!/bin/sh # backup script # 2007-03-11 # Ilari Scheinin # firstname.lastname@helsinki.fi # MIT License # config section DESTINATION_VOLUME=MyExternalHD # name of the backup drive BACKUP_NAME=Backup # name of the disk image on the backup drive RDIFF_BACKUP_STATISTICS=/opt/local/bin/rdiff-backup-statistics # end of config section VOL=/Volumes/$DESTINATION_VOLUME IMAGE=/Volumes/$DESTINATION_VOLUME/$BACKUP_NAME.sparseimage if [ ! -d $VOL ]; then # can we find the backup volume? echo 1>&2 Error: Backup volume not found. exit 1 fi if [ ! -f $IMAGE ]; then # can we find the backup image? echo 1>&2 Error: Backup image not found. exit 1 fi # mount the image and use grep and awk to grab the path to the mounted volume MOUNT=`hdiutil attach -readonly $IMAGE | grep Apple_HFS | awk '{print $3}'` if test ! -d $MOUNT; then # did the image mount successfully? echo 1>&2 Error: Could not mount backup image. exit 1 fi if test $# -eq 0; then # no arguments given ARGS="" elif test $# -gt 1; then # more than one argument ARGS=$* elif test $1 = "last"; then # get only the last backup ARGS="--begin-time 1D" else # one argument ARGS="--begin-time $1" fi # get the backup statistics $RDIFF_BACKUP_STATISTICS $ARGS $MOUNT # unmount the disk image hdiutil detach -quiet -force $MOUNT exit 0

Summary

So that is my backup setup. As I said in the beginning, when it comes to backing up your data, you should really know what you are doing. So if you are not familiar with the tools and techniques used above, you might want to look for other alternatives. Also, I am by no means an expert in writing shell scripts, so my scripts are likely to suffer from poor design and errors. But anyway, I hope this description can be helpful for other people when they are designing their backup systems.