Backing Up Using Expect and Rsync

Rsync alone is a powerful tool for moving data within and among local and remote servers. When combined with Expect, interactive sessions can be automated on the command line for very useful purposes.

I find the most powerful application being the process of hot backups to remote servers each evening. While optical disk backups are run for offline archiving and data preservation, I find most the most useful backups being those held on secured remote servers managed via Expect and Rsync.

Expect – found at http://expect.nist.gov/ – simply enables automated responses to prompts at the command line when user intervention is not necessary or reasonable. Thus, overnight backups can be run, enabling remote authentication and a password entry when prompted (when ‘expected’ – hence the name). More information can be found by running man expect on the command line.

These files are appended with the .exp extension and as you will see in the scripts below, use a shebang at the top to call the Expect program. I set permissions on these scripts as executable by root only.

Rsync – found at http://samba.anu.edu.au/rsync/ is a tool for data synchronization, copying of files and so on. More information can be found by running man rsync on the command line.

Rsync can be advantageous as it will perform differential backups once a master copy is completed, backing up files that have changed by reviewing the datestamp and file size, minimizing bandwidth and time frames in which backups need to run. Running a tape or optical backup drive can be configured with the remote backup server to further free up overnight resources for your production web server.

When used in conjunction, a straight forward backup scheme can be implemented for web servers (among other systems).

Here are the scripts in use. They are called from the backup server to the target server and data is stored in ‘hot backup’ for easy access:

Script 1 – MySQL Database backup


#!/usr/bin/expect
set timeout 19900
spawn /bin/bash
expect -re "]# "
send "rsync -avzb -e ssh root@domain.com:/var/db/mysql/dumps/ /backup/servers/comain.com/mysqlr"
expect -re "password:"
sleep 2
send "rootpasswordr"
expect -re "total size is"
expect -re "]# "
send "exitr"

Now for backing up domain data (web sites), a second script is run:

Script 2 – Domain backups


#!/usr/bin/expect
set timeout 19900
spawn /bin/bash
expect -re "]# "
send "rsync -avzb -e ssh root@domain.com:/home/sites /backup/servers/domain.com/homer"
expect -re "password:"
sleep 2
send "rootpasswordr"
expect -re "total size is"
expect -re "]# "
send "exitr"

To deconstruct this:

1) Expect is called (your path may be different – reveal it by issuing ‘whereis expect’ on the command line)

2) send the command to the remote server over ssh:

rsync options are -a (archive which preserves permissions and links among other items), v (verbose), z (compress) and b (backup – adds a ~ to preexisting destination files, a sort of versioning of backups)

3) an ssh password is ‘expected’, or in other cases, authorized keys can be copied over to the remote server to bypass a password prompt for ssh.

4) a password is expected – which is sent via Expect to the server for authentication and the operation is carried out.

5) The script completes and closes.

I have named these files mysql.backup.exp and domains.backup.exp and entered them into cron to run on a daily schedule.

While this should not be the only backup procedure – it can be very useful to have a live server available for quick restores or review of backup data.

Additional scripts can also be written to backup critical configuration files (i.e. the /etc directory) and other areas of the server(s).