Backing Up Using Expect and Rsync

By Blane Warrene
Mila is a prolific book cover designer on 99designs. Work with her today!
I'm led by "Work hard. Play fair. Be nice." That's a key to success.
Mila Milic

Rsync alone is a powerful tool for moving data within and among local and remote servers. When combined with Expect, interactive sessions can be automated on the command line for very useful purposes.

I find the most powerful application being the process of hot backups to remote servers each evening. While optical disk backups are run for offline archiving and data preservation, I find most the most useful backups being those held on secured remote servers managed via Expect and Rsync.

Expect – found at http://expect.nist.gov/ – simply enables automated responses to prompts at the command line when user intervention is not necessary or reasonable. Thus, overnight backups can be run, enabling remote authentication and a password entry when prompted (when ‘expected’ – hence the name). More information can be found by running man expect on the command line.

These files are appended with the .exp extension and as you will see in the scripts below, use a shebang at the top to call the Expect program. I set permissions on these scripts as executable by root only.

Rsync – found at http://samba.anu.edu.au/rsync/ is a tool for data synchronization, copying of files and so on. More information can be found by running man rsync on the command line.

Rsync can be advantageous as it will perform differential backups once a master copy is completed, backing up files that have changed by reviewing the datestamp and file size, minimizing bandwidth and time frames in which backups need to run. Running a tape or optical backup drive can be configured with the remote backup server to further free up overnight resources for your production web server.

When used in conjunction, a straight forward backup scheme can be implemented for web servers (among other systems).

Here are the scripts in use. They are called from the backup server to the target server and data is stored in ‘hot backup’ for easy access:

Script 1 – MySQL Database backup

set timeout 19900
spawn /bin/bash
expect -re "]# "
send "rsync -avzb -e ssh root@domain.com:/var/db/mysql/dumps/ /backup/servers/comain.com/mysqlr"
expect -re "password:"
sleep 2
send "rootpasswordr"
expect -re "total size is"
expect -re "]# "
send "exitr"

Now for backing up domain data (web sites), a second script is run:

Script 2 – Domain backups

set timeout 19900
spawn /bin/bash
expect -re "]# "
send "rsync -avzb -e ssh root@domain.com:/home/sites /backup/servers/domain.com/homer"
expect -re "password:"
sleep 2
send "rootpasswordr"
expect -re "total size is"
expect -re "]# "
send "exitr"

To deconstruct this:

1) Expect is called (your path may be different – reveal it by issuing ‘whereis expect’ on the command line)

2) send the command to the remote server over ssh:

rsync options are -a (archive which preserves permissions and links among other items), v (verbose), z (compress) and b (backup – adds a ~ to preexisting destination files, a sort of versioning of backups)

3) an ssh password is ‘expected’, or in other cases, authorized keys can be copied over to the remote server to bypass a password prompt for ssh.

4) a password is expected – which is sent via Expect to the server for authentication and the operation is carried out.

5) The script completes and closes.

I have named these files mysql.backup.exp and domains.backup.exp and entered them into cron to run on a daily schedule.

While this should not be the only backup procedure – it can be very useful to have a live server available for quick restores or review of backup data.

Additional scripts can also be written to backup critical configuration files (i.e. the /etc directory) and other areas of the server(s).

  • http://simon.incutio.com/ Skunk

    If you use SSH public/private key authentication you don’t need to use Expect for this at all, which is good because it means you don’t have to have script fiels lying around containing your unencrypted root password.

  • http://www.rideontwo.com z0s0

    Damn Simon, beat me to it ;-P

  • http://www.practicalapplications.net bwarrene

    Yes – it is a trade-off in some cases. I have setup ssh keys for root on some servers with host and command restrictions for passwordless actions (limiting usage of private keys to one host limited to one command) – however – it became a management pain for some folks managing multiple root keys. This above worked better as both servers (although on Internet) have firewalls, intrusion detection and change management layers which allowed us to breathe easier using Expect.

  • niranjan

    Good one.I was making search to identify functioning of rsync.Simple and short explaination

  • KTK

    Hi ,
    Can anybody help me with this problem in “EXPECT” script.

    The expect script works fine when called locally,but when i try to call the script from a remote machine through a SSH ,it fails.
    send: spawn id exp0 not open

    Please let me know what can be done.

Get the latest in Front-end, once a week, for free.