If you don’t do backups… well, you should.

An example of a good backup system could be:

  • External drive backups: your computer is backed up at an external drive;
  • CD/DVD backups: burn your backups to a CD or DVD;
  • Offsite backups: your backups are located at a different site, ie, another place other than your home or office.

A good idea is to have all three backup systems in place. External drivers can brake and DVDs are misplaced, so a offsite location for your backups are a good idea.

I started using rsync.net for my offsite backups. It’s a paid service, but it gives you monthly payments for GB (minimum 7 GB) and ssh, ftp, sftp, webdav, https, and… rsync. Check the site for full features.

A great way to use rsync.net for your backups is using duplicity. Duplicity is an encrypted bandwidth-efficient backup that uses the rsync algorithm.

I will show you how I set up my system to create encrypted backups in a rsync.net server.

More info about this setup can be seen at rync.net own site.

Installing duplicity

sudo apt-get install duplicity

Generate an ssh key to connect to rsync.net

  • Generate the ssh key using the defaults (don’t use a password when asked):
ssh-keygen -t rsa
  • Copy your key to rsync.net:
scp ~/.ssh/id_rsa.pub user@server.rsync.net:.ssh/authorized_keys
  • If you need to add more keys from different computers, generate the key and:
cat ~/.ssh/id_rsa.pub | ssh user@server.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

Generate a gpg public and private key to encrypt your backups

  • Generate the gpg keys (get down your private key, example: 123AbcH123BB4321):
gpg --gen-key
  • You can check the keys in your system:
gpg --list-keys
  • Look in the output something like this. Look for the public key (in this example your public key is 1AAB123A)
pub  1239A/**1AAB123A** 2011-10-09 [expires: 2012-10-08]
 uid          Your Name (name) <mail@mail.com>
 sub  54321/1234BBCC 2011-10-09 [expires: 2012-10-08]

Bash scripts to backup your files

I created two bash scripts, one that implements what I want to do with duplicity, and another one that uses the first one to backup my directories.

It uses the following duplicity commands (more at duplicity website):

  • Clean up and remove older backups
duplicity cleanup -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir
duplicity remove-older-than 30D -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir
  • Backup (full or inc)
duplicity full --encrypt-key="1AAB123A" /home/user/Documents scp://user@server.rsync.net/bck_dir
  • Verify the backup
duplicity verify --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir /home/user/Documents

First script: using duplicity to backup a given directorie

Notes about this script:

  • Has 4 parameters:
    • backup type: inc for an incremental backup, and full for a full backup
    • backup dir: origin for the backup
    • destination dir: rsync.net directorie
    • exclude filename: a filename to exclude
  • Careful: Has the private key hardcoded in the script (!)
  • It starts by cleaning up and remove older backups
  • Performs a full or incremental backup
  • Verify the backup

In the script you need to change the following to fit your needs:

  • The SERVER variable
  • The SCP_SERVER variable
  • The PUB_KEY variable
  • The PASSPHRASE variable (this is your private key)
  • In the remove-older-than command you may want to change the number of days until remove the backup

rsync.sh

#!/bin/bash

###############################################################
# rsync.sh
#
# Parameters:
#   $1 - backup type ( full | inc )
#   $2 - ori dir
#   $3 - backup dir
#   $4 - exclude filename (example: "file.zip")
#
# example: rsync.sh full /home/user/mydir backups/bckdir file.zip
#
###############################################################

BCK_TYPE=$1
ORI_DIR=$2
DEST_DIR=$3

if [[ -z $4 ]]; then
  EXCL=""
else
  EXCL="--exclude **"$4
fi

SERVER=user@server.rsync.net
SCP_SERVER=scp://user@server.rsync.net
PUB_KEY=1AAB123A

# secret
export PASSPHRASE=123AbcH123BB4321

RUNNING_SCRIPT=`basename "$0"`
CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'`
LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.`echo "${DEST_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log

# create backup dir

if ssh ${SERVER} test -d ${DEST_DIR}; then
  echo "'"${DEST_DIR}"' already exists at '"${SERVER}"'" >> $LOG
else
  ssh ${SERVER} mkdir ${DEST_DIR}
fi

# -- BACKUP --

echo "" >> $LOG
echo "<<<<<<========================================INI====================================>>>>>>" >> $LOG
echo "" >> $LOG
echo "==============================> backing up '"${ORI_DIR}"' to '"${DEST_DIR}"'" >> $LOG
echo ">" `date "+%Y-%m-%d %H:%M:%S"` >> $LOG

# clean up
echo "" >> $LOG
echo "==============================> cleanup ('"${ORI_DIR}"')" >> $LOG
duplicity cleanup -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG
echo "" >> $LOG
echo "==============================> remove-older-than 30D ('"${ORI_DIR}"')" >> $LOG
duplicity remove-older-than 30D -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG

# backup
echo "" >> $LOG
echo "==============================>" ${BCK_TYPE} "backup ('"${ORI_DIR}"')" >> $LOG
duplicity ${BCK_TYPE} --encrypt-key="${PUB_KEY}" "${ORI_DIR}" ${EXCL} ${SCP_SERVER}/${DEST_DIR} >> $LOG

# verify backup
echo "" >> $LOG
echo "==============================> verify ('"${ORI_DIR}"')" >> $LOG
duplicity verify --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} "${ORI_DIR}" >> $LOG

# rsync.net quota
echo "" >> $LOG
echo "==============================> rsync.net quota" >> $LOG
ssh ${SERVER} quota >> $LOG

echo "" >> $LOG
echo "<<<<<<========================================END====================================>>>>>>" >> $LOG
echo "" >> $LOG

#eof

Second script: use the previous script to backup your directories

Notes about this script:

  • It uses the notify-send command for desktop notifications
sudo apt-get install libnotify-bin
  • It’s divided into groups that run in parallel
  • It only have one parameter: if it’s a full backup or an inc one

In the script you need to change the following to fit your needs:

  • The BCK_MAIN_DIR variable (it’s the directorie in rsync.net where your backups go; it will be created automatically)
  • In each group add as much func_run_duplicity functions as you want, changing the folder/file to backup and the name of the folder to keep that backup in rsync.net (this folder will be created automatically - example backups/BCK1)
  • You can add new groups
  • In Group 2 there is an example of a file to exclude from the backup

run_rsync.sh

#!/bin/bash

###############################################################
# run_rsync.sh
#
# Parameters:
#     <type> full | inc
#
# example: run_rsync.sh full
#
###############################################################

BCK_TYPE=$1

BCK_MAIN_DIR=backups

RUNNING_SCRIPT=`basename "$0"`
CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'`

MAIN_LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.${BCK_TYPE}.log

# -------------------------------------------------------------
# func_run_duplicity ()
#
# function that runs duplicity
#
# Parameters:
#   $1 - group
#   $2 - ori dir
#   $3 - backup dir
#   $4 - exclude
# -------------------------------------------------------------

func_run_duplicity() {
  GROUP=$1
  ORI_DIR=$2
  BCK_DIR=${BCK_MAIN_DIR}/$1$3
  EXCL=$4
  LOG=${CUR_DIR}/logs/`date "+%Y%m%d%H%M%S"`.`echo "${BCK_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log

  echo "Group" $GROUP "INIT:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}

  $CUR_DIR/rsync.net.sh \
      ${BCK_TYPE} \
      ${ORI_DIR} \
      ${BCK_DIR} \
      ${EXCL}

  wait

  echo "Group" $GROUP "END:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
}

echo "<<<<<<========================================INI====================================>>>>>>" >> ${MAIN_LOG}

notify-send "Running $BCK_TYPE rsyn.net duplicity sync"

{
  echo "Group 1 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
  func_run_duplicity 1 "/home/user/FolderA" "BCK1"
  func_run_duplicity 1 "/home/user/FolderB" "BCK2"
  echo "Group 1 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
} &

{
  echo "Group 2 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
  func_run_duplicity 2 "/home/user/FolderC" "BCK3" "File.zip"
  echo "Group 2 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
} &

wait

echo "<<<<<<========================================END====================================>>>>>>" >> ${MAIN_LOG}

notify-send "$BCK_TYPE rsyn.net duplicity sync ended"

#eof

Setting up the scripts and running them automatically

Create a folder for your tow scripts and a backup folder:

mkdir ~/rsync.net
mkdir ~/rsync.net/logs

I use cron to run a daily inc backup and weekly full backup (Sundays):

crontab -e

Add to cron:

00 21 * * * ~/rsync.net/run_rsync.sh inc
00 16 * * 0 ~/rsync.net/run_rsync.sh full

Restoring your backups

To restore your backups:

mkdir ~/restore
duplicity --encrypt-key="1AAB123A" scp://user@server.rsync.net/backups/BCK1 ~/restore

Please tell me about your setup, experiences and suggestions!