Offsite backups with rsync.net

09 Oct, 2011

If you don’t do backups… well, you should.

An example of a good backup system could be:

External drive backups: your computer is backed up at an external drive;
CD/DVD backups: burn your backups to a CD or DVD;
Offsite backups: your backups are located at a different site, ie, another place other than your home or office.

A good idea is to have all three backup systems in place. External drivers can brake and DVDs are misplaced, so a offsite location for your backups are a good idea.

I started using rsync.net for my offsite backups. It’s a paid service, but it gives you monthly payments for GB (minimum 7 GB) and ssh, ftp, sftp, webdav, https, and… rsync. Check the site for full features.

A great way to use rsync.net for your backups is using duplicity. Duplicity is an encrypted bandwidth-efficient backup that uses the rsync algorithm.

I will show you how I set up my system to create encrypted backups in a rsync.net server.

More info about this setup can be seen at rync.net own site.

Installing duplicity

1sudo apt-get install duplicity

Generate an ssh key to connect to rsync.net

Generate the ssh key using the defaults (don’t use a password when asked):

1ssh-keygen -t rsa

Copy your key to rsync.net:

1scp ~/.ssh/id_rsa.pub user@server.rsync.net:.ssh/authorized_keys

If you need to add more keys from different computers, generate the key and:

1cat ~/.ssh/id_rsa.pub | ssh user@server.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

Generate a gpg public and private key to encrypt your backups

Generate the gpg keys (get down your private key, example: 123AbcH123BB4321):

1gpg --gen-key

You can check the keys in your system:

1gpg --list-keys

Look in the output something like this. Look for the public key (in this example your public key is 1AAB123A)

pub  1239A/**1AAB123A** 2011-10-09 [expires: 2012-10-08]
 uid          Your Name (name) <mail@mail.com>
 sub  54321/1234BBCC 2011-10-09 [expires: 2012-10-08]

Bash scripts to backup your files

I created two bash scripts, one that implements what I want to do with duplicity, and another one that uses the first one to backup my directories.

It uses the following duplicity commands (more at duplicity website):

Clean up and remove older backups

1duplicity cleanup -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir
2duplicity remove-older-than 30D -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir

Backup (full or inc)

1duplicity full --encrypt-key="1AAB123A" /home/user/Documents scp://user@server.rsync.net/bck_dir

Verify the backup

1duplicity verify --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir /home/user/Documents

First script: using duplicity to backup a given directorie

Notes about this script:

Has 4 parameters:
- backup type: inc for an incremental backup, and full for a full backup
- backup dir: origin for the backup
- destination dir: rsync.net directorie
- exclude filename: a filename to exclude
Careful: Has the private key hardcoded in the script (!)
It starts by cleaning up and remove older backups
Performs a full or incremental backup
Verify the backup

In the script you need to change the following to fit your needs:

The SERVER variable
The SCP_SERVER variable
The PUB_KEY variable
The PASSPHRASE variable (this is your private key)
In the remove-older-than command you may want to change the number of days until remove the backup

rsync.sh

 1#!/bin/bash
 2
 3###############################################################
 4# rsync.sh
 5#
 6# Parameters:
 7#   $1 - backup type ( full | inc )
 8#   $2 - ori dir
 9#   $3 - backup dir
10#   $4 - exclude filename (example: "file.zip")
11#
12# example: rsync.sh full /home/user/mydir backups/bckdir file.zip
13#
14###############################################################
15
16BCK_TYPE=$1
17ORI_DIR=$2
18DEST_DIR=$3
19
20if [[ -z $4 ]]; then
21  EXCL=""
22else
23  EXCL="--exclude **"$4
24fi
25
26SERVER=user@server.rsync.net
27SCP_SERVER=scp://user@server.rsync.net
28PUB_KEY=1AAB123A
29
30# secret
31export PASSPHRASE=123AbcH123BB4321
32
33RUNNING_SCRIPT=`basename "$0"`
34CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'`
35LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.`echo "${DEST_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log
36
37# create backup dir
38
39if ssh ${SERVER} test -d ${DEST_DIR}; then
40  echo "'"${DEST_DIR}"' already exists at '"${SERVER}"'" >> $LOG
41else
42  ssh ${SERVER} mkdir ${DEST_DIR}
43fi
44
45# -- BACKUP --
46
47echo "" >> $LOG
48echo "<<<<<<========================================INI====================================>>>>>>" >> $LOG
49echo "" >> $LOG
50echo "==============================> backing up '"${ORI_DIR}"' to '"${DEST_DIR}"'" >> $LOG
51echo ">" `date "+%Y-%m-%d %H:%M:%S"` >> $LOG
52
53# clean up
54echo "" >> $LOG
55echo "==============================> cleanup ('"${ORI_DIR}"')" >> $LOG
56duplicity cleanup -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG
57echo "" >> $LOG
58echo "==============================> remove-older-than 30D ('"${ORI_DIR}"')" >> $LOG
59duplicity remove-older-than 30D -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG
60
61# backup
62echo "" >> $LOG
63echo "==============================>" ${BCK_TYPE} "backup ('"${ORI_DIR}"')" >> $LOG
64duplicity ${BCK_TYPE} --encrypt-key="${PUB_KEY}" "${ORI_DIR}" ${EXCL} ${SCP_SERVER}/${DEST_DIR} >> $LOG
65
66# verify backup
67echo "" >> $LOG
68echo "==============================> verify ('"${ORI_DIR}"')" >> $LOG
69duplicity verify --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} "${ORI_DIR}" >> $LOG
70
71# rsync.net quota
72echo "" >> $LOG
73echo "==============================> rsync.net quota" >> $LOG
74ssh ${SERVER} quota >> $LOG
75
76echo "" >> $LOG
77echo "<<<<<<========================================END====================================>>>>>>" >> $LOG
78echo "" >> $LOG
79
80#eof

Second script: use the previous script to backup your directories

Notes about this script:

It uses the notify-send command for desktop notifications

1sudo apt-get install libnotify-bin

It’s divided into groups that run in parallel
It only have one parameter: if it’s a full backup or an inc one

In the script you need to change the following to fit your needs:

The BCK_MAIN_DIR variable (it’s the directorie in rsync.net where your backups go; it will be created automatically)
In each group add as much func_run_duplicity functions as you want, changing the folder/file to backup and the name of the folder to keep that backup in rsync.net (this folder will be created automatically - example backups/BCK1)
You can add new groups
In Group 2 there is an example of a file to exclude from the backup

run_rsync.sh

 1#!/bin/bash
 2
 3###############################################################
 4# run_rsync.sh
 5#
 6# Parameters:
 7#     <type> full | inc
 8#
 9# example: run_rsync.sh full
10#
11###############################################################
12
13BCK_TYPE=$1
14
15BCK_MAIN_DIR=backups
16
17RUNNING_SCRIPT=`basename "$0"`
18CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'`
19
20MAIN_LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.${BCK_TYPE}.log
21
22# -------------------------------------------------------------
23# func_run_duplicity ()
24#
25# function that runs duplicity
26#
27# Parameters:
28#   $1 - group
29#   $2 - ori dir
30#   $3 - backup dir
31#   $4 - exclude
32# -------------------------------------------------------------
33
34func_run_duplicity() {
35  GROUP=$1
36  ORI_DIR=$2
37  BCK_DIR=${BCK_MAIN_DIR}/$1$3
38  EXCL=$4
39  LOG=${CUR_DIR}/logs/`date "+%Y%m%d%H%M%S"`.`echo "${BCK_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log
40
41  echo "Group" $GROUP "INIT:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
42
43  $CUR_DIR/rsync.net.sh \
44      ${BCK_TYPE} \
45      ${ORI_DIR} \
46      ${BCK_DIR} \
47      ${EXCL}
48
49  wait
50
51  echo "Group" $GROUP "END:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
52}
53
54echo "<<<<<<========================================INI====================================>>>>>>" >> ${MAIN_LOG}
55
56notify-send "Running $BCK_TYPE rsyn.net duplicity sync"
57
58{
59  echo "Group 1 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
60  func_run_duplicity 1 "/home/user/FolderA" "BCK1"
61  func_run_duplicity 1 "/home/user/FolderB" "BCK2"
62  echo "Group 1 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
63} &
64
65{
66  echo "Group 2 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
67  func_run_duplicity 2 "/home/user/FolderC" "BCK3" "File.zip"
68  echo "Group 2 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
69} &
70
71wait
72
73echo "<<<<<<========================================END====================================>>>>>>" >> ${MAIN_LOG}
74
75notify-send "$BCK_TYPE rsyn.net duplicity sync ended"
76
77#eof

Setting up the scripts and running them automatically

Create a folder for your tow scripts and a backup folder:

1mkdir ~/rsync.net
2mkdir ~/rsync.net/logs

I use cron to run a daily inc backup and weekly full backup (Sundays):

1crontab -e

Add to cron:

100 21 * * * ~/rsync.net/run_rsync.sh inc
200 16 * * 0 ~/rsync.net/run_rsync.sh full

Restoring your backups

To restore your backups:

1mkdir ~/restore
2duplicity --encrypt-key="1AAB123A" scp://user@server.rsync.net/backups/BCK1 ~/restore

Please tell me about your setup, experiences and suggestions!

#Linux #Ubuntu #Rsync.net