If you don’t do backups… well, you should.
An example of a good backup system could be:
- External drive backups: your computer is backed up at an external drive;
- CD/DVD backups: burn your backups to a CD or DVD;
- Offsite backups: your backups are located at a different site, ie, another place other than your home or office.
A good idea is to have all three backup systems in place. External drivers can brake and DVDs are misplaced, so a offsite location for your backups are a good idea.
I started using rsync.net for my offsite backups. It’s a paid service, but it gives you monthly payments for GB (minimum 7 GB) and ssh, ftp, sftp, webdav, https, and… rsync. Check the site for full features.
A great way to use rsync.net for your backups is using duplicity. Duplicity is an encrypted bandwidth-efficient backup that uses the rsync algorithm.
I will show you how I set up my system to create encrypted backups in a rsync.net server.
More info about this setup can be seen at rync.net own site.
Installing duplicity
sudo apt-get install duplicity
Generate an ssh key to connect to rsync.net
- Generate the ssh key using the defaults (don’t use a password when asked):
ssh-keygen -t rsa
- Copy your key to rsync.net:
scp ~/.ssh/id_rsa.pub user@server.rsync.net:.ssh/authorized_keys
- If you need to add more keys from different computers, generate the key and:
cat ~/.ssh/id_rsa.pub | ssh user@server.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
Generate a gpg public and private key to encrypt your backups
- Generate the gpg keys (get down your private key, example: 123AbcH123BB4321):
gpg --gen-key
- You can check the keys in your system:
gpg --list-keys
- Look in the output something like this. Look for the public key (in this example your public key is 1AAB123A)
pub 1239A/**1AAB123A** 2011-10-09 [expires: 2012-10-08]
uid Your Name (name) <mail@mail.com>
sub 54321/1234BBCC 2011-10-09 [expires: 2012-10-08]
Bash scripts to backup your files
I created two bash scripts, one that implements what I want to do with duplicity, and another one that uses the first one to backup my directories.
It uses the following duplicity commands (more at duplicity website):
- Clean up and remove older backups
duplicity cleanup -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir
duplicity remove-older-than 30D -v9 --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir
- Backup (full or inc)
duplicity full --encrypt-key="1AAB123A" /home/user/Documents scp://user@server.rsync.net/bck_dir
- Verify the backup
duplicity verify --encrypt-key="1AAB123A" scp://user@server.rsync.net/bck_dir /home/user/Documents
First script: using duplicity to backup a given directorie
Notes about this script:
- Has 4 parameters:
- backup type: inc for an incremental backup, and full for a full backup
- backup dir: origin for the backup
- destination dir: rsync.net directorie
- exclude filename: a filename to exclude
- Careful: Has the private key hardcoded in the script (!)
- It starts by cleaning up and remove older backups
- Performs a full or incremental backup
- Verify the backup
In the script you need to change the following to fit your needs:
- The SERVER variable
- The SCP_SERVER variable
- The PUB_KEY variable
- The PASSPHRASE variable (this is your private key)
- In the remove-older-than command you may want to change the number of days until remove the backup
rsync.sh
#!/bin/bash
###############################################################
# rsync.sh
#
# Parameters:
# $1 - backup type ( full | inc )
# $2 - ori dir
# $3 - backup dir
# $4 - exclude filename (example: "file.zip")
#
# example: rsync.sh full /home/user/mydir backups/bckdir file.zip
#
###############################################################
BCK_TYPE=$1
ORI_DIR=$2
DEST_DIR=$3
if [[ -z $4 ]]; then
EXCL=""
else
EXCL="--exclude **"$4
fi
SERVER=user@server.rsync.net
SCP_SERVER=scp://user@server.rsync.net
PUB_KEY=1AAB123A
# secret
export PASSPHRASE=123AbcH123BB4321
RUNNING_SCRIPT=`basename "$0"`
CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'`
LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.`echo "${DEST_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log
# create backup dir
if ssh ${SERVER} test -d ${DEST_DIR}; then
echo "'"${DEST_DIR}"' already exists at '"${SERVER}"'" >> $LOG
else
ssh ${SERVER} mkdir ${DEST_DIR}
fi
# -- BACKUP --
echo "" >> $LOG
echo "<<<<<<========================================INI====================================>>>>>>" >> $LOG
echo "" >> $LOG
echo "==============================> backing up '"${ORI_DIR}"' to '"${DEST_DIR}"'" >> $LOG
echo ">" `date "+%Y-%m-%d %H:%M:%S"` >> $LOG
# clean up
echo "" >> $LOG
echo "==============================> cleanup ('"${ORI_DIR}"')" >> $LOG
duplicity cleanup -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG
echo "" >> $LOG
echo "==============================> remove-older-than 30D ('"${ORI_DIR}"')" >> $LOG
duplicity remove-older-than 30D -v9 --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} >> $LOG
# backup
echo "" >> $LOG
echo "==============================>" ${BCK_TYPE} "backup ('"${ORI_DIR}"')" >> $LOG
duplicity ${BCK_TYPE} --encrypt-key="${PUB_KEY}" "${ORI_DIR}" ${EXCL} ${SCP_SERVER}/${DEST_DIR} >> $LOG
# verify backup
echo "" >> $LOG
echo "==============================> verify ('"${ORI_DIR}"')" >> $LOG
duplicity verify --encrypt-key="${PUB_KEY}" ${SCP_SERVER}/${DEST_DIR} "${ORI_DIR}" >> $LOG
# rsync.net quota
echo "" >> $LOG
echo "==============================> rsync.net quota" >> $LOG
ssh ${SERVER} quota >> $LOG
echo "" >> $LOG
echo "<<<<<<========================================END====================================>>>>>>" >> $LOG
echo "" >> $LOG
#eof
Second script: use the previous script to backup your directories
Notes about this script:
- It uses the notify-send command for desktop notifications
sudo apt-get install libnotify-bin
- It’s divided into groups that run in parallel
- It only have one parameter: if it’s a full backup or an inc one
In the script you need to change the following to fit your needs:
- The BCK_MAIN_DIR variable (it’s the directorie in rsync.net where your backups go; it will be created automatically)
- In each group add as much func_run_duplicity functions as you want, changing the folder/file to backup and the name of the folder to keep that backup in rsync.net (this folder will be created automatically - example backups/BCK1)
- You can add new groups
- In Group 2 there is an example of a file to exclude from the backup
run_rsync.sh
#!/bin/bash
###############################################################
# run_rsync.sh
#
# Parameters:
# <type> full | inc
#
# example: run_rsync.sh full
#
###############################################################
BCK_TYPE=$1
BCK_MAIN_DIR=backups
RUNNING_SCRIPT=`basename "$0"`
CUR_DIR=`echo "$0" | awk -F"$RUNNING_SCRIPT" '{ print $1 }'`
MAIN_LOG=${CUR_DIR}/logs/`date "+%Y%m%d"`.${BCK_TYPE}.log
# -------------------------------------------------------------
# func_run_duplicity ()
#
# function that runs duplicity
#
# Parameters:
# $1 - group
# $2 - ori dir
# $3 - backup dir
# $4 - exclude
# -------------------------------------------------------------
func_run_duplicity() {
GROUP=$1
ORI_DIR=$2
BCK_DIR=${BCK_MAIN_DIR}/$1$3
EXCL=$4
LOG=${CUR_DIR}/logs/`date "+%Y%m%d%H%M%S"`.`echo "${BCK_DIR}" | awk -F'/' '{ print $2 }'`.${BCK_TYPE}.log
echo "Group" $GROUP "INIT:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
$CUR_DIR/rsync.net.sh \
${BCK_TYPE} \
${ORI_DIR} \
${BCK_DIR} \
${EXCL}
wait
echo "Group" $GROUP "END:" ${ORI_DIR} `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
}
echo "<<<<<<========================================INI====================================>>>>>>" >> ${MAIN_LOG}
notify-send "Running $BCK_TYPE rsyn.net duplicity sync"
{
echo "Group 1 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
func_run_duplicity 1 "/home/user/FolderA" "BCK1"
func_run_duplicity 1 "/home/user/FolderB" "BCK2"
echo "Group 1 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
} &
{
echo "Group 2 INIT" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
func_run_duplicity 2 "/home/user/FolderC" "BCK3" "File.zip"
echo "Group 2 END" `date "+%Y-%m-%d %H:%M:%S"` >> ${MAIN_LOG}
} &
wait
echo "<<<<<<========================================END====================================>>>>>>" >> ${MAIN_LOG}
notify-send "$BCK_TYPE rsyn.net duplicity sync ended"
#eof
Setting up the scripts and running them automatically
Create a folder for your tow scripts and a backup folder:
mkdir ~/rsync.net
mkdir ~/rsync.net/logs
I use cron to run a daily inc backup and weekly full backup (Sundays):
crontab -e
Add to cron:
00 21 * * * ~/rsync.net/run_rsync.sh inc
00 16 * * 0 ~/rsync.net/run_rsync.sh full
Restoring your backups
To restore your backups:
mkdir ~/restore
duplicity --encrypt-key="1AAB123A" scp://user@server.rsync.net/backups/BCK1 ~/restore
Please tell me about your setup, experiences and suggestions!