Network backup

Automated Backup Tools

Please refer to the documentation at Backup Tools and use them when possible.

The below instructions are preserved for reference in instances where backups and network transfers need to be performed manually.

Manual Backup (step by step)

N.B. Instructions have been updated

You will need to be on a wired connection for this to work

On the old machine/drive run the following commands in the terminal. If you are using a recovery CD you will likely not need to use "sudo" before commands as you will be root.

Read this before proceeding

• Use reverse ISO format for the date (e.g. 20121223 for Dec 23 2012) not American date format. Failing to do so may result in files being accidentally deleted.
• Always rsync the entire home directory even if there is only one user otherwise files and users may not be restored correctly.
• Errors are logged in the file rsync.log.

All these commands will need to be run as root (via sudo). Because sudo does not effect operations to the right of '>', dpkg --get-selections > /home/dpkg.out will fail if you try to run it as sudo on its own. To get around this out put the file and move it in place .

 sudo cp /etc/passwd /etc/shadow /etc/group /home/
 dpkg --get-selections > dpkg.out
 sudo mv dpkg.out /home/
 sudo tar -czf /home/etc.backup.tar.gz /etc/
 sudo rsync -avzh --rsync-path="sudo rsync" --exclude =".gvfs" /home/ tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 2>rsync.log
   or  
 sudo rsync -avzh --rsync-path="sudo rsync" --exclude =".gvfs" --exclude="*/.cache/*" --exclude "*.mozilla/firefox/*/Cache/*" /home/ tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 2>rsync.log

Do not include brackets in the ISO format!

This should back up enough to create an effective clone of the old machine. The alternative rsync commands excludes cache files Note: on the data recovery box you will need to add /mnt/sdb1 in front of /home and /etc. For the second command use

  dpkg --get-selections --admindir=/mnt/sdb1/var/lib/dpkg >dpkg.out

this avoids the need to chroot.

When backing up a Windows Machine, unless you have a good reason to do otherwise, only backup the "Documents and Settings" folder in XP and "users" folder in Vista and Windows 7. This means you can omit the first four steps and run

 sudo rsync -avzh --rsync-path="sudo rsync" /mnt/sdb1/users/ tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 2>rsync.log

or

 sudo rsync -avzh --rsync-path="sudo rsync" "/mnt/sdb1/Documents and Settings/" tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 2>rsync.log

in place of the last step. Note the full path with /mnt/sdb1 and the use of quotes in the second example you can also escape the spaces with back slashes like this:

  sudo rsync -avzh --rsync-path="sudo rsync" /mnt/sdb1/Documents\ and\ Settings/ tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 

Its easiest to type sudo rsync -avzh "/mnt/sdb1/Do and hit the tab key to let the shell fill in the full path with the spaces correctly escaped.

On Backup Rack

Install the hard drive on one of the backup machines on the Backup Rack

Boot up to the live environment:

• Live Boot
• Debian Live
• Debian Live Rescue

upon reaching the prompt, issue the following commands:

 sudo su
 fdisk -l

Check the output of fdisk -l and note where the disk appears (e.g. /dev/sd* or /dev/hd* then enter:

for Linux file system:

 mount /dev/sd* /mnt/

for NTFS (windows) file system

 mount -t ntfs /dev/sd* /mnt/

Once the drive is mounted, then continue on with the backup procedure

cp /mnt/etc/passwd /mnt/etc/shadow /mnt/etc/group /mnt/home/
dpkg --get-selections --admindir=/mnt/var/lib/dpkg > dpkg.out
mv dpkg.out /mnt/home/
tar -czf /mnt/home/etc.backup.tar.gz /mnt/etc/
rsync -avzh --exclude =".gvfs" /mnt/home/ tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 2>/home/user/rsync.log
   or  
rsync -avzh --exclude =".gvfs" --exclude="*/.cache/*" --exclude "*.mozilla/firefox/*/Cache/*" /mnt/home/ tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 2>rsync.log

The --admindir option in the second command ensure you are getting the package info from the external drive, not the live image. There is also an --installdir=[targetdir] option that will allow you to install packages elsewhere. The directory you point to is treated as the root of the file system. The --root option affects both the admindir, where package info is taken from, and the target directory for installs.

To mount an external drive

If its a linux disk/partition you can go ahead and mount it by using

mount /dev/sdb1 

etc with the appropriate location taken from the output of fdisk -l. If this fails run

sudo fstab-fix

and try again.

If its an ntfs disk run

sudo mount-ntfs 

instead and it will be automatically mounted. The output from the command will tell you where.

Manual Restoration (step by step)

On the new machine/drive run the following commands in the terminal.This does not apply to Windows machines.

All these commands will need to be run via sudo:

 sudo rsync -avzh  --rsync-path="sudo rsync" tsbackup@oscar:/srv/tsbackup/[date-ticketnumber]/ /home
 sudo apt-get update
 sudo dpkg --set-selections < /home/dpkg.out
 sudo apt-get -u dselect-upgrade

***Updated***

(Note: In order to successfully perform the following operations, they should be done in a root shell.)

Then open up /home/passwd and do sudo vipw copy over any lines that refer to users that need to be recreated, They will be at the bottom of this file.

Next open up /home/shadow and do sudo vipw -s and repeat. Make sure you have not created any duplicate lines in the last two steps.

Finally open up /home/group and do sudo vigr. Make sure users are in all the groups they were before (eg. adm:x:4:oem,paul).

Do not blindly copy over the password shadow and group files it will breaks things

The last part restores the original password files etc. Before that you can operate as oem (as root) but it might not work properly after this stage. Don't forget to check to see if file ownership on the home directories matches properly. It might break things. To make sure it doesn't compare the version in /home and the installed version before copying over. You will need to make sure that UID and GUID (the numbers in passwd and group(users start at 1000) match up, especially for system users.

Reference

The sections below are included only to explain the preceding commands; please consider the above procedure definitive.

Backing up User Accounts

The three files you usually need are:

• /etc/passwd (list of users and home directories)
• /etc/shadow (the encrypted password file)
• /etc/group (list of groups and what users are in them)

Copy these into the home folder to make sure you remember to back them up:

sudo cp /etc/passwd /etc/shadow /etc/group /home/

It can be good practice to back up the whole /etc directory and /var/www as well, but its beyond the scope of what we do. The best practice is to keep the drive around for the users to check afterwards if possible.

Backup and Restore Installed Packages

You can also generate a list of the user's installed packages so you can reinstall them all at once later using a single command.

sudo dpkg --get-selections > dpkg.out 

Don't forget to include this file in the backup!

Then, when you have installed a new system/replaced the box/etc, you can do:

sudo dpkg --set-selections < dpkg.out

then

sudo apt-get -u dselect-upgrade

on the new box to install the same set of packages as before.

(This can be done over a network using ssh, but setting up ssh is beyond the scope of this document).

Backup and Restore User Data

To back up the home folder(s) over the network using rsync (don't actually be root or else funny things happen).

sudo rsync -avzh /home/ tsbackup@oscar:/srv/tsbackup/[date-ticketnumber] 2>rsync.out

N.B.

• Use reverse ISO format for the date (e.g. 20121223 for Dec 23 2012) not American date format. Failing to do so may result in files being accidentally deleted.
• Always rsync the entire home directory even if there is only one user otherwise files and users may not be restored correctly.
• This is our standard: the date part so that we don't accidentally overwrite existing directories on the backup box; the entire home directory so that we can ensure that all of the files that were backed up are restored.
• Errors will be logged to file rsync.out

To copy it back again:

sudo rsync -avzh  tsbackup@oscar:/srv/tsbackup/[date-ticketnumber]/ /home

N.B. The trailing slash on the source (first part) is important in rsync. A trailing slash means copy the contents of this folder without creating the folder itself. If you are uncertain about anything you can test it using the -n option. This will do a dry run and nothing will actually be copied over. It is a good habit to do a sanity check first with rsync; it is an immensely powerful tool and you want to make sure it is going to do what you think it is.

Rysnc is a more efficient protocol to use than scp. It will ensure file permissions and ownerships are preserved and will compress data on the fly to make for a faster transfer. (it will also only copy the needed files so if you add files, then you can run the command again and rsync will only copy the additional files. You can also remove files and use the --delete option and it will remove the files on the remote host).

If you don't want to copy hidden files add --exclude=".*/" as an option. Its a good idea to at least copy over users .mozilla files, and any .evolution or .thunderbird folders as a minimum. It shouldn't cause too many problems copying over all hidden files.

For even more information, you can have a look at

man rsync

Backup and Restore User's Desktop Background

You can also restore the user's desktop background (as long as you have the file backed up as well. To check, run the command without >desktop.pict).

gconftool-2 --get /desktop/gnome/background/picture_filename >desktop.pict

copy over desktop.pict and on the new machine

gconftool-2 --type string --set  /desktop/gnome/background/picture_filename "$(cat desktop.pict)"

N.B. copying over a user's /home folder wholesale with all the hidden files and folders will do this anyway.

Recreating User Accounts and Restoring Permissions

If you backed up the User Accounts from the old system as detailed above, you can simply restore those files to /etc now, and you'll have the same user/group setup as on the old system. Just do:

• • • Updated***

Open up /home/passwd and do sudo vipw copy over any lines that refer to users that need to be recreated, They will be at the bottom of this file.

Next open up /home/shadow and do sudo vipw -s and repeat. Make sure you have not created any duplicate lines in the last two steps.

Finally open up /home/group and do sudo vigr. Make sure users are in all the groups they were before (eg. adm:x:4:oem,paul).

Do not blindly copy over the password shadow and group files it will breaks things!!!

The last part restores the original password files etc. Before that you can operate as oem (as root) but it might not work properly after this stage. Don't forget to check to see if file ownership on the home directories matches properly. It might break things. To make sure it doesn't compare the version in /home and the installed version before copying over. You will need to make sure that UID and GUID (the numbers in passwd and group(users start at 1000) match up, especially for system users.

If you didn't back them up, you will need to recreate those accounts manually. For each username, do:

adduser [username]

You might need to change the ownership to match that of the user if you find that the ownership and groups have changed to a numeric value (check using ls -lh).

• For multiple users, do:

find /home/username/ -type f -exec chown -R username.username {} \;

'Remember to include the semicolon at the end.'

• For a single user, you can just do:

chown -R username.username /home/username/

You might also need to apply proper permissions, though this step should be unnecessary if you used rsync.

find /home/username/ -type f -exec chmod 644 {} \;
find /home/username/ -type d -exec chmod 754 {} \;

or, for more private users:

find ./ -type f -exec chmod 640 {} \;
find ./ -type d -exec chmod 750 {} \;

When you are finished with a backup and you've made sure that all the data is properly transferred and that the machine works, you can (optionally) remove your backup from tsbackup:

rm -r /var/tsbackup/[date-ticketnumber]