Lets make a killer backup tool

[jkerr82508]

When I started trying to write bash scripts, I read somewhere a recommendation always to use the full path to any file (especially executables) used in the script. It may not always be necessary, but it does no harm.

Jim

[viking60]

I use the full path in viking.backup and I have now added the full path to ex.txt. Running it without cron to see if it still works....
And that did hurt The exclude list comes up with errors. There is a lot of "relative" stuff regarding rsync.
Installed gnome-schedule to make sure I have not fubared my syntax with spaces etc....
Nope I am getting frustrated here so I will now install dropbox.....
Yes that helped 2 GB ready for unimportant files
And important ones:
http://dl.dropbox.com/u/12631335/viking.backup
http://dl.dropbox.com/u/12631335/ex.txt
http://dl.dropbox.com/u/12631335/backup.sh

Hey we can all become friends here and do this together -how very nice
Now where do I put the nude photos of my mistress

[rolf]

I was thinking along the lines of cron needing to be told what is the working directory, something like:

Code: Select all

#!/bin/bash
cd /home/thomas
for i in `cat viking.backup`;
do rsync -azrvv --delete --delete-excluded  --exclude-from 'ex.txt'  $i /media/My?Book/thomas-pc_backup/;
done

or, alternately

Code: Select all

#!/bin/bash
for i in `cat /home/thomas/viking.backup`;
do rsync -azrvv --delete --delete-excluded  --exclude-from '/home/thomas/ex.txt'  $i /media/My?Book/thomas-pc_backup/;
done

It doesn't look to me that you followed me, there. (one too many Fleinsopp, probably )
Whether that's it, idk but there's a lot about cron and path and environment here at google:

[viking60]

I was to busy playing with Dropbox there. OFCOURSE that works - caramba! Sometimes I just need to make life difficult.
I have been rewriting the script for python for the last hour due to that stupid mistake
I'll promise you my next pizza will be without Fleinsopp
Your first option works like a charm

[rolf]

[viking60]

You really know how to pick them.
OK time to sum it up then:
We have a fully functional and flexible backup system that can be set as a cron job (which is a must for backup systems really).
We never have to touch the code in backup.sh after the initial setup. Unless your mount point for the hard disk varies.
We decide what is to be backed up in the file viking.backup and we can easily alter it every time we feel the need for it.
We can pick entire directories for our backup in viking.backup because it now is easy to exclude what we do not want.
We can make a list in the file ex.txt that will be excluded from the backup directories i.e tmp/.
We can exclude directories, files and file patterns.
Backups containing files and directories that later have been excluded are "cleansed", meaning that the old, now excluded files and directories, are deleted on the backup.
I am getting tears in my eyes here - What a success!

The files needed to achieve the wonders described above are:
backup.sh

Code: Select all

#!/bin/bash
cd /home/<your_name>
for i in `cat viking.backup`;
do rsync -azrvv --delete --delete-excluded  --exclude-from 'ex.txt'  $i /media/<your_HD>/<your_backup_directory>/;
done

This is the script that makes the magic happen. We need to tell it what to back up in:
viking.backup

Code: Select all

/home/<your_name>
/etc

You can put whatever you want in here as long as you get the syntax right:
/home/<your_name>/ will put the content of your home directory directly in your backup directory.
/home/<your_name> will put the folder with contents in your backup directory
/home/Documents/my_loveletter.txt will put one love letter in your Documents directory on the backup.etc. etc.

After we have backed up the entire home directory we might want to exclude some things we do not need. This can be temp files and cache etc.
We do that in the shape of a list in the file:
ex.txt

Code: Select all

*/tmp
*/.config/gsmartcontrol/*
*/.cache
*/.gvfs

Again this is an example you can exclude whatever you want. With one exception: You must exclude <your_name>/.gvfs if you run the backup as root (sudo or su). Or else the backup will not finish correctly. The effect being that nothing old is deleted on the backup.
In general if you test run the backup and get an error message it makes great sense to put the file on the exclusion list to ensure a smooth backup.
If you run a backup as a normal user you might get one or two files or directories that you cannot access. exclude them or chmod them!

Put all the files above in your </home/<your_name>/ directory and you can run it by typing sh backup.sh in your ~/ (home directory).
once you have tuned the lists to your liking you can automate your backup with a cronjob:

Code: Select all

sudo EDITOR=nano crontab -e

and

Code: Select all

15 20 * * * /bin/bash /home/<your_name>/backup.sh

This will start your backup a quarter past 8 pm - every day. (This is a root cronjob because we need to backup /etc)

Obey by these rules from Valhalla and you will live to fight another day
Puh

[viking60]

OK b1o we have the functionality! How about a nice applet or something to make this one fool proof?
Ah my first scheduled backup just started How do I know? Here is a trick: I have excluded the entire tmp directory from the backup so if it exists it will be deleted from the backup.
So i just created a tmp directory on the backup HD. It will be removed (first thing) when the backup starts.

[viking60]

OK I have managed to adapt the script to work with SSH too:

Code: Select all

#!/bin/bash
for i in `cat /home/thomas/viking.backup`;
do rsync -azr --delete --delete-excluded  --exclude-from 'ex.txt' ssh $i thomas@thomas-pc:/media/My?Book/viking60-server_backup/;
done

This is my viking-server connecting to my HD on thomas-pc and backing up.
Here I meet a special challenge because ssh is not to keen to transfer file names with øæå (Norwegian letters). So I figure I have to implement

Code: Select all

LANG=nb_NO.utf8

somewhere. And that is the problem: where should it go?

[rolf]

I recall some talk about setting environment values in bash scripts popping up at the previous google search, ^^^up there^^^: . Look at ->this one<-, also.

Maybe you should put a line at the beginning of your script:

export LC_ALL="nb_NO.utf8"

or

export LANG=nb_NO.utf8

idk just from looking at some of that dialog @ google...

[viking60]

Thanks I'll experiment with the placement. It does not help when putting it on top of the script. The backup runs but the deletion is skiped and the files containing exotic letters come up with errors:

Code: Select all

rsync: recv_generator: failed to stat "/media/My Book/viking60-server_backup/thomas/Dokumenter/atester/s\#370knad GF.doc": Invalid or incomplete multibyte or wide character (84)

[rolf]

Don't know if you've the time or inclination, FL, but I thought it might be a fit for you to post your killer bash app over here: http://code.google.com/p/universal-bash-script-gui/

[viking60]

Hey great finding!
I am not finished I have made a two python versions - well not even versions but still backups. One of the advantages is that I do not get the ssh lang problem.
I thought I should tidy up the variables a bit with prompting etc. too
And every time I am almost there. Gizmo walks over my keyboard. And then I start thinking that the discussion around the script and functions is more important that the fancy inputs in the script.
I will post it over there when I am "finished"