12.09.2008

On the computing front, got around to trying a couple of things.

A while back, I set up some simple software to use talk to X10 plugs from the computer so that I could schedule things (such as the Christmas tree turning on and off). I detailed it on this project page.

But now that my windows machines on all hibernate and use power saving features, the only machine that is on all the time is my linux server. I needed to move over the scheduling to the linux machine with some simple cron jobs.

And with this small software called BottleRocket, this was super easy. Now, turning on the Christmas tree (device a1) is as simple as:

> /usr/local/bin/br a1 on

Awesome.

Next up is to fix another issue caused by my windows machines constantly hibernating. Backups.

My current backup solution uses a really sweet piece of software called rsnapshot which uses a backup concept of rsync combined with hard links for super fast and super efficient backups. The only problem is, if the machines aren't on, then the backups fail. I have some of them using Wake-On-Lan (WOL) so that I can wake them up before I back them up, but if one machines takes a long time to backup, then the others could go back to sleep. Also, I have one machine which does not support WOL.

So, I started looking into BackupPC. This is a very highly rated backup system that is open source and very useful in enterprise environments. It supports the same concept as rsnapshot (rsync and hard links to de-duplicate data), but it also is set up to find machines when they are available on the network and back them up, which is what I need.

Long story longer, I fought with this ALL weekend. I actually got everything working with BackupPC. It does what I needed. But, the last part of my backup solution that I need is access to the snapshots so that I can duplicate it to an external hard drive when we evacuate for a hurricane, or just for archival.

This is where the BackupPC finally convinced me to go back to rsnapshot. BackupPC mangles the filenames in the backups. They have a reason for it: it is so they can differentiate between backed up files and their own attrib files. Plus, since their access to the data is supposed to be through the website, this should be a non-issue.

But for me, I don't even need the website. I'm the only one at home accessing backups. Also, in a lab environment, it is nice to let people mount the backup volume as read only and have access to it. With the file name mangling that BackupPC does, this does not work.

So, after an entire weekend of configuring, messing, trying, and testing, I decided to go back to rsnapshot, which of course left me with my original problem. So, I had to add some wrappers around rsnapshot to make it smart.

For the machine that doesn't support WOL, I needed to do something exactly like BackupPC. That is, I need to run a job often and check and see if the machine is available on the network. If it is, and it has been a while since it was backed up, then it needs to get backed up. If it isn't online, or if it was backed up recently, then it needs to get skipped.

In case anyone is interested, here is the csh wrapper script I am using in a hourly cron job to accomplish this. Since it is being run from cron, anything printed to stdout will get mailed to the system administrator email. So, if I detect that it hasn't been backed up in quite a while, I alert the administrator. (NOTE: The script needs a little cleaning... a few of the items in here should be in configurable variables, such as the log file to append to and the machine name. Apologies.)

rsnapshot_wrapper_some_machine.csh:
#!/bin/csh -f

if $1 == "" then
echo "usage: $0 "
exit
endif

##########################################
# #
# some_machine #
# #
##########################################

set current_time_string = `date`
echo " " >> /var/log/rsnapshot
echo " some_machine $1 attempt at $current_time_string" >> /var/log/rsnapshot

if (-f /var/log/rsnapshot_some_machine_last) then
set last_success=`cat /var/log/rsnapshot_some_machine_last`
else
set last_success=0
endif

set current_time=`date +%s`
@ threshhold_secs = 60 * 60 * 23
@ warn_secs = 60 * 60 * 72
@ time_since_last_success = $current_time - $last_success

if ( $time_since_last_success > $threshhold_secs ) then
# Wake up machine
echo "Waking up some_machine..." >> /var/log/rsnapshot
wakeonlan 00:00:00:00:00:00 >> /dev/null
echo "Sleeping ..." >> /var/log/rsnapshot
sleep 35
echo "Done sleeping. Attemping to find via ping..." >> /var/log/rsnapshot
ping -qnc 1 some_machine > /dev/null
if ($?) then
# If it has been more than a couple of days, alert the admin
if ( $time_since_last_success > $warn_secs ) then
echo "some_machine failed. Consider manual run"
echo "some_machine failed. Consider manual run" >>/var/log/rsnapshot
else
echo "some_machine not awake. Exiting..." >> /var/log/rsnapshot
endif
else
echo "Found ping. Attemping to automount..." >> /var/log/rsnapshot
cd /misc/some_machine/
touch foo
echo "Found automount... Attempting backup..." >> /var/log/rsnapshot
cd -

# Run the rsnapshot job
rsnapshot -c /etc/rsnapshot.conf.some_machine $1

# Save the successful backup time
date +%s > /var/log/rsnapshot_some_machine_last

set current_time_string = `date`
echo "some_machine $1 done - $current_time_string" >>/var/log/rsnapshot
endif
else
echo "Not enough time has passed for some_machine." >> /var/log/rsnapshot
endif


Seems to be working. I'm a little less stressed now.

4 comments:

Keith said...

If you ever decide to use backuppc you could write a "demangle" script.

Maybe a mirrored filesystem with links that point to all the files in the backuppc repo with demangled names.

Frankie said...

that's a great idea... i wonder if the backuppc folks have considered that as a feature... it wouldn't really take any (noticable) disk space.

Serge said...

It's a very old thread - 4 years ago - but i've just installed backuppc this week-end. It seemed at first glance quite impressive, but now that it's configured i'm too quite worried by this "file name mangling", for the same reasons as yours : i would like to access directly files from the command line or from a remote computer (sshfs and so on), and not only by the web front-end, but with the names mangled, it's quite a mess.

I really find this "filename mangling" thing a very poor decision. There were probably a lot of unintrusives solutions instead of that one, i'm really surprised they've choosen that one.

I'm quite deceipted by rsnapshot too for a lot of reasons ...

So in the end, after all these years, what are you running these days as a backup solution ?

Frankie said...

thanks for the comment.

actually, 4 years later I am using the following for a backup solution.

1. Crashplan - solves most of my needs, and with an arrangement I've made with my dad, we both back up to each other's houses to satisfy off-site backup in case of flood, fire, theft, etc.

2. rsnapshot - I'm still waking machines and taking a snapshot of things. This is easy for linux machines, but for the windows machines, it can be a bit of a pain to setup. But now that it is setup, the windows machines backup without me thinking about it.

3. Windows 7 backup - For windows boxes, I also use the built in backup program in windows 7. This has a number of advantages. One is that it integrates in such that I can easily right click on a file and then click "previous versions" to see a history. The other is using an image backup to restore the entire machine.

Of all these, crashplan is really the software I live by these days. The other solutions are just handy nice to haves, but crashplan is where my real risk reduction is. I can set up multiple targets for machines to backup to, and I get a regular report showing all the machines and targets and the percentage of them that are backed up. Crashplan works on almost every OS, and I've found the quality of the software to be outstanding.