Affordable Home Backups for 10-100G Systems?

MichaelJames asks: "Ok, I have my MP3's streaming, all our digital pictures up, and a file server running on one machine in the basement. What would be the best way to do simple backups of the system and data? Get a tape drive Get a CDRW or DVDRW to backup the MP3 and pics, but use the old Zip drive for the file server data?" With drives in the 10-20 gig range only getting smaller and less expensive, what are we to do for backups, that have yet to scale well in the same range. For home systems with up to 100G of storage, what do you use to back up that much data, with a solution that's affordable to the average computer user? Have DVD writers become cheap enough for serious consideration as a backup media?

Tapes are still the way to go

[still cynical]

CD-R/CD-RW are too slow and too small, plan on spending a day or so swapping disks. You can always mirror to another hard drive, get a basic RAID card or just use a Ghost-like program to do manual backups. But tape is still cheaper per megabyte and more reliable. Sure, you can damage a tape, but it's harder to do than with a hard drive. SCSI tape drives are more expensive than another drive, but fast enough, and allow you to keep multiple versions or copies of your backup. Try that with hard drives and you need arrays. Tape starts looking REAL cheap then.

Get a USB drive

[biscuit67]

I got an external usb drive for $248 for an 80G unit. It's the cheapest, fastest, least hassle way of backing up your data. Yeah, of course, you can buy internal drives at a much lower cost but I also needed a way of carting the data to other machines easily. Even though I don't have a USB2 interface, backing up my 40G or so was complete overnight. I'm sure if I had usb2, it would have only taken 2 or 3 hours. Sure as hell beats tape. Tape is a thing of the past. It's NOT cheap.

Actually...

[JanneM]

First, on a typical system, not all data is really worth backing up; the OS and all applications can be reinstalled in the event of a crash (for Linux, it might even be slightly beneficial, as you'll reinstall newer versions and get rid of various cruft you've forgotten you have). Some data has been saved just because it's convenient or simply less bother than having to actively remove it (for me, I tend to collect old logs, various mails I never will look at again and documentation that's several revisions old). A lot of mp3:s and movies may already be burned onto CD:s. That filled 40Gb drive may actually 'only' contain 4-5 Gb of data that actually needs to be backed up.

The data I actually need to back up I manage by having the important stuff an specified directories, then mirroring them over the net to my machine at work. By doing it incrementally, there is little time or bandwith wasted.

/Janne

Another solution

[Outlet of Me]

Another solution might be to pair down what you backup. It isn't strictly necessary to backup everything on your hard drive. OSes and programs can be reinstalled, but the data that you create with them is the more precious commodity. Of course, figuring out exactly what you need to backup is the problem, and you still lose some information that isn't easily backed up or restored (like settings for programs in the Windows registry). It's taken several reinstalls of Windows before I figured out exactly what I needed to save...

Backup solutions

[Haywood68]

Go onto ebay... you can pick up a DLT (15/30GB) drive for around $150. scsi card $50. media are a little expensive (~$25 ea), but for around $275 and a little opensource backup software, you can get reliable backups.

Re:Hard Drive != Long Term Backup

[Sxooter]

In Linux, and many other flavors of unix, you could buy three drives, set them all three up as a mirror set, then pull one of the drives and take it offsite. If one drive dies, you can replace it without interuption, if both fail, you can just bring in the one from offsite and plug it in. Given the speed / price / performance of modern IDE hard drives, you could have two offsite drives AND a mirror set for less than most medium to large tape drives, and as far as I know, hard drives have a pretty long shelf like (usually >10 years) assuming they are treated well.

Re:Hard Drives

[c.r.o.c.o]

I just hate it when somebody posts exactly what I was going to post. Had to erase a good two minutes of typing because of it. :)

Anyway, that is exactly what I am doing. I have a 13Gb hdd for the system, and a 40Gb for storage (mp3s, movies, etc). Also I have a burner and one of the removable racks you mention. And in it, there's an identical 40Gb hdd used solely for backups. I keep it safe, and I plug it in every few days to copy the new stuff to it, remove the old, etc. I know that ideally I should have more backup space than hdds I'm using, but I never really run out of space. I am always writing the very important files to CDs, sometimes in duplicate. Call me paranoid, but after losing 3 years of data because of a hdd crash and a cheap CD which refuzed to be read, I'm not taking any chances. Also, all the stuff I don't need often (less than once a month) goes to CD.

One very, very important thing though. Don't cheap out on the removable racks. Make sure that at least the lid on the one you get is mettal, and there's at least a fan in the hdd tray. All racks have one fan on the rack itself (the part that gets mounted in the case). But make sure you have another one in the tray.

I used to have a rack made out of plastic completely, and with only one fan. My Maxtor 7200rpm drive was getting HOT. And I do mean hot! Then one day I ripped the IDE cable from its mount, and I had to buy another rack. This one is with metalic lid, and 2 fans inside. Now the hdd doesn't even get warm. And the difference in price between the racks was $10 canadian (about US$7.5).

Those few extra bucks are probably going to prolong your hdd life by quite a bit.

Re:Recycle and save the environment!

[yakovlev]

100GB of 5.25 inch floppies is like 28 cubic feet of floppy disk.

At:
2 MB/Floppy=.002GB/floppy
5x5x1/25 Floppies/inch^3 = 1 Floppy/inch^3
12x12x12=1728in^3/ft^3

100GB of 5.25in floppies is:

100/.002=
50,000 floppies or

50,000/1=
50,000 cubic inches or

50,000/1728=
28.9 cubic feet

That's a heck of a lot of floppy disks, espesically to back up one 5x3x10 = 150 cubic inch hard drive.

802.11 solution

[swordboy]

I'm sharing my cable modem via 802.11 with all the neighbors and since I am the local "neighborhood helpdesk technician", they often come to me for advice. Recently, one of them wanted to know how to go about backing things up properly. It dawned on me that hard drive space is abundant and most people are buying much more than they need (the person in question has an 80 gig at about 20% capacity). So I worked out a deal so that everyone is backing up to each other's PC at night on a weekly basis. The 802.11b connection keeps drive thrashing to a minimum yet provides enough speed for complete backup on an overnight basis.

I should start charging for these ideas... Can't wait for the proliferation of freenet!

Doing backups

[digitalhermit]

One thing to do first is to separate the data from the OS. I.e., create a backup set with all your reinstallation files. For my systems this is a copy of the Linux installation iso and a directory of all the patches to it. Backup your system files (depends on if you're running Winders or something else). This takes a couple CDs, but is a lifesaver if you need to recreate the system or even upgrade it.

Bringing up the system is less of a problem with newer OSes, since you can usually, at minimum, get to your data. Configuring the database, webserver, and firewalling depends on how good you are with the OS. However, when I worked at a former company there was no real plan to get a working system back in place. We were using Novell with Arcserve -- unfortunately, you couldn't get to the data without a working system.

Next I usually try to segregate rapidly changing stuff versus things that are pretty much static. E.g, my mp3 collection is relatively static. I occasionally buy a fresh CD and rip it, but I'm pretty much satisfied with my collection as it is. I put these on CDROM. It takes a while to create them, but it's cheap and safe. If you want to keep everything up to date, you can run a script to save only files not included on the CDROM.

Finally, I back up my constantly changing stuff such as CVS, MySQL database, etc. to 4MM tape. It's cheap (hardware and tape) and most drives are pretty well supported.

How I back up a File server of 320 Gigabytes

[AgTiger]

I'm classifiable as an audio addict, having taken my entire personal
collection of CD's and ripped them to MP3's at 320 bit, and wanted to
have them stored in a central place, accessible from any machine in my
home. Currently this collection is at approximately 620 full CD's of
music, and I'm pushing right at, or just above the 80 gigabyte limit.

Now when you factor in personal files, financial records, games,
downloaded material, installation software you don't want to lose,
etc...etc... Well, see for yourself. Here's my space breakdown for the
partitions on my main file server Fumus (Smoke, in Latin):

fumus:/pub/mp3 # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 3.0G 2.1G 804M 72% /
/dev/hda1 129M 6.8M 115M 6% /boot
/dev/hda5 9.8G 1.8M 9.3G 1% /home
/dev/hda6 20G 13G 6.3G 67% /pchome
/dev/hda8 40G 22G 17G 57% /pub
/dev/hdb1 75G 38G 33G 53% /pub/mp3
/dev/hda7 1.9G 20k 1.8G 1% /scratch
/dev/hdc1 74G 34G 40G 46% /pub/mp3_2
/dev/hdd1 74G 36G 37G 49% /pub/software

So, here's what I looked at:

Tape: For the size I'd need: Way WAY too expensive. When I brought
the media down into the range I'd afford, I'd be swapping tapes all week
to get a backup done. Not time effective.

CD-R: Faster, yes, but at 650 megabytes per media, same problem as
tape, only you've traded magne tic for optical.

Extra hard drives in the same machine: Originally, this is exactly what
I had done with a single file server running Reiser file systems in the
more experimental days. I got the scare (and lesson) of my life when
Reiser went a bit nuts, and started corrupting some of my data. I only
lost about one percent, but I vowed, never never NEVER again would I
backup data on a critical machine on live media in the same machine.

Okay, so here's what I finally DID select as my solution: A second
machine called Ignis (Fire in Latin) that uses the absolutely identical
configuration, right down to the types and number of drives, partition
sizes, everything. They both connect into my 100Mb network switch, and
Ignis rsync's from Fumus every hour on the hour thanks to scripts in
/etc/cron.hourly

In fact, here's Ignis' /etc/cron.hourly/rsync_with_fumus script:

rsync -arul --one-file-system --quiet fumus:/pub/mp3_2 /pub
rsync -arul --one-file-system --quiet fumus:/pub/mp3 /pub
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub/software /pub
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub /
rsync -azrul --one-file-system --quiet --delete --force fumus:/pchome /

Is this a bit extreme? Yes. But... if, gods forbid, Fumus really does
let out its magic smoke, or Ignis does catch on fire, and the physical
media were actually damaged, hopefully the damage would be limited to
*one* case, and wouldn't end up taking both machines out. Then I really
would be crying the blues.

Oh yes, and each machine is on their own 900VA UPS. I'm not playing
THAT game. :-)

Re:The question I have to ask...

[geekoid]

digital movies can be hard to replace.
Obsoleted OS's. If someone is usinf win95, and its got all the patches, you may want to back up everything, do to lack of support.
is replacing 100's of megs of .mp3s really trivial? certainly not impossible, but still a pain.
The ability to get everything back w/o reinstalling and downloading Service packs and patches is a huge plus to most people.

Backups

[4pksings]

First you must give consideration to what you really need to backup. I only do my home directory. I have a tar/gzip of /etc there. I burn my mp3's to cdrom as necessary. How much stuff do you really need to backup after that?

RAID 5 is not foolproof. I just had a server lose one disk which caused a second disk to go offline which toasted the whole array. I restored from tape.

Extra disks are not foolproof. I tried to boot SCO unix the other day and the harddrive no longer functions.

Tape has been the most reliable as long as it was written with a tape program that still exists. I have read 5 year old BRU tapes successfully.

I still use a 4MM DAT. It's small, so I don't backup much, only the necessities. But I know I can get it back!

PK

Offsite backup to large disk with rsync

[leto]

for i in `cat rsync.list| egrep -v "^#"`
do
HOSTNAME=`echo $i| awk -F: '{print $1;}'`
DIRECTORY=`echo $i| awk -F: '{print $2;}'`
DATE=`date +%A`
install -d /vol/backup/$HOSTNAME/$DATE
rsync --numeric-ids --compress --rsh=/usr/bin/ssh --recursive --archive --relative --sparse --one-file-system --compare-dest=/vol/backup/$HOSTNAME/current $HOSTNAME:$DIRECTORY /vol/backup/$HOSTNAME/$DATE
done

Then once a week we run a similar script that updates the 'current' directories and uses --delete

(rsync.list contains entries like "hostname:/some/mounted/partition")

Some Metrics...

[pbryan]

To backup a 100GB drive, you require...

- 6 DVD+RW (18 GB) discs, or
- 20 DVD-RAM (5.2 GB) discs, or
- 158 CD-R discs, or
- 72,818 HD 3.5" floppy discs

Re:Tarballz!

[BigBir3d]

one problem with tarball or gzip or whatever, if it is corrupted in transfer, you just lost everything in the tarball (1000 files?), instead of maybe one file.

hard drives fail far to often... tapes break... cd-r/cd-rw only good for a few years... Linus is right, have others mirror it for you.

me, i keep it on a website, which is not on my machine :-) (not much data though, just a few pictures)

Re:Hard Drive != Long Term Backup

[Trepidity]

But I can't see any better solution affordable for the casual home user. CD-R's are even shorter-term media - I've already had 5-year-old CD-R's become unreliable to read, while my 8-year-old hard drive (it's not even an expensive one - some cheap Connor Peripherals thing that came with my Packard Bell) is working 100% perfectly.

I'd call hard drives semi-permanent media that can be taken off-site easily, especially if they are mounted in a removable rack, as suggested. If a hard drive is used solely for backup (say, once a week?), MTBF should not be less than 10 years, even for a Maxtor.

Consider Fire, Flood.. also Electromagnetic Pulse

[GPS Pilot]

When devising a backup strategy, most people don't consider the threat of Electromagnetic Pulse (EMP).

A quick google will show you that EMP would be a cheap, relatively easy-to-build terrorist weapon, having devastating effect on our electronics-dependent economy.

A strong EMP would wipe any kind of magnetic media -- tape or disk.

I am by no means an expert. Does anyone know how to defend against it? Would placing your backup media inside a heavy metal safe provide sufficient RF shielding to prevent damage?

Freenet!

[Cryptnotic]

Use Freenet! Encrypt all your data and publish it on Freenet, using an unpublished key that only you know. Then keep requesting the files every day from various Freenet sites so that it never leaves the Freenet cache.

I'm not sure if this violates the Freenet code of ethics. But it probably does. And if everyone started doing it, it would probably kill Freenet very very quickly.

Cryptnotic

Re:Hard Drives

[comcn]

0 * * * * scp -r / [e-mail]:/home/myaccount

Great idea, but why use scp? rsync does exactly the same thing, but in a far more efficient way, as most of the files in your whole directory tree (/) are unlikely to change every day.

Something like (and I also wouldn't run it every hour for my size of fs...)

0 1 * * * rsync -aze ssh / account@backupmachine:/home/bigstoragearea/

Re:Tarballz!

[spreadthememe]

Fireproof safes are for paper, not for plastic or any sort of sensitive equipment. If a hard drive is heated above (say) 140F for more than a few minutes, the data is gone. Allowing for the fact that home fires typically reach temperatures in excess of 1200F, many of the drive components will melt.

Fireproof safes protect combustible material from combustion. They do not protect meltable (?!) material from melting. Put your stock certificates and your marriage license in the safe. Put the backup harddrive somewhere else.

Re:Hard drive . . . rsync

[kalinh]

yeah yeah, I was trying to figure out how you'd do this in the back of my mind since I posted it. I had to say "may" jsut to avoid someone complaining about not being able to trust any of their friends.

I was thinking that you might be able to use something like the Cryptographic File System to encrypt the entire volume and that that level of abstractin would still let rsync be used effeciently. I have *no* experience with CFS though, no idea how it would affect rsync, or what sort of access controls it uses or are available for it. I still wouldn't quite call it impossible.

Seriously though, good friends are still an easy alternative for most people.

The problem is also potentially the solution

[smashy76]

The real problem here is the absolutely insane rate of hard drive size increases. Hard drive capacity has been increasing at greater than 100% per year since IBM's GMR heads came out. You can now buy a 70TB(yes, terabyte) emc drive array (384 * 180GB seagates)! Maxtor is coming out with a 160GB! ATA drive (once they finish addressing the 28 bit sector address limit). I agree, this poses the issue of "do we really need all of this space?", but data needs will always scale.

Therefore, since the fault of the problem lies with hard drives hugely outpacing every other form of recording medium in rate of capacity increase, the only reasonable solution will soon be (if it isn't already) to use hard drives as the backup medium. Yes, I know, hard drives combine the media with the mechanism and that is normally a big no-no, but in this case I think the monetary facts must be faced. In order to get around the media/mechanism issue as well as the off-site storage issue while not emptying our wallets, I think multi-site dual-hard drive-backup is in order.

At any given site, one would use large ATA drives in the backup server (most likely in an external hotswap cage for the corporates) in place of a tape library/cdr/etc. But, in this case, we should make two copies (use two drives!) to hopefully get around the combined media/mechanism issue. Trust me, the cost of doubling up will be far less than an equivalent media based solution. Off-site fire-proof backup companies could start taking hard drives (being considerably more careful than with tapes, of course). If you want, you could encrypt whatever was on the drive. This scheme scales from the smallest home needs(keep the drives in the safety deposit box) to the largest corporates, and makes the most sense monetarily.

RAID 5 + replicated backups on disks

[aoliva]

Being one of the maintainers of Amanda (www.amanda.org), I'd always been of the opinion that tape backups were the only way to do backups seriously.

The recent explosion in disk capacities and decrease in prices got me to rethink this, just when it came the time for me to set up a home office. When I compared the cost of a reasonably-good tape drive and a number of tapes large enough for me to get at least a month of backups in rotation, and computed how many 60GB disks I could buy with that money, the solution was clear.

I ended up setting up 3 machines with 4x60GB each. They're all on RAID 5, such that if any single disk fails, the machine keeps running (actually, I have /boot on RAID 1 over the 4 disks and / on RAID 1 over 2 of the disks and an alternate root to test upgrades over the other 2, but you get the point). This got me blazingly fast disk access, that tapes would never help me get :-)

I get all my backup-worthy data rsynced over to the other machines daily or so. I plan to start playing with Inter-Mezzo soon, so that I don't have to remember to run these backups, and so that I don't run these backups on the wrong direction.

But that's not all. With the mind-boggling amount of disk space I could afford, I could (actually, I will, but you get the idea) set up Amanda to backup interesting portions of my home directory to disk, and also replicate this to at least another of my local machines. Such backups can use software compression, such that they don't take as much space as live data. Also, I intend to use another form of compression: instead of backing up CVS trees (I've got loads of check outs), I'm going to back up only local changes to files, so that, in case of disaster, I can still download the original CVS tree and re-apply patches. But this is still a plan, not something I've got running.

Finally, I've got yet another disk on a remote site, to which I rsync not only the interesting portions of my data, but also my backups. I could convince someone else to run this remote backup site for me by offering this person the speed up of RAID 0 over two disks (one of those mine). As for keeping the secrecy of the data on this remote backup site, I'd just get the backup files encrypted, no big deal.

I can strongly recommend this solution: I got pretty much as much data safety as could be expected from a tape-based backup, without any of the hassle of having to switch tapes and moving them off-site and back on-site, and with the bonus of very fast access to local data, unlikely donw-time and fast recovery except in case of total disaster (i.e., having all of my local machines failing, in which case I'd have to either download my backups from the remote site over the net or, more likely, take a replacement machine over to the remote backup site and copy files over a fast local network connection, or from disk to disk.

As for getting 4 IDE disks into a single machine, don't even think of using only the 2 IDE controllers that come on most motherboards these days (for RAID set-ups, you really want one IDE disk per controller). There are a few good motherboards that come with 4 IDE controllers, so that you can even have a CD-ROM and/or a CD-RW in addition to the 4 disks. If you can't find such a motherboard that suits your needs, you can always get one of those PCI cards that adds 2 IDE controllers to your machine.

As for the problem of fitting so many disks in a standard ATX chassis, it can be done. Cooling may be a problem, but a good cooler has been good enough.

All in all, I'm very happy with this arrangement. It was not cheap, but it was not as expensive as a tape-based solution, and it's far more flexible, way faster and it doesn't require any baby-sitting after you get it going. And I can keep far more backup history than I thought it was going to be possible.

Re:Hard Drive == Long Term Backup

[evilviper]

The chances aren't as bad as you make them sound...

Let's just say you have an internal RAID system with, oooo, 4 drives, along with a removable drive to backup everything. The problem lies in the whole trusted/shared medium concept. If there is a surge passed along the case, the SCSI/IDE cable, or through the power-supply cabling, not only will ALL of your drives get toasted, but if you have the backup-harddrive connected to the system (actively archiving your data, finished archiving and waiting for you to remove it, or just because of a BAD practice of never actually removing the removable hard drive) you will loose your backup hard drive as well.

While RAID is a good thing, multiple hard drives are still at the mersey of everything they are connected to. Using such a system as your only backup is a bad idea that happens too often. Having a removable hard drive is an option, but the work involved really makes other solutions much more viable, especially on a large-scale (and on a small-scale, people are lazy!).

I propose a network-based automatic backup system for most people. You simply have your main system automatically backup it's data over the network to another system (systems with low number-crunching capabilities can be put back to work here). Of course, you would want to maintain at least 2 concurrent backups in-case the main system dies during the said backup. The benefit of network backup are that human intervention is not required (the user and administrator don't need to do much of anything after initial setup) and off-site backups can happen transparently (just send the data to the other office down the street, across town, whatever.

Speed of the network may appear to be a problem, but 100 GigaBytes (UNCOMPRESSED) can be transfered in 2.22... hours over 100Base-Tx. First of all, it's likely you'll be compressing that data, which on average halves the size, and so the time is halved as well. Secondly, Gigabit over Cat-5 is available at $45 per NIC, making backups take one-tenth that time. And finally, an encrypted SSH, IPSec, PPTP, (etc) tunnel could be established that would ensure the data is kept private. Data security is much more difficult when you have multiple copies of it unencrypted in a conviently sized package. You are just saying 'steal me'.