Affordable Home Backups for 10-100G Systems? 690
MichaelJames asks: "Ok, I have my MP3's streaming, all our digital pictures up, and a file server running on one machine in the basement. What would be the best way to do simple backups of the system and data? Get a tape drive Get a CDRW or DVDRW to backup the MP3 and pics, but use the old Zip drive for the file server data?" With drives in the 10-20 gig range only getting smaller and less expensive, what are we to do for backups, that have yet to scale well in the same range. For home systems with up to 100G of storage, what do you use to back up that much data, with a solution that's affordable to the average computer user? Have DVD writers become cheap enough for serious consideration as a backup media?
Tough problem (Score:3, Informative)
One thing some people do is back up their HD to a second HD.
Zip disks seem practically useless these days--recordable CD is just too cheap and universal by comparison.
Tape drives are the high-end solution, but expensive.
raid 1 (Score:1, Informative)
Tarballz! (Score:5, Informative)
Another hard drive (Score:3, Informative)
Actually, what I do is make the new (largest one I own) drive the backup drive, put the old backup drive into use as the primary drive, and retire the smallest one. Just make sure the new drive is as large as the others added together.
CD-R's are OK, but why bother with the hassle? Just run a cron job to copy the files every evening/hour/whatever.
Onstream (Score:4, Informative)
Pros:
Can be found for under $100
Linux Support!
Cons:
Tapes are expensive
Tape didn't quite catch up... (Score:3, Informative)
Re:Hard Drives (Score:4, Informative)
The drive enclosure was a bit more expensive than the rack mentioned above (under $100 with shipping) however it did come with a two sets of enclosures - one for the drive itself to use externally, and another to put in a PC cabinet if you want to hot swap the drive with it instead of using the card slot.
Re:Hard Drives (Score:3, Informative)
Does anyone have a better bang/VS/buck solution? I know CDRW's are quite cheap, but one has to factor in speed, as well as human intervention. Swapping disks requires attention, which requires a wage. The backup system simply is launched every friday after work, and does it's thing on it's own time, without need for people. And with 100GB, no sanely priced tape drive comes close.
Here (Score:1, Informative)
Once a week, remote dump your house to a central backupserver. No only does it offer compression, its fast and very easy.
DV Camcorder + firewire (Score:3, Informative)
http://dvbackup.sourceforge.net
Cheap CD backup. (Score:1, Informative)
Given that MP3's and digital pictures don't change with time like databases, documents etc, why not do what we do and back the whole lot up once. Then as you download new MP3's, pictures, whatever. Put them into a directory say Not_Backed_Up, and burn that to CD also when you have enough to fill a CD. Then migrate this with the rest of the data and start fresh with the next CD to be filled.
FWIW if you have the money to buy 2 identical drives RAID 1 might be the way to go.
Incremental backups... (Score:4, Informative)
However, I would suspect that most users don't change a huge percentage of their HD's content on a daily basis, unless you are routinely d/l'ing or ripping MP3s and MPGs on a daily basis (and I note that when I do generate that kind of traffic, it is usually because I am making a compilation CD, and while this does generate a few GB of "new" files on my HD that day, that data doesn't need to be backup up because I've got the original CDs anyway).
As a result, it seems to me that a reasonable solution is to create a "baseline" backup, say to a CD or DVD, at system install time, when there is (relatively) little on the disk, and then each day (or week, depending on needs), do an incremental backup of changed data only to another CD.
This approach is obviously quite inefficient if you have a complete HD failure, in that you have to recreate a new drive by starting with the first backup CD and then restore EACH ONE thereafter until the final CD restores the disk to it's last backed-up state, but for a more common problem of losing or corrupting an individual file, since that is more likely to happen with a recently modified than a remotely modified file, you are likely to be able to restore a last good version within only a few CD's of the most recent incremental backups.
Hard drive . . . rsync (Score:5, Informative)
I agree.
What hasn't been mentioned is rsync [samba.org], which makes disk to (local or remote) disk backups fast and easy.
It is trival to set up a second disk that is a "stale" mirror of your primary disk(s) that backs up nightly, and will boot off a floppy. This captures some of the advantage of RAID (quick recovery) while being an actual backup, not just fault tolerance.
Rsync can use ssh as a transport, so you can securely back up remote disks as well.
-Peter
RAID is _not_ backup ! (Score:5, Informative)
RAID offers good protection for some things: hardware failure (ie: HD crash) and uptime. That aren't the only woes, however... You can loose data in a lot of ways:
Disaster (fire, quake, flood, nuff said)
Hardware failure (disk, controller, ...)
OS failure (FS corruption, ...)
Application failure (User space applications malbehaving, virii, ...)
User failure (accidental deletes, experimental children - trust me on this one ;-)
:)
...) are a plus, but more cumbersome.
RAID will protect you from the second, but will happily add nothing in case of any of the other failures. Backing up to another media is a necessity.
Adding an extra disk (or two, or three), and some tar/cpio cronjobs will add basic protection. (No disaster recovery for you, unless it's off-site
Removable harddrives (firewire, frames,
Tape is considered a more 'trustworthy' backup medium because the mechanism and data storage are separated (ie: tape drive / tape), while in a HD it's in one single package, and it's not as easy to replace the logic board/stepper motor if this flunks. With tape it's easier: just get a new tapedrive.
Anyhow: don't rely on RAID to save your data - it won't.
Re:What I did - Linux Software RAID-5 (Score:3, Informative)
Cygwin has RSync for Windows. Works really well.
Back up hard drives not a good idea (Score:1, Informative)
If you a backing up things that would be inconvenient to loose but not mission critical then this method may be fine.
If your are backing up irreplacible data then I would suggest using a different method.
I have experienced the meltdown of two systems where in both cases it was the power supply that failed. Unfortunately, the when the supplies failed they took out pretty much every component in the systems (including the hard drives). I can only guess at the voltage that was pumped through everything but between the two systems I found 5 chips that were cracked or blown apart (1 of them on a hard drive controller board)!
These incidents happened at different times, in different locations, on different types of computers.
If you do use hard drives for backup consider removable hard drive bays and don't leave the back up hard drives in your system.
Just my 0.02.
--
Can't be bothered to login.
Try DLT... (Score:5, Informative)
Pick up a DLT2000XT (15gb native) off ebay for about $200. Tapes are dirt cheap, about $5/ea and the media is extremely durable, nearly indestructible.
Pick up a DLT1 (40gb native) off ebay, about $500. Tapes are moderately expensive at around $20/ea, but again the media is extremely durable.
DLT is industrial strength backup, the drives are built like tanks and the tapes can take incredible abuse.
Its all standard SCSI and works great with linux, no problems whatsoever.
I considered buying hard drives for backups, but they are far too fragile for long term backup and off-site storage. Most drives arent designed to be spun up and down lots of times either.
Last thing you need is for your backup harddisk to go splat when youre trying to power it up to restore your main system from a data loss.
With DLT, this isnt likely to happen.
Re:Hard Drives (Score:2, Informative)
That's actually pretty funny - the reliability metric isn't reliable.
I agree, don't trust hard drives. They're a bad solution for backups, as they don't store as well as real backup media. However, RAID systems are also not suitable because you can't archive data in case your system burns, and it doesn't protect you from filesystem corruption or inadvertent "rm -rf" commands.
They're steep in price, but DLT or AIT are optimal. The newer models are capable of 100GB, except you can't get that much on there if the data is uncompressible (like MP3s are). If you wanted to be able to back up less than 100GB, you could probably get an older DLT/AIT for a grand or less. I've seen 8 or 10-slot AIT changers for less than $3kUS, a config which can store almost a terabyte. If you can afford that kind of cash, you probably won't have to worry about backups for a long time to come.
Re:Damn Good Question- Tape works better than CDRW (Score:1, Informative)
Dantz Retrospect or Veritas to a yamaha cdrw, to a sony cdrw, and to a creative cdrw. IMHO, the firmware in these drives seems to hang if there is a media problem, and dantz just acts like its waiting. Veritas blue screens.
If you are going to try this, use cdrw's or you may end up with 35 coasters.
On the other hand, I shelled out $140 for Dantz because it supported my Onstream drive. This works the best, but sometimes the $250 computer i dedicated to doing backups hangs, expecially when I back up the full 120 gb.
raid backup for physics (Score:1, Informative)
Re:Tough problem (Score:4, Informative)
This brings the price of a DVD burner down to $150, since the drive is 600 coon skins before rebate. At that price, and if the playstation 2 drops in price this christmas, i sense lots of burned games in my future...
Digital Camcorder? (Score:3, Informative)
This is not exactly a dirt cheap solution, but if you have/want a digital camcorder anyway, there's only the cost of extra tapes.
Make sure the camcorder works with dvbackup before buying one though. It doesn't work with any JVC's that I know of, or at least not mine
Disk or Amanda (Score:1, Informative)
If your data changes quite often, and you need the ability to do "point in time" recoveries of particular pieces of data, try Amanda [amanda.org].
Re:Tapes are still the way to go (Score:3, Informative)
Tapes are 113$US (shipped) for a 4 pack.
For those of you following along at home, that is ~300$US for 120GB of backup.
400$US for two sets of 120GB.
Not bad, and it is a DAMN FAST drive...
ZIP, PGP, FTP (Score:2, Informative)
Its not the best of all solutions, but I don't have anything I would die without and it was really cheap (free).
Re:Hard Drives (Score:3, Informative)
Some thoughts:-
1) IME, Racks cause heat build-up and kill HDDs. I won't use them and I recommend others don't either.
2) If you're going to permanently mount and power the drive, then it stands as much chance of dying through power cycles and usage as your main drive.
3) If you're going to leave the drive unpowered most of the time, then you probably won't bother backing up because it'll require a shutdown, an opened case, a boot and another shutdown.
IMHO, If you're going to use another drive, you might as well go for RAID and have the backups done automatically and continuously.
--
Re:What about fire? (Score:1, Informative)
3ware Escalade = 420 gigs of RAID5 for $1100 (Score:2, Informative)
It's a PCI card with *8* IDE drive slots, which you can configure in a RAID 5 array for huge, failsafe backups.
I've got 8 60-gig IDE drives on it, in a RAID 5 array. Gives aboout 420 gigs. Shows up as a SCSI device in OpenBSD. Works great with Linux.
Churning away wonderfully.
I've backed-up 200 gigs of files on it so far.
420 gigs of RAID5 storage = $1100 USD. ($300 for the card. $800 for 8 60-gig drives.)
Here's my post on the OpenBSD list about it [monkey.org]
Re:Hard Drive == Long Term Backup (Score:5, Informative)
Your best option is to put all data on a 2-disk mirrored RAID and use another drive as a removable for an off-site or fire-safe backup. The probability of 3 hard disks failing simultaneously, one not in use, is so incredibly small it's laughable. And for that non-zero chance, if it happens, you can pay to have the spindle of one of the failed drives transferred to a new drive in a clean room.
NNTP backup (Score:3, Informative)
Advantages:
- Standardized TCP-based protocol, uses authentication.
- Lots of free "backup software" available - e.g. pan, newspost, tin...
- Centralized backups over the network! Only one server directory to back up.
- Backups are automatically dated by the server, and can be signed and even encrypted if you integrate pgp into your posting script.
Disadvantages:Re:Hard drive . . . rsync (Score:2, Informative)
Then rsync over ssh at night, use RSA "passwordless" Authentication as explained here [ibm.com] , set up a nice little script, cron it you've got reasonably accessable cheap off-site backups.
Kalin
/usr/share isn't for user sharing of files (Score:2, Informative)
You know, the funny thing about /usr/share is it is supposed to be where you put "data" which can be shared by users, right?
No. /usr/share and /usr/local/share are for files installed by an application and shared between the different binary architectures the application can be compiled for (such as map files for a game). To share data among users, use /home/johndoe/pub, or use groups.
Re:/home/dir (Score:3, Informative)
I believe this was the original purpose of
The way I do it here.... (Score:4, Informative)
I have a "backup" hard drive in my server. This drive is always unmounted so that there is no chance of filesystem corruption from the operating system.
I just use a crontab to run a simple script that mounts the drive and coppies whatever specified backup files to it, then unmounts it. The same method slightly modified could be used to back up this same backup disk to another location on the network on regular intervals.
Iomega Peerless (Score:2, Informative)
rdiff-backup (Score:1, Informative)
http://www.stanford.edu/~bescoto/rdiff-backup/
From the web page:
What is it?
rdiff-backup is a script that backs up one directory to another. The target directory ends up a copy of the source directory, but extra reverse diffs are stored in a special subdirectory of that target directory, so you can still recover files lost some time ago. The idea is to combine the best features of a mirror and an incremental backup. rdiff-backup also preserves subdirectories, symlinks, special files, permissions, uid/gid ownership (if it is running as root), and modification times. Finally, rdiff-backup can operate in a bandwidth efficient manner over a pipe, like rsync. Thus you can use rdiff-backup and ssh to securely back a hard drive up to a remote location, and only the differences will be transmitted.
RAID5 (Score:1, Informative)
real "off-site" solutions (Score:2, Informative)
Too bad iDrive & Freespace.com went offline
Re:Hard drive . . . rsync (Score:3, Informative)
I rarely use the word impossible, but I think this is. He, presumably, is root. The system has to write to the disk . . . I don't see how to overcome this.
OTOH, you could encrypt the files prior to transmission. This creates the new problems that 1. the efficiency of rsync is lost unless you do some kind of "chunky" encryption and 2. there is no obvious way to do this.
I guess you could use dump [sourceforge.net] to do a full backup periodically, then encrypt and upload that, then do incremental dumps nightly and encrypt and upload that. Not pretty.
-Peter
DVD-R(W)/CDRW incremental backups (Score:2, Informative)
I've got all the hardware, but I can't seem to find a piece of software that's built for incremental backup to non-tape media. The closest I've found is NTI BackupNOW [ntibackupnow.com], but after much frustration I discovered that even their software won't support DVD-R(W) for some months to come.
Has anyone actually **SUCCEEDED** in setting up such a backup system? Bonus question - has anyone had to restore data from this kind of setup?
Re:Hey old-timer... (Score:3, Informative)
Of course you'll still need something like 150 CDRs to get everything, inceremental or not - and that's just the first dump that will keep growing.
BTW, to the other replies to this - one of the companies I worked for, we did full, weekly backups of all the important systems (ie the actual OS and software - not the data) like the domain controller, database servers, fileservers, firewall, the backup servers themselves, etc. and then daily incrementals on them.
For the actual data on the file and database server, plus everyone's personal workstations (something like 100 people) it was a full backup once a month (maybe less) and nightly incrementals, which worked great because here people usually wanted just a couple of files their dumb asses deleted a week ago (usually perfectly timed to want the tape that just went off to off site storage :) ) All in all that achieved very good granularity. Of course the incremental tapes were recycled after about two months or so.
The database servers were a bit more tricky so additional full dumps were done manually, when, let's say, 500,000 new compounds were imported. There were also databases attached to data collection robots that had their own separate system going.
All in all, this took three backup servers (one for the data collection databases, one for individuals PCs and one for the other stuff) with their respective tape loaders, plus individual tape drives for the main database servers (we also had things like SGI workstations, but the scintists on those did their own thing), and the full backups did take up almost the entire weekend. But it was setup well enough that it took only a few minutes a week to maintain the whole thing - get a new tape, write on it what the server tells you to, put it in the loader, take the specified one out, send it off - you are done. And it did work - they had to recover after a fire once (before my time) and did it without any problems.
Anyway, the point of the story was this - it's not as simple as "do incrementals" or "no! do full backups" when you are talking about actual companies, it's a bit more tricky than that.
Ecrix VXA-1 (Score:3, Informative)
The drives support three different flavours of media - 12GB, 20GB and 33GB, and come with IDE, SCSI, or Firewire interfaces. The IDE is cheapest, at $699, with the media costs being $80, $45, or $35 depending on the capacity.
Is it the absoulte top of the line as far as tapes go? No. But the cost can't be beat. And you get a reasonably fast (3MB/sec) drive with very nice reliability (take a look at the independent testing on their site; e.g. soaking a tape in hot coffee for a minute, rinsing it, drying it and reading the data off.)
They also recently merged with Exabyte, who will be positioning it as their new value solution. Hopefully the Exabyte name will expand the market enough to drive the prices down on these even further.
Matt
reduendancy consists of many levels (Score:2, Informative)
1) Use resierfs -- it'd stable now and recovers better then ext2 in small "incidents"
2) RAID 5 array
3) Run a nightly script that hard links all the files into a . (hidden) directory -- protects against rm -rf''s
4) Run mirror in every directory from cron-- if you lose or mangle one file you cna recontruct it from the contents of the mirror and the other files (works a lot like XOR in raid 5 arrays -- aka a RAID for files).
5) DDS3 incremental back-up; complete backup at regular intervals -- one set of tapes stored off site.
This protects you from multiple levels of failures -- with the catastrophic redundancy being the tapes. You don't always want to rely 100% on the tapes for all your redundancy.
DLT (Score:2, Informative)
Re:tapes vs. second hard disc drive (Score:2, Informative)
Don't listen to him. Tapes always let you down!
Some of the problems with tape are:---
Granularity -- i.e. you accidentally deleted "xmascard.lst" how do you know which tape it was on and how long will it take to scan several tapes to find your 2K file?
How can you be sure the tape is readable. In my experience something like 5% of backup tapes cannot be read by any drive except the one that wrote the tape.
No standard format, no backward compatabilty, short market life for thevarioustecnoligies. You may be very smug that you have an "acme" format backup stored in a safe deposit box, but after your house is flooded (mentioning the f*r* word upsets some people) you discover "acme" drives are no longer sold anymore, you just have to hope someone is trying to unload a working drive on e-bay.
Expense -- the drives, the tapes etc. will all cost several times as much as a couple of external hard drives.