Affordable Home Backups for 10-100G Systems? 690
MichaelJames asks: "Ok, I have my MP3's streaming, all our digital pictures up, and a file server running on one machine in the basement. What would be the best way to do simple backups of the system and data? Get a tape drive Get a CDRW or DVDRW to backup the MP3 and pics, but use the old Zip drive for the file server data?" With drives in the 10-20 gig range only getting smaller and less expensive, what are we to do for backups, that have yet to scale well in the same range. For home systems with up to 100G of storage, what do you use to back up that much data, with a solution that's affordable to the average computer user? Have DVD writers become cheap enough for serious consideration as a backup media?
Hard Drives (Score:5, Insightful)
Given that a 100G hard drive is cheaper than any removable media solution, why not just buy another hard drive and install it in a removable (not hot-swappable, just removable) rack?
Racks are $20 at my local Fry's, and inserts for other hard drives are $10.
Re:Hard Drives (Score:4, Informative)
The drive enclosure was a bit more expensive than the rack mentioned above (under $100 with shipping) however it did come with a two sets of enclosures - one for the drive itself to use externally, and another to put in a PC cabinet if you want to hot swap the drive with it instead of using the card slot.
Re:Hard Drives (Score:3, Informative)
Does anyone have a better bang/VS/buck solution? I know CDRW's are quite cheap, but one has to factor in speed, as well as human intervention. Swapping disks requires attention, which requires a wage. The backup system simply is launched every friday after work, and does it's thing on it's own time, without need for people. And with 100GB, no sanely priced tape drive comes close.
What about fire? (Score:2, Insightful)
Re:What about fire? (Score:2)
Re:Hard Drives (Score:2)
How about this one? (Score:2)
Supposedly, it performs just like a SCSI drive, but it's IDE. A couple of these in the aforementioned rack in a mirrored RAID combo would make a perfect backup.
I'm definitely swapping out my current configuration for two of these once I can afford the $600.
Re:How about this one? (Score:2)
Hard Drive != Long Term Backup (Score:2, Redundant)
Real backup is done on semipermanent media (>10 year storage) in a format that can be taken off site easily.
Re:Hard Drive != Long Term Backup (Score:2, Interesting)
RAID != Backup (Score:4, Insightful)
However, a HDD in an external enclosure could be considered a valid backup, however, for true redundancy, you better have two drives you swap, and you better be doing surface tests regularly. A drive, properly treated, should last many, many years. Also, you could combine a drive with monthly or quad-yearly backups to CD-R, just make sure you do your research on the inks used in CD-R disks, some don't last as long as others.
Just my $.02
Re:Hard Drive == Long Term Backup (Score:5, Informative)
Your best option is to put all data on a 2-disk mirrored RAID and use another drive as a removable for an off-site or fire-safe backup. The probability of 3 hard disks failing simultaneously, one not in use, is so incredibly small it's laughable. And for that non-zero chance, if it happens, you can pay to have the spindle of one of the failed drives transferred to a new drive in a clean room.
Re:Hard Drive == Long Term Backup (Score:3, Interesting)
Let's just say you have an internal RAID system with, oooo, 4 drives, along with a removable drive to backup everything. The problem lies in the whole trusted/shared medium concept. If there is a surge passed along the case, the SCSI/IDE cable, or through the power-supply cabling, not only will ALL of your drives get toasted, but if you have the backup-harddrive connected to the system (actively archiving your data, finished archiving and waiting for you to remove it, or just because of a BAD practice of never actually removing the removable hard drive) you will loose your backup hard drive as well.
While RAID is a good thing, multiple hard drives are still at the mersey of everything they are connected to. Using such a system as your only backup is a bad idea that happens too often. Having a removable hard drive is an option, but the work involved really makes other solutions much more viable, especially on a large-scale (and on a small-scale, people are lazy!).
I propose a network-based automatic backup system for most people. You simply have your main system automatically backup it's data over the network to another system (systems with low number-crunching capabilities can be put back to work here). Of course, you would want to maintain at least 2 concurrent backups in-case the main system dies during the said backup. The benefit of network backup are that human intervention is not required (the user and administrator don't need to do much of anything after initial setup) and off-site backups can happen transparently (just send the data to the other office down the street, across town, whatever.
Speed of the network may appear to be a problem, but 100 GigaBytes (UNCOMPRESSED) can be transfered in 2.22... hours over 100Base-Tx. First of all, it's likely you'll be compressing that data, which on average halves the size, and so the time is halved as well. Secondly, Gigabit over Cat-5 is available at $45 per NIC, making backups take one-tenth that time. And finally, an encrypted SSH, IPSec, PPTP, (etc) tunnel could be established that would ensure the data is kept private. Data security is much more difficult when you have multiple copies of it unencrypted in a conviently sized package. You are just saying 'steal me'.
Re:Hard Drive != Long Term Backup (Score:4, Insightful)
True -- but given the article's "affordable", "home", "10-100GB" parameters, I'd be quite happy regarding hard drives as a real solution.
Don't expect one hard drive to last you 10 years, because 10 years from now, systems with 40-pin IDE won't exist. (And likewise, neither will readers for the tapes you purchase today. When was the last time you saw a DC600 cartridge tape drive available?)
If you're talking longterm storage, leave your "backup" drive somewhere secure, and expect to replace it every 3-5 years. (That'll probably be a 500G serial IDE drive 5 years from now, a terabyte-range solid-state device 10 years from now, and a petabyte-range holocube 20 years from now.)
Re:Hard Drive != Long Term Backup (Score:3, Interesting)
I'd call hard drives semi-permanent media that can be taken off-site easily, especially if they are mounted in a removable rack, as suggested. If a hard drive is used solely for backup (say, once a week?), MTBF should not be less than 10 years, even for a Maxtor.
Re:Hard Drives (Score:2, Redundant)
Sure if it's removable then you're probably mostly OK, but if some power accident fries your box, and your backup hard drive is in there, oh well.
Another principle of good backups is to have another copy in another location, since having an extra hard drive won't help you if your house burns down.
For average users, I think the only real solution to backups is going to come when network storage is dirt cheap and bandwidth is just as cheap. God how I'd love to backup my data with one entry in a crontab:
0 * * * * scp -r / myaccount@bigcheapstorage.com:/home/myaccount
(Using the SSH keysystem to avoid entering passwords, of course)
Re:Hard Drives (Score:2)
Of course, the poster was looking for a home backup, so data retention probably isn't high on his/her list.
I have way too many god damn machines on a home network (windows, linux, Solaris/Sparc, OSX, etc) so simply backing up to a seperate drive doesn't look like it would cut it. Tapes would be best for a med/large *home* network, but it ain't exactly the cheapest. It might be worth looking at e-bay, dot bomb auctions to see if you can pick up a tape drive for cheap.
he thought of that, you know. (Score:2)
Another principle of good backups is to have another copy in another location, since having an extra hard drive won't help you if your house burns down.
OK, so buy more than one back up HD for each application and don't leave it in the box. If you want, take the extra copy home or put it in a safe deposit box. His point is that the extra HD is still cheaper than removable media for most people right now.
This is all lost on me right now. The largest projects I have easily fit on a CD, source executable and data files uncompressed. My photos are largely segregated by where and when they were taken and are archived that way. The one or two home movies I've made are no more than 10 minutes long and of such pathetic quality that they are less than 100MB each. This may change in time, but right now the 10 cent CD and a central FTP server are more than adequate. Thingy goes to FTP, then gets CDed several times and removed.
I doubt I'll ever pay to have someone else paw through^H^H^H^H^H^H^H^H^H store my data.
Re:Hard Drives (Score:3, Interesting)
Anyway, that is exactly what I am doing. I have a 13Gb hdd for the system, and a 40Gb for storage (mp3s, movies, etc). Also I have a burner and one of the removable racks you mention. And in it, there's an identical 40Gb hdd used solely for backups. I keep it safe, and I plug it in every few days to copy the new stuff to it, remove the old, etc. I know that ideally I should have more backup space than hdds I'm using, but I never really run out of space. I am always writing the very important files to CDs, sometimes in duplicate. Call me paranoid, but after losing 3 years of data because of a hdd crash and a cheap CD which refuzed to be read, I'm not taking any chances. Also, all the stuff I don't need often (less than once a month) goes to CD.
One very, very important thing though. Don't cheap out on the removable racks. Make sure that at least the lid on the one you get is mettal, and there's at least a fan in the hdd tray. All racks have one fan on the rack itself (the part that gets mounted in the case). But make sure you have another one in the tray.
I used to have a rack made out of plastic completely, and with only one fan. My Maxtor 7200rpm drive was getting HOT. And I do mean hot! Then one day I ripped the IDE cable from its mount, and I had to buy another rack. This one is with metalic lid, and 2 fans inside. Now the hdd doesn't even get warm. And the difference in price between the racks was $10 canadian (about US$7.5).
Those few extra bucks are probably going to prolong your hdd life by quite a bit.
Re: (Score:2)
Re:Hard Drives (Score:2, Insightful)
Re:Hard Drives (Score:3, Informative)
Some thoughts:-
1) IME, Racks cause heat build-up and kill HDDs. I won't use them and I recommend others don't either.
2) If you're going to permanently mount and power the drive, then it stands as much chance of dying through power cycles and usage as your main drive.
3) If you're going to leave the drive unpowered most of the time, then you probably won't bother backing up because it'll require a shutdown, an opened case, a boot and another shutdown.
IMHO, If you're going to use another drive, you might as well go for RAID and have the backups done automatically and continuously.
--
Re:Hard Drives (Score:2, Informative)
That's actually pretty funny - the reliability metric isn't reliable.
I agree, don't trust hard drives. They're a bad solution for backups, as they don't store as well as real backup media. However, RAID systems are also not suitable because you can't archive data in case your system burns, and it doesn't protect you from filesystem corruption or inadvertent "rm -rf" commands.
They're steep in price, but DLT or AIT are optimal. The newer models are capable of 100GB, except you can't get that much on there if the data is uncompressible (like MP3s are). If you wanted to be able to back up less than 100GB, you could probably get an older DLT/AIT for a grand or less. I've seen 8 or 10-slot AIT changers for less than $3kUS, a config which can store almost a terabyte. If you can afford that kind of cash, you probably won't have to worry about backups for a long time to come.
Tough problem (Score:3, Informative)
One thing some people do is back up their HD to a second HD.
Zip disks seem practically useless these days--recordable CD is just too cheap and universal by comparison.
Tape drives are the high-end solution, but expensive.
Re:Tough problem (Score:2)
Re:Tough problem (Score:4, Informative)
This brings the price of a DVD burner down to $150, since the drive is 600 coon skins before rebate. At that price, and if the playstation 2 drops in price this christmas, i sense lots of burned games in my future...
Maybe it's not fancy, but... (Score:5, Funny)
Re:Maybe it's not fancy, but... (Score:5, Funny)
Re:Maybe it's not fancy, but... (Score:2, Redundant)
Recycle and save the environment! (Score:5, Funny)
Hey, I've got all these 5 1/4" floppy disks sitting in boxes in a back closet. I bet if I added them all up, they would amount to close to 100 GB.
Re:Recycle and save the environment! (Score:2)
I can just see this as a backup solution
Re:Recycle and save the environment! (Score:2)
I bet they wouldn't! I remember seeing the stack of floppies that = 1 Travan tape.. It was 6 feet high.
Re:Recycle and save the environment! (Score:2, Interesting)
At:
2 MB/Floppy=.002GB/floppy
5x5x1/25 Floppies/inch^3 = 1 Floppy/inch^3
12x12x12=1728in^3/ft^3
100GB of 5.25in floppies is:
100/.002=
50,000 floppies or
50,000/1=
50,000 cubic inches or
50,000/1728=
28.9 cubic feet
That's a heck of a lot of floppy disks, espesically to back up one 5x3x10 = 150 cubic inch hard drive.
Not big enough (Score:3, Insightful)
Personally, I backup to a second harddrive.
Tapes are still the way to go (Score:5, Interesting)
Re:Tapes are still the way to go (Score:3, Informative)
Tapes are 113$US (shipped) for a 4 pack.
For those of you following along at home, that is ~300$US for 120GB of backup.
400$US for two sets of 120GB.
Not bad, and it is a DAMN FAST drive...
Tarballz! (Score:5, Informative)
The question I have to ask... (Score:5, Insightful)
Out of 100gig, how much do you really NEED to back up?
The vast majority of my space is taken up by MP3s (where I converted my CD collection), but that could easily be replaced. To tell you the truth, of the things that I would need (documents, pictures, etc), I could easily fit it all onto a CDR. Well, maybe two. (I take lots of pictures)
Basically it boils down to, do you really need to shell out the money for that extra drive?
:^)
Re:The question I have to ask... (Score:2)
My backups consist of web design work, and that's basicly it. I have a rolling backup on a CDRW and it works great.
/home/dir (Score:5, Insightful)
Not to distro-bait, but Debian in particular shines here because apt makes it so damn easy to bring a system back to the state you wanted. For myself I have created a meta-package (.deb) which does nothing but depend on the applications I want installed on every desktop system: galeon, gnucash, xchat, gaim, xmms, vim-gtk, and a handful of others. Then I back up my meta-package, all of 10k including a few shell scripts I wrote for myself. Install my meta-package on a new system, and voilá, apt fetches and installs every app, that I need to continue working, dependencies included.
Re:/home/dir (Score:3, Informative)
I believe this was the original purpose of
Re:The question I have to ask... (Score:4, Interesting)
Obsoleted OS's. If someone is usinf win95, and its got all the patches, you may want to back up everything, do to lack of support.
is replacing 100's of megs of
The ability to get everything back w/o reinstalling and downloading Service packs and patches is a huge plus to most people.
Another hard drive (Score:3, Informative)
Actually, what I do is make the new (largest one I own) drive the backup drive, put the old backup drive into use as the primary drive, and retire the smallest one. Just make sure the new drive is as large as the others added together.
CD-R's are OK, but why bother with the hassle? Just run a cron job to copy the files every evening/hour/whatever.
Re:Another hard drive (Score:4, Funny)
So, is this Fibonacci backup method?
Onstream (Score:4, Informative)
Pros:
Can be found for under $100
Linux Support!
Cons:
Tapes are expensive
Get a USB drive (Score:2, Interesting)
Tape didn't quite catch up... (Score:3, Informative)
Actually... (Score:3, Interesting)
The data I actually need to back up I manage by having the important stuff an specified directories, then mirroring them over the net to my machine at work. By doing it incrementally, there is little time or bandwith wasted.
/Janne
Another solution (Score:2, Interesting)
LZip Compression (Score:2, Funny)
Why bother? (Score:5, Insightful)
That may seem like a stupid question, but you need to consider the reasons you want to have a backup before you settle on a method.
Are you afraid of your drive failing? If so, then using a RAID solution should cover you.
Are you afraid of losing your whole system (perhaps due to lightning or theft)? If so, then your backup must be kept physically isolated from your system.
Are you afraid of accidentally deleting files (such as `rm -rf
Are you afraid of having your system down for an hour or two while you replace a drive? If so, then regardless of other issues, you need a RAID setup.
Do you want to use your MP3s with some other device? If so, you probably want CD-R copies.
Of course, there are other considerations that I haven't mentioned or thought of.
Exactly. Poster Meets Terrorist Profile IMO (Score:4, Funny)
Why do you want to keep your data safe?
Is an encryption device utilized with a harddrive or an application?
Where did you obtain all of your software?
Are you looking to copy to a device that has the ability to encrypt files?
If you are looking for a portable back-up device, why do you need it to be portable?
Do you travel extensively?
When you do travel, do you primarily travel by air?
Do you have a digital camera?
Do you have a mobile phone?
Have you ever encrypted an email message?
Have you ever deleted an email message?
If so, have you had data rewrite over the sector(s) containing such message?
What was the title of the last book you purchased?
Re:Why bother? (Score:2, Insightful)
1) I'll never get another six weeks of vacation all at once to replace my digital pictures of Europe.
2) Sure, I still have all the CDs I made
Some of it can't be replaced, some of it isn't worth my time and effort to replace it when a hundred bucks solves the problem.
I've been working with the Addonics ExDrive stuff. PCMCIA, USB 2.0, or Firewire enclosures for normal hard disks. I don't know about Linux support; I went this route since I know it works on my Windows laptop and on my iPaq... It does kind of take a different mindset from what I had been using (read: "It's disk! Install it in something!") but it does make sure I've got redundant copies of, say, digital pictures that aren't on-line.
-JDF
What I did - Linux Software RAID-5 (Score:2)
So I got 4 x 60 GB Maxtors (cheapest MB/$ ratio when I purchased), two Promise Ultra TX2-100 controllers, and set up a 180 GB software RAID-5 under Linux. I'm running ReiserFS on it, so I don't have to fsck 180 GB if I crash the system running an experimental kernel, and I keep the whole thing on a big UPS. Total cost for hardware was about $500.
I mirror data onto the RAID using rsync from my other computers, (or just drag-and-drop to the Samba server from the Win2K box). I think this is cheaper and more effective than a tape backup system for 180 GB of data.
The Linux software RAID gives me the reliability - I've inadvertently tested it when a power connector popped loose from one of the hard drives - I didn't even notice until I read the kernel log (for a different reason). After powerering down, I plugged the drive back in, restarted, did the "raidhotadd" command, and away I went again. No data loss, no hassle.
For stuff that I really, really want backed up beyond the reach of thieves and fire, I use CDR's and a safety deposit box. Luckily there is not too much stuff that falls into this category - source code and documents for projects I've worked on, my email, stuff like that - it still all fits on one CDR.
If you don't use Linux, I think Win2K can do software RAID too. Never checked though.
Re:What I did - Linux Software RAID-5 (Score:3, Informative)
Cygwin has RSync for Windows. Works really well.
Obligatory comment (Score:5, Funny)
Re:Obligatory comment (Score:5, Funny)
DV Camcorder + firewire (Score:3, Informative)
http://dvbackup.sourceforge.net
Incremental backups... (Score:4, Informative)
However, I would suspect that most users don't change a huge percentage of their HD's content on a daily basis, unless you are routinely d/l'ing or ripping MP3s and MPGs on a daily basis (and I note that when I do generate that kind of traffic, it is usually because I am making a compilation CD, and while this does generate a few GB of "new" files on my HD that day, that data doesn't need to be backup up because I've got the original CDs anyway).
As a result, it seems to me that a reasonable solution is to create a "baseline" backup, say to a CD or DVD, at system install time, when there is (relatively) little on the disk, and then each day (or week, depending on needs), do an incremental backup of changed data only to another CD.
This approach is obviously quite inefficient if you have a complete HD failure, in that you have to recreate a new drive by starting with the first backup CD and then restore EACH ONE thereafter until the final CD restores the disk to it's last backed-up state, but for a more common problem of losing or corrupting an individual file, since that is more likely to happen with a recently modified than a remotely modified file, you are likely to be able to restore a last good version within only a few CD's of the most recent incremental backups.
Hard drive . . . rsync (Score:5, Informative)
I agree.
What hasn't been mentioned is rsync [samba.org], which makes disk to (local or remote) disk backups fast and easy.
It is trival to set up a second disk that is a "stale" mirror of your primary disk(s) that backs up nightly, and will boot off a floppy. This captures some of the advantage of RAID (quick recovery) while being an actual backup, not just fault tolerance.
Rsync can use ssh as a transport, so you can securely back up remote disks as well.
-Peter
Re:Hard drive . . . rsync (Score:2)
Re:Hard drive . . . rsync (Score:3, Informative)
I rarely use the word impossible, but I think this is. He, presumably, is root. The system has to write to the disk . . . I don't see how to overcome this.
OTOH, you could encrypt the files prior to transmission. This creates the new problems that 1. the efficiency of rsync is lost unless you do some kind of "chunky" encryption and 2. there is no obvious way to do this.
I guess you could use dump [sourceforge.net] to do a full backup periodically, then encrypt and upload that, then do incremental dumps nightly and encrypt and upload that. Not pretty.
-Peter
RAID5 (Score:2)
If you are concerned with recovering deleted files, simply use tar or something similar to backup to either a separate directory, or create a separate partition on the RAID array. Another advantage is that you can always increase your storage by slapping in an additional drive, partitioning it the way you want, and then adding it to the current array.
Backup solutions (Score:2, Interesting)
RAID is _not_ backup ! (Score:5, Informative)
RAID offers good protection for some things: hardware failure (ie: HD crash) and uptime. That aren't the only woes, however... You can loose data in a lot of ways:
Disaster (fire, quake, flood, nuff said)
Hardware failure (disk, controller, ...)
OS failure (FS corruption, ...)
Application failure (User space applications malbehaving, virii, ...)
User failure (accidental deletes, experimental children - trust me on this one ;-)
:)
...) are a plus, but more cumbersome.
RAID will protect you from the second, but will happily add nothing in case of any of the other failures. Backing up to another media is a necessity.
Adding an extra disk (or two, or three), and some tar/cpio cronjobs will add basic protection. (No disaster recovery for you, unless it's off-site
Removable harddrives (firewire, frames,
Tape is considered a more 'trustworthy' backup medium because the mechanism and data storage are separated (ie: tape drive / tape), while in a HD it's in one single package, and it's not as easy to replace the logic board/stepper motor if this flunks. With tape it's easier: just get a new tapedrive.
Anyhow: don't rely on RAID to save your data - it won't.
Network backups, disk images, etc (Score:2)
Of course, data file should not be installed on a local drive, so that you can implement some sort of a disk imaging solution for the base installation. The disk image should be of the main drive with the core installation folders or mounts.
This way if someone screws up the system, you can blow out the main drive, replace it with a known good config, and then add the two or three apps you need, with the datafiles safely someplace else. This could even be done from a bootable CD, if needed.
802.11 solution (Score:5, Interesting)
I should start charging for these ideas... Can't wait for the proliferation of freenet!
Re:802.11 solution (Score:2)
Re:802.11 solution (Score:5, Funny)
I'm so sorry to hear that...
So I worked out a deal so that everyone is backing up to each other's PC at night on a weekly basis.
...and I'm really sorry to hear that. I hope you have a good lawyer and some written authorization from your neighbors.
Just the thought of what you've setup is enough to give me shivers.
What were you thinking? You're setting yourself up for a huge lawsuit and/or war with your neighbors. What if someone gets pissed at you because your dog leaves a surprise on their lawn? (just an example). You're the one who's responsible here, and you get no benefit. Your neighbors are leeching off your cable modem and using you for your computer knowledge.
Geeze, if I were you I'd be going insane! At least charge them for cable modem usage. And charge them for backups, but back it up to your own server. Get a tape drive and keep a recent set offsite (perhaps at a friendly neighbor's place, or a friend's place). And most of all... have a full contract signed by your neighbors clearing you of any responsibility!
Excuse me while I have a heart attack...
It's all a matter of compromise (Score:2)
Magnetic storage : the data is sensitive to electromagnetic fields, heat, and the magnetism in the ferrite particules decays over time (i.e. the data has a reasonable chance to get corrupted after 5 to 10 years, even if the media is properly stored). Also, reading a magnetic medium can wear out the medium, but that's not an issue if you just want backups. The bit density on these media is good, and the price per megabyte is excellent.
Optical storage : there is no decay in theory, but I read somewhere that pressed CDs actually have a lifespan of 20 to 30 years even when properly stored, and CDR(W) even much less, especially when stored exposed to sunlight. I'm not certain there is much real-life data on the durability of DVDs. The bit density is good, but not as good as magnetic storage, and the price per megabyte, I guess, is comparable, but mastering optical media requires more effort.
I'm not considering the fact that magnetic storage media are re-writable, since you're talking about doing backups. So I guess, the question really boils down to whether or not you want to have backups that are more reliable over time, and whether or not you want to spend the time and effort to create CDs or DVDs of your data.
Then of course, there is the question of obsolescence : CDs have been around for years, and I don't think they're going to disappear any time soon. However, tapes for example can become unreadable because nobody makes readers anymore (ever tried to re-read a 80Mb Colorado tape recently ?). Same goes for hard-drives, although I'll admit IDE and SCSI interfaces will be around for quite a while longer.
Then of course, if you really badly want your data stored safely for the longest time, you can get yourself an old punchcard deck, a very large box of punchcards, and there is a fair chance that some archeologist in the year 4000 will find them in a dig just above where you house used to be ;-)
Moderating tips... (Score:2)
Try DLT... (Score:5, Informative)
Pick up a DLT2000XT (15gb native) off ebay for about $200. Tapes are dirt cheap, about $5/ea and the media is extremely durable, nearly indestructible.
Pick up a DLT1 (40gb native) off ebay, about $500. Tapes are moderately expensive at around $20/ea, but again the media is extremely durable.
DLT is industrial strength backup, the drives are built like tanks and the tapes can take incredible abuse.
Its all standard SCSI and works great with linux, no problems whatsoever.
I considered buying hard drives for backups, but they are far too fragile for long term backup and off-site storage. Most drives arent designed to be spun up and down lots of times either.
Last thing you need is for your backup harddisk to go splat when youre trying to power it up to restore your main system from a data loss.
With DLT, this isnt likely to happen.
Large Disk Backup (Score:2)
It would be nice if I could hot-plug the drives instead of having to reboot. If I was more thorough (or paranoid), I'd take the drive off-site and put it in a safe deposit box. Dataports are made by CRU-INC [cruinc.com], but there are other similar products.
Cluster (Score:2)
They are clustered. First created with a 100mb crossover link. Then moved one offsite with a modem link between them (direct modem to modem). AS the files don't change much there is not a lot of traffic. (1 file in 3 days maybe).
There is also software link doubletake and surviveIT [ca.com] for the corporate world.
Doing backups (Score:4, Interesting)
Bringing up the system is less of a problem with newer OSes, since you can usually, at minimum, get to your data. Configuring the database, webserver, and firewalling depends on how good you are with the OS. However, when I worked at a former company there was no real plan to get a working system back in place. We were using Novell with Arcserve -- unfortunately, you couldn't get to the data without a working system.
Next I usually try to segregate rapidly changing stuff versus things that are pretty much static. E.g, my mp3 collection is relatively static. I occasionally buy a fresh CD and rip it, but I'm pretty much satisfied with my collection as it is. I put these on CDROM. It takes a while to create them, but it's cheap and safe. If you want to keep everything up to date, you can run a script to save only files not included on the CDROM.
Finally, I back up my constantly changing stuff such as CVS, MySQL database, etc. to 4MM tape. It's cheap (hardware and tape) and most drives are pretty well supported.
obligatory whine (Score:2)
actually, most of the points made in the earlier story apply here.
and as i'm sure others will/have jumped in to say, there's a big gap between failover/redundancy afforded by a raid setup or external hard disk and tape backups. it, like so much else, depends on what you need.
i'm pretty discouraged personally. i don't really see an affordable way to do real backups of a couple hundred gigs of data. it's probably going to have to be a mixed setup. most of the data is static: flac & mp3's. so maybe that just needs one offsite backup that's done to a couple hard disks--basically a mirror. then get some kind of tape system for the other 40gigs or so of slowly or quickly changing data that i also want to keep a couple snapshots of for historical purposes, as well as rotate offsite. still not sure what the latter solution should be.
i have an old dds2 drive. but that's 8gb max compressed. dlt or ultrium is way out of budget. maybe an onstream. but i don't know anyone who's actually got one--any firsthand reports on these? and how likely is whatever onstream uses likely to exist/be supported in say 5 years or beyond?
What is your true need? (Score:2)
However, if you need to backup data with the ability to rollback changes or deletions then you are most definitely looking at a tape system.
Here's a similar question. (Score:2)
Currently we've got a storeroom full of boxes and boxes of CDRs. I want to make an archival backup of them.
Negatives will last for a hundred years if they're stored properly. Just the other day we made reprints of one of the first sessions our studio did, more than 20 years ago. However, CDRs do not last this long. Assuming nothing catastrophic (fire, CDs breaking) happens; CDRs are only made to last a few years at most. (Rough estimates put the shelf life of CDRs at 10 years or so.)
What form of storage is the most archival?
Consider that we've got probably a thousand CDRs sitting in boxes. Some way of doing a batch backup (CD towers?) would be great.
How I back up a File server of 320 Gigabytes (Score:5, Interesting)
collection of CD's and ripped them to MP3's at 320 bit, and wanted to
have them stored in a central place, accessible from any machine in my
home. Currently this collection is at approximately 620 full CD's of
music, and I'm pushing right at, or just above the 80 gigabyte limit.
Now when you factor in personal files, financial records, games,
downloaded material, installation software you don't want to lose,
etc...etc... Well, see for yourself. Here's my space breakdown for the
partitions on my main file server Fumus (Smoke, in Latin):
fumus:/pub/mp3 # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 3.0G 2.1G 804M 72% /
/dev/hda1 129M 6.8M 115M 6%
/dev/hda5 9.8G 1.8M 9.3G 1%
/dev/hda6 20G 13G 6.3G 67%
/dev/hda8 40G 22G 17G 57%
/dev/hdb1 75G 38G 33G 53%
/dev/hda7 1.9G 20k 1.8G 1%
/dev/hdc1 74G 34G 40G 46%
/dev/hdd1 74G 36G 37G 49%
So, here's what I looked at:
Tape: For the size I'd need: Way WAY too expensive. When I brought
the media down into the range I'd afford, I'd be swapping tapes all week
to get a backup done. Not time effective.
CD-R: Faster, yes, but at 650 megabytes per media, same problem as
tape, only you've traded magne tic for optical.
Extra hard drives in the same machine: Originally, this is exactly what
I had done with a single file server running Reiser file systems in the
more experimental days. I got the scare (and lesson) of my life when
Reiser went a bit nuts, and started corrupting some of my data. I only
lost about one percent, but I vowed, never never NEVER again would I
backup data on a critical machine on live media in the same machine.
Okay, so here's what I finally DID select as my solution: A second
machine called Ignis (Fire in Latin) that uses the absolutely identical
configuration, right down to the types and number of drives, partition
sizes, everything. They both connect into my 100Mb network switch, and
Ignis rsync's from Fumus every hour on the hour thanks to scripts in
/etc/cron.hourly
In fact, here's Ignis'
rsync -arul --one-file-system --quiet fumus:/pub/mp3_2
rsync -arul --one-file-system --quiet fumus:/pub/mp3
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub/software
rsync -azrul --one-file-system --quiet --delete --force fumus:/pub /
rsync -azrul --one-file-system --quiet --delete --force fumus:/pchome /
Is this a bit extreme? Yes. But... if, gods forbid, Fumus really does
let out its magic smoke, or Ignis does catch on fire, and the physical
media were actually damaged, hopefully the damage would be limited to
*one* case, and wouldn't end up taking both machines out. Then I really
would be crying the blues.
Oh yes, and each machine is on their own 900VA UPS. I'm not playing
THAT game.
"Affordable" backup? Surely you jest. (Score:2)
1) Go cheap (i.e. Zip drive, MO drive, CD-RW, etc.) and only backup the files you NEED (i.e. home directory, "My Documents" folder, etc.)
2) Shell out for tape. This way you will be able to make multiple backups, keep them offsite, maintain them long-term, and back up most of your system. I find that the sweet-spot is usually to go 1-2 generations old (i.e. right now I'm using a DAT24 drive), that way you only have to pay about $400-600 instead of $1500+ for the mechanism.
Don't fall for the "just use hard drives" trick. Hard drives have a number of problems.
1) Mechanism + media in one unit. I've seen hard drives whose heads STUCK to the platters, rendering them useless, after only sitting around for a couple of months. Oops! Data gone! They are also sensitive to static and to environmental changes -- if your backup drive gets zapped accidentally, you can't just plug the media into a new mechanism! Data gone!
2) Backups limited. If you backup a 100GB system onto a 100GB hard drive, you're limited to one physical backup. This creates all kinds of problems... Many's the time I've had to go back four or five generations in backups to find an old file that I didn't realize I'd deleted months ago. Not to mention that you are limited to one backup stored in one location -- no redundancy.
3) Even RAID has problems -- I've had bad SCSI cables that filled a RAID filesystem with corruption before we were able to track down the problem and switch the cable. If that happens when you're just depending on RAID to preserve your important data... oops! There's a reason why many RAID-enabled datacenters also maintain backups of critical data in a second medium!
So that's my personal take. I think either tape, maintaining at least 5-10 backup generations at a time, or MO/CD-RW just for keeping your critical files. Or both, even.
Digital Camcorder? (Score:3, Informative)
This is not exactly a dirt cheap solution, but if you have/want a digital camcorder anyway, there's only the cost of extra tapes.
Make sure the camcorder works with dvbackup before buying one though. It doesn't work with any JVC's that I know of, or at least not mine
Offsite backup to large disk with rsync (Score:3, Interesting)
do
HOSTNAME=`echo $i| awk -F: '{print $1;}'`
DIRECTORY=`echo $i| awk -F: '{print $2;}'`
DATE=`date +%A`
install -d
rsync --numeric-ids --compress --rsh=/usr/bin/ssh --recursive --archive --relative --sparse --one-file-system --compare-dest=/vol/backup/$HOSTNAME/current $HOSTNAME:$DIRECTORY
done
Then once a week we run a similar script that updates the 'current' directories and uses --delete
(rsync.list contains entries like "hostname:/some/mounted/partition")
Some Metrics... (Score:3, Interesting)
- 6 DVD+RW (18 GB) discs, or
- 20 DVD-RAM (5.2 GB) discs, or
- 158 CD-R discs, or
- 72,818 HD 3.5" floppy discs
Hey old-timer... (Score:5, Insightful)
That worked really well for backing up our 80MB drives onto stacks of 1.44MB floppies, since you would really only need to insert about 5-10 floppies during your weekly backup, just to get the files that changed.
So why not just do incremental backup onto CD-Rs? Even with 100GB of archives, most of those are static. You probably won't need to use more than one CDR per week (maybe two) to track the changes. It's cheap, relatively painless if you've got the right software (and it wouldn't be hard to throw together incremental backup/recovery scripts in Perl if you're into that sort of thing.) and you've probably already got a CD burner.
If less than 650MB of files change in a week, the rest of the CDR can be filled up with files that were on earlier CDRs (this way your backup set can remain finite and you can throw out the earlier CDRs as they become obsolete. Or if you keep them all, you can reconstruct that state of your hard drive at *any* time, not just at the last backup.) This seems ideal to me--why is everyone else talking about expensive solutions like tape drives, DVD-RWs, and second hard drives?
Re:Hey old-timer... (Score:3, Informative)
Of course you'll still need something like 150 CDRs to get everything, inceremental or not - and that's just the first dump that will keep growing.
BTW, to the other replies to this - one of the companies I worked for, we did full, weekly backups of all the important systems (ie the actual OS and software - not the data) like the domain controller, database servers, fileservers, firewall, the backup servers themselves, etc. and then daily incrementals on them.
For the actual data on the file and database server, plus everyone's personal workstations (something like 100 people) it was a full backup once a month (maybe less) and nightly incrementals, which worked great because here people usually wanted just a couple of files their dumb asses deleted a week ago (usually perfectly timed to want the tape that just went off to off site storage :) ) All in all that achieved very good granularity. Of course the incremental tapes were recycled after about two months or so.
The database servers were a bit more tricky so additional full dumps were done manually, when, let's say, 500,000 new compounds were imported. There were also databases attached to data collection robots that had their own separate system going.
All in all, this took three backup servers (one for the data collection databases, one for individuals PCs and one for the other stuff) with their respective tape loaders, plus individual tape drives for the main database servers (we also had things like SGI workstations, but the scintists on those did their own thing), and the full backups did take up almost the entire weekend. But it was setup well enough that it took only a few minutes a week to maintain the whole thing - get a new tape, write on it what the server tells you to, put it in the loader, take the specified one out, send it off - you are done. And it did work - they had to recover after a fire once (before my time) and did it without any problems.
Anyway, the point of the story was this - it's not as simple as "do incrementals" or "no! do full backups" when you are talking about actual companies, it's a bit more tricky than that.
100GB? Whew! (Score:4, Insightful)
I have to wonder whether (first of all) why in the heck anyone would need to have 100GB of disk space on a home system. But then I have five systems networked together and have more storage than I would have thought sane a few years a go though I have a bit of a ways to go before I will run into the poster's backup problem. It wasn't too long ago that, if you could afford 100GB, you could probably afford a SCSI array controller that would let you do a lot of RAID, hot swapping, automatic drive replacement, etc. With today's cheap disk prices you don't have to be wealthy to have an ocean of disk space. (I can remember the days when we thought having 900MB on a MicroVAX II was extravagant.)
You could always do it the traditional way and get some tape drives. Unfortunately, they're much more expensive than you might think when you have to backup that much disk space. You certainly wouldn't want to go cheap and be feeding 90m DAT cartridges into a drive all night (it'll start feeling like you're backing up to floppies before long). A good high capacity tape drive can get, what, 20GB onto a single cartridge? Not bad. And I think that at this point in time, tape is more cost effective than DVD-R. (Something tells me that the MPAA, and maybe the RIAA, will try to keep it that way too.)
Mirroring disks can be helpful. Hard disks are getting cheaper and cheaper. Heck it's almost scary mow much disk space you get in a typical PC sold at Best Buy nowadays (and without a backup device; it's almost criminal). If you're running mirrored disks you'll forestall the inevitable disk crash that takes all your data with it. Question for the Linux folks using the `md' driver: Does it allow adding a third member to a mirrorset? And, if so, can it be done while the system is `live'? (The third member gets removed and taken offsite in case there's a disaster.)
One final thought: The poster wasn't actually running a 100GB filesystem were they? I'm thinking that a power glitch could cause a world record to be set for the longest fsck-on-reboot run. Plus I'd think that backing up such a beast would be a challenge. I tend to keep my filesystem sizes no larger than what I can fit on a single tape cartridge... just to make life simple. (I'm used to having to pipe `df' commands through `more' at work so I don't mind lots of mount points. :-) )
NNTP backup (Score:3, Informative)
Advantages:
- Standardized TCP-based protocol, uses authentication.
- Lots of free "backup software" available - e.g. pan, newspost, tin...
- Centralized backups over the network! Only one server directory to back up.
- Backups are automatically dated by the server, and can be signed and even encrypted if you integrate pgp into your posting script.
Disadvantages:The way I do it here.... (Score:4, Informative)
I have a "backup" hard drive in my server. This drive is always unmounted so that there is no chance of filesystem corruption from the operating system.
I just use a crontab to run a simple script that mounts the drive and coppies whatever specified backup files to it, then unmounts it. The same method slightly modified could be used to back up this same backup disk to another location on the network on regular intervals.
Work smarter, not harder! (Score:3, Insightful)
Instead of dumping all 100GB of files every time, 95% of which haven't changed since the last backup, use an incremental backup program to write only the 5% that actually changed. After the initial archive, the backup files will be significantly smaller, and could potentially saved on CDs.
Ecrix VXA-1 (Score:3, Informative)
The drives support three different flavours of media - 12GB, 20GB and 33GB, and come with IDE, SCSI, or Firewire interfaces. The IDE is cheapest, at $699, with the media costs being $80, $45, or $35 depending on the capacity.
Is it the absoulte top of the line as far as tapes go? No. But the cost can't be beat. And you get a reasonably fast (3MB/sec) drive with very nice reliability (take a look at the independent testing on their site; e.g. soaking a tape in hot coffee for a minute, rinsing it, drying it and reading the data off.)
They also recently merged with Exabyte, who will be positioning it as their new value solution. Hopefully the Exabyte name will expand the market enough to drive the prices down on these even further.
Matt
I'm not sure why this is so hard... (Score:3, Funny)
RAID 5 + replicated backups on disks (Score:4, Interesting)
The recent explosion in disk capacities and decrease in prices got me to rethink this, just when it came the time for me to set up a home office. When I compared the cost of a reasonably-good tape drive and a number of tapes large enough for me to get at least a month of backups in rotation, and computed how many 60GB disks I could buy with that money, the solution was clear.
I ended up setting up 3 machines with 4x60GB each. They're all on RAID 5, such that if any single disk fails, the machine keeps running (actually, I have
I get all my backup-worthy data rsynced over to the other machines daily or so. I plan to start playing with Inter-Mezzo soon, so that I don't have to remember to run these backups, and so that I don't run these backups on the wrong direction.
But that's not all. With the mind-boggling amount of disk space I could afford, I could (actually, I will, but you get the idea) set up Amanda to backup interesting portions of my home directory to disk, and also replicate this to at least another of my local machines. Such backups can use software compression, such that they don't take as much space as live data. Also, I intend to use another form of compression: instead of backing up CVS trees (I've got loads of check outs), I'm going to back up only local changes to files, so that, in case of disaster, I can still download the original CVS tree and re-apply patches. But this is still a plan, not something I've got running.
Finally, I've got yet another disk on a remote site, to which I rsync not only the interesting portions of my data, but also my backups. I could convince someone else to run this remote backup site for me by offering this person the speed up of RAID 0 over two disks (one of those mine). As for keeping the secrecy of the data on this remote backup site, I'd just get the backup files encrypted, no big deal.
I can strongly recommend this solution: I got pretty much as much data safety as could be expected from a tape-based backup, without any of the hassle of having to switch tapes and moving them off-site and back on-site, and with the bonus of very fast access to local data, unlikely donw-time and fast recovery except in case of total disaster (i.e., having all of my local machines failing, in which case I'd have to either download my backups from the remote site over the net or, more likely, take a replacement machine over to the remote backup site and copy files over a fast local network connection, or from disk to disk.
As for getting 4 IDE disks into a single machine, don't even think of using only the 2 IDE controllers that come on most motherboards these days (for RAID set-ups, you really want one IDE disk per controller). There are a few good motherboards that come with 4 IDE controllers, so that you can even have a CD-ROM and/or a CD-RW in addition to the 4 disks. If you can't find such a motherboard that suits your needs, you can always get one of those PCI cards that adds 2 IDE controllers to your machine.
As for the problem of fitting so many disks in a standard ATX chassis, it can be done. Cooling may be a problem, but a good cooler has been good enough.
All in all, I'm very happy with this arrangement. It was not cheap, but it was not as expensive as a tape-based solution, and it's far more flexible, way faster and it doesn't require any baby-sitting after you get it going. And I can keep far more backup history than I thought it was going to be possible.
Re:Cheapest way might be another hard drive... (Score:4, Insightful)
I once worked at a place where we had a lightning storm. Within a week, about half of the hard drives had failed, out of about a dozen. RAID won't save you then. And how fast can you get replacement hard drives installed, anyway?
All the affected machines were plugged into good UPSes, too.
Moral of the story: Always use offline backups.
Re:Cheapest way might be another hard drive... (Score:2)
Re:raid (Score:2, Insightful)
Another thing that some of us are looking towards is finding a trustworthy friend to share capacity with. If you each buy the extra hard drive (or have space to spare), and rsync nightly with each other, you can get reasonable coverage, and offsite backup. Just pick one reasonably geographically far from you so your data doesn't get sucked up in the same tornado.
Or if you don't trust the "friend", use an encrypted filesystem, or crypt the files first.
Re:raid (Score:2)
RAID won't protect against those types of accidents. Indeed, RAID will happily mirror your screwup for you, automatically. Real backups, on removable media, are the only way to keep a working copy of your system. A separate, permanent hard drive would work too (although it doesn't provide the mental separation between "secure backup" and "live system" that DVDs or tapes would). The important thing is that you perform your backup manually, at a point when your system is in a "known good" configuration.
Re:More Drives (Score:2)
No work at all if you go removable.
(Removable drive rack review [makeitsimple.com])
All of the removable IDE (or SCSI) racks are pretty much the same as the gadget in the review, with minor variations on the theme.
As a bonus, they make futzing around with other operating systems and/or distros (benchmarking, porting, fooling around) a piece of cake, and are a great way to "use up" those old Other uses - sneakernet with 20G removable media. If you live in an apartment and can't h4x0r j00r w4llz with cable runs, it makes loading content onto your "MP3/DiVX jukebox" computer a snap.
I've got two on my "main" machine (one to boot from, one to use as a backup / "gigabyte floppy drive"), and one on each of my "media playback" machines.
Re:raid 1 (Score:2)
The difference becomes clear when you say ">" when you mean ">>"!
-Peter
Re:Easy (Score:2)
Re:Cheap CD backup. (Score:2)
Also look at this comment [slashdot.org] which offers very similar advice.