Best Backup Server Option For University TV Station? 272
idk07002 writes 'I have been tasked with building an offsite backup server for my university's television station to back up our Final Cut Pro Server and our in-office file server (a Drobo), in case the studio spontaneously combusts. Total capacity between these two systems is ~12TB. Not at all full yet, but we would like the system to have the same capacity so that we can get maximum life out of it. It looks like it would be possible to get rack space somewhere on campus with Gigabit Ethernet and possibly fiber coming into our office. Would a Linux box with rsync work? What is the sweet spot between value and longevity? What solution would you use?'
Re:Done to death. (Score:5, Informative)
Cue usual discussion about defining the problem correctly, choose the right tool for the job, etc.
Specifically:
"Would a Linux box with rsync work?" - It depends on the objective business requirements you've defined or been given. If those requirements include "has to be implemented on Foo operating system", then those requirements are not just for a backup solution.
"What is the sweet spot between value and longevity?" - Simple: Graph accumulated TCO/time based on quotes from internal and external service providers. Throw in some risk/mitigation. Find the plot which best meets your cost/time/business requirements.
"What solution would you use?" - Almost certainly not the solution you would use, because my needs are different. What is your backup strategy? What are your versioning requirements? What are your retention requirements? (How) do you validate data? Who should have access? What is an acceptable speed for access to archived data? What's an acceptable recovery scenario/timeline, etc.
If you do not already know the answers to those questions, or how to find reasonable answers, ask neighboring university TV stations until you find one which has implemented a backup solution with similar business requirements as your's, and copy and paste the appropriate bits. You'll likely get better answers from people who have solved your exact problem before if you search Google for the appropriate group/mailing list for your organization's level of operating complexity, and ask there instead of asking generalists on slashdot, and hoping that someone from your specialist demographic is also here.
Re:FreeBSD/Linux + Rsync (Score:5, Informative)
Actually, I'd suggest using OpenSolaris so that you can take advantage of ZFS. Managing large filesystems and pools of disks is *stupidly* easy with ZFS.
You could also do it with Linux, but that would require you to use FUSE, which has a considerable performance penalty. I'm not sure about the state of ZFS on FreeBSD, although I imagine that the Solaris implementation is going to be the most stable and complete. (For what it's worth, I've been doing backups via ZFS/FUSE on Ubuntu for about a year without any major problems)
Re:Build a Backblaze Storage Pod. (Score:4, Informative)
lose the drobo (Score:2, Informative)
BackupPC (Score:4, Informative)
What I use is BackupPC [sf.net]. It's a very nice web front end to tar over ssh.
For linux, all the remote servers need are sshd listening somewhere, and with the backuppc servers public key in an authorizedhosts file. It will pipe tar streams over an ssh connection.
For windows, it can use samba to backup over SMB
I run a copy on my home file server, which backs up all the machines in the house, plus the couple servers I have out in colo.
When it performs an incremental backup, after it is done it will populate its timestamped folder with hardlinks to the last full backup for duped files. so restoring from any incremental will still get the full version no matter when it was last backed up.
Also after each backup, it will do 2 hashes on every file and the previous backup. If the files match, it deletes the second copy and again hardlinks it to the first copy of the file.
I have nearly 3 months worth of backup retention, backups every 3 days (every day on a couple), but for the base system and files that rarely change, each 'copy' does not take up the same amount of disk space.
It is very good at saving disk space.
Heres some stats from its main page as an example
There are 7 hosts that have been backed up, for a total of:
* 26 full backups of total size 38.34GB (prior to pooling and compression),
* 43 incr backups of total size 0.63GB (prior to pooling and compression).
Pool is 10.11GB comprising 108499 files and 4369 directories (as of 9/16 01:00),
Restoring gives you a file browser with checkboxes. after you tell it what you want, it can send you a tar(.gz) or .zip file, OR it can directly restore the file via tar over ssh back to the machine it was on, by default in the original location but that can be changed easily too.
The main downside is the learning curve. But once you get things down, you end up just copying other systems as templates, updating the host/port/keyfile/etc settings.
Also, with all those hard links, it makes it a pain to do any file/folder manipulation on its data dir.
Most programs won't recognize the hard link and just copy the file, easily taking up the full amount of storage.
But works just as well with only itself and one remote server.
schedule it to start at night and stop in the morning, set your frequency and how much space to use before it deletes old backups, and let it run.
Don't use rsync â" at least, not vanilla (Score:5, Informative)
Don't use rsync to make backups. Because you don't just want to backup against spontaneous combustion â" inevitably, there will be accidental deletions and the like occurring in your studio. If you use rsync (with --delete, as any sane person would, otherwise your backup server will fill up in days, not years), then when some n00b runs `rm -rf ~/ReallyImportantVideos`, they'll be deleted from the backup too.
Remember that pro photography website that went down, because their "backup" was a mirroring RAID setup? Yep â" they lost all their data on one fell swoop when somebody accidentally deleted the whole lot. Don't make the same mistake.
Use an incremental backup tool. Three that come to mind are rdiff-backup [nongnu.org], Dirvish [dirvish.org], and BackupPC [sourceforge.net].
I would think that rdiff-backup would suit your needs best. I currently use BackupPC at home, which is great for home backups, but I think that it's overkill (and possibly a bit limited) for what you want.
Hope this helps!
Different Solutions (Score:1, Informative)
My company is developing a local backup and co-location data center, and I have been one of the major forces in decided what software we go with. If you are looking for linux style freedom, as mentioned before, rsync is all you need. If you happen to be looking for something more professionly supported, there are many options, but I will tell you some of what I have seen. At significant cost, the primary system I run into is EVault, which works ok, is very stable, and doesnt have too many crazy features. Offsetting that is the horrible, and I mean horrible, cost. Acronis just (as in like less than a month ago) came out with their new backup product, which they even give a free trial for. It does bare metal restore among other things, and I was very impressd with it, but it didnt meet some of my requirements and I didnt get to play with it much more. On the cheaper more jenky side of things, I have tried NovaStor backup products with overall horrendouse results, stay away completely from them. (things like being able to export data directly to a removable drive for first time transfer is ridiculous!) I am very impressed with a completely off the wall solution called RBackup. It seems at first very "made in india" but it has tons of features that are easy to understand (being brandable is a big plus) and generally can be setup quickly or very granularly. If your using a windows system you should check it out.I have also looked at symantecs and other things, but these so far are a few of the major players in the "I want to remote backup my own data to my own servers" category (which excludes lots of stuff) Since I am still in the review process, I am also curious to see what other people say. I can also tell you that I have setup almost 4 drobos now and they really rock, so your doing good on that front!
Re:Amazon S3 (Score:4, Informative)
Why do anything when you can pay someone else twice as much? 12TB from Amazon will be an order of magnitude more expensive [backblaze.com] than just running a storage server, and you have to pay for internet bandwidth instead of just running a wire.
Re:Need more information (Score:5, Informative)
The parent is absolutely right. We don't have enough details to really make a recommendation, but if the question is 'can rsync replicate 12 TB with an average rate of churn over a 1 Gbps link reliably'? The answer is an emphatic and resounding YES!
I used to maintain an rsync disaster recovery clone that was backing up multiple NetWare, Linux, Unix, and Windows servers to a central repository in excess of 20 TB over primarily 100 Mbps links. We found that our average rate of churn was 1% / day which was easily accomplished. It was all scripted out with Perl and would notify on job status each night or failures. Very easy to slap together and rock solid for the limited scope we defined.
When you get into more specifics on HA, DR recovery turn around times, maintained permissions, databases and in use files, versioning, etc.. things can get significantly more complicated.
Re:Don't use rsync â" at least, not vanilla (Score:4, Informative)
rsync actually includes an option to make hardlinked snapshots as part of the syncing process, nowadays.
Personally, I don't trust it and always do that part manually, then let rsync do what it does best... But yeah, even "vanilla" rsync contains exactly the functionality you mention.
Different Solutions (Score:3, Informative)
My university is developing a local backup and co-location data center, and I have been one of the major forces in decided what software we go with. If you are looking for linux style freedom, as mentioned before, rsync is all you need. If you happen to be looking for something more professionly supported, there are many options, but I will tell you some of what I have seen. At significant cost, the primary system I run into is EVault, which works ok, is very stable, and doesnt have too many crazy features. Offsetting that is the horrible, and I mean horrible, cost. Acronis just (as in like less than a month ago) came out with their new backup product, which they even give a free trial for. It does bare metal restore among other things, and I was very impressd with it, but it didnt meet some of my requirements and I didnt get to play with it much more. On the cheaper more jenky side of things, I have tried NovaStor backup products with overall horrendouse results, stay away completely from them. (things like being able to export data directly to a removable drive for first time transfer is ridiculous!) I am very impressed with a completely off the wall solution called RBackup. It seems at first very "made in india" but it has tons of features that are easy to understand (being brandable is a big plus) and generally can be setup quickly or very granularly. If your using a windows system you should check it out.I have also looked at symantecs and other things, but these so far are a few of the major players in the "I want to remote backup my own data to my own servers" category (which excludes lots of stuff) Since I am still in the review process, I am also curious to see what other people say. I can also tell you that I have setup almost 4 drobos now and they really rock, so your doing good on that front!
Re:FreeBSD/Linux + Rsync (Score:1, Informative)
So far, it works very well -- the advantages of Debian's packaging system with advantages of OpenSolaris (e.g. ZFS)
It's a good stepping stone for those used to linux (you just have to relearn some system command usage, system config, service management, etc)
http://www.nexenta.org [nexenta.org]
Re:BackupPC (Score:4, Informative)
Re:Need more information (Score:3, Informative)
I second that motion....
We do something similar with rsync, backing up about 6-8 TB of data. We have php scripts that manage it all and version the backups, keeping them as long as disk space allows. Heck, you can even have a copy of our scripts [effortlessis.com] free of charge!
With these scripts, and a cheap-o tower computer with huge power supply and mondo cheap, SATA drives, we manage to reliably backup a half-dozen busy servers off-site, off-network, to a different city over the Internet automagically every night.
Yes, more information is needed, blah blah blah. But it's definitely a feasible idea.
Re:Don't use rsync â" at least, not vanilla (Score:3, Informative)
*nod*, at least for various definitions of "manually."
I have a script which makes a hard-linked clone of the latest backup, and then rsyncs to that (with some manner of special commandline switch which is made for this scenario and that I can't be bothered to look up right now). It's easy, and it lets me have layered backups not totally unlike (though nowhere near as slick as) Netapp's snapshots.
I have done bare-metal restores of Linux boxen from backups made like this. Works just fine, with an iota of bootstrap knowledge.
Re:Don't use rsync â" at least, not vanilla (Score:4, Informative)
rsync makes it pretty easy to implement a bargain-basement backup system if you're willing to do a bit of hacking around with scripts and soft/hard links. Make your backups into e.g. /backups/2009/09/17/* and update the symlink for /backups/latest to point to that dir; when the next backup comes along, use the --link-dest=/backups/2009/09/17/ to hardlink all files that have stayed the same, but copy over the newer versions into your /backups/latest. This way you get a) the absolute minimum space taken up without resorting to snapshots and b) and easy way of looking at and restoring individual files or the whole tree from a given date/time. For bonus points set up a vacuum script that automagically deletes the oldest backups whenever your backup partition gets to 90% full or whatever. Run your set of scripts every hour or so (but don't forget to include lock files/semaphores so you don't end up running nine instances of the script simultaneously).
As far as syncing large amounts of data, firstly use rsync 3 if you can - it's a hojillion times faster with large numbers of files and much easier on your memory. If you're going over the internet, tunnel through SSH using inline compression (if your data is easily compressible that is) - heck, tunnel through SSH on your private network, rsync makes it ridiculously easy. Using this technique I managed to keep a mirror of a 2TB file server over a 2Mbps SDSL link no more than an hour or two out of date.
That's how I remember it working anyway - don't have a box I can try it out on here, but in all honesty rsync and a bit of bash/python/whatever is capable of reproducing all sorts of "enterprisey" backup features for zero cost and almost zero effort (and, I'll almost certainly say, zero approval from your boss). IMHO it's one of the killer apps of UNIX.
Disclaimer: I am not an employee of Rsync Overlord Corp, just a satisfied customer ;)
Re:redundancy, anyone? (Score:3, Informative)
Why bother?
OpenSolaris will run rsync just fine and it is also free.
There are a lot of good solutions out there so I wouldn't limit myself to just Linux.
You have OpenFiler running on Linux.
You have FreeNAS on BSD.
And you could roll your own on OpenSolaris and use ZFS with fancy gui tools if you really want to.
Re:Different Solutions (Score:3, Informative)
In my experience, getting open files backed up is the hardest thing in a 24/7 environment.