Which OSS Clustered Filesystem Should I Use? 320
Dishwasha writes "For over a decade I have had arrays of 10-20 disks providing larger than normal storage at home. I have suffered twice through complete loss of data once due to accidentally not re-enabling the notification on my hardware RAID and having an array power supply fail and the RAID controller was unable to recover half of the entire array. Now, I run RAID-10 manually verifying that each mirrored pair is properly distributed across each enclosure. I would like to upgrade the hardware but am currently severely tied to the current RAID hardware and would like to take a more hardware agnostic approach by utilizing a cluster filesystem. I currently have 8TB of data (16TB raw storage) and am very paranoid about data loss. My research has yielded 3 possible solutions: Luster, GlusterFS, and Ceph."
Read on for the rest of Dishwasha's question.
"Lustre is well accepted and used in 7 of the top 10 supercomputers in the world, but it has been sullied by the buy-off of Sun to Oracle. Fortunately the creator seems to have Lustre back under control via his company Whamcloud, but I am still reticent to pick something once affiliated with Oracle and it also appears that the solution may be a bit more complex than I need. Right now I would like to reduce my hardware requirements to 2 servers total with an equal number of disks to serve as both filesystem cluster servers and KVM hosts.""GlusterFS seems to be gaining a lot of momentum now having backing from Red Hat. It is much less complex and supports distributed replication and directly exporting volumes through CIFS, but doesn't quite have the same endorsement as Lustre."
"Ceph seems the smallest of the three projects, but has an interesting striping and replication block-level driver called Rados."
"I really would like a clustered filesystem with distributed, replicated, and striped capabilities. If possible, I would like to control the number of replications at a file level. The cluster filesystem should work well with hosting virtual machines in a high-available fashion thereby supporting guest migrations. And lastly it should require as minimal hardware as possible with the possibility of upgrading and scaling without taking down data."
"Has anybody here on Slashdot had any experience with one or more of these clustered file systems? Are there any bandwidth and/or latency comparisons between them? Has anyone experienced a failure and can share their experience with the ease of recovery? Does anyone have any recommendations and why?"
I know this isn't what you asked but... (Score:5, Interesting)
No ZFS? (Score:5, Interesting)
How about ZFS with your RAID controllers in single drive mode (or worst case JBOD)? Let ZFS handle the vdevs as mirrors or raidz1/2 as you wish. ZFSforLinux is rapidly maturing and definitely stable enough for a home nas. Or go the OpenIndiana route if that's what you're comfortable with.
My 4TB setup has actually been a joy to maintain since committing to ZFS, with BTRFS waiting in the wings. The only downside is biting the bullet and using modern CPUs and 4-8GB memory. Recommissioning old hardware isn't the ideal way to go, ymmv.
Just a thought.
Thoughts on OCFS (Score:4, Interesting)
We have been using OCFS [oracle.com] (Oracle Cluster File System) for some time in production between a few different servers.
Now, I am not a sysadmin so can't comment on that aspect. I'm like a product manager type, so I only really see two sides of it: 1) when it is working normally and everything is fine 2) when it stops working and everything is broken.
Overall from my perspective, I would rate it as "satisfactory". The "working normally" aspect is most of the time; everything is relatively seamless - we add new content to our servers using a variety of techniques (HTTP uploads, FTP uploads, etc) and they are all magically distributed to the nodes.
Unfortunately we have had several problems where something happens to the node and it seems to lose contact with the filesystem or something. At that point the node pretty much becomes worthless and needs to be rebooted, which seems to fix the problem (there might be other less drastic measures but this seems to be all we have at the moment).
So far this has JUST been not annoying enough for us to look at alternatives. Downtime hasn't been too bad overall; now we know what to look for we have alarming and stuff set up so we can catch failures a little bit sooner before things spiral out of control.
I have very briefly looked at the alternatives listed in the OP and look forward to reading what other reader's experiences are like with them.
Re:You Should... (Score:4, Interesting)
I'll go out on a limb and say it is just hoarding behavior. I wouldn't be surprised if slaker (53818) has a whole bunch of other stuff, besides data, but at least the data hoarding takes up less room than books, and isn't as sick as animal hoarding...
Having observed some hoarders, first hand, I think something goes off in their head that is like a "gotta collect them all" flag. It usually is concentrated on a favorite subject, but it could even be set off with garbage, like tearing open a package and setting down the wrapper... one is trash, but, if it is not discarded, the second one is the "start of a collection", and off they go.
Ill fit... (Score:4, Interesting)
Those filesystems are not designed primarily with your scenario in mind. If you want a hardware agnostic support, use software RAID or a non-cluster filesystem like ZFS.
Distributing your storage will probably not enhance your ability to survive a mishap. In fact, the complexity of the situation probably increases your risk of messing up your data (I have heard more than a couple of instances of someone accidentally destroying all the contents of a distributed filesystem, but in those professional contexts they have a real backup strategy. You'll be pissing away money on power to drive multiple computers that you really don't need to power.
If you care about catastrophic recovery, you need a real backup solution. This may mean identifying what's "important" from a practical home situation. If you don't mind downtime so long as your data is accessible in a day or two (e.g. time to get replacement parts) without going to your backup media and without suffering the loss of non-critical data, then also having a software raid or ZFS is the way to go. If you want to avoid downtime (within reason), get yourself a box with basic redundancy designed into it like a tower server from Dell/HP/IBM. If Intel, you would sadly want to go Xeon to get ECC, on AMD you can get ECC cheaper. In terms of drive count, I'd dial it back to 4 3TB drives in a RAID5 (or 5 in RAID6 if you wanted), safe on power and reduce risk in the system.
Re:Drobo? (Score:4, Interesting)
STAY AWAY FROM DROBO!
I had a client ask me to set one up for them. You don't partition it like a standard raid array, you format it to some predetermined size that may be larger then the physical disk space in the machine (through their drobo dashboard). If you have three 1 TB disks you will have around 2TB of actual storage but you can format it for 16TB under Win 7. This is achieved via their "beyond raid" technology which fools the OS into thinking there is more disk space than there actually is. This lets the user make one large volume now and then add disks in the future, even disks of different sizes can be mixed and matched. If you start to go beyond the physical capacity, the array degrades and goes offline until you add another disk and wait hours or days for the disks to reorganize. My client was consolidating her photography library to the drobo when it just crapped out. Turns out she ran over the physical limit.
Then if your lucky, your computer be it Apple or Windows will take upward of 30 to 45 minutes to boot and shutdown if the fucking thing is plugged in and powered on during either of those two procedures. Drobo recommends you move your data to another set of disks and re-format your drobo. As if people have a few spare TB of disk capacity just sitting around, that's the reason they bought your shit box to begin with, assholes. Its a known issue.
I have personally used hardware raid 5, software block level raid 5 and ZFS. If you ask me id rather have the file system do the RAID work at the file system level, not the block level where the file system is ignorant of what lies beneath. ZFS is the way to go until BTRFS is fully stable and feature competitive with ZFS. Then you do incremental backups offsite, either to a family or friends house or to a commercial off site backup provider.
And what is your 16 TB consist of? If its movies and the like then don't bother spending money backing it up. If its self make video and other personal large files then that makes sense. I know of people who spent oodles of cash to backup silly crap like downloaded movies that can easily be replaced or rented from Netflix.
With the way things are going in the storage world, SSD's will eclipse mechanical disks at the desktop level and mechanical disks will be relegated to backup duty where they far outstrip SSD's in capacity. It reminds me of when tape drives were the king of capacity, often tapes were several orders of magnitude larger than current hard disks and tapes were cheap. They were slow but my god did they have capacity. Now it looks like SSD's will assume the role of desktop storage and to some degree server storage while mechanical disks will be used for large backup systems and file servers. Mechanical hard drives of today will be tomorrows tape drives and then obsolete when SSD's begin to overtake then in capacity. By then we might have something even higher in capacity like holographic or some other sci-fi sounding storage.