New Linux Petabyte-Scale Distributed File System

An anonymous reader writes "A recent addition to Linux's impressive selection of file systems is Ceph, a distributed file system that incorporates replication and fault tolerance while maintaining POSIX compatibility. Explore the architecture of Ceph and learn how it provides fault tolerance and simplifies the management of massive amounts of data."

History

[Alcoholic Dali]

Ceph was designed by Sage Weil (of WebRing fame), who is also one of the founders of DreamHost. They will likely be using it internally soon, if they aren't already. http://en.wikipedia.org/wiki/DreamHost [wikipedia.org]

Is it ready for primetime?

[Meshach]

The headline in the Ceph wiki [newdream.net]: Ceph is under heavy development, and is not yet suitable for any uses other than benchmarking and review.

Re:History

[TooMuchToDo]

http://www.dreamhost.com/jobs.html [dreamhost.com]

FILE SYSTEMS SOFTWARE ENGINEER
Los Angeles, CA

New Dream Network has a vacancy for a Senior File Systems Software Engineer in Los Angeles, CA. Minimum requirements – Master’s degree in Computer Science or Computer Engineering, minimum of 2 years experience in storage programming, and background in Linux kernel programming, file systems development, network programming and Operating Systems design.

Qualified applicants should send a plain text resume to [e-mail]

Re:Is data integrity really necessary for large da

[CoderJoe]

Google's BigFile/BigTable architecture is a distributed filesystem. if a node goes down, the data that was on that node gets copied to other nodes to keep the replication count up.

Facebook is using apache cassandra, which adopts similar designs.

Re:Is data integrity really necessary for large da

[ProfMobius]

First, Facebook & Google data are not possible to regenerate, as they are personal things, like emails, messages, posts, etc.

Second, you have other sectors producing large amount of data beside your favourite networking website. One example is the LHC. It is going to produce terabytes of data per DAY (15 petabytes per year). Another are space telescopes. Those data can't just be 'regenerated'. 1 day worth of data is incredibly expensive to produce.

Distributed file systems are already there, and people use them. Maybe not on your level of computer usage.

When you don't know what you are talking about, I think it is better to just keep quiet.

Re:Linux®

[tomhudson]

Definitely looks weird. I always write it in all-lowercase. But apparently the trademark is either all-caps ("LINUX®") or the standard capitalized form ("Linux®") [linuxmark.org]

Someone should remind them to register "linux®" (all lowercase), before Darl tries to. A capital first letter just doesn't look right.

Re:Linux®

[John Hasler]

A word mark is always registered as all upper case. Lower and mixed case are still covered.

Nope

[avm]

Nothing special at all. It only means Taco used sequential instead of randomised integers for user ids, which in turn can be viewed as a very loose chronology of user registrations.

In other words, no.

Re:"Enterprisey" design? Yet no scrubbing?

[Anonymous Coward]

Did I miss it, or did they really forget that crucial part?

You missed it. There is a scrubbing mechanism in ceph.

Re:pet-a-byte?

[SlothDead]

Tera -> Tetra -> 4 -> 1000^4
Peta -> Penta (like Pentagram) -> 5 -> 1000^5
Exa -> Hexa (like Hexagon) -> 6 -> 1000^6
Zeta -> Setta (like 7 in many languages) -> 7 -> 1000^7
Yotta -> Otta -> 8 -> 1000^8

Or use 1024 if you don't like IEEE/IEC norms...