Linux Clustering

An anonymous reader writes "Beowulf clustering turns 10 years old, and, in this interview, creator Donald Becker talks about how Beowulf can handle high-end computing on a par with supercomputers."

On par? Yes and no

[grape jelly]

Beowulf clusters have never been the fix-all solution to pricey supercomputer needs. Traditional mainframe supercomputers will forever have their niche in computing that can't just be muscled through sheer volume of vector processes (i.e., processes in which good latency is essential). Even the creator of the Beowulf cluster agrees:

Quote from the article: *snip!*

Supercomputer vendor Cray has created a new product that is designed to compete with some Linux clusters. Cray Canada CTO Paul Terry said that Linux clusters really can't compare to a supercomputer. What is your take on Cray's moves against Linux?

Becker: They are simultaneously saying that Linux clusters are not high-performance computing systems while introducing a product to compete with Linux clusters. They clearly saw that a large part of their customer base was moving toward commodity clusters, Beowulf-class clusters, to do high-end computing.

Clusters can't replace all of the workload being done by supercomputers today, but it can replace the bulk of the traditional vector supercomputers. There is always that 10% of the market that won't run well on clusters, and that is the market that Cray is in. We are happy to solve most of the problems of the world and run most of the applications and play in our part of the marketplace.

Re:On par? Yes and no

[EnderWiggnz]

yeh - sometimes, you actually need the pure performace that a vector processor will give you, without the initial overhead of paralelizing a process to run loosely-coupled on a beowulf.

It's not just about speed and massively parallel

[Wizzy Wig]

processing...

To be considered a "supercomputer," it also needs enough CONTIGUOUS MEMORY SPACE to hold the massive amounts of data associated with true "supercomputing." So far, no cluster has met that requirement.

Cluster Schedulers

[Anonymous Coward]

And GridEngine is free and opensource:

http://gridengine.sunsource.net/

Otherwise known for...

[jaylee7877]

Donald Becker also has done a large amount of work on Linux Network drivers. Grep through linux/drivers/networking and you'll find he's done work on Intel NICs, Realtek 8139s, even the ne2000 (I think he said he puked a few time while working on that one). Thanks for all your hard work Donald!

Re:Just out of curiosity...

[Anonymous Coward]

ClusterKnoppix [bofh.be] may be just what you're looking for...

openMosix

[241comp]

Beowulf isn't the only game in town folks. A much easier to maintain and balance cluster can be built using openMosix [openmosix.org]. openMosix is a single-image-cluster extension for Linux.

Re:It's not just about speed and massively paralle

[vsack]

There are certain classes of problems that clusters don't map to well. Applications with a very high cost of inter-processor comminucation or that demand a huge piece of contiguous memory are probably always going to be outside the realm of clusters.

However, problems that are embaressingly parallel can be handled by a cluster very adequately for a fraction of the cost of a traditional supercomputer. I don't know that you can ignore this class of problems and say that clusters aren't "true 'supercomputing'".

FUD, FUD, and wrong. Crey stockholder eh?

[Rhys]

Sad to see this little knowledge about parallel computing on slashdot: blatantly wrong information marked as informative. +5 no less.

Let's address this first: there are two common memory architectures, distributed memory (a cluster) and shared memory (a 'traditional' supercomputer). Each can emulate the other. Saying a cluster doesn't have enough memory, presumably at each node, is really saying: "I don't really understand message passing."

This would be more important if datasets were actually large. Unfortunatly for your argument they aren't. A handfull of nodes and they'll hold the whole simulation easily in memory (albeit it'd take years to run because there's so few CPUs at work.)

How would I know? Well, I work with the Center for Simulation of Advanced Rockets aka CSAR at UIUC, one of five DoE ACSI sites in the country. I manage their supercomputer, which is getting upgraded from 200 P3-class dual proc PCs to 640 dual proc Xserve G5s. Before that I was a grad student working with them, albeit not on the CSAR simulation but instead on a related grant, the CPSD.

Now, there are computing problems which clusters aren't good at (or at least that's the traditional claim. My master's thesis and advisor would seem to dispute that this is actually the case.) However, most problems as the interview says, run just fine on clusters. Physical simulations (which covers CSAR's rockets to the national labs nuclear weapon research to hurricane/weather simulation, all the way down to protein folding and atomic and sub-atomic scale crystal formation simulation) need to know about what's in the area you're working on, and what's in nearby areas.

Occasionally you'll find an oddball like galactic simulation (or molecular dynamics) that needs to compute gravity across the whole universe. Fortunatly we have multigrid methods and a friendly gravity equation to solve this problem: get real data from those near you. Average those far from you and use that instead.

Then of course there's the idea that even "traditional" supercomputer problems that don't run well on clusters can be run efficiently on clusters IF you move beyond 1 process per CPU. Load up 10, 20, 100, 1000 little workers on a processor. Get fast context switching between them (not OS level!). Use message passing rather than shared memory (locking, ick!) to communicate. One worker blocked waiting for network data? Process the next one! If you've tuned things right you'll find you always have work to do.

Sounds crazy? Supercomputing '02 didn't think so: http://charm.cs.uiuc.edu/research/moldyn/

clusterknoppix

[Yenhsrav_Keviv]

ClusterKnoppix [bofh.be] is pretty cool. Its got all the auto detect hardware features of regular knoppix, and also aautomatically adds itself to the cluster.

Re:Just out of curiosity...

[kst]

You might take a look at Rocks:
http://www.rocksclusters.org/ [rocksclusters.org].

Quite a few people have built Rocks clusters out of a bunch of old computers.

Disclaimer: I work with the folks who created this.

SearchEnterpriseLinux

[scoobrs]

I noticed that this article by the PenguinComputing CTO appears to answer the article by the Cray CTO and contradicts it. All I want to know is this: how much did PenguinComputing and Cray spend on advertising banners on SearchEnterpriseLinux to have these articles made? Let's let vapor settle before it gets to our heads.