North America's Fastest Linux Cluster Constructed

SeanAhern writes "LinuxWorld reports that 'A Linux cluster deployed at Lawrence Livermore National Laboratory and codenamed 'Thunder' yesterday delivered 19.94 teraflops of sustained performance, making it the most powerful computer in North America - and the second fastest on Earth.'" Thunder sports 4,096 Itanium 2 processors in 1,024 nodes, some big iron by any standard.

Very great and all...

[irokitt]

But why did they use itanium processors? Were they acquiring parts before Opterons were availabel? Did they have a problem with Xeon processors? Or did they have too much cash lying around?

"Most" powerful

[Alomex]

Look, any way you cut it the 100K computers Google is reputed to have is the most powerful Linux cluster anywhere in the world.

vs google

[docl]

This is probably a stupid question, but would anyone care to explain how this is different than a really large cluster. For example, if people estimate google to approach 100K nodes, how does this compare?

Another Article

[Flashbck]

And only 55 people [com.com] were needed to build it!

Re:"Most" powerful

[irokitt]

4,096 iTanium processors versus ~8,000 boxes sporting Pentium II, III, and 4 processors. But remember that the interest Google has is in disk access and redundacy, not complex mathematical computation. So it isn't configured as a 'supercomputer' per se.

Re:I'd hate to be the guy...

[MrRuslan]

You know what would be cool...if it would be posible to somehow recycle part of the heat from this clust into energy.perhaps tunneling all of it into one source and it boil water into steam to make energy...kidding about that last part ;)

Second fastest on earth?

[Stevyn]

yeah, that we know about. I remember the article on google a few weeks ago that made everyone think just what they hell they're running over there. I wouldn't be surprised if governments kept other supercomputing clusters secret. I don't mean anything tin-foil-hatish here, I'm just thinking that some governments have test facilities that they don't let the public know about.

Re:apple's response will be interesting

[Jeff DeMaagd]

I love G5s, but Virginia Tech's cluster IMO can't say much until they get the G5 Xserves, because the PowerMac G5s don't have ECC memory. ECC is very important for such a large scale project that runs simulations where data is stored in RAM for any meaningful duration.

Re:apple's response will be interesting

[gerardrj]

There's also the difference in the interconnects, that has a lot to do with the efficiency of the system as a whole.
Lets see what the VTech system does with ECC RAM installed when some node's aren't double-checking other node's results.

Re:apple's response will be interesting

[System.out.println()]

And don't forget that the current round of G5's are currently almost a year old... and long due for an upgrade. I hope some other instituion builds a 1,500 G5 2.6 GHz cluster :) (Or something to that effect.)

Re:"Most" powerful

[Boone^]

You're right, but this still only uses an off-the-shelf interconnect from Quadrics. Quadrics bills themselves as the "price/performance leader", not the performance leader.

There are many purpose-built supercomputers coming up (like Sandia's Red Storm) that use custom yet pricy interconnects that end up smoking anything Quadrics can put together. Anytime your interconnect relies on a PCI-type bus, you take a latency penalty on each end. Real supercomputers access memory on other nodes directly, not through a generic shared bus to a fancy network card.

Read some of this if you're bored, it goes through Sandia's entire thought process. http://www.lanl.gov/orgs/ccn/salishan2003/pdf/camp .pdf [lanl.gov]

Re:apple's response will be interesting

[fmorgan]

I heard a presentation from VTech on why they selected the G5 over the Itanium (for scientific calculations, with lot of floating poing operations, both are faster than AMD chips; not a big problem for AMD, of course; how many of us need to simulate nuclear explosions in our desktops? well, at least until the next generation of strategy games, of course).

At the time - this was a study done in July/Aug 2003, remember - the speed of the G5 and the Itanium2 were similar for the same clock speed (for scientific calculations; before someone flames me with something off topic, remember that this is a very particular kind of application); then what happened was that Intel was simply "out-clocked"! Kind of funny when Intel was the big champion of "clock-speed" over AMD, Motorala and IBM.

This was in a presentation by VTech at an O'Reilly conference; coverage for this with several articles (including /. discussions) can be found here:
http://www.macdevcenter.com/mac/osx2003/

Re:Did hell freeze over?

[Ralph Wiggam]

Nuclear weapons are dinosaurs. They did their job from 1945 to 1991. Who are we going to nuke now? The North Koreans, who are proposing a peace treaty? Canada? Nukes are weapons of deterence. Osama isn't sitting in his cave thinking, "We shouldn't mess with the US, they might nuke us."

-B

Probably OT, but...

[Trogre]

... if you want a practically guided tour of LLNL, watch TRON sometime. They filmed it there (the science-lab live action stuff anyway).

Of course...

[MaestroSartori]

...you realise that it isn't a linear scale. Trying to make a G5 cluster which achieved 4.8 gigaflops per processor would take more than the 4400 processors, and thus would easily take more than 300 more processors than are used for the Itanium cluster.

300 processors. Thats 150 dual-processor boxes. I can't be bothered working it out now, but how far that goes to eliminating the power & heat advantage the G5 has would be interesting to find out...

Re:Very great and all...

[Too Much Noise]

Check SpecFP benchmarks - Itanium2 smokes pretty much everything else. Reason? it was meant to be a fp monster from the beginning. Integer math is weak (Opterons kick Itanium on that pretty hard), but FP math, especially vector FP math is Itanium's selling point. Why do you think the vast majority of I2 sales were to scientific research groups? (check the target profile for SGI's I2 clusters - research and defense)

Re:Did hell freeze over?

[NEOtaku17]

I don't know if people realize this but one of the major reasons computers exist as they do today is because of military research. Yeah building nukes was one of the driving forces behind more complex and powerful computers. Both nuclear power and computers grew up as a result of WW2. Now stop complaining about military research because you wouldn't even be typing messages on Slashdot if it weren't for the need of more advanced military technology pushing science and application of science further ahead.

Re:Very great and all...

[tap]

I wish those graphs were easier to read! From the looks of it, at ~32 processors the 1.5 GHz It2 has a 12 second run time and the 1.8 GHz Opteron is 24 seconds. The It2 cluster has Quadrics high end Elan4 interconnect, while the interconnect of the Opterion cluster isn't listed. It might have GigE for all we know.

The It2 probably cost around 5 times as much as the opterons, so a real comparison would be 32 It2 processors vs 160 Opterons. With the scaling shown for that model, the Opterons of equilivent cost would be 2-3 times faster than the Itaniums.

Re:Nah, but who knows what the NSA has cooking.

[DAldredge]

The NSA has their own small scale fab to make their own custom chips. Hell, they do not every try to keep it secret and have let more than one file crew film part of it.

They can design all the custom chips they want with their rather large budget.

Your forgetting

[IAmAMacOSXAddict]

It also has twice the processors, to generate the X2 times speed that they claim. Something tells me, now that VA is recieving the XServer G5 cluster nodes, that they may want to add some more units. they can put 48 units in each rack now, rather than 12 of the full size G5 Desktop form factor. According top my primitive calculations that would allow them to run 4 times as many machines in the same space (would be over 8000 CPUs. I figure that will likely kick the crap out of this new linux cluster...

Re:before everyone starts shouting at once...

[ajp]

Speculation: very cool. Lots of registers, yes, but the Register Stack Engine makes it feel like you never run out: there are automagically more. EPIC: bundles are actually between 3 and 5 instructions. Rotating registers: most important for passing parameters. Predicated instructions? 128 of them.

However your post, and my post, are wasted on /. These people will always hate Itanium because it's not kewl. But they're not the ones who will be buying the processors.