Ask Slashdot: Best Use For a New Supercomputing Cluster?

Supp0rtLinux writes "In about 2 weeks time I will be receiving everything necessary to build the largest x86_64-based supercomputer on the east coast of the U.S. (at least until someone takes the title away from us). It's spec'ed to start with 1200 dual-socket six-core servers. We primarily do life-science/health/biology related tasks on our existing (fairly small) HPC. We intend to continue this usage, but to also open it up for new uses (energy comes to mind). Additionally, we'd like to lease access to recoup some of our costs. So, what's the best Linux distro for something of this size and scale? Any that include a chargeback option/module? Additionally, due to cost contracts, we have to choose either InfiniBand or 10Gb Ethernet for the backend: which would Slashdot readers go with if they had to choose? Either way, all nodes will have four 1Gbps Ethernet ports. Finally, all nodes include only a basic onboard GPU. We intend to put powerful GPUs into the PCI-e slot and open up the new HPC for GPU related crunching. Any suggestions on the most powerful Linux friendly PCI-e GPU available?"

Lost some funding?

[turkeyfeathers]

Start with the cheapest backend that'll get the system up and running, then use your supercomputer to mine Bitcoins for a few days, then use all the money you'll make to buy the InfiniBand backend (you'll probably have enough money left over to buy Monster cables to hook everything up).

Best Use For a New Supercomputing Cluster?

[Jodka]

Generating Bitcoins

Re:So little detail

[oobayly]

Indeed, it's a bit like somebody writing in to Dear Deirdre and saying "I've a 13 inch cock, how can I make girls aware of this, and what's the best way to make use of it?"

Re:Riiiiight

[PPH]

Happens to me when I visit Costco all the time.

Re:I call Shenanigans!!!

[Anonymous Coward]

Yes, it probably is a tax scam. It is now the US Federal Year End. Someone wrote a really good funding proposal and got it approved to get money for a HPC cluster to do *something*. Doesn't really matter. The grant application will have focused on broad ideas like # of cores and what not and not the details. A bit surprising that the network wasn't spec'd because that is such a major cost item, but whatever, maybe the grant application's work loads are not network bound.

So, now that the money is approved the task to build the thing falls to the inexperienced IT group who make all kinds of dumb choices, then will claim they are massively over worked/underfunded trying to get the thing to run and end up with shitty performance and a lot of wasted time and money. Oops. Your Money At Work.

The system you spec'd should be around the 250TF range, if you set it up properly with QDR IB and do all the work to get MPI optimized. If you are good. The correct way to design the network is to match a 36 port IB switch with 18 servers, and then correctly spread the resulting 1200 uplink ports across a pair of 686 port core switches. The cost of IB cables alone will be shocking and you'll regret not using a HPC blade server arrangement for this.

Considering the questions you are asking, you should have gone with an HPC focused integrator that could provide the full system for you. 4 GIGE's on every box? Waste of money. The IB equipped blade designs from SGI/Bull, etc are very nice, space and power efficient and much more cost effective. They even come with a pre-integrated and tested OS ready to go, working boot over IB and other long term cost saving features.

Gosh I hope you bought a few PB of HPC focused storage as well, otherwise you won't find anyone who can even use your machine for their problems.

Re:Yes, this is legit and no, we're not idiots

[Anonymous Coward]

"I've got 1200 servers shipping to me and my two best engineers are gone and we're not sure what to do with them when they get here."

Best. IT horror story. Ever.

OS, duh!

[ThurstonMoore]

The obvious answer is Windows Server 2008 HPC.