Forgot your password?
typodupeerror
IBM Software Hardware Linux

Building The MareNostrum COTS Supercomputer 187

Posted by timothy
from the available-at-every-pharmacy dept.
karvind writes "IBM Power Architecture Community Newsletter has a story about making a supercomputer (Number 4 on top 500 list) from easily available components (like BladeCenter and TotalStorage servers, 970FX PowerPC processors, and Linux 2.6). A joint venture between IBM and the Spanish government, it is named MareNostrum: the Latin term meaning 'our sea.' Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers."
This discussion has been archived. No new comments can be posted.

Building The MareNostrum COTS Supercomputer

Comments Filter:
  • specifically (Score:5, Informative)

    by bersl2 (689221) on Wednesday February 16, 2005 @04:42AM (#11686916) Journal
    Mare Nostrum refers to the Mediterranean Sea.
  • Mare Nostrum (Score:5, Informative)

    by nebaz (453974) on Wednesday February 16, 2005 @04:47AM (#11686932)
    Mare Nostrum literally means "our sea". It is what the Romans called the Mediterranean Sea during the Empire. As you can see [dalton.org], it was an apt name.
  • by Anonymous Coward on Wednesday February 16, 2005 @04:47AM (#11686940)
    I am an African Grey parrot, and I can tell you that while you humans are celebrating this achievement, I and my fellow Greys are laughing at you. Supercomputers are old news to us; in fact, one of my friends solved the halting problem while taking a crap the other day. Seriously, people, we like you 'cause you feed us, but leave this kind of stuff to us.

    (I tried to register an account but /. thought my user name was too long)

  • by Anonymous Coward on Wednesday February 16, 2005 @04:48AM (#11686941)
    but does it run Linux? Oh crap, never mind.
  • Couldn't they just "me too" it and run xserves >_

    Now im going to cry!
    • Well since it was IBM, they do get a volume discount on buying IBM equipment.

      Besides, last I checked (back when I bought mine) there is a 6-8 week waiting list for Xserves. (Confessions of an Apple Fan Boy.)

  • Top 500? (Score:4, Funny)

    by Ubi_NL (313657) <joris&ideeel,nl> on Wednesday February 16, 2005 @04:51AM (#11686956) Journal
    (Number 4 on top 500 list)
    ...while being Number 6 on top 300 list, and Number 65 on top 2000 list.

    This is like those CDs that have 'best of the Top40' and not contain the top10 list of that

  • and in 32 years (Score:3, Insightful)

    by headlessspider (859133) on Wednesday February 16, 2005 @04:59AM (#11686973) Homepage
    and after about 16 (or 32) years we'll have that power in our desktops...
  • Isn't this basically a Beowulf-type cluster with just many many nodes? Exactly where lies the innovation? The fact that a cluster of many processors have a lot of computing power is not exactly new.
    • It probably has to do more with the fact that as you increase the number of nodes, your increase in performance decreases on a per node basis. To get that many nodes working together takes an incredible amount of resource management. It makes you wonder where the limit currently is for if it is worth adding an extra node, or if the resource management requirements negate the extra nodes computational power.
      • Re:Beowulf cluster? (Score:4, Interesting)

        by tomhudson (43916) <barbara...hudson@@@barbara-hudson...com> on Wednesday February 16, 2005 @09:13AM (#11687724) Journal
        It makes you wonder where the limit currently is for if it is worth adding an extra node,
        If you're the vendor, there is no limit. Every extra node == $$$

        If you're the buyer, there is no limit. Every extra node == _MORE_STATUS_

        If you're the guys writing code for it, there is no limit. Every extra node == job security++

        If you're the people administering this, there is no limit. Every extra node == bigger budget next year

        See, citizen? Size does count.

        • Re:Beowulf cluster? (Score:4, Informative)

          by antifoidulus (807088) on Wednesday February 16, 2005 @09:20AM (#11687750) Homepage Journal
          Eh, not really. In a lot of the algorithms, there comes a point when adding nodes will make it SLOWER because the increases in communication time are greater than the decrease in computation time. Now granted this does depend a lot on a) what you are doing with the machine and b) the machines themselves, but just thinking that people who make these things love to just pile on hardware is a bit naive....
          • > just thinking that people who make these things love to just pile on hardware is a bit naive

            Or a bit brazenly honest. There's no limit to the benefit from scaling your cluster up, since

            (1) there are always applications that run better on more cpus, even with a crappy network design.

            (2) there are always better network designs that improve the remaining applications.

            (3) there is always the option of partitioning it into mutiple sub-clusters if you're too dumb for (1) and too cheap for (2).
    • It reminds me of how 3Dfx went down... They wanted faster cards, but rather than make something innovative, they just made the cards bigger so they could fit more stuff on them. The things were huge, it cracked me up when I saw them on ZDTV (before it was called TechTV). This seems like what IBM is trying to do here.
    • by commodoresloat (172735) on Wednesday February 16, 2005 @05:27AM (#11687041)
      Yeah but it's made from "easily available components." Hmmm, I think I've seen that before, at the University of Virginia. Don't G5s qualify as "easily available"?
      • > Don't G5s qualify as "easily available"?

        Not at the time.

        The cluster was announced as G5 but they used G4 for the first incarnation and later performed a system-wide upgrade (or two, who knows .
        http://www.thinksecret.com/news/virginiatech3.h tml

        Note how Apple's site (http://www.apple.com/education/science/profiles/v atech/) talks about 2.0GHz, and both 2.3GHz and later 2.5GHz processors were used...
        And this* "These systems were custom built by Apple for Virginia Tech utilizing dual 2.3GHz G5 proces
        • Re:Beowulf cluster? (Score:1, Informative)

          by Anonymous Coward
          VT got Dual 2.3Ghz XServes a couple of months before they were publically released, but they are publically available now.
          You can go to the online Apple store and order some if you like, it says they'll ship it on the same business day.
        • Wrong, the origional cluster used G5's, they just used dual G5 towers. Later the cluster was upgraded to dual G5 Xserver's running faster than publicly available G5 chips (as you noted).
      • Insightful?!?

        You should be lucky I chose to berate you for mistaking Thomas Jefferson Community College for Virginia Tech instead of moding you down.

        Aside from that gaff, however, you are essentially correct. The VT cluster may as well have been OTS, even if the units were built especially for VT. I think most of the "custom" part of the custom G5s was simple packaging and the fact that VT got the first run of chips. (don't quote me on that one...I just know what I've read in the newsletters).
    • Re:Beowulf cluster? (Score:2, Informative)

      by photon317 (208409)
      It's really hard to scale these kinds of clusters beyond certain limits, once you start getting into all the practical details.

      There's the basic facilities stuff: fitting enough power and cooling density into a single datacenter to cram that many nodes within a reasonable distance of each other for cabling purposes.

      There's the network architecture. A single switched network between your nodes doesn't get you all that far. Depending on the characteristics of the expected workload and all that jazz, there
      • I beg to differ about inherent scalability limits. Perhaps relative to a given application, such as one which requires random routing superlinear in node count, but for the broad range of applications,
        for which logarithmic cross-section bandwidth growth is sufficient, there are well-understood architectures that scale without bound.

        This stuff was cute and clever in 1992, but these days it's a pretty well-trodden path.

        • At least in the world I play in, logarithmic cross-section bandwidth is nowhere near sufficient. The interconnect design and technology is still very critical for us, as is storage.
  • Off who's shelf? (Score:4, Insightful)

    by identity0 (77976) on Wednesday February 16, 2005 @05:05AM (#11686994) Journal
    I love the first line in the article, which ends, "is constructed of such totally off-the-shelf parts as IBM BladeCenter JS20 servers, 64-bit 970FX PowerPC processors, TotalStorage DS4100 storage servers, and Linux 2.6. This is its story."

    Right, like I regularly go to Fry's to stock up on some DS4100s and Bladecenters. I'd love to be the geek for whom that stuff is "off-the-shelf". Can you even buy bare PPC CPUs and mobos?
    • by Anonymous Coward
      IBMs shelf. The stuff in itself is nothing particular, just ordinary computer stuff sold by IBM to hundreds or thousands of other customers.

      Just because you "can't" buy the stuff in pieces (IBM will probably gladly sell you the stuff as spare parts, if you prefer to assemble it yourself) doesn't mean it's not off the shelf.
    • Re:Off who's shelf? (Score:1, Informative)

      by Anonymous Coward
      Can you even buy bare PPC CPUs and mobos?

      Not 970-based yet, but anyway:
      Pegasos [pegasosppc.com].
      Terons [mai.com] (which are also marketed, by raping the corpse of the Amiga, to a bunch of clueless zealots as "AmigaOnes". The CX and PX models are discontinued due to hardware flaws, the jury is still out on the newer Mini model).
      • by PayPaI (733999)

        Pegasos.

        $775 [ultraspec.us] for a G4 (1GHz?) on an ATX board with specs comparable or less than a $112 [newegg.com] Athlon64 motherboard.

        Terons

        $3,900 [mai.com](!) for a board with a 750FX processor (unknown speed) and technology comparable to what was going out of style 4 years ago. (USB1.1? 10/100 Ethernet? PC133 Memory???)
        I'm not holding my breath for these. Call me when I can get a decent motherboard with a 1GHz processor for less than $200.

    • by lachlan76 (770870)
      You don't need your own fab, and you don't need to rent one. If you don't need custom components, it's off-the-shelf.
    • "off the shelf" means the technolgy has already been developed, not that any dork can get one at the local CompUSA. This generally means lower cost since you don't have to pay for custom chip development.
      • Thank you.

        Another example would be ordering a Dell. While there are Dells that are available retail, most of them can only be gotten directly from the manufacturer. The same is true for High-end Macintoshes, most mainframes, and a good deal of tape library market.


    • Right, like I regularly go to Fry's to stock up on some DS4100s and Bladecenters. I'd love to be the geek for whom that stuff is "off-the-shelf". Can you even buy bare PPC CPUs and mobos?


      Let alone Linux 2.6! Where the hell does one find that?!?

    • When you but this many at once, you can get it any way you want. Painted bright pink, some bizarre packaging, whatever.
    • Re:Off who's shelf? (Score:3, Informative)

      by blamanj (253811)
      Well, the info is right here [ibm.com] for quantity 1, and there a button that says "Configure and order a JS20," so if you're willing to order a few thousand of them (they're about $4K/ea with 2.5G RAM), you can build your own.

      Actually, that's a reasonable price, considering it's IBM, who aren't usually considered a bargain brand.
      • Re:Off who's shelf? (Score:3, Informative)

        by clem.dickey (102292)
        You'll need a chassis for that blade, of course. Luckily, they're half-off through the end of March. Buy the entry model [ibm.com] and get change back from your $1000 bill. Oh, and you'll want a rack [ibm.com] to put the chassis in. But at $1489 for the rack, check the local surplus store first. And while you there pick up a display, mouse and keyboard.
  • ...about it still not being sufficient to catalogue ones porn collection.
  • by flopsy mopsalon (635863) on Wednesday February 16, 2005 @05:16AM (#11687012)
    I happened to look at the Top 500 supercomputers [top500.org] site and I couln't help noticing out of the top 5 supercomputers almost half are in non-US countries like Spain and Japan. This is not to beat some kind of patriot act drum. Instead, it got me to thinking.

    With supercomputing powers now avaible to any country or group with a few readily available components, it is only a matter of time before these supercomputing powers may be used by a rogue state [korea-dpr.com] or radical group to cause havoc among electronic communications using methods like denial of service attacks, spyware, and crapflooding message boards.

    I think it is high time the nations of the world put their heads together and addressed this issue. For example, I don't think the US Federal Government even has any cabinet-level position like Secretary of Information Technology or something like that. When are they going to get with the times? It will probably take another terrorist attack or something.
    • Uh oh.
      Someone's been watching too much 24.
    • Yeah especially with all the bandwith that north korea has, im sure they will launch a very succesful DoS attack on Whitehouse.com as the north koreans are introduced to pr0n for the first time!
    • by Anonymous Coward
      and crapflooding message boards

      Yep, that's gonna cause mass mayhem and planet-wide panic. Them terrorists, messing up with our patriotic message boards! ... or something

      Kid, when you grow up you'll relize there are some differences between a Beowulf cluster and a botnet of crappy PCs.
    • by grozzie2 (698656) on Wednesday February 16, 2005 @05:35AM (#11687062)
      These things are already being used by rogue states. The us military has a bunch of them dedicated to modelling nuclear events.

      With regard to denial of service attacks, there's a cluebox over in the corner, you need to go grab a couple out of the box. DOS attacks dont require a big computer, they require massive bandwidth with massive routing diversity available. The actual computer power required borders on insignificant. A supercomputer like this is useless for that kind of thing, by necessity, it will have an internal networking and communications environment, and likely only a relatively low speed interconnect to external networks.

      But look on the bright side, the knee jerk 'terrorist behind every lamp post' reaction is just what the american government has been trying to instill in the population for the last few years. Your post here shows, it's been an effective campaign, money successfully spent, and the objective achieved. It's become the 'trendy' response to just about everything these days.

      • "denial of service attacks, spyware, and crapflooding message boards.

        Bit of hint there that the grandpappy was sending up the 'terrorists are everywhere' idiots ;-)

        J.

      • But look on the bright side, the knee jerk 'terrorist behind every lamp post' reaction is just what the american government has been trying to instill in the population for the last few years.

        While the parent poster may have gotten the details wrong (you are correct in that massive DOS requires distribution and bandwidth, not computing power), the manner in which you dismissively respond was unwarrented.

        Yes, the US Government has certainly been doing a lot to enforce the idea that Terrorists are Out Ther
        • The US and its allies may have spies working covertly in foreign rogue nations. Presumably, any intelligence or orders passed between them may involve internet-routed signals. A supercomputer would allow rogue states to brute-force crack the key and compromise the agent's mission.

          Not if the algorithm is good. Time and energy for a brute-force crack increases exponentially with key length; the energy output of the Sun over its entire lifetime isn't enough to even count to 2^256, let alone crack a 256-bit k
    • DOS's, spyware and crapflooding don't require supercomputing power. Now, nuclear simulations and the like- that's probably more worrisome. You very appropriately linked to NK, the state most likely at this point to be running such simulations. Although, they might just test their bomb for real, as a show of power. Whatever the case, I don't think any rogue states give a flying fuck about crapflooding /.
    • With supercomputing powers now avaible to any country or group with a few readily available components, it is only a matter of time before these supercomputing powers may be used by a rogue state or radical group to cause havoc among electronic communications using methods like denial of service attacks, spyware, and crapflooding message boards.

      You are SO informed! That is exactly what you can NOT use a supercomputer for! Supercomputers are used for scientific calculations. Sure you can use them to calcul

      • > As for being concerned about rogue states getting their hands on a supercomputer... that is what the US export regulations are for! I just have to say: There are more things between earth and sky than the Americans can think off.
    • I can't believe ours is smaller than the Japanese...
    • crapflooding message boards.

      This got me into thinking, too - with posts like this, rogue states will quickly realise they can save some of that computing power just by leaving slashdot alone.

    • First and formost, that is NSA's directive; That is to secure our systems and networks.

      Next there have been several CS security czars, but all have quit because they have not had the response from the admin that they thought was needed.

      Will it take a terrorist hit to get us really thinking about all this? Nope. In spite of 9-11, we are still not really any more "secure" than we were on 9-10.

      What it will take is lawsuits against companies. A good one that is going through now, is the one where a guy got r
    • it is only a matter of time before these supercomputing powers may be used by a rogue state (for) crapflooding message boards.

      Look, North Korean secret agent guy, just because you have a supercomputer doesn't mean you need to go posting this ton of crap to Slashdot thanks very much.

      That's almost as ludicrous as Intel's faster CPU to surf the web faster theory. The trolls on slashdot are ample proof that you need nothing more than a low end pentium and lots of time to keep the drivel flowing all day.
    • by Anonymous Coward on Wednesday February 16, 2005 @07:14AM (#11687316)

      Keep in mind that these are Top500 KNOWN supercomputers.

      It's quite possible that many more computers that exist but are operating under classified conditions. For instance I would be suprised if the NSA had something for breaking crypto that rivaled some of the machines in the top 50 or so super computers, but it's not something that would appear on this list. This is for boasting rights only, if a place chooses not to publicize their computer, it won't end up on the list.

      American computers pretty much dominate. Some like the Spainish one are built by American companies (like IBM), but just happen to be installed in foreign places.

      Now the NEC Earth Simulator is a BIG exception. It dominated the top 500 for a considurable period of time and is completely japanese built, designed and owned. It is a huge technological acheivement and the pinnacle of technology for it's day. It's hard to relate to how significant it was. It is the top of the line when it comes to old-school massively parrallel supercomputers, blew American supercomputers out of the water.

      But along came Linux clusters (which in many ways is competely unsuitable for some of the things that Earth Simulator can do) and now Blue Gene. Which can possibly do twice the work as Earth Simulator, but consumes a fraction of the power and space needed to house these types of computers.
      • There is a lot of stuff you can find out about the NSA but there is no way to know just how much computing power they have, but as for basic idea. They have around 50 ACRES or more of supercomputing floor space. They run there own fab creating custom CPU's. And they dropped off a 200CPU system to their museum because it got 3 years old and so was not worth keeping... So being conservative lets say they have around 10 200CPU systems per acre and 50 acres so that's around 100,000CPU's. So my guess is they
    • Please don't let there be a Secretary of Information Technology. The creation of cabinet level posts is like 1984. We create a Homeland Security Secretary to instill fear and insecurity with his "Terror Alert" system. We have a Secretary of Defense who invades other countries. Our Secretary of Education is wrapped up in testing to the point of interfering with actual education.

      I shudder to think of what a "Secretary of Information Technology" would be. In my tinfoil sheilded skull I imagine a jack boot on

    • Back before the US government had Terrorists to scare us with, there were COMMIES! COMMIES UNDER THE BEDS! Commies Plotting to corrupt our Precious Bodily Fluids! Ohhhh, Nooooooo!!! And instead of Weapons of Mass Destruction, they had NU-CU-Lur Bombs!

      Anyway, the Feds had a bunch of export control laws to prevent Commies from getting Big Computers that they could use to design better Nukes, as well as laws to prevent Commmies (and Americans) from getting crypto.

      • Those of us who were on Usenet 20 yea
  • by bandix (184495) <bandixNO@SPAMgeekpunk.net> on Wednesday February 16, 2005 @05:17AM (#11687017) Homepage
    This is all about timely and focused execution. The speed at which this project was realized is important. Consider: from the initial concept in late December of 2003 to assembling the computer in Madrid took less than a year. Normally, this kind of supercomputer projects take years.

    Lame!

    SGI had NASA AMES' Columbia online in 120 days, and landed #2 on the Top500.
    • "This is all about timely and focused execution"

      SGI had NASA AMES' Columbia online in 120 days, and landed #2 on the Top500.

      I am sure that what they mean is that every phase of the project was according to schedule, specifically including the PR efforts.

  • Sounds like (Score:5, Funny)

    by jim_v2000 (818799) on Wednesday February 16, 2005 @05:20AM (#11687021)
    Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers.

    Sounds like the specs of Microsoft's Xbox 3...
  • by Anonymous Coward
    I'm sorry, but why are we giving IBM free press again? For god's sake, the very first sentence of the freaking article is utter rubbish:

    The MareNostrum supercomputer at the Barcelona Supercomputing Center, ranked number four in the world in speed in November 2004, is constructed of such totally off-the-shelf parts as IBM BladeCenter JS20 servers, 64-bit 970FX PowerPC processors, TotalStorage DS4100 storage servers, and Linux 2.6

    That's it - the first sentence of the article, if you exclude the title, the
  • ...from easily available components...

    Screw easily available. Anyone with a budget to buy 2,282 servers and 4,564 processors can afford custom parts.

    Call me when it's also easily affordable and I can pick up the parts at my local Fry's, or better yet, my local supermarket.

    The only real item of interest was that it was made with all IBM parts and Linux.

    ...

    Come to think of it, if Fry's also sold groceries I won't have to shop anywhere else...

    • Well, if you really need a supercomputer, the first step is not to get the computer, but instead the funding. As super computers go, if your needs are served by a distributed computing environmet, an "entry level" supercomputer does not really cost all that much compared to "traditional" supercomputers. Yes we are still talking about hundreds of thousands of dollars. (there seems to still be 256 processor 2.8ghz xeon based computers with gig ethernet connectivity on the latest top 500 list) If you need to
    • You can design and manufacture all your components as easily and quickly as you can buy servers from IBM???
    • not a plug but try aldis cheap groceries and they sell medion computers and accessories I got a 200 gig external hard drive for 69 dollars in a town where the going rate is over 150 for 160 gig drives so maybe that will solve some of your problems
    • In the long run, these will be too cheap to charge.
      Ray Kurzweil told me so.
  • by Anonymous Coward
    Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers.

    It's just the thing to find SHA1 collisions of ISO images in 2^56 operations...

  • ...but how fast can it open Photoshop CS?

  • but I bet Windows still runs slow on it.
  • by Anonymous Coward
    REALY, DOES IT????
  • Given enough money how is that impressive anymore? What's the best single-thread-performance machine today?
    • Yeah, that's the point. Also, they don't even do any advance in programming models. Just MPI/PVM with my old friend fortran or C.

      Haha, but the funny thing is that they'll end doing high performance computing based on Web Services!!

      Best regards,

      diego.

  • I just love how every time someone writes about another grid or supercomputer or beowulf cluster they always say "easily available components" as if I could find most of them in a standard IT closet or just run down to the local computer shop and pick them up with my corp. AMEX.

    In what world is 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers easily available??? Maybe if you are Big Blue, but then, why would it be more difficult for
    • If you have the money, the components are readily available and tested. It is allmost allways much more difficult to create everyting from scracth than using components that are allready designed and working. Thus the money needed to build it from scracth would be much higher and timeframe longer.
  • by nicc777 (614519)
    ...does it have an AGP slot?
  • Get a load of Xserves, install Xgrid or something else [vt.edu], plug in, process (it seems it takes about no time to assemble [vt.edu] and it runs osX).

    Add more Xserves to get more power...
  • by woah (781250) on Wednesday February 16, 2005 @07:40AM (#11687366)
    Mare Nostrum is Latin for "our sea"?

    But I thought it was Ouray Easay!

    What's going on?

    This is all so confusing! I need to take a nap.

  • No memory specs? (Score:3, Interesting)

    by PornMaster (749461) on Wednesday February 16, 2005 @09:09AM (#11687703) Homepage
    Am I the only one finding it curious that nowhere in the specs do they mention how much RAM there is per node or in the aggregate?

    It mentions how many nodes, how many CPUs, how many racks, how much storage, but not how much RAM.
  • Some one get these guys [msu.edu] one of these computers, see how long it takes for their virii to evolve enough to take over the world!
  • CELL Supercomputer (Score:2, Interesting)

    by Omegalomaniac (26573)
    The scary thing is, that the CELL with its potential for 25 GFlops of double precision floating point, could rival this system with just 1600 8 SPE units.

    Granted, the CELL isn't exactly off the shelf, and I'm willing to bet 4,564 970FXs will be cheaper than 1600 CELLs for quite some time, so the project still has merit.
  • from easily available components (like BladeCenter and TotalStorage servers, 970FX PowerPC processors

    Sounds to me like building the biggest supercomputer today is nothing more than throwing more money at it than your compeition. No real technical skill needed beyond beyond that of wiring up your new home entertainment center a few hundred times over.

  • I will chip in 1 slightly used 3 pronged US power cable, black.

MATH AND ALCOHOL DON'T MIX! Please, don't drink and derive. Mathematicians Against Drunk Deriving

Working...