Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Linux Software Hardware

2.5.65 On 32-way NUMA-Q with Preempt Enabled 131

Posted by timothy from the linux-kernel-2.5.65-that-is dept.
_iris writes "I think the subject speaks for itself. Here is the link to the story on KernelTrap." In case you have a spare 32-processor machine munching grass in the back 40.
This discussion has been archived. No new comments can be posted.

2.5.65 On 32-way NUMA-Q with Preempt Enabled More

2.5.65 On 32-way NUMA-Q with Preempt Enabled

Comments Filter:
  • Good to see the Linux kernel making such leaps and bounds.

    Keep at it guys!
    • Such leaps and bounds, ah yes. A quote from one of the followups to the article:
      These machines have been in production since something like 1996 and were EOL'd around 2001. i.e. this is not just 5 years old, its entire product line is 2 or 3 years dead.
      Yup, Linux, so up to date it's just beginning to suport hardware that hasn't been built for 2 years.

      • Re:Congratulations to the Linux Developers (Score:5, Informative)

        by Hellkitten ( 574820 ) on Wednesday April 09, 2003 @07:44AM (#5692239)

        Yup, Linux, so up to date it's just beginning to suport hardware that hasn't been built for 2 years

        The point is that 2.5.65 booted with preemtion patches on a 32 processor machine

        That is preemtion of kernel threads. If there is a deadlock or race condition it would be more likely to show up un a beast like that than in your average dual athlon. So this is really not about supporting 32 processors (which is old news) but about the quality of the work that has gone into kernel preemtion

        I have no idea if any other OSes out there support preemtion of kernel threads running on multiple cpus. Anyone care to enlighten me?

        • Re:Congratulations to the Linux Developers (Score:5, Informative)

          by julesh ( 229690 ) on Wednesday April 09, 2003 @08:47AM (#5692457)
          I have no idea if any other OSes out there support preemtion of kernel threads running on multiple cpus. Anyone care to enlighten me?

          Solaris. I believe Mach does as well. There are probably others that aren't as well known.
          • Irix comes to mind as an immediate example, and probably every single hard real time OS out there like VxWorks. You can't support hard real time without preemption of kernel threads as your user service may be more important than said kernel thread (think about a program that decides when to lower the flaps on an airplane vs. the kernel thread that flushes dirty buffers to disk - clearly, you'd want the "lower flaps" thing to have priority!)

            As an asside, Linux has been running on large NUMA systems before

    • Yep, maybe Linux is ready to step out of kindergarten soon. SGI IRIX supports 1024 CPU's in NUMA if I recall correct, so Linux has alot of progress to make then it comes to single-image scalability. Well Microsoft is still stuck in kindergarten too, and have not yet managed to move beyond basic 32bit which most UNIX'es did 10 years ago.:)

  • Beowoulf (Score:2, Funny)

    by rf0 ( 159958 )
    I would like a ... oh wait it is :)

    Rus

    • Re:Beowoulf (Score:1, Offtopic)

      by Andy Tanenbaum ( 655028 )
      Paraphrased from the FAQ on beowulf.org, a beowulf cluster is a "kind of high-performance massively parallel computer built primarily out of commodity hardware components, running a free-software operating system like Linux or FreeBSD, interconnected by a private high-speed network." These 32 processor machines are single machines, and not nodes connected by a network. Definitionally, these machines are not Beowulf clusters.
      • hwaet, we gardena in geardagum . . .

        ENG 301 best class ever. a few weeks ago someone dropped an info sheet for beowulf cluster development in his mailbox. apparently he thought it would be a good idea to ask in class 'so, anyone know what this is?'

        ha

  • NUMA-Q (Score:1, Funny)

    by duplo ( 253071 )
    Nuns Under Management of Al-Qaeda

  • 32 Proc ? (Score:4, Funny)

    by MosesJones ( 55544 ) on Wednesday April 09, 2003 @06:47AM (#5692089) Homepage

    Sheesh, I'm sitting here with a 64 Way and two 32-way boxes just waiting for decent to run on them.

    Does this mean that FINALLY I can shift Quake Server off the clustered S80s in the basement ?

  • Munching grass? (Score:4, Funny)

    by Eric Ass Raymond ( 662593 ) on Wednesday April 09, 2003 @06:47AM (#5692090) Journal
    In case you have a spare 32-processor machine munching grass

    I'd rather have a girlfriend who is also into muching carpet.

  • Cool (Score:3, Informative)

    by rf0 ( 159958 ) <rghf@fsck.me.uk> on Wednesday April 09, 2003 @06:48AM (#5692097) Homepage
    Now this is cool. I know that SGI can scale the Altix to 64 CPU's running 2.4 with their own additions in an SSI. However not sure about. 2.5. Its nice to see it in the main kernel anyway and the only way is up

    Rus

    • Re:Cool (Score:1, Informative)

      by Anonymous Coward
      You can expect the 2.5 to be way way better than 2.4 + SGI addons.
    • What I see this as is good news for SGI. If the std kernel tree can support NUMA @ there spec level, it gives them one more reason to have to dump more time and $ into development and allows them to spend more elsewhere. Esp since they would like to move away from IRIX in the long run.
    • In fairness that's without a metarouter. Throw in that, and it scales all the way up to 512 processors (with 1024 planned for the future). This isn't some dodgy clustering crap, it does turn it into a 512 processor shared memory box. Now if only it wasn't Itanium...

  • Man and i though i had it good (Score:3, Funny)

    by tokaok ( 623635 ) on Wednesday April 09, 2003 @06:49AM (#5692100)
    i thought i my daily 3-ways were good but it seems ive been missing out :(

  • Complete article (Score:5, Informative)

    by blackcat++ ( 168398 ) on Wednesday April 09, 2003 @06:49AM (#5692102)
    Here is the complete article, the fscking lameness filter made it quite a struggle to get it posted here. Anyway:

    Zwane Mwaikambo announced today on the lkml that he's successfully boot the 2.5.65 development kernel on a 32-way NUMA-Q server with -preempt enabled. Speaking to Robert Love [interview], the kernel preemption maintainer, he began his announcement saying, "Robert, I suppose you can add another notch on your erm.. bedpost(?) and congratulations to all the kernel developers!" NUMA awareness in the scheduler was added into the development kernel in late January [story].

    William Lee Irwin III [interview] explained the significance of this achievement:

    "This has had a hard time historically. I'm really glad NUMA-Q's are now immune (in the sense of correctness) to this config; previously it was believed that preemption points in printk(linux_banner) would take out the machine early in boot if preemption was enabled. Congratulations rml! If you're booting without issues on these things, you are a _very_ long way toward being race-free. This is incredibly good news, both for the preemption support, and for the general stability of the i386 bootstrap."

    Read on for the full thread.

    From: Zwane Mwaikambo
    Subject: 2.5.65-preempt booting on 32way NUMAQ
    Date: Sun, 6 Apr 2003 06:48:33 -0400 (EDT)

    Robert i suppose you can add another notch on your erm.. bedpost(?) and congratulations to all the kernel developers! It survived some local networking stress tests, but there is more fun stuff like tty layer to completely obliterate ;)

    (Hardware courtesy of OSDL)
    Running configuration
    32 Processors, PIII 500
    32G RAM

    Patches required:
    2.5.65 (only because isp1020 decided to get huffy)
    Purge assign_irq_vector panic - Zwane Mwaikambo

    [boot messages]

    From: Robert Love
    Subject: Re: 2.5.65-preempt booting on 32way NUMAQ
    Date: 06 Apr 2003 14:28:42 -0400

    On Sun, 2003-04-06 at 06:48, Zwane Mwaikambo wrote:
    > Robert i suppose you can add another notch on our erm.. bedpost(?)
    > and congratulations to all the kernel developers! It survived some
    > local networking stress tests, but there is more fun stuff like tty
    > layer to completely obliterate ;)

    Excellent, Zwane.

    Congratulations! Good work.

    Robert Love

    From: William Lee Irwin III
    Subject: Re: 2.5.65-preempt booting on 32way NUMAQ
    Date: Sun, 6 Apr 2003 04:23:40 -0700

    On Sun, Apr 06, 2003 at 06:48:33AM -0400, Zwane Mwaikambo wrote:
    > Robert i suppose you can add another notch on our erm.. bedpost(?)
    > and congratulations to all the kernel developers! It survived some
    > local networking stress tests, but there is more fun stuff like tty
    > layer to completely obliterate ;)

    Wow!

    This has had a hard time historically. I'm really glad NUMA-Q's are now immune (in the sense of correctness) to this config; previously it was
    believed that preemption points in printk(linux_banner) would take out the machine early in boot if preemption was enabled.

    Congratulations rml!

    If you're booting without issues on these things, you are a _very_ long way toward being race-free. This is incredibly good news, both for the preemption support, and for the general stability of the i386 bootstrap.

    All that's really left is driver and non-i386 arch coverage if I'm right.

    -- wli

    From: Zwane Mwaikambo
    Subject: Re: 2.5.65-preempt booting on 32way NUMAQ
    Date: Sun, 6 Apr 2003 07:25:09 -0400 (EDT)

    On Sun, 6 Apr 2003, William Lee Irwin III wrote:

    > This has had a hard time historically. I'm really glad NUMA-Q's are now
    > immune (in the sense of correctness) to this config; previously it was
    > believed that preemption points in printk(linux_banner) would take out
    > the machine early in boot if preemption was enabled.

    Which kernel version was that from

  • That sounds really cool! (Score:4, Funny)

    by wheany ( 460585 ) <wheany+sd@iki.fi> on Wednesday April 09, 2003 @06:57AM (#5692115) Homepage Journal
    What the hell are you guys talking about?

  • Superb (Score:2, Funny)

    by vesamies ( 240247 )
    Yes, it is now possible to launch 32 preemptive NUMA-Q missiles strikes simultaneously using the Linux kernel. Excelent!!!

  • Hidden Meaning (Score:4, Funny)

    by evilviper ( 135110 ) on Wednesday April 09, 2003 @06:58AM (#5692119) Journal
    I think the subject speaks for itself.

    Translation: I'm going to bed, and the editors are lazy.
  • I think the subject speaks for itself
    If only it spoke in a language I can understand

  • what does that mean. (Score:5, Interesting)

    by leuk_he ( 194174 ) on Wednesday April 09, 2003 @07:04AM (#5692132) Homepage Journal
    NUMA, ok, that i understand.

    (Instead of one big shared memory pool it uses processors that each have their own pool, and can access other memory with a timing penalty)

    but what does "-preempt " have to do with this. what does this option do? Int unix always preemtive?

    • Re:what does that mean. (Score:5, Informative)

      by platypus ( 18156 ) on Wednesday April 09, 2003 @07:09AM (#5692144) Homepage
      Preemt means preemtive in kernel space, you are talking about userspace. kerneltrap has an interview IIRC with Robert Love where the ins and outs are explained, if not, try google.

    • Re:what does that mean. (Score:1, Informative)

      by Anonymous Coward
      The Linux kernel usually isn't. That leads to high latency when the kernel does something which takes a long time. With the preemptive kernel patch, the kernel can no longer make certain assumptions, which means it has to synchronize some things which were guaranteed to complete without interruption before. The usual problems arise: Deadlocks, missing synchronization, etc.
    • but what does "-preempt " have to do with this. what does this option do? Int unix always preemtive?

      Short answer:
      Userland processes can always be preempted. This patch allows the kernel to be preempted (as a side effect the kernel also has to be reentrant). It may help some latency issues (don't get blocked in the kernel so much) though decrease overall throughput (more time spent on context switches instead of real work). Not quite usre how it helps massive parallelism, probably allows the kernel to b
  • Well this is great to hear in my opinon but I'm just wondering if this will add to the Law Suit that SCO has on the plate for Linux. It's pretty sad when someone put's a lot of time and work into a project and then someone, like myself, questions if it will hurt more then help Linux. I love hearing about such innovation though and I'd like to say great job by all...
    • The SCO lawsuit has nothing to do with this at all. If anything, SCO will get money out of IBM, they aren't threatening to illegalize the kernel source or anything. You're blowing this waaaay out of proportion. The source is in the clear.
      • Blowing it out of proportion by asking if their will be any reprocussions? I understand :)

        I'm not saying the source is not "in the clear" or IP of SCO but if I remember correctly the ability to handle 32 processors was a ability that was present in UNIX not Linux, right? I was just curious if anyone thought if it would affect the lawsuit in anyway at all. With that in mind did IBM contribute in any way to this project? I'm just being curious :)
      • Not really. You seem to forget that if IBM is caught "being guilty" of IP violations, they may turn around and ask for royalties from all Linux users. In other words, the IBM/SCO case would at as a legal precedent to allow them to enforce their IP and gleen royalties from it.

        Having said that, someone asking a question is hardly blowing something out of proportion!

  • Stop stealing ideas from SCO! (Score:4, Funny)

    by Billly Gates ( 198444 ) on Wednesday April 09, 2003 @07:16AM (#5692160) Journal
    We all know that SCO invented NUMA and SMP. Jeesh.

    • Re:Stop stealing ideas from SCO! (Score:1, Interesting)

      by Anonymous Coward
      Why was this modded up as funny? SCO did actually invent SMP on x86, which has lately been some touchy subject on Mandrake forums [mandrakeforum.com]. The final decision was the copyrighted SCO code will be removed from both the Linux kernel and drakconf in the next release.

  • Inevitable (Score:5, Funny)

    by Ed Avis ( 5917 ) <ed@membled.com> on Wednesday April 09, 2003 @07:20AM (#5692166) Homepage
    Taco's Law: any story about massive scalability will be posted on a web server which craps out due to 'too many connections'.

    Anyone got a mirror?

  • Correct URL (Score:5, Informative)

    by lemmen ( 48986 ) on Wednesday April 09, 2003 @07:20AM (#5692167) Homepage
    It seems the URL isn't working because of the session ID. Use this link instead if you get a "to many connections" error.

    http://www.kerneltrap.org/node.php?id=628 [kerneltrap.org]

    Have fun!

  • *sigh* (Score:2, Interesting)

    by Anonymous Coward

    Would you have all been as interested in this story if you'd known:

    • It's been on KernelTrap for a *couple days* now
    • The machine in question is about 5 years old
    • Linux already has been booting on large-number-of-CPU machines for awhile. This story is about the CONFIG_PREEMPT subsubfeature.
    • This has very little to do with Linux's scalability. If I booted Linux 2.5.x on a bazillion CPU monster tomorrow, it wouldn't be a bazillion times faster than my single-CPU desktop system.

    *sigh*

    • ..the fact that IBM's eServer xSeries 440 is a NUMA-Q box that can scale to 16 Processors now? It is a NUMA-Q box...
    • Well, while you are right that *this* has very little to do with scalability of linux, you might be delighted to see [tldp.org] that $big_number-cpu systems are profiting quite well from newest linux scalability work (read till the end of that page).

      105.02user 14.50system 0:04.83elapsed 2474%CPU
      (0avgtext+0avgdata 0maxresident)k0inputs+0outputs
      (394245major+570713minor)pagefaults 0swaps

      isn't too shabby for compiling a whole kernel, is it?
    • It's been on KernelTrap for a *couple days* now

      Oh no!! Not a couple of days! And I almost read it, too - I feel so dirty now. To think that a number of actual days has passed since the story first appeared and I have failed to read it. Damn.

  • Here is a Mirror List [man.ac.uk]

  • Kernel 2.6 release (Score:2, Interesting)

    by Anonymous Coward
    when?

  • The greatest thing about Slashdot... (Score:5, Funny)

    by mofolotopo ( 458966 ) on Wednesday April 09, 2003 @08:12AM (#5692306)
    Is that occasionally there are headlines like this that I can read, re-read, and still have no clue what the article's actually about. I don't know what ANY of that stuff means.

    • Re:The greatest thing about Slashdot... (Score:1, Funny)

      by Anonymous Coward
      The meaning of the article is quite simple: Not only is the 2.5 version of the Linux kernel on the way, it is so much on the way that it is on 32-way.
    • NUMA == Non-Uniform Memory Access. From the NUMA FAQ [sourceforge.net]: Non-Uniform Memory Access means that it will take longer to access some regions of memory than others. This is due to the fact that some regions of memory are on physically different busses from other regions.

  • ./ server (Score:2, Interesting)

    by mad27 ( 630687 )
    Would be nice if /. mirrored the stories it links to. This way only news >1 day old is accessible :(
    • I've been discussing this with another /. user. Perhaps if wget fired up and did a recursive web-suck on all articles that are linked to in the abstract, tar'd and gzip'd each one, and then mirrored them with BitTorrent [bitconjurer.org]? That way the OSDN doesn't have to pay for the bandwidth.

      The other solution I've heard is posting the articles to FreeNet [freenetproject.org] and linking to the key. More convenient (you don't have to de-tar the articles) but much slower.
  • We're just about to mothball a 32 processor SP, maybe I'll drag it home. It could do double duty as a furnace.

  • This is mucho needed than you may think... (Score:1, Informative)

    by Anonymous Coward
    IBM's x440 shipping 16-way NUMA/SMP system will benefit from this significantly...this machine will also soon be going to 32-way capability...so for those of you thinking there's no hardware out there today to take advantage of this...guess again.

    IBM x440 [ibm.com]

  • More info (Score:2, Informative)

    by mdw162 ( 654188 )
    It refers to preemptable work on BSD, but here [bsdvault.net] is a good general description of kernel preemption.

  • Stability vs Features (Score:4, Insightful)

    by the-dude-man ( 629634 ) on Wednesday April 09, 2003 @09:20AM (#5692640)
    Its good they finally got it to boot...but still...i think there are far to many bugs in printk. I've had the 2.5's barf on me quite a bit because of this, and it only seems to get worse as it spans out over more proccessors. I think we need to proritize here. The kernel devlopers should be focusing more on stablilzing the 2.5.x kernels rather than adding loads and loads of new features. The recent benchmarks show the 2.5.x kernels are lagging way behind 2.4 and even 2.3 kernels. I think we need to stop loaded all the pretty new features for a minute and focus on getting what we have right now to work. I still have problems with ntfs writing out malformed blocks :|

    There is alot of cool stuff in the new 2.5.x kernels i will admit, and i look forward to using it, but as it stands i cant put a 2.5 kernel anyweres but on my home machine because once it hits a production envoirnment it craps itself. I know its just a devlopment release, but lets get it speed up a little before we start working on features for distrubted systems :)
    • Maybe things have changes since the last time I followed Linux development with any care, but I seem to recall that the odd kernel releases - that is, 2.1, 2.3, etc - are all development kernels. That is, that is the point where they add "loads and loads of new features". The even numbered kernels, 2.2, 2.4, etc, are the release kernels. That is, the ones where things have been stablized to the point of being put on "industrial grade" systems. So the fact that your 2.5 kernel starts crapping out when yo
  • the kernel is getting kind of bougoise these days. With one name and all. They don't even bother to say what 2.5.65 refers too!

    I guess Linux is going mainstream. Maybe I'll stop calling it RedHat or Mandrake, and just call it "2.5..."
    • > I guess Linux is going mainstream. Maybe I'll stop calling it RedHat or Mandrake, and just call it "2.5..."

      Bah! The Mandrake OS is already at 9.1, therefore it's 3.5 times as advanced as Linux!

  • ...but I fail to realize the signifikance of this news item. Anyone of the in-the-know crowd: care to enlighten me? 32 bla NUMA-Q what?
  • I found Unisys has some 32-way IA-32 some machines [unisys.com] but they only support Windows (and for this kind of money the vendor is going to damn well support my OS).

    Who else makes a 32-way system? Does anybody have recommendations? What do these things cost? With a good set of scripts and/or something like MPI a rack of 16 2-way servers is nearly trival to manage and utilize, so the integrated systems neeed to be around $25K to be interesting.
    • Actually, I just looked closer at the spec sheet (pdf) and it mentions both unixware and SCO OpenLinux support. Not that I'd run either one of those on it. I bet Gentoo linux could do some interesting things on that, tho.

Slashdot Top Deals

I have hardly ever known a mathematician who was capable of reasoning. -- Plato

Close