Linus And Alan Settle On A New VM System

stylewagon writes: "ZDNet are reporting that Linus Torvalds and Alan Cox have finally agreed on which Virtual Memory manager to include in future kernel releases. Both have agreed to use the newer VM, written by Andrea Arcangeli, from kernel version 2.4.10 onwards. Read more in the article."

sensation seekers

[selmer]

There never really was a big dispute about the VM subsystem, read alan's diary [linux.org.uk] for an account of what happened in his opinion.

I cite November 2nd: The great VM dispute really isn't. It went something along the lines of "Putting a new vm in 2.4.10 is crzy", "Probably it was but its done so lets make it work" and at 2.4.14pre8 "See it works" "Yep".

NUMA?!

[ssimpson]

It's previously been argued that Andrea's VM doesn't work with NUMA architectures, hence work should continue on Rik's 2.4.x design

Not a problem now, but it's one of the major aims of 2.5, according to Linus. Anyone know how they are going to square this circle?

Minor point

[Anders]

From the article: Torvalds is close to handing over the stable 2.4 kernel to Cox.

...and I thought that Marcelo Tosatti was going to maintain 2.4 [slashdot.org].

Re:NUMA?!

[Alan Cox]

Getting NUMA to work well with Andrea/Marcelo's VM might be more interesting. On the other hand Martin and the other IBM folks don't seem duly perturbed on list so I'm not that worried

Re:This article is a joke...

[Syberghost]

It also says that Alan Cox will take over 2.4 once 2. 5 is opened which is wrong...

Just in case somebody doesn't believe you, here's the proof [advogato.org].

Re:So what?

[stilwebm]

For one, the new VM is less likely to begin killing processes or swapping randomly. Also, the new VM works in a more simple (relatively speaking) way that uses about 1/2 the swap space as the previous 2.4 VM. Like the VM system in 2.2.x kernels, having 128MB of RAM and a 128MB swap file will result in about 256MB of virtual memory available to the system. In the previous 2.4.x VM system, a computer with 128MB of RAM needed 256MB of swap space to effectively use 256MB of virtual memory.

Rik van Riel on the Future of VM Work

[rmadmin]

I ran across an archive at OSDN.com that had a video from the 2.5 Kernel Summit (March 30th and 31st 2001). On the list of videos is 'Future VM Work presented by Rik Van Riel. Its a 1 Hour 4 Minute Video clip, but after listening to it for 5 minutes I knew it was WAY too technical for me. =P Anyways, if you want to see what he said about improving VM, head over to:

http://www.osdn.com/conferences/kernel [osdn.com]

They have Real format in both 56K and 128K streams, Mpeg, and Mp3 of his speach. Looks interesting if you've got the bandwidth and the time.

virtual memory

[Lethyos]

In brief and ONLY the basics... Modern operating systems today handle memory addressing in a virtual sense so that "fences" can be placed around memory owned by the OS and different applications. These fences serve to protect memory from being overwritten by other rouge programs. This works by making each program think that it's start of memory IS the actual start of physical memory. For example. Program "foobar" may be located at memory address 0xFF44 bytes, and have 0xFF bytes allocated to it. Instead of addressing its memory in the base:offset format as 0xFF44:0xFF, it thinks that 0xFF44 is actually address 0x00 and the top of memory is 0xFF. That way, it can't write to physical addresses at 0xFF43 or anything else lower. This range of memory can be broken into fragments and scattered through memory so that if other programs have been allocated since foobar started, it's not trapped.

Bear in mind, this is only the basic gist of what virtual memory is all about. This particlar subsystem will also handle memory paging (which is part of swapping out to disk), amongst other tasks.

Before you really get determined to start hacking the kernel tomorrow, I suggest you start with something a little more meager. You need to get some experience in computer arcitecture fundamentals, then really basic OS design. Read a few books. Learn Motorola or IA32 Assembly language. Learn to write some old DOS programs (a number of DOS emulators with free, open source DOS distros are available, so some searching) where you have to allocate every byte and word by hand, and not just say "Foo *f; f=new Foo();". Next, start to learn C and figure out what malloc() is all about. Then try coding a kernel module. This is obviously not an extensive road map, but computers and their operating systems are sophisticated. You can't really (unless you're someone like Cox or Torvalds) just dive right into systems programming and know what you're doing. It may take years of experience before you start to tinker with code in the kernel and actually write something that works.

Re:This article is a joke...

[dangermouse]

It claims that SuSE is a US company (funny how I could have sworn it was German ;o)

Just for the record (I've pointed this out elsewhere), SuSE Inc. is a U.S. company... SuSE has done the general equivalent of incorporating in several countries, including the U.S. SuSE Inc. is, in fact, in Oakland as the article reports.

Of course, it might not be SuSE's U.S. corporation that's involved here, but that's a completely different possible source of inaccuracy. ;)

My biggest problem with Linux was the old VM.

[TurboRoot]

We run on alot of small systems, we are talking 8-16 megs of ram here and pentium 75ish processors. We tried to use linux once, but when linux runs out of memory with the old VM, it sucked HARD. I mean, I had processes being swaped out of memory compleatly that were ACTIVE!

Why do we use such small systems? Because we want them to perform under extream load when placed on larger systems. Its smart really, its easy to benchmark a few functions on a pentium 75, than a 2 ghz pentium. If your application doesn't run peppy with one usuer on a pentirum 75, it sure as hell won't support 1000 users on 2 ghz pentium.

Thats why we have used FreeBSD for all this time, in FreeBSD the VM manager is perfect, and isnt' even slated for upgrade in the near future due to the fact it works like it should. If you are using telnet on a FreeBSD machine, and _one_ applications uses a ton of resources, that one application will run slow. But your telnet will continue on fine. Try putting 12 megs of RAM in your machine, than compiling PostgreSQL while using a telnet session. You won't even notice the compile on FreeBSD, but you will with Linux.

Funny enough, this also ties into the article earlier regarding why Linux isn't used for alot of large scale databases. Databases consume HUGE amounts of RAM and the OS under it has to be peppy about it. Linux in the past has been tuned for desktop/single user performance and not what those databases need. They need TONS of resources, and quick _CONSISTANT_ access to it.

That said, I am very happy to see them getting a better VM. Because my biggest problem with FreeBSD is its crappy java support, the most recent stable JDK it supports is 1.2.2. And thats in Linux emulation mode!

So if things work out, and Linux supports java well, and doesn' crap out when it runs out of resources. We will defiently switch to Linux, and life will be good!

Re:Details please

[Anonymous Coward]

They both had stability problems at 2.4.10. Both are fairly stable at this point.

Rik's VM is considerred "smarter." It works harder to balance things out and keeps track of all kinds of info.

Andrea's is not very well documented.

Andrea's is simpler.

Andrea's uses "class zones" which are not good for NUMA. We want to have NUMA in 2.5.

Andrea's is faster.

Linus is considerred the leader and Alan Cox is considerred the second in command. If it's a really tough call to make then probably going with Linus is the less contraversial thing to do.

When Linus switched VM's we had gone through 10 point releases of the kernel and the VM still wasn't as stable as it should have been. If Andrea's VM took 10 releases to become stable then that would have sucked. A lot of people pointed this out to Linus in a blunt way. People thought it might take nearly that long but it only took 4.

So it worked out in the end.

Between a rock and a hard place

[renehollan]

Oh, it is SOOO tempting to patch what you know, rather than start over with something new, and relatively untested, espescially when deadlines loom.

Of course, this might not fix the problem: either the patch doesn't fix enough, or the design is flawed to start with (I have not delved into either Linux VM to competently present an opinion here, just speculation on what might be wrong). But there is something to be said for "the devil you know". At least the problems with the old VM were fairly well known. Moving to a new VM could potentially introduce new ones. Not something you want to do close to release.

Now, those plagued by problems with the older VM might, in exasperation, think anything would be better. Enter Linus with Andrea's alternative: "looks good, ship it!" [my paraphrasing]. Those of us who'd tremble at the thought of new, untested code, could, well, stick to an older kernel, even if it meant giving up some new features.

But wait! It's Alan to the rescue!! Picking up all the relatively easy fixes and enhancements, and giving me a choice. Leave the contentious parts till later..

It strikes me that the minor, temporary -ac fork served both camps of users until the issues were resolved.

Some might argue for a more disciplined approach, and not make major changes so close to release. (But, if it isn't "ready", why not just postpone release? Is Linus feeling pressure to release prematurely? Or just trying to release "often enough".) But that stands in the way of progress -- I've seen managers crippled by fear of changing anything. Sometimes you gotta take a chance, even if some things break while (many) others get fixed.

While smaller, digestable kernel changes might be more palatable, they're already available in interim development releases. Sometimes a change is sufficiently profound that it touches everything (yes, that's an argument to refactor, but hindsight is 20/20, so I won't argue that point too much) and takes a while.

I am not one to say which approach is globally better -- I can only comment on what works better for me. I can say, though, that when a community is really split over a course of action, and any single choice will not satisfy a large number of people, a fork, even if temporary, is probably the least painful route in the short term, even as the long term consequences are undesirable if it goes on too long.

After all, all progress is a mini-fork from "stable" to "untested".

Re:NUMA?!

[tkrotchko]

If you're wondering what NUMA stands for (as I did), you can get a good 2 sentence definition at the Webopedia [internet.com]

SuSE Inc. is a sub of SuSE AG, Germany

[heilbron]

just for the sake of correctness:

SuSE is a wholly owned subsidiary of SuSE AG, Germany. Although its US web site doesn't explicitly state that it may be deduced from
http://www.suse.com/us/company/legal/index.html
...