Byte Benchmarks Various Linux Trees 270
urbanjunkie writes: "Moshe Bar has an interesting article, essentially benchmarking the standard kernel (with aa VM) against the -ac kernel (with Rik's VM)." He also raises some very interesting points about how patches (and entire development trees) interact.
(with a VM) (Score:1)
Re:Seti@home (Score:2)
I don't know you guys, but my all-time favorite benchmark tool is Seti@Home. =) The distro which spits out more work units, that's what I'll take. And my linux box does 5 times as fast as my w32, mind you. But you already knew that, did you? =)
You don't happen to run commandline client on Linux and the 'gui' client on Windows, do you? ;) The difference between gui and commandline client is huge on both Windows and Linux.
Interesting conclusion... (Score:3, Interesting)
While I personally may not have agreed with this synopsis prior to reading the article (and am still not completely sold...), there are certainly some interesting facts and figures to ponder the next time you reload your system or update your kernel...
Re:Interesting conclusion... (Score:3, Interesting)
I think perhaps that we should start having "kernel" races if you will, and we could have various categories (ie, I/O races, Stress test races) in which the various trees would compete and by-and-by the best kernel trees would become known.
Please, I would like to hear your comments!
Re:Interesting conclusion... (Score:5, Funny)
Correct me if IM wrong, but dont races in the kernel cuase horrible security problems?? And who would decide the race conditions?
Re:Interesting conclusion... (Score:1)
As for who would decide the race conditions, Im sure that woulnd tbe too hard to come up, of course the obvious downpart would be if the racers themselves choose the race conditions but i think that could be solved no?
Re:Interesting conclusion... (Score:5, Interesting)
One minor nitpick though ... I never released an -rmap VM [surriel.com] against 2.4.18-pre3, the latest is still against 2.4.17. I suspect that the crashes Moshe saw are due to some change in 2.4.18-pre3 conflicting with the -rmap VM patch, especially since rmap-11c has survived the kernel torture lab at RH. ;)
If you support forks so much... (Score:1)
Or is the article incorrect?
Re:If you support forks so much... (Score:5, Informative)
My -rmap [surriel.com] VM is a patch against marcelo's standard 2.4 kernel, because that is the thing people have. It just doesn't make sense to release patches against kernels nobody has.
Also note that -rmap replaces pretty much all parts from the -aa VM I don't agree with, while at the same time integrating some parts from the -aa VM that I do like.
question... (Score:2)
I compiled and ran the "memory hogger" application
and it did not eat more than 424K on FreeBSD..
I know you have a lot more to do thatn to answer silly simple questions BUT.. why is that
Thx in advance..
thx (Score:2)
Re:Interesting conclusion... (Score:1)
[quote]
while Rik restricted himself to question whether the standard Linux kernel would ever even finish a stress test
[/quote]
Of course I'm far from serious with this remark, I'm glad to see a Dutchman "die graag zijn kop boven het maaiveld uitsteekt" (let's see how google translates that
PS. Actually, this is a test to verify whether our average
Re:Interesting conclusion... (Score:1, Funny)
"who willingly one's pate upstairs the mowing field uitsteekt"
I'm guessing it kinda looses something in the translation...
Re:Interesting conclusion... (Score:2, Insightful)
Immediately, I see a great deal of benefit that could be derived from such a venture for the common Linux user. This information could not only render which kernel is the "fastest", but could also provide information on which kernel design is best optimized for a given task. Effectively, we would have "Open Information on Open Source" (TM).
Alternatively, however, I would never want to see this go to the extreme of a kernel tree being abandoned nor neglected as a result of these "Kernel Olympics" - variety is the spice of life, and IMHO the strength and diversity that drives Open Source!!!
Red Hat 2.4.9 is a very good kernel, fast...WHY? (Score:4, Interesting)
Re:Red Hat 2.4.9 is a very good kernel, fast...WHY (Score:5, Informative)
Re:Red Hat 2.4.9 is a very good kernel, fast...WHY (Score:4, Interesting)
Re:Red Hat 2.4.9 is a very good kernel, fast...WHY (Score:3, Informative)
RH 2.4.9 is a lot like the ac kernels. Mainly because, Alan runs that part of the RH distro. But, there are many concerns that RedHat addresses. They work with corperate customers and partners (like Oracle) to make sure the kernel they ship is as stable and fast as they can get it. So the RH kernel does diverge from the "plain" AC kernel.
Of course they submit all their patches back to Linus. But Linus just hasn't been keeping up with them. Linux accepts patches based on three factors: his previous experience with the developer submitting the patch, the "correctness" of the patch, and the phase of the moon. And the phase of the moon is the dominant factor, because even Alan Cox complains that Linus won't accept his patches.
So the RH kernel is excellent because Alan Cox, RedHat, and RedHat's corperate partners make sure the kernel is fully fleshed out. This is the kind of vetting that Linus doesn't do.
"If it compiles it is Good, if it boots it is Perfect!" -Linus Torvals.
Because of the patches... but WHICH patches? (Score:2)
It seems like this may be the key source of competitive advantage for a Linux distribution vendor: the know-how to optimize the kernel and other software to make a fast, stable system.
Pournelle ? (Score:1, Redundant)
> As Jerry Pournelle pointed
I always though he's a sience-fiction author?
---
Re:Pournelle ? (Score:1, Offtopic)
He also is(was?) the author of the "User's Column" in BYTE magazine.
I haven't seen nor heard of him in the last decade, although I did talk to him on the phone once. (I did tech support and called him back.)
Re:Pournelle ? (Score:2)
He's still writing - good classic Science Fiction, and still writing Chaos Manor, his collumn for Byte. He's also got a site that is similar to Slashdot in "umm... vaguely geeky stuff, lots of visitor feedback and opinionated as hell". It is located (in all its hellishly difficult to navigate, 'spend a few hours finding the good stuff buried deeply' glory) here [jerrypournelle.com].
--
Evan "Many many years spent with Byte" E.
Re:Pournelle ? (Score:2)
Re:Pournelle ? (Score:1)
[ot] So what's the best kernel to get right now? (Score:4, Interesting)
I'm sitting at home with my fresh install of RH 7.1 and I'm wondering what kernel to upgrade it to. Any suggestions? Is there a stable one in there somewhere that I should go with? Should I stick with the default kernel that's on their now?
If I'm regularly compiling new programs using gcc or g++, is it safe to go from one tree to another, as long as they're all 2.4.x, or what? Do I need to recompile with a new kernel? Or is that a red herring?
Re:[ot] So what's the best kernel to get right now (Score:4, Informative)
You should use RH 7.2 instead. It comes with kernel 2.4.7 or something, but can (and should) be upgraded to RedHat's kernel 2.4.9 via up2date. The RedHat kernel is quite stable and fast.
Re:[ot] So what's the best kernel to get right now (Score:2)
rpm -Uvh kernel-2.4.17-athlon.rpm or whatever it was and rebooted. I'm sure everyone will chime in on how 'evil' it is to install a kernel by rpm, but for my typical desktop box, it's no big deal.
I think you'd need to get the new modutils package, but i don't recall.
Re:[ot] So what's the best kernel to get right now (Score:2)
The current kernel supported by Red Hat for Red Hat Linux 7.1 is 2.4.9-21, which you can see does a good job in this test.
2.4.17 of course! (Score:2)
Before 2.4.17, I couldn't get sound to work with Return to Castle Wolfenstein... trying to run the game with sound would just put the machine into a probable race state... I'd try to run a program and it would just hang... switch to another virt term login run top and hang...
I'd have to restart the machine because I couldn't even kill the processes. But with 2.4.17, NO MORE PROBLEMS.
Me is a happy camper.
Lemme Guess... (Score:2)
Switch to netgear cards / ng_tulip driver. I've never tried to gauge them for performance, but at least the driver is solid.
The folly of Open Source (Score:1, Flamebait)
However, as was discussed on
With closed source, a single person can make all the final decisions about what will make it into the final release, but Open Source has no such system. A person who feels that his ideas are not being taken seriously can fork the project and create his own possibly incompatible tree. This seems to be what is happening now with Linux.
At the early stages of tree forking, there is usually little problem. Most of the forks coexist with each other peacefully with only minor manual tweaking to fix any conflicts. Over time, though, further development along one of the forks will inherently diverge further from the root tree.
The argument that good patches will make it back into the root tree, thus preventing fairly deep splitting, is fallacious. Many of the reasons for forking are precisely because the root project refuses to take a fix from a contributor. What happens afterwards is that users will get 'locked into' a particular strain of the project and will be unable to incorporate other good features that are present in other strains which, for whatever reason, can't be merged into their chosen strain.
Kernel hacking just isn't as fun as it used to be.
Do you think the BSD projects reflect this? (Score:4, Insightful)
The advantage here is that with three BSDs you have three separately-tuned operating systems that attack different problems very well, yet maintain a certain level of commonality and compatibility.
Call me a starry-eyed optimist, but my exposure to this open source fad started with the Wired article in the autumn of 1997. In it the writer painted a picture of a 'computing epic', one collectively started and maintained. I still think that metaphor is accurate and useful.
Re:The folly of Open Source (Score:2)
The process of having these trees sprout up and played with by businesses and advanced users provide a sort of natural selection for improved evolution of the system. While all this complicated mess plays itself out among the bleeding edge people, the common users have the stock redhat or what have you kernel to play it safe with.
Re:The folly of Open Source (Score:2)
Keeping the realtime code in the main code tree would be counter productive -- confusing the development of the larger Linux tree, but not providing much of anything to the 98% of us who have little need for hard realtime in our systems.
Re:The folly of Open Source (Score:2, Interesting)
Can you provide some high profile forking examples that have occured in the recent history of Open Source, out of interest? Or evidence of similar forks in Linux. Not, to coin a phrase(?), "soft forks" as I have mentioned here, but "hard forks". Fundemental changes in ideology or personality clashes that have seriously caused a split in the community?
Free vs. Open vs. Net BSD (Score:1)
Re:The folly of Open Source (Score:2, Insightful)
Value in diversity (Score:1)
You have *diversity*. Unlike the M$ world where you don't.
Sure, it is hairy at first... but once you get used to the fact that Linux is all about diversity... having choices... many, many choices... you come to like it. In fact, the monolithic Windows really gets on your nerves once you fully appreciate Linux.
Re:The folly of Open Source (Score:2, Insightful)
All Rob was suggesting was a formalisation of a de facto practise. Hardly a fundamental paradigm shift in kernel development.
It did unearth a number of gripes people had; eg. Linus dropping maintainer patches. The thrust seems to be that Linus wants to trust ten or so people, and only accept patches from them. So if Linus is dropping your patches, push them on to a 'trusted' maintainer instead.
IMHO Linus needs to step forward and deal with this a little more assertively. But everyone loves a good bit of controversy, and the Linux kernel is the tall poppy of the open-source community that everyone wants to chop down ...
hmm (Score:1, Funny)
Re:hmm (Score:3, Informative)
Why is this funny? (Score:5, Insightful)
C'mon guys. Show a little open-mindedness. One of the things I really missed from the Windows world when I switched to Linux was the "Windows Update" feature. Want to install the latest security or feature patches? Click a check box and hit "install". No dependencies, no patch conflicts, no esoteric config options, it just worked. Admittedly Ximian's Red Carpet comes close, but it's still a little quirky sometimes.
I know there will always be those people who want to manually tweak their kernel (god bless 'em!). There's a lot of us, though, that don't want to deal with it. I'd rather have one-click shopping for all of my patch needs so that I can spend more time writing code or playing Quake. MS understands this. Apple understands this. Why doesn't the Linux community understand?
Re:Why is this funny? (Score:2)
That means 'Aunt Tilly' never even has to even update her box, ever.
Granted the typical 'desktop user' wouldn't know how to do this
Re:Why is this funny? (Score:1)
That's bullshit. Latest security patches find their way to windows update several days, if not weeks, after their initial release. Windows Update is great service but unfortunately Microsoft uses it mainly for distributing the latest version of IE/MP/Messenger. I'm sure that one of the factors causing long delays is that I don't use an english language version of Windows, but my friedns who do say it isn't much better. I hope to see more security patches in Windows Update now that Microsoft has promised to take the matter seriously, we'll see how it goes.
And before anyone says this is just FUD from a linux fanboy let me say that the only OS in my PC is Windows 2000 and I have no plans to change that.
Re:Why is this funny? (Score:2, Interesting)
freebsd has it
in fact both of those also upgrade third party software that's not part of the OS
windows update will never upgrade mozilla for you or KDE whereas apt-update & cvsup-portsupgrade upgrade EVERYTHING you've installed (if installed via apt-get or
[my understanding of the debian process is through the grapevine so if I'm off the mark, be cool; not a fool]
Re:Why is this funny? (Score:2, Interesting)
I installed windows 2000 on my new Inspiron 8100 yesterday. And I used the so cool 'Windows update' function.
Note that it requires to use IE (No IE, no windows update? At least the help doesn't describe any command line method.)
Until I had the drivers installed, everything was done with IE 5.0, in 640x480 mode, and (I think) 8 bits; so ugly!
And if you do that from home on a modem, you have to restart your connection at every reboot. That's optimum.
And when this is finished, you can now start installing non-windows applications!
I forgot to mention that the first time I installed windows 2000 on a logical partition, is wasn't able to boot it after I installed Linux. Impossible to repair as my disk was not detected correctly.
[Perhaps because w2k doesn't support DMA 100.
When I face this kind of problems I am pretty happy to be able to recompile a new kernel.]
I completely reinstalled it on a new Primary partition and, after I installed the drivers, I had a crash (blue screen) during reboot. [Note I was updating the win modem driver while using it to download the driver from the web]. Anyway, it was impossible to restart (crashing even in FailSafe mode or with 'last known good configuration'), and impossible to repair even with an up-to-date ERD. Had to restart the install from scratch for the third time! Damn!
On the other hand, I have installed Debian 3.0 (aka testing) from floppies on the same machine. I rebooted once during install, and then another once again after I installed and tweaked the new kernel.
Of course, you are not obliged to compile the kernel. I bet that if you use Mandrake or Red Hat, you will have a pretty good hardware detection process. I just like Debian and like to tweak the kernel.
I did everything on the command line. Didn't have to have a full graphical installation (I don't like too much laptop built-in mouses).
To summarize:
Windows: requires graphical environment and a mouse to install correctly plus requires having at least 9 non-necessary reboots (with modem disconnections
Debian GNU/Linux: two reboots (one optional). All in command line, with a 15 seconds boot. No problem. There are perhaps GUI front-end to apt, but I do not need them. Can something be simpler than 'apt-get upgrade'?
I let you decide who is the easier and who is my winner.
So to come back to the subject of the thread: download kernel and patches.
I could go on...
Riels rmap is nice...... (Score:3, Informative)
It makes a big difference on loading the hell out of my woefully under memoried workstation at work. My home machine, used mostly for surfing has seen a dramatic imporvment in free memory and is no longer swappin , it has 512 meg, so it really shouldnt swap too much, but was constantly with the stock redhat kernel, as well as 2.4.17 plain vanilla, trhe 2.4.18 -rpma12a has been ROCK solid.
On my server however with 256 megs , the stock redhat roll has done nicley with the minimal load its under.
I am a little leary about using the rmap in prouction as of yet, it seems to be killing things each nigh, (no shit) that dont drop with 2.4.17 or 2.4.9
I would like to see an option at configure to select a VM, I think the preemptive added would be fun too, I know its a pain because of the way it all intergrates to the other code, but thats my desire, it seems to be alot of other peoples desire as well, its funny how I bitched and moaned about the Riel VM , that was in the kernel prior to AA's , but since then and all the patching that was done I think Riels would give it a run for its money
Re:Riels rmap is nice...... (Score:5, Insightful)
Interesting. I've not managed to run into bugs like that on my computers here, so you must be running a very different workload to trigger such a bug.
Would you have the time to help me debug this problem and is it still happening with the latest rmap VM [surriel.com] ?
Stable kernel? (Score:4, Interesting)
So far the VM has been replaced twice, and now the rmap patch is apparently going to be added despite the fact that "something is seriously messed up in the reverse-map implementation".
Have they saved any experimental code for the 2.5.x kernels, or will that now be stable?
Re:Stable kernel? (Score:5, Informative)
(and personally, I'd prefer to keep -rmap separate for quite a while more ... development is much more efficient in a fork)
I stand corrected (Score:1)
Re:Stable kernel? (Score:1)
Sheepishly,
Jason.
Re:Stable kernel? (Score:2)
Sad.
Re:Stable kernel? (Score:2)
Re:Stable kernel? (Score:2)
Now, by keeping the interfaces stable, they usually make the kernels more stable (in the uptime sense) as well, but that's not exactly the goal.
If you blindly upgrade to the latest kernel, you will be bitten. It'd be like grabbing Mozilla nightlies, or if MS released IE nightly builds, or whatever. There will be a few stinkers.
Frankly, if you're the type to complain when a kernel is buggy instead of sending in bug reports, you probably should be at least two full releases behind the latest. (When 2.4.17 comes out, upgrade to 2.4.15) The exception being when there's been a big hole found in a line of kernels. And if you need the stability, consider downgrading to the kernel before the problem one instead of upgrading to a less-tested one.
Why use an unstable patch? (Score:3, Informative)
There are various patches like the Robert Love's [tech9.net] preempt patch which might be considered production quality. And perhaps some collections of production quality patches exist out there. But I wouldn't say -ac or -dj are in that category.
Or any of the patches marked 'preXYZ'. They're 'pre' for a reason you know. I'd be thrashing them on test servers, then giving feedback to the maintainer of that series. Let the maintainer declare them stable first.
You'll find in environments ambivalent to Linux that you really need to prove its stability to management first. Trying a new whiz bang kernel can have unforeseen side effects, in meetings that you'll never be invited to; and whose outcome you will only learn when it's too late to change it. "We let Bill convert server X to Linux and then it corrupted the filesystem. Clearly Linux carries more business risk than expected."
AA vs stock kernel (Score:1)
It's about the uptime, stupid! (Score:5, Insightful)
If a user compiles 35 gigs of code on a 6 processor box and it takes 5 minutes longer, he is not going to complain. If he compiles 35 gigs of code on a 6 processor box and it crashes half-way through the compile, your going to here it from your boss.
Benchmarking kernels is plain pointless. Take a machine for each kernel, put it under real load and tell me how many times it crashes in 100 days, and I will you which kernel I want to use.
Number of crashes in 100 days? (Score:3, Interesting)
I can't have multiple crashes in 100 days. If you are doing real load, spend real money, get real systems.
Don't build machines with your screw driver, get QA'd servers. Don't roll your own kernel, let Redhat test it.
These types of tests are useful as commentary and recommendations for what people should do in the development process.
It's about the frames per second, stupid! (Score:2, Funny)
This is nearly impossible to measure (Score:3, Interesting)
You could try to take a statistical run at it, of course, but I suspect the number of machines required to give meaningful results would be on the order of Google's farm.
Re:It's about the uptime, stupid! (Score:2)
A friend of mine actually forced Win2k down to the minimum amount of swap allowed (2MB IIRC) to avoid this problem.
In contrast, my FreeBSD box (my home desktop machine) feels much snappier (although I use Windowmaker and no desktop) and the cacheing works wonderfully. I've noticed the cache helping me along on several occasions (generally when I'm organizing directories or programming and lots of things start happening instantaniously as the files are still in cache).
Re:It's about the uptime, stupid! (Score:2)
Really? I never have this problem at all. Win2k has always been snappy as it comes for me.
This is wild speculation, but there's an option (in conrol panel->system, I believe) to switch between optimized for applications and and optimized for background services. You may want to check and make sure that's set correctly.
Re:It's about the uptime, stupid! (Score:2)
You must not run Win2k with very large (40-200MB resident) applications that get sent to the background much do you? Even Mozilla triggers this behavior on a consistant basis.
And don't get me started on the sounds in Win2k. Every time that stupid beep plays the kernel chugs for a moment to read in the sound effect, slaughtering any hope you have of maintaining say a bitstream to a CD-Burner.
The VM is one area where MS really pooched Win2k IMHO.
Re:It's about the uptime, stupid! (Score:2)
Alltogether, though, I think we can both agree that VMs and kernel speed *is* important, especially in a desktop enviroment. We've both had bad experiences with GUIs due to kernel problems. Which is really the point I was trying to originally make.
Rik van Riel soaking up the karma (Score:2, Funny)
1. Write a VM.
2. Get a fine article written about your work
3. Have somebody post the article on
4. Post. Post. Post.
Rik's put in at least 3 comments in this tree and they're all being mod'ed up.
I'd better start on my own VM!
(Or, I'll write a reverse disk-sector mapper)
Hopefully Someone Has an Answer... (Score:3, Interesting)
Since I've seen posts from at least one kernel developer in response to the attached story, I figured that this might be a good place to ask the following question:
A little while ago, I wrote an application that uses an incredible amount of memory... A very space inefficient implementation of Eratosthenes' Sieve [utm.edu]. In essence, what the algorithm does is cycle through the entire contents of memory sequentially many, many times (not a completely correct description). What I found with the following three kernel versions:
My question for the Kernel gods out there is as follows: are there any stable 2.4.x kernel releases out there that would handle this type of stress without the performance degradation that I've experienced with these kernels?
Re:Hopefully Someone Has an Answer... (Score:1)
As soon as you fill up physical memory, you're using swap space. The kernel will either swap other programs to disk to make room for yours, or it will swap parts of yours out that aren't being used. Either way, disk i/o is very expensive - hence the performance decrease. No kernel can help that, really. Once you're out of memory, you're out.
Re:Hopefully Someone Has an Answer... (Score:1)
I'm sorry that I wasn't clearer in the original post... With all three kernel versions, what I experienced was not only performance degradation, but a change in runtimes from about 15 minutes to over 8 hours. Unfortunately, I never managed to really quantify times or run a thorough set of benchmarks, but it seems that such a drastic difference in runtimes would imply that the VM's in all of these kernels are not handling a worst-case scenario correctly.
Re:Hopefully Someone Has an Answer... (Score:2)
Besides, think about the access time differences between RAM and your hard drive... you're going from a few nanoseconds to a few milliseconds; that's a thousand times slower. Your test went from 15 minutes to 480 minutes, which is only an increase of a factor of 32. That means it was talking to swap what, 30% of the time?
I'm not entirely sure how the math breaks down here, exactly, but it really doesn't seem too bad considering the OS itself (and anything else you might have had running) was taking up some of that physical RAM, too.
A suggestion (Score:1, Offtopic)
<speculation> AFAIK kernel uses a hairy buddy system (but I did not check that myself), if that is the case, not whole memory is accessible in an arbitrary allocation sequence. So your analysis of exceeding 1024Megs of memory by less than 10 Megs is incorrect. You have to allocate progressively smaller page sizes that follow fibinocci(sp) series (or something like that, see your local kernel hacker for more info) if you want whole your data in memory. </Speculation>
I thought about it some more... (Score:3, Interesting)
If everything goes according well, your kernel should swap out and swap in exactly M amount of memory for each preaccesing pass, where M is the amount of data that does not fit in the main memory. So if you have total memory T, total data T+M and K*J size prefetch, total swapped data per whole process would be M*(T+M)/(K*J).
Re:I thought about it some more... (Score:2)
If your algorithm is doing something bloody quick (optimized CRC, etc) you'll probably work faster than the disk I/O. If however you're calculating weather patterns or Conway's Life, or anything involving a bit of CPU and reprocessing the block of data in a non-linear fashion, you'll likely be okay.
If the routine involves looping through 1GB of data 100 times and you don't have the RAM, it will end up reading 100GB from disk, eventually. Your process may be dead simple and go from 15M for a 990MB dataset to 8Hr for 1GB, or it might be complex and go from 7:50 for 990MB to 8HR for 1GB, but if it takes 7:45 to read the 100GB, you've established a lower-bound on how long it'll take to process.
What were you doing to this data as you looped through it?
You might want to consider doing your own VM. If you're just 5% or so over available RAM you could manage this by writing to a file and deallocating two 6% chunks of memory back and forth. Make then each as far apart (in the dataset) as possible so that after loading and processing one, you have as much time as possible to let the VM load and precache the next set without blocking.
Re:Hopefully Someone Has an Answer... (Score:3, Insightful)
The reason VMs usually work well is because most programs don't actively use all the memory they allocate at once, so the VM swaps out the memory that isn't being used. If your program uses all the memory it allocates, the VM has no choice but to use the slow disk to store some of it. No magic VM will solve your troubles.
Re:Hopefully Someone Has an Answer... (Score:2)
Please, please, give us the algorithm that will detect this usage pattern without either using great gobs of memory to record past usage, or using a lot of CPU time. Until someone does this, the LRU algorithm is simple and works fairly well for the most programs.
On a case-by-case basis, it's possible to revise most of the exceptional algorithms so as not to fight with LRU memory management:
Multimedia playing, where the most recently used (played) page is never needed again: free() it! De-allocated pages aren't saved, I hope. Only an idiot would keep all the objects generated and used just once - but there must be a lot of idiots writing programs.
Sieve of Eratosthenes:
(1) Compress the array as much as possible so as to keep as much in memory as possible. You don't need to store the numbers, just whether they are crossed out yet or not. Use an array of bits, initialized to zero, where b(i) is set to 1 when number i is "crossed out." Maybe there are even more clever storage schemes?
(2) Change the bottom to top sweep to a back and forth sweep. That is, the sieve takes the next prime number p and "crosses out" numbers p+p, p+p+p, and so on to the end. Then it comes back to the beginning, finds the next prime number (not crossed out) p1, and crosses out all the multiples of p1. So let's do this sweep in reverse. Integer divide t = n/p1, where n is the last number in your array. Cross off t (which is in the 2nd to most recently used page), t-p1, t-p1-p1,
(3) Rearrange the algorithm to take a whole block of primes and apply all of them to one block (page) of the array at a time. Or, if you want a _really_ big array, write it to apply several thousand primes to one track of the hard drive at a time...
#1 is obvious and easy, if you are using a language that supports 1 bit objects. #2 doubles the code, but the program is so short that it doesn't really matter. #3 takes some thought and clever coding to run efficiently and accurately.
The problem is, there aren't enough really good programmers out there to figure out how to optimize every program so as to fool every VM into allocating the memory efficiently. So it certainly would help if there was a way to _tell_ the VM to switch to a different allocation scheme for particular objects. Problem: objects (to the programmer) and pages (to the VM) are not in any particular correspondence. The Sieve program, for instance, really needs one page for scalar variables (p, n, i, etc.), which is always being referenced, and so must stay in memory under the LRU method, while the array takes many pages which are best swapped as MRU. What you actually get from a C compiler is something like the first page containing both scalars and the beginning of the array. So you cannot tell the VM to swap all pages containing the array by MRU...
Re:Hopefully Someone Has an Answer... (Score:2)
The first is to have algorithms for five or six different VM algorithms, a LRU, an MRU, and some more complex ones optimized for various tasks. Then, start everything off with the one that proves best in a random sampling of programs. But, while actually using the LRU (for instance) to swap, let the other routines simulate their own swapping. When one starts to pull ahead of the others, use it, but keep simulating.
Now, as long as there is plenty of swap space, regardless of free physical memory, it may be a win to keep large statistical studies of memory usage in different processes. Of course, this sort of thing would get turfed when the system actually started to run out of memory, but in the meantime the slight slowdown due to having less available RAM might be offset by swapping less, or with proper speculative swapping, not having to halt a process.
Also of use would be memory pools, to avoid having to kill userspace processes just to let the kernel alocate enough memory to keep running.
Of course, there will always be pathological conditions which render even the best VM incapable of dealing. (Imagine an FTP server that didn't write out uploaded files to disk, keeping it all in memory, eventually it'd run out or physical and virtual memory. Or the case of a process running some study on the data which necessitated accessing pages in an effectively random manner.)
I think we should work on putting an upper limit on swap time instead of making it a little snappier for every day usage. (I'd sacrifice 3fps in Quake3 to keep Photoshop from effectively locking my computer when I manipulate 200+ MB images.) We should also consider killing processes to be the very last resort before forbidding the kernel any more memory, and this should come about only from a bug in the kernel. (All parts should be designed as not to grow unbounded.)
But then, I'm an armchair VM designer. It's an educated guess, but still only a guess.
Re:Hopefully Someone Has an Answer... (Score:2)
How about a hybrid strategy: exempt the 25% of pages that were most recently used from swapping, but then pick the one(s) to swap out randomly from the other 75%. For the Sieve, this would mean that the few pages used for the program and the continually accessed scalar variables would stay in memory, along with OS pages, and the rest of the top 25% would be waste space. But the other 75% would be managed by random swapping, which is a lot better than what happens with the normal algorithms. And for more normal programs -- with 256M RAM becoming standard, anything that is in frequent use should stay in the most recently used 64M and not be affected by the randomizing.
Or is it practical for an OS to recognize when VM is thrashing, and shift strategies from the normal (LRU or some approximation) to random?
Re:Hopefully Someone Has an Answer... (Score:2, Informative)
(The real answer is less simplistic, but this is close enough to the truth.)
Re:Hopefully Someone Has an Answer... (Score:2)
But yes, it would easily solve the program of accessing 1.1GB of data on a 1GB system. You simply tell the VM that
This could be implemented as an out-of-band request to the VM, or could be handled by writing a new malloc() that's backwards compatible.
You'd call newmalloc() in the program, specifying how much RAM, what swap pattern is likely to work, and if any of it should be considered higher priority than the rest. Ideally this would be processed by system libraries and the improved VM would read it, but on a system without those libraries the _newmalloc() function in your program is used, and it simply sums up what you've asked for and places a simple call to malloc(), giving you the ram, but without the performance options.
I'm sure a few high-end systems have this sort of thing, I can't imagine writing weather simulations to run on a Cray for instance, and having to use your own manual hacks for memory management, so I'd assume they have something... I doubt they bothered with the fallback though, most things written for a Cray wouldn't be worth running elsewhere.
Don't understand Moshe's conclusion (Score:4, Interesting)
"From the above figures it seems that the old van Riel VM is somewhat faster (considerably faster in the case of 2.4.9) than the new Arcangeli VM..."
Is my math wrong? The RVR VM in 2.4.9 is ever so slightly faster on the 2nd run and slower on the 5th, and the slowest of all is the newer one in 2.4.18pre3rmap. What's my mistake?
Moshe's politely indicating that van Riel was an ass when asked for comments; we can conclude either that Moshe didn't have a proper recent RMAP kernel to test with (as a result), or that the recent RMAP kernels are hit and miss.
From looking at van Riel's comments here, he vehemently believes his kernel is perfect and Moshe just got it wrong... The problem is that lots of people seem to "get it wrong" with that VM, including Linus... Overall in Rik I get the sense of an aggressive person who may have trouble admitting mistakes or accepting failure; not good traits in a programmer, since it's humility and communication skills which can often be the critical factors in a team programming effort... and lack of them can cause exactly the kinds of problems we've observed.
Re:Don't understand Moshe's conclusion (Score:3, Insightful)
ehh... when i think "top rank" open-source programmer, I think: Theo de Raadt (his early work on netbsd/sun alone is brilliance), RMS (GCC, emacs, etc), Jordan Hubbard (of FreeBSD fame), Linus Torvalds, Alan Cox, etc.
Of that group, Only Jordan and Alan strike me as having "humility and communications skills." Linus is often curt, sometimes cordial, and sometimes very rude. Theo is almost always rude. One look at RMS' TCL outburst and 'communications skills' obviously don't belong in the same sentance with that guy. I'm not trying to troll or flame here, I'm just saying that expecting open source (or Free Software, whatever floats your boat) programmers who run insanely popular projects and who get treated like gods whenever they show up in Slashdot (ooh.. it's Alan, +5 - not that he doesn't usually deserve it), at a Con, or on IRC or something... expecting these guys to have normal-sized human egos is asking a lot. It's clearly asking too much of most of them.
also please note that i'm not slotting rick in next to AC, Linus and John... yet. Just making a comparison.
Re:Don't understand Moshe's conclusion (Score:2)
come on... the guy takes pride in his work, enjoys working creatively in the linux kernel code. he not believe his kernel is perfect (why would he conclude the rmap is still under development?). how about a little love? you know a show of hands for AA, RVR, Alan, hell even Linus. they spend their time writing, developing, arguing over a linux kernel so i can have a choice in the OS i put on my machine. so i can spend my time developing wannabe c++ applications for KDE. thanks guys for the work and dedication.
why didn't author measure page faults? (Score:4, Informative)
So how do you measure page faults? Be sure your kernel is configured with "BSD process accounting". Then use a shell like tcsh. The man page of tcsh describes the "time" variable, you can set it to report the number of major/minor page faults that occured during the lifetime of the process.
I did my own unscientific test back in November. I ran 32 simultaneous instances of mpg123 on a just-booted machine. Among other things I measured the number of page faults. The results for the then-current kernels I had were:
those number are the means of the 32 values from each process. Anyway, you get the idea.
This is all well and good (Score:2, Interesting)
Kernel Test Suites Article (Score:4, Informative)
Yes, I post the link to this here all the time, I think it's useful to people.
Rik van Riel Chat (Score:4, Funny)
Re:Rik van Riel Chat (Score:2)
He's the fucking pimp.
50 karma in a single story.
Re:Rik van Riel Chat (Score:2)
You've got one of the leading mojo's in Linux Kernel development prettymuch being interviewed by "the people" here and answering Q's etc, and you decide to lambast him as a "whore" because people are actually interested in what he has to say. Note that Linus, RMS , ERS , A-Cox or any of them pretty much have never(as far as I remember) have shown up in here, so that's a pretty nifty thing IMHO
And for christ sake, promoting interesting responses what Karma is for you idiots. (and Karma cap stops it going nuts). Get a life.
Comparision of evolution.. (Score:2, Interesting)
I have to disagree with the notion that this isn't Exactly how evolution occurs. Evolution isn't Random, Virus are the vector for almost all radical natural genetic manipulation. The purpouse of a virus is to use the resources of a host organism to replicate it's DNA sequences because Virus are too small to replicate themselves. In doing so It injects it's DNA inside a host cell and basically exploits the RNA strands inside to replicate it's own DNA. While doing this The virus can in fanct find new DNA within the host and borrow it for it's own protection. In many cases this is to make it resistant to antibodies produced by the body (HIV.) Now, not all Virus destroy the host cell especially when antibodies destroy it before it can complete it's task. In some instances the Virus may act as a Vecor Borrowing the DNA from one species, and inserting that code in another. Many virus can cross infect species. For example humans and pigs can catch influenza from each other. Geese and pigs can also catch influenza from each other, while humans and geese cannot infect each other with influenza.
Virus are acting for thier own self promotion and preservation. When a DNA stand from one species makes that species less able to destroy them they would try to splice that DNA into as many species as they can. Comparing that to kernel developers
is pretty straight forward. They try to 'infect' the kernel tree with the 'code' they've produced for any number of reasons. Being known for coding on linux, to get promotions at a linux friendly workplace, for the challenge and fun of contributing to the linux kernel, or just to fix something so that they can do something with linux that they were trying to do but couldn't.
This introduces variation along the same analog as virus changed DNA. As for 'uncorrected errors' in the DNA strand the only thing we can prove comes from that is cancer. Thus that type of 'mutation' is analog to 'bugs' in the code of the linux kernel. Unfortunately humans aren't anywhere near as good as the roughly 99.7% error correction rate of replicating double helix DNA strands, so code tends to get a lot more malignant tumors (root exploits) to cut out.
server this, server that (Score:3, Interesting)
Re:server this, server that (Score:2)
picking nits (Score:3, Insightful)
I'm going to disagree with this notion of evolution. Evolution is not undirected. The current environment gives a good deal of direction to the sorts of evolution that occurs. For example: evolution appropriate to tropical beaches is unlikely to occur in the arctic tundra.
Similarly, Evolution in the Linux world is also mostly in reaction to environmental needs. Where the difference in randomness comes is that the mutations that lead to biological evolution are generally random in nature -- but environment and statistics choose which mutations lead to enhanced viability.
For Linux, patches are generally in direct response to specific needs. The nature of these changes are directed by nature but generally random in form -- ranging from the icky to the elegant. Fork maintainers like Torvalds and Cox are more like the social interactions which can shift the survivability of an otherwise brilliant mutation/patch. Although this social rejectin will deeply affect survivability, an especially bright change can still give a survivabillity edge that makes up for the rejection
This is really the pleasant aspect of the Linux community; exactly those people who are busy and sought after by many journalists and hackers are also those who take time to answer questions with enthusiasm and a very positive attitude. Thank you Andrea, thank you Alan.
This is something of a chicken and egg proposition. Those people "who take time to answer questions with enthusiasm and a very positive attitude" are precisely those who will likely be sought out by Journalists and others. I mean -- come on! Are you going to take your question to someone who regularly beats into submission anybody who comes to them with (what they consider) a non-profound queston?
Journalists, especially often need their question answered today , and can't be bothered to wait for someone who berates them for half an hour for asking a straightforward question -- especially if they then forget what the origina question was (no known anectdote comes to my mind).
Re:picking nits (Score:2)
Re:More nits (Score:2)
Evolution is directed by the needs generated by (or arising from) the immediate environment.
and -- yes, I agree. The earth and nature have no needs, per se. As an environmentalist, I would say that we have needs, and if we sate our needs and desires in a way inconsistent with what Mother nature 'needs' (i.e. what would be required for it to stay stable and supportive of thriving human life), we're going to pay bigtime, in the future.
Re:what's the point? (Score:1)
That's where Linux shines compared to the *BSDs. Sure, use it on your headless servers, but for a b0x at home, most simply prefer Linux.
Re:what's the point? (Score:3, Offtopic)
There are only two advantages I see to Linux that makes it a better "desktop" OS. First, they put DRI in the kernel. They're working on this now for FreeBSD, but there's a lot of resistance to keep userland stuff out of kernelland. If all you care about is running games, then FreeBSD is not it. Go get a PS2 or an XBox.
Second, the popularity of Linux gives it a greater pool of developers to draw from. So Linux gets new device drivers faster. But you still will *not* get Linux shoehorned into this week's premium super buy at CompUSA. With Linux you have to wait around three months to get a driver for brand new hardware. With FreeBSD you have to wait about six months. If you buy computers more often than once every six months, stick with Linux. As for myself, I had no problems with FreeBSD on a stock Dell Optiplex GX240.
In terms of desktop software, don't worry a bit about it. They're the same on both platforms. Identical. Staroffice, GNOME, KDE, Xmms, Gimp, Mozilla, etc.
Re:what's the point? (Score:2)
I venture to say I am not alone, in that when I upgrade, I always upgraded to hardware that was already out at least 6 months anyway (even when I ran windows), that is where the sweet spot is in price/performance ratio usually.
So, one could buy a new system every 6 months, with no driver hassles at all, if they just trail the market, like I'm sure millions do. (at least everyone who isn't rich and wants the most for their money).
Re:what's the point? (Score:1)
There is just no such problem, imho.
Re:Ummm no. (Score:1)
Following your way of thinking, Windows would have to be more mature than both, Linux and BSD....
I don't see how 'maturity' and 'time of existence' are possibly related.
Try XP (Score:1, Troll)