Behind the Scenes in Kernel Development 139
An anonymous reader writes "Some interesting changes took place in the way the Linux kernel is developed and tested. In many ways, the methods used to develop the Linux kernel are much the same today as they were 3 years ago. However, several key changes have improved overall stability as well as quality. This article takes a look behind the scenes at the tools, tests, and techniques -- from revision control and regression testing to bugtracking and list keeping -- that helped make 2.6 a better kernel than any that have come before it." We might as well mention here (again) that a couple of new kernels are out: leif.singer writes "2.6.3 and 2.4.25 are out, fixing another vulnerability in do_mremap()."
Kernel development interests me terribly (Score:5, Interesting)
Would these tests be a good starting place?
Re:Kernel development interests me terribly (Score:5, Informative)
Much time is spent teaching people how to write code but never really reading it. This is a perfect example of how to do it and why you would.
Re:Kernel development interests me terribly (Score:5, Insightful)
Reading code can be a huge help in becomeing a better coder. You see how other coders do things. Learning from their bad on what not to do and seeing new good methods you may not have come up with on your own.
Re:Kernel development interests me terribly (Score:5, Informative)
He talks about browsing code, package structures, adding features or fixing bugs in a large codebase, and so on. It's a good read - well worth the money.
Re:Kernel development interests me terribly (Score:5, Funny)
Does the fact that Diomidis Spinellis has repeated won [ioccc.org] the International Obfuscated C Code Contest (IOCCC [ioccc.org]) make him more or less qualified to write such a book :-)? Check out his "best abuse of the rules" entry from 1988 that is my all-time favorite. BTW, the contest is currently open.
Re:Kernel development interests me terribly (Score:3, Informative)
> repeated won the International Obfuscated
> C Code Contest (IOCCC)
Heh
Then he goes on to suggest that preprocessor usage be minimized in 'normal' programming activities
Re:Kernel development interests me terribly (Score:3, Insightful)
Re:Kernel development interests me terribly (Score:2)
I'm a firm believer that all CS curriculum should have two classes for the sophomore year entitled "Real World Programming Part 1 & 2". Part 1 consists of an entire semester writing a user application to meet a loose set of design specifications.
In part 2, the students drop their own code and inherit another student's codebase from part 1. Part 2 will consist of dropping 30% of the functionality, altering another 40% of it and then adding in another set of features about half the size of the
Re:Kernel development interests me terribly (Score:1)
I manage a department, which includes about 20 programmers and some do it right from the start (and right out of a school), and some don't.
What is interesting, while not maybe surprising, is that women tend to document their code better and also write more maintainable code. This observation is prone to errors (most of the prog
Re:Kernel development interests me terribly (Score:1)
Re:Kernel development interests me terribly (Score:5, Funny)
You could contribute some work to SCO: I hear they're very interested in having someone sprinkle several "printk("(c) SCO\n");" lines here and there in init/main.c, since they can't do it themselves, having no technical department, being a law firm and all...
Re:Kernel development interests me terribly (Score:3, Informative)
"Linux Kernel Development" [amazon.com] is a nice introduction book by kernel hacker Robert Love, and it already covers the 2.6 Kernel.
It doesn't go into too much detail, but it gives a very good overview and basic understanding of the issues you have to deal with in the kernel! I'm currently reading it and getting enlightened :-)
Recommended book on the Linux kernel (Score:2, Informative)
Re:Kernel development interests me terribly (Score:2)
It's a bit expensive, since it's also apparently a textbook, but I think it is worth it. Reads easy, too.
Re:Kernel development interests me terribly (Score:5, Interesting)
Very recently, I've been writing some low-level code. There was a long while I'd thought this was out of my league. Then I realized several things:
So I think the moral of the story is to just be fearless/persistent. If you're not confident, there are plenty of ways you can improve without even involving anyone else:
And then, if you're still not comfortable talking on the linux-kernel list, I think you have at least another couple choices:
As for myself, I'm taking my own advice to make sigsafe [slamb.org] - an alternate set of system call wrappers (libc level) that eliminate a couple race conditions involving signals, without a performance penalty. It's going well - the code works, and I have a race condition checker and microbenchmark to prove it. I just released my first version. Now I'm working on the documentation; it still needs a lot of work. (I could use plenty of help with this project! If you want to try low-level programming, it's a great way. It requires writing assembly for each combination of operating system and architecture. I've only written it for two systems. There are plenty left, and public systems to do it on if you don't have access to exotic machines of your own. Plus, you can hopefully gain some low-level understanding by proof-reading and helping me write the documentation.)
Once I have that polished, I've got a couple projects I might try in the Linux kernel (and/or other kernels):
Re:Kernel development interests me terribly (Score:4, Insightful)
An addendum: your experiments don't have to be anything spectacular. Here are a few:
Another project I've considered is adding likely() and unlikely() equivalents to boost [boost.org]. A lot of people use this code, so increasing its performance a little would be pretty beneficial.
Re:Kernel development interests me terribly (Score:2)
Is there any nifty way to speed up the compile->execute cycle? The way I see me coding is:
a)code
b)compile
c)reboot test machine and wait 1-3 minutes for it to come up
d)see if it worked
e)goto (a)
step C could be frustrating, is there a quicker way to go about it?
Also, what is the likely() and unlikely() functions you speak of. Google shows a lot of unrelated info.
Thanks
Re:Kernel development interests me terribly (Score:4, Informative)
[...]
c)reboot test machine and wait 1-3 minutes for it to come up
[...]
step C could be frustrating, is there a quicker way to go about it?
You could making your changes to a User-mode Linux [sourceforge.net] kernel to avoid the reboot. Or running it inside a virtual machine [vmware.com]. That way you only have the kernel's boot time, not the main system BIOS, ATA-100 BIOS, SCSI BIOS, etc.
Also, what is the likely() and unlikely() functions you speak of. Google shows a lot of unrelated info.
They're macros that tell the compiler if the expression contained within is likely to be true or false. There's an article [lwn.net] about them here. If you've ever seen any code that mentions __builtin_expect, it's the same thing with better names:
Re:Kernel development interests me terribly (Score:1, Informative)
Safe compiler cache if you do "make clean" a lot.
distcc - http://distcc.samba.org/
Simple, fast compilation cluster software.
Both are great.
Re:Kernel development interests me terribly (Score:2)
While testing is one way, you might try doing janitorial work or some of the simple stuff. [kernelnewbies.org]
If you have good coding skills, try newbies, otherwise, you may find testing is up your ally.
Re:Kernel development interests me terribly (Score:1)
Automatic Testing (Score:5, Interesting)
I can't say how much I appreciate the automatic tests. This is applying computers to a thankless task that they're suited for.
Now if they only had a web dashboard portal showing the latest results in an easily-assimilated color coded HTML table....
If only... (Score:4, Funny)
Then you'd become a PHB.
Re:Automatic Testing (Score:2, Interesting)
> results in an easily-assimilated color
> coded HTML table
So true. I've set up one of these for a project I work on, and it's really helpful to be able to see at a glance the status of a bunch of projects [ultralog.net].
Seems like some lintish tools could be incorporated into a kernel daily build, and maybe something like CPD [sf.net] as well...
Re:Automatic Testing (Score:3, Informative)
http://www.osdl.org/lab_activities/kernel_tes
Just look around there.
Whatever it is, it's working... (Score:3, Informative)
Kernel quality (Score:5, Interesting)
I have a suggestion : how about not calling development kernels with an even version number?
- 2.6.0-beta-something kernels were bad (okay fair enough, it was beta, and Linus admitted having called a 2.5.x kernel 2.6 in order to lure early adopters and get them to test it).
- 2.6.0, 2.6.1 and 2.6.2 were unstable for me, with doozies such as oopses while rmmoding and random crashes using ide-scsi (yes I know it's deprecated, but some of us need it).
I now run 2.6.3-rc3 and it's the first time it seems stable enough to be called a 2.6 kernel. There are some problems left, but overall it's getting decent. But then why are the others "2.6" kernel called 2.6 at all? they were really 2.5 kernels imho.
This has happened before, with the beginning of the 2.4 serie. I only felt it was getting good enough at version 2.4.6 and above (I'm not counting the failed 2.4.11 release). When 2.4.0 went out, I thought it meant it was ready for prime time, like 2.2.0 was, or at least was more, but no it was crap. I was slightly annoyed with Linus then, but I thought he had been pressured by commercial Linux shops and that he wouldn't do it again. But no, he did it again with 2.6.
It's really quite annoying, because those who follow Linux know the first "stable" kernels aren't stable at all, therefore avoid it, therefore defeat the point of testing it for Linus, but beginners think "cool, a new stable kernel", try it and are disappointed, giving a bad name to an otherwise great kernel. Too bad
Re:Kernel quality (Score:5, Interesting)
Re:Kernel quality (Score:5, Interesting)
I realize that, but what I'm saying is that those in the know get burnt a couple of times, then see through the bullcrap and silently renumber the kernel versions. In the end, early adopters are pissed off because they've been lied to a little, and swear never to try newer stable versions again, newcomers get disgusted by the quality of early stable releases, and Linus doesn't get the testing he wanted that made him bend the truth in the first place, therefore everybody loses.
I'd much rather see Linus say : "here, there's this 2.5, we call it 2.5.xx-RC-something. It's close to 2.6, but not quite. *PLEASE* test it for us *PLEASE*!, that'll allow 2.6.0 to be good". He could even have a "best testers" or "more devoted QA volunteers" list prominently displayed on the main page at kernel.org, to appeal to people's sense of ego.
At least that would be a more honest approach to testing new kernels than lying to people.
Re:Kernel quality (Score:1, Insightful)
Name me one piece of software that worked as expected with a
Re:Kernel quality (Score:5, Insightful)
I can't think of a x.y.0 release of any software project that's been properly stable. It's not just linux, it's the way the world is. You could argue that software never ever becomes perfectly stable: marking a series as "stable" is really just shorthand for good enough that further development is largely maintenance, therefore we expect the structure and codebase to remain stable", not some guarantee that they'll never go wrong. It's more a development term than a performance or reliability term, though the stability of development generally arises from the performance and reliability being sufficient to obviate the need for large changes to the code.
Even quite late in stable kernel release cycles there's occasionally a shocker - anyone remember 2.0.33?
If you don't like those kernels, just stick with 2.4 until a distribution ships with 2.6.8 or so. For what it's worth 2.6.(1+) has been fine for me.
Nobody's lying to you - there has to be some cut-off where a kernel series is declared stable, and by and large I think Linus judges it pretty well.
Re:Kernel quality: OpenBSD (Score:2)
OpenBSD comes to mind. I faithfully rely on their
Re:Kernel quality (Score:2)
That leaves people like me who use it on the desktop (and I haven't had a single problem from 2.6.0 onward) and geeks who test it on non-vital server boxen.
I don't think it's much of a problem, frankly.
Re:Kernel quality (Score:5, Funny)
Re:Kernel quality (Score:3, Informative)
Linus Torvalds: Linux 2.6.0-test1
Jul 14 2003, 09:16 (UTC+0)
Ok, the naming should be familiar - it's the same deal as with 2.4.0.
One difference is that while 2.4.0 took about 7 months from the pre1 to the final release, I hope (and believe) that we have fewer issues facing us in the current 2.6.0. But very obviously there are going to be a few test-releases before the real thing
Re:Kernel quality (Score:2)
Out if curiosity, why do you need ide-scsi? Most people in the past used it for CD burning under linux, but ide-scsi is no longer a requirement for using ATAPI cd burners.
There are other uses as well (tape drives for example), but they're generally a lot less common.
Re:Kernel quality (Score:2)
I need(ed) it because I could not, for the hell of me, get a POS Mitsumi CD burner to work any other way. The other reason I want(ed) it was because I tend to dislike breaking things that work, such as init scripts, fstab, symlinks and various script thingy I made that expect the burner to be a scsi device, and other scsi devices to appe
Re:Kernel quality (Score:5, Informative)
disable ide-scsi and use latest cdrecord with
dev=ATAPI:x,x,x
instead of
dev=x,x,x
and everything is cool. Don't forget to check lilo.conf. Stop using cdrdao. Ignore xcdroast's "This would be faster is you had ide-scsi enabled" dialogs.
This should apply to almost every kind of ide-scsi use.
Re:Kernel quality (Score:2, Informative)
Also, I'm not sure what hardware the original poster has, but I had no problem with 2.6.1 or 2.6.2. As always, YMMV.
Re:Kernel quality (Score:2, Informative)
dev=/dev/hdc works for me. Seems simpler.
Re:Kernel quality (Score:1)
From my perspective,
2.2.0? (Score:5, Interesting)
(I remember this because we were waiting for 2.2.x to come out, having just gotten a dual P-II 350 server [2.0.x didn't have SMP support]. Fortunately, we managed to hold off for the first few revisions.)
It's not as if this problem is unique to the Linux kernel. "Never use a Red Hat
Re:Kernel quality (Score:1)
In a nutshell; The problem occurs when I switch back and forth between machines my mouse stops working or goes erratic. (the work-around is to restart gpm and then X) I dropped back to 2.4.24 because it doesn't have this proble
Re:Kernel quality (Score:1)
Of course, I use a USB mouse now, so I don't know.
Versions of Windows up through ME had the same problem. I'm not sure, but I'd guess that PS/2 mice need to be bootstrapped.
Re:Kernel quality (Score:1)
Re:Kernel quality (Score:3, Insightful)
It's really quite annoying, because those who follow Linux know the first "stable" kernels aren't stable at all, therefore avoid it, therefore defeat the point of testing it for Linus, but beginners think "cool, a new stable kernel", try it and are disappointed, giving a bad name to an otherwise great kernel.
The problem here is not with the kernel numbering, it's with your understanding of the kernel numbering. An even-numbered kernel is "stable" in the sense that the developers promise they will not mu
Re:Kernel quality (Score:2, Informative)
I thought rmmmod support was removed or at least deprecated in the latest kernel? Any kernel hackers care to clear this up?
Why would beginners be installing a new kernel? The distros would have the latest features patched into their supported kernels. In any case, if you are installing kernels, you should at least read kernel traffic. [kerneltraffic.org]
Re:Kernel quality (Score:2)
There's no guarantee a module from 2.5.x will work on 2.5.x+1, where a module (ignoring bugs) from 2.6.0 will work in 2.6.60.
If you want crash-proof, wait for RedHat or Suse or someone to package the kernel for you. If you want stable interfaces so you can begin to develop your kernel module, or whatever, then grab 2.6.0 straight off the presses, but maybe don't put your valuable data on it
Re:Kernel quality (Score:1)
Re:Kernel quality (Score:3, Interesting)
My friend, not everybody uses Linux just for shits and giggle, and for the heck of saying "Windoze sux0rs" to buddies. I actually need to get work done with my Linux boxes, and so when a stable kernel isn't stable, it pisses me off.
Also, FYI, I reckon I have a certain right to be annoyed because I contribute code back to the free software community, in the form of userland sof
Re:Kernel quality (Score:5, Insightful)
two things (Score:2)
2. I don't think that "stable" in this context means what you think it means.
Post: -1, Redundant (Score:5, Informative)
The announcement for 2.6.3 and 2.4.25 was yesterday, and the vulnerability to which the link in the text above refers was with mremap, not munmap; there's also another vulnerability with mremap mentioned yesterday as an *update* to the kernel announcement.
Re:Post: -1, Redundant (Score:2)
Szo
SCO (Score:5, Funny)
The only match was "a misconfigured system".
ACPI cure for 2.4.25 HOW-TO (Score:5, Informative)
do_mremap (Score:4, Funny)
In defense of Mr. Emap (Score:2)
Framebuffer? (Score:5, Funny)
Bitkeeper (Score:5, Interesting)
[the author] wrote:
I grant that sometimes you have to simplify history to avoid digressing in an article, but this is a bit too inaccurate to let stand.Bitkeeper wasn't suggested by anyone; it didn't have to be. It was developed from the ground up to Linus' requirements. Larry McVoy had a discussion about source control with Linus years ago, in which Linus said "none of the products are good enough" and Larry said, "ok, I'm going to write one that is". Apparently he had this on his mind anyway, and so he started Bitmover Co. As bitkeeper became a usable product Larry continued to take Linus feedback and improve it until it was good enough for Linus to use... at which point Linus started using it.
This is still a simplification of course, but it's closer... and as you can see, there were no third party suggestions involved.
Re:Bitkeeper (Score:5, Informative)
Interesting read (Score:5, Insightful)
Re:Interesting read (Score:3, Interesting)
Without beeing too sure, I believe that Linux developer's in the beginning focused more on fixing bugs than keeping things clean and structured. That's mainly because the base of the system needed to be developed, and little attention was drawn to factors like speed and optimization. "
Re:Interesting read (Score:4, Informative)
It's still amazing to me that a project as large as Linux was able to be so successfull BEFORE the changes that were made to the development process. It lacked a centralized CVS, coherent bug tracking, automated testing... These are all things I use in the smallest of professional projets.
CVS is an interesting one. I don't really understand his rationale, but Linus says that for Linux development, CVS is not only inadequate, it is bad. Linus has said that if he ever has to quit using Bitkeeper, he'll go back to tarballs, patches and manual version control, because every other available system is worse.
I can certainly see weaknesses in CVS, and for Linux development I can see how the lack of changesets would be problematic, but the notion that manual version control could be better is... startling.
Re:Interesting read (Score:3, Interesting)
SVN is much more efficient though. I'm not sure that comment aplied only to CVS or to avaliable versioning systems in general.
Re:Interesting read (Score:2)
Re:Interesting read (Score:1)
Re:Interesting read (Score:2)
Re:Interesting read (Score:2)
Re:Interesting read (Score:2)
It lacked a centralized CVS
It still lacks a centralized revision control system, since one of the prime virtues of BitKeeper is that it is decentralized.
changes, huh (Score:2)
cp -R
ducks
But why is it in so few distros? (Score:5, Interesting)
Re:But why is it in so few distros? (Score:2)
Re:But why is it in so few distros? (Score:2)
scripsit lavalyn:
Yuck, yuck, yuck. Seriously, though, 2.6.2 is in Sid right now, and 2.6.0 is already in Sarge. Sure, it's not the default kernel, but it's sure as hell available -- four years early ;)
Re:But why is it in so few distros? (Score:2, Informative)
Re:But why is it in so few distros? (Score:1)
However I get the feeling that the new scheduler still needs some finetuning for desktop apps.
It seems to be great already for servers.
But glxgears dropped by almost 50% for me - and tvtime drops frames all the time.
Surprisingly the kernel is even SLOWER, when I disable powermanagement and framebuffer support.
I also looked forward for the Sis 746 AGP support - but it is still very unstable [gentoo.org].
Unfortunatly, Dave Jones, the main agpgart developer does not seem to be [codemonkey.org.uk]
Re:But why is it in so few distros? (Score:3, Informative)
Re:But why is it in so few distros? (Score:2)
Has any JVM been used in this production? (Score:1)
Maybe real kernel hackers do not use something that is half the speed of C but maybe the odd exception likes java.lang.Exceptions.
Offtopic but I just like to know where a 5 year java programmer might fit in besides maybe jakarta from apache.
Java kernels (Score:1, Offtopic)
Beautiful (Score:5, Insightful)
Can anyone take a guess how many low-level memory exploits are in Windows XP, 2k, or others? Perhaps it's irrelevant. Who needs to crack the low mem when there are so many ways into the system at the document level?
Re:Beautiful (Score:1)
Because Karma to effort ratio is too low (especially if you post more than 5 minutes after the story got posted)?
BitKeeper? (Score:2, Insightful)
Re:BitKeeper? (Score:2, Insightful)
Troll Alert! (Score:1, Informative)
Wrt. the "must-not-develop-a-competitor" angle: This does not prevent people contributing to Linux. It simply means that they cannot use BK to do it! (If they plan on being able to work on any RCS). You may argue the fairness of such a stipulation in the BK license, but the fact remains: Linus chose BK.
Re:BitKeeper? (Score:4, Informative)
a) not use BitKeeper
b) buy a BitKeeper licence
Second, RCS doesn't support concurrent development. That's why we have CVS.
Third, why BitKeeper?
Though CVS has lots of shortcommings (and thats why Subversion exists) and Subversion (SVN) is still labeled "alpha" by it's developers (though in practice it's stable enough to be self hosted and widely used), the real reason has to do with the basic model of CVS and SVN. Two main issues, in my opinion:
a) In CVS/SVN you need write access to the central repository or you can't make proper use of versioning control. Giving write access is a problem for Linux's contribute based development model. BitKeeper doesn't need it.
b) CVS/SVN know about branches but they don't know about merges from one branch into the other. Their view of the repository is a pure spanning tree. Subversion has a "merge" command, but a merge is commited as any other change into the repositoty. BitKeeper knows about previous merges and where they were merged from and uses that information to be smarter at resolving conflicts when you do a merge.
In contribute based development every change to the project has to go through one of few maintainers who can write into the main repository (in Linux's case, there's only one), so proper merging support becomes very important. At some point before BitKeeper, Linus was having trouble keeping up with all the patches people were sending him and people were getting angry with that.
If you don't believe me, you can check the GNU Arch website: http://wiki.gnuarch.org/
They're developing a Free versioning control system very similar to BitKeeper.
Re:BitKeeper? (Score:2, Informative)
Just a nit on Subversion's status [tigris.org]: it's in beta now, with a release candidate scheduled for tomorrow and the official 1.0 release set for Monday.
Re:BitKeeper? (Score:1, Interesting)
2.4.x and 2.6.1 and external USB devices (Score:3, Interesting)
I've encountered many problems with external hard drives using USB 1 and 2 interfaces. Locking up the entire system on large file copies was the main issue. (Copying small numbers of files was never an issue. Lockups occured on different drives, different external chipsets, different 2.4.x kernels though supposedly fixed in the latest 2.4.x releases.)
I've finally gotten the nerve to run a few days of tests on 2.6.1 to see if this has been really resolved, and I'm happy to report that this now works like a charm.
If you've encountered similar problems with 2.4.x, give 2.6.x a try.
Linux? Yeah. But GCC is also interesting. (Score:2)
GCC is also an amazing feat of software and creating custom backends would make many experiments possible (porting to old architectures, porting to virtual machines, c++ on virtual machines) etc.
Re:Linux? Yeah. But GCC is also interesting. (Score:1, Informative)
Re:The Tao of Linux (Score:1)
Re:Linux in C++ (Score:1)
Re:Linux in C++ (Score:1)
Well, I've reads these 'covers' in the kernel mailing lists, and they have just ended in brainless flamewars. Mostly because C developers don't like C++ and assume things.. and perhaps C++ developers have a hard time understanding that OO design with C isn't that hard.
the bottom line is; the code is complicated and it takes a lot of time to get into it! Even though you get nice coupling between systems with modular C programming, it does not mean that it couldn't be done better. If the slab memory allocat