Cox on Torvalds and Linux Kernel Development

sebFlyte writes "Alan Cox' speech at FOSDEM sounds like it was interesting... according to this ZDNet report on it he has some interesting views. For one, he says: 'Linus is a good developer, but is a terrible engineer.' He also has a few digs at Torvald's methods surrounding security fixes, and some other interesting insights in the kernel development process: 'Sometimes you see a fix and think "this is perfect, move my fix into the kernel tree." Later you think, "I must have been drunk. Don't apply that patch."'"

Also for your perusal

[by Anonymous Coward]

An Interview with Linus himself [linuxmagazine.com]

Re:Also for your perusal

[TrekCycling (468080)]

That's interesting. Thanks for the link!

The most interesting part to me is that he uses a Mac running Linux now. For a long time the story was that he ran SuSE (I'm assuming on x86). I'm a SuSE user, and some of the enthusiasts in SuSE-land used to use this as proof of why SuSE was "the best". I just laughed. I wonder what distro he runs now.

Re:Also for your perusal

[Linker3000 (626634)]

I wonder what distro he runs now.

Hurd?

Re:Also for your perusal

[antime (739998)]

Heh, wonder why they cut out part of Linus' post quoted at the beginning? (The original post read "just a hobby, won't be big and professional like gnu".)

Re:Also for your perusal

[impaler (78415)]

Seeing as GNU is mentioned zero times in the article, they probably cut it off to avoid having to explain to people what GNU was. Even though it is in Linux magazine I guess some readers may not know what GNU is, and a discussion of GNU is probably (depends on how you look at it) irrelevant to an interview with Linus Torvalds about kernel development.

the BSD TCP/IP was exploited by MS

[FreeUser (11483)]

Without GNU's GPL MS would've openly pirated various free implementations of UNIX.

What about the ones that are not released under GPL, like *BSD?

The BSD TCP/IP network stack was openly "pirated" by Microsoft, if by "pirated" one means "used while drowning in hypocracy by decrying the very free movement one is exploiting," which is really the only definition of "piracy" I can imagine would apply to using a Free product in compliance with its license. I suspect that is what the original poster meant.

I wouldn't have chosen the word "pirate" as it implies copyright violation, which is clearly not the case if you're adhering to a license. "Exploited" would have been more appropriate.

Microsoft openly exploited the BSD TCP/IP network stack because of the liberal BSD license, something the authors of FreeBSD have absolutely no problem with, and in fact encourage. As to whether this is strategicly wise of them or not, well, that is a flamefest reserved for typical GPL/BSD arguments. I personally think the GPL is what has made Linux viable and protected it against many of the worst depridations by Microsoft...though of course it won't hold up to a patent assault once Bill Gates finishes ramming software patents down the Europeans' throats (one may speculate on which appendage Mr. Gates is using to do the ramming), but as Apple, OpenBSD, FreeBSD, ogg-vorbis, and numerous other projects have shown, the BSD license has its strengths as well, and can be quite ideal for other projects.

Odd

[Kip Winger (547075)]

It's funny how petty squabbles between key developers could tear even what is now a major, corporation-funded project apart that millions of machines and companies depend on.

I'm willing to be if such things continue, some entity, perhaps IBM, will set down their foot and use pressure put maintenance of the kernel project under the jackboot of a truly dictatorial manager, making Linux more an open source Cathedral than a bazaar.

Re:Odd

[Soko (17987)]

I'm giving up mod points to try and dispell what seems to me a sensationalised headline.

Alan Cox is not showing disrespect to Linus here, read the whole quote:

"Linus is a good developer, but is a terrible engineer," said Cox. "I'm sure he would agree with that."

Alan and Linus have been working together for a very long time, so I'm sure Linus wouldn't give this statement a second thought. Each must know they compliment the other and make the whole of the Linux kernel better - even if they have the odd disagreement - or the kernel would have been truly forked (no pun intended) a long time ago. As it is, they work together on patches and ideas. We don't have much to worry about, I think, since Linus' sense of who's a good dev and who fits into his team well is uncanny.

Soko

Re:Odd

[LnxAddct (679316)]

Good points. But for those who didn't rtfa, or have no idea about kernel development, Alan Cox generally makes sure the kernel is stable. Linus likes to innovate and throw new ideas in without necessarily testing them thoroughly. Innovation and stability are usually separated by quite a time period in any development process, so all Alan was more or less saying was that (as the parent stated) they compliment each other. OSDL pays Linus to hack up new stuff thats needed, Red Hat pays Alan to make sure that new stuff is stable and can be effectively used to its full potential. You really do need them both, as one without the other won't achieve much, and giving those tasks to one man alone would be quite a burden and errors would be abundant. Both Alan and Linus are absolute geniuses at what they do and no one is arguing that. Since when did OSS need to sensationalize headlines?
Regards,
Steve

Re:Odd

[Kjella (173770)]

so all Alan was more or less saying was that (as the parent stated) they compliment each other.

I don't think being called a "terrible engineer" is a compliment. Def: 1. An expression of praise, admiration, or congratulation. Versus a complement: 1c. Either of two parts that complete the whole or mutually complete each other.

Kjella

compliment is ying and yang

[oo_waratah (699830)]

Mathematically compliment is opposite sides of an angle. if one is acute then the other must be obtuse. They compliment each other like the chinese ying and yang.

They balance one another because they are different or compliment each other. THis is different from the compliment that sings the praise of someone else.

Re:Odd

[Abcd1234 (188840)]

Wow, someone hasn't grown up. Those who don't understand their own limitations, or get pissy when they're discussed/joked about in public, are nothing more than egotists and/or immature morons. Then again, I'm not surprised you wouldn't understand this, as the concept of humility is lost on a very large number of people in this world...

Re:Odd

[by Anonymous Coward]

squabbles? go read al viro's lkml posts.

these folks flame and flame well. Similar fireworks seem to be an important hallmark of a healthy project.

You are speaking, in ignorance, from the wrong orifice.

Re:Odd

[servognome (738846)]

Anyone want to guess which branch would be more popular?
For the geeks... Linus, for the companies with money... IBM
The money trail will probably end up deciding the winner.

Think selection, not direction.

[AJWM (19027)]

Linus isn't running the show. He's not paying anybody, he can't fire anybody, he can't make anybody drop one project or idea to work on another.

He can direct some developers to do something and they can tell him to take a hike, or they can do it because they think it's a good idea.

More often, though, there are just many ideas (patches, development threads, what have you) to choose from and Linus "rules" by choosing which goes into his kernel.

The cathedral is about direction. That isn't what Linus does -- he just selects what is best from what the bazaar has produced.

(Sure, he may also make suggestions and remarks that indicate what his selection criteria are, and that may in turn influence kernel developers, but that doesn't prevent someone from coming up with an even better idea that Linus hadn't considered before and changing Linus's mind. That doesn't happen in a cathedral -- do you think some workmen with a brilliant but different idea for St. Paul's would have been paid attention to by Christopher Wren?)

Christopher Wren and the Cathedral/Bazaar

[kahei (466208)]

Actually, the design of St Paul's and the surrounding area was the result of a complex interplay between Wren, the king, the church and the people:

Wren -- wanted to create baroque-style city center
King -- wanted to save money
Church -- gothic all the way, baybee
People -- wanted convenience and dense business development

In the end, the group that came closest to getting what they wanted was the people; that's why London has no great big boulevards like those of Paris (the people valued lots of living space above ease of riot control). Wren, the man with the 'brilliant but different idea' used the King's negotiating weight to slip change after change past the Church, but he couldn't change the design completely which is why it's a hybrid gothic/baroque.

Thus the design evolved by consensus, negotiation, and balancing bright ideas against established needs. Incidentally, I never used to like St. Paul's, but now that they have cleaned it all up and redeveloped the area north of it with some excellent postmodern work, it's looking really good. Weather still bad though.

Going back to the subject of software, I really don't think this whole Cathedral/Bazaar analogy was well chosen in the first place (especially as gothic cathedrals were striking examples of community efforts, as has been pointed out elsewhere). Major projects and projects that depend on a central authority can be developed in just as fluid a way as small, distributed ones -- if you have the right people. If you don't have people like Wren and the King, if you don't have people who change their minds when they find a brilliant idea, there can be problems.

Linus is authoritarian?

[weighn (578357)]

I'm not trolling, and am a little ignorant of kernel development - so bare with me, but surely Linus isn't the be-all-end-all overseer of what ends up in the kernel? Why target him exclusively?

Re:Bare with me?

[Grey Ninja (739021)]

Why thank you. Don't mind if I do. [twinks4all.com] :)

Re:Linus is authoritarian?

[updog (608318)]

but Linus dictates main-line completely.

Uh, not really - Andrew Morton is the official maintainer of the 2.6 kernel, and Marcelo Tosatti is the maintainer for 2.4.

Obligatory Peanuts reference...

[tktk (540564)]

Linus probably keeps all the secret fixes under his security blanket.

Interesting take on Linus

[Gopal.V (532678)]

"Linus has this bad habit of fixing security holes quietly," said Cox. "This is a bad idea as some people read all the kernel patches to find the security holes."

I wouldn't advertise my mistakes either ... neither do the OpenBSD folks or any ego driven engineer :)

The article paints Linus as the typical Flawed Hero of contemporary literature. He's good and yet he's not perfect - at least that's what comes out of it for me. (and no digs on BitKeeper .. hmmm..)

Re:Interesting take on Linus

[perseguidor (777194)]

I *know* this is offtopic, but as far as I know the concept of the Flawed Hero comes from the greek tragedy, and I wouldn't ever call it characteristic of contemporary literature, which is more fond of anti-heroes methinks.

Linus' Security Practice

[alanlke (685520)]

Cox may have had a good point on Linus' methods for security patches, but fortunately the community has spawned sites such as this http://www.securityfocus.com/ [securityfocus.com] to publicly announce when people find security flaws from poking through the patch code.

Even if Linus tries to keep these things secret, they'll get out quite quickly.

Re:Linus' Security Practice

[bonch (38532)]

Slightly off-topic: LinuxSecurity [linuxsecurity.com] shows the weekly security advisories for all distro software, including the BSDs (it's actually quite a lot).

ZD states..

[Creepy Crawler (680178)]

That Linus is a person, and not a GOD as some people worship him as.

Its actually pretty damned nice to see a bunch of people get together and make something as big as the Linux Kernel. Linus started it, but we all will finish it.

Still, I fail to see how some bugs would be super-bad, as the article seems to say. Id rather have a crash bug, rather than a SUID change bug.. STill, not all security comes from the Kernel. Some security comes from network filter drivers, some com from the application, which many hackers target, and whatnot. Though, the kernel is a great place to attack if you have that guest acct and "want" root ;P

Re:ZD states..

[Stanistani (808333)]

Calling Linus an engineer is like calling Gandhi a politician... you need to look up just a little to judge how badly you underestimated the impact of the man and his followers.
Gandhi had his moments of pettiness and just plain tom-foolishness, but the sum of his efforts changed the way people gain power back from those who would usurp it for their own.
In a different, yet no less trivial way, so did Linus (although I would not call him Mahatma).

nothing new

[sewagemaster (466124)]

the /. headline makes it looks like there's quite a bit of fued between cox and torvalds, which isnt really the case if you RTFM.

different people have different working styles, no matter whether it's kernel coding, software apps, or ASIC designs. if either group/individuals are too giving to the other group, there can never be enough feedback/ constructive critisisms between them. having yes-men surrounding you isnt the best thing. and it's not like that they're arguing so much they've halted any soft of development progress.

[offtopic]
gives me an idea though, maybe when job interviewers start asking me those behavioural questions about "a time when you've had disagreements and a way of resolving them", there's no need to bring up something too dramatic.
[/offtopic]

Re:nothing new

[creimer (824291)]

maybe when job interviewers start asking me those behavioural questions about "a time when you've had disagreements and a way of resolving them", there's no need to bring up something too dramatic.

I had a job interview a number of years ago at a video game company. The guy asked me with a straight face what I would do if two of my co-workers was having a fist fight out in the hallway.

I almost blurted out, Does that happen a lot around here? Instead, I gave a safe answer that I would go get a superviso

Re:nothing new

[mvdw (613057)]

I was once given some advice about interviews from my Dad, who said something along the lines of: often, the interviewer is looking at whether you know your own limitations. For example, they will often continue to probe about a particular issue, and if the interviewee keeps saying that (s)he'd keep trying to fix the problem, that a Bad Thing(tm). The Correct Answer(tm) is to say, after a while, "That's when it becomes your problem". After a certain amount of time with a problem, you should always defer it

Count your blessings...

[DesScorp (410532)]

...at least it didn't say "Cox IN Torvalds"

Who cares about Linus anymore?

[mnmn (145599)]

Linux has become much bigger than Linus now. The kernel alone has its parts maintained by other people, many of whose patches are applied without much checking to the main tree because they're 'responsible' for it, like certain architectures, driver trees etc.

Apart from the name, Linus currently has the final say of what goes in. Thats just officially. In real life it seems far more is delegated to others for different parts of the kernel, and Linus is one of the developers, far from the most active, and not really exercising his right to block patches against the majority's will.

Re:Who cares about Linus anymore?

[bonch (38532)]

"Who cares about Linus anymore?"

Slashdot, apparently, since it usually posts an article on just about every mention of Linus, every minor activity, every little comment, and every little speculation. People glorify him as the guy who single-handedly wrote Linux, when Linux is really the work of thousands of developers who just send patches to him (and now other delegates too).

Re:Who cares about Linus anymore?

[nysus (162232)]

I'm interested in Linus for purely selfish reasons: I can learn from him. I learn from him even though I've barely looked at any Linux source code or do any programming in C or do any kind of low-level programming for that matter. But as someone who plays a small role on a small open source development project, it's fascinating for me to hear from a person who leads such a hugely successful project and the problems and obstacles he and other developers have to overcome to be successful. It helps me put perspective on my own work.

That's not glorification, that's taking advantage of somebody else's insights and experience.

Re:Who cares about Linus anymore?

[oconnorcjo (242077)]

when Linux is really the work of thousands of developers who just send patches to him (and now other delegates too).

Every leader depends on the efforts of those working for them but that does not mean that great leaders are any less great. Linus LEADS. He is the "queen bee" and as such he makes sure the Linux kernel is a great piece of software though he does little "honey gathering" himself.

Re:Who cares about Linus anymore?

[michaeldot (751590)]

The more I learn about Linus from day to day, the more he sounds like a great human being with his priorities in the right place... "Other than software, I've been spending a few hours a week for the last month building a small playhouse for the kids in the backyard." A great role model for young geeks once they grow up.

Tabloid fluff

[rafael_es_son (669255)]

The article is yet another clear piece of filler that pretends to build antagonism between two important figures of the kernel project. How does this stuff keep getting accepted by Slashdot?

Is this the way Slashdot supports open source, fostering internal divides in exchange of ad eyeballs?

Sign of a Strong Working Team

[kmactane (18359)]

I'm sure everyone who doesn't bother to RTFA will now think, "Oh, no, Linus and Alan are bitching each other out in public." That's nothing like what's going on here. For one, the submitter quotes only half of one particular line from the article:

"Linus is a good developer, but is a terrible engineer," said Cox. "I'm sure he would agree with that." [emphasis added]

So it sounds like Alan and Linus have discussed this particular difference in their talents before, either over beers at a pub, or over email or something.

Second, the article makes clear that part of what's going on is that Alan and Linus each have very different responsibilities in keeping Linux going, and so they necessarily focus on different things. Alan points out that as the dev tree maintainer, Linus is trying to keep the code maintainable, while Alan's trying to keep it stable.

And both of these things are necessary. It sounds to me like rather than being "at loggerheads", or "ready to call off the working relationship", instead Linus and Alan are a very well-matched and complementary team, both of whom contribute enormously to Linux's success and quality.

Each of them has strengths that make up for the other one's weaknesses, and it sounds like they have a good enough working relationship to give each other constructive criticism when needed.

The Developer and his Code ...

[foobsr (693224)]

'Sometimes you see a fix and think "this is perfect, move my fix into the kernel tree." Later you think, "I must have been drunk. Don't apply that patch."'

Nothing has changed. In 1983 one cause of coding (programming) errors had been described as a misfit of perceived and actual reality (Zemarek in Psychologie des Programmierens, S. 111-129, Hrsg. H. Schauer, M. Tauber, R. Oldenbourg, Wien, 1983).

CC.

Re:Engineer, programmer, developer, same thing

[by Anonymous Coward]

no they are not.

programmer = code monkey
developer = higher level design and algorithms.
engineer = full system design, makes the important decisions.

Re:Engineer, programmer, developer, same thing

[DoctorMO (720244)]

Engineers make thing from whats avaiable to produce the best design for the end result Developers create new ideas and methods that seem really cool Programmers code the idea's into a working product It's even better when you can get a person who good at all 3, more likly there good at 1 and 3 or 2 and 3

Re:With all due respect to Alan Cox,

[bloo9298 (258454)]

That was quite funny, and I see that you managed to fool somebody into giving you an "Insightful" mod.

Mods take note, the parent post is deliberately nonsensical. For example, "It's been shown that creating working kernel based on a register machine like most modern microprocessors is NP hard".

let the market decide.

[mckwant (65143)]

So fork the kernel. Unless I'm missing something, OSS is rife with opportunity to make something else happen. Lord knows, there are plenty of branches of the kernel that are available.

If microkernel is the way to go, then write the code, and we'll see what happens in the OSS marketplace. I'm not going to say "best technology wins," as that's rarely been the case historically.

There is, after all, a reason that the HURD hasn't taken over yet. I might well be missing something, as this isn't really my area.

Linus ballances bugs & features CORRECTLY

[ron_ivi (607351)]

The article misses the fact that Microsoft and Linux may be successful BECAUSE they make the Right tradeoffs between bug fixes and features and time to market.

from TFA: "Cox revealed that although Linus is good at developing code, he does not enjoy some of the other jobs that go along with software development such as bug fixing and beta testing."

Note that Linus and Microsoft are both being accused of the same thing.... some purists are arguing that they don't focus enough on the bug fixing. The reality is, that no matter what focus you will NEVER have a bug free system. All software makes such feature/stability tradeoffs perhaps the most important challenge of any software project is balancing the tradeoffs of perfectionism vs. time-to-market of bleeding edge features that is best for their market. Other operating systems that do focus excessively on security for their "core" offering tend to fall behind in features (like the old mainframe software banks use, etc). Sure such software has its place; but it's not the mainstream market.

Note that for the security consious customer, though, Red Hat and SuSE both have higher-security releases of their own (like the Common Criteria vesrions like this one [novell.com]; and releases like Debian Stable that also focus on security and bug fixes only. By people who don't understand that those releases are targeting a different market, those branches are often criticised for being filled with obsolete software.

It doesn't have to be Linus's job to handle the most conservative customer's needs.

Re:Linus is a hacker

[happyslayer (750738)]

I was going to mod, but decided to jump in, instead... I don't know Linus, I'm not a kernel-demigod, and you may know a lot more about him than I do. And while I'm a linux-enthusiast (and therefore an admirer of all the work that goes into it), I'm not a groupie who automatically jumps up to defend the Order of the Penguin. With that said, I don't see how "contemporary ideas" have anything to do with his ability to manage and guide the development of an OS. I've read correspondence about kernel issues (as I've come across them), and it always seems to me that he tries to keep it simple and direct. "Does it work?" and "Will it screw things up later?" appear to be the underlying themes...very admirable ones, in my opinion. Even more to the point: Why should anyone care if he has little or know knowledge outside his project? (And it appears to me that he has a lot of experience...but I can't/won't try to rattle off his resume. See above.) If I have to have brain surgery, I don't give a damn whether or not my surgeon knows how to do an appendectomy; he's got one job to do, and that's all I care about. Well-rounded educations, backgrounds, etc. are great when your project has to cover a wide range of issues. (Ever get involved in a government software project? It's a nightmare!) But if your needs are specific, then the more of an expert you are in that one area, the better off you'll be. To me, he's a smart guy doing a pretty good job of herding cats. 'Nuff said.

Re:Linus is a hacker

[Dahamma (304068)]

Oh you have asked him about "OO" yourself, "PimpDawg"?
Have you ever even looked at the Linux kernel source? The VFS is a better OO abstraction than anything you will ever come up with in your lifetime.
How did this get modded above -1?

Slight exaggeration?

[antientropic (447787)]

The VFS is a better OO abstraction than anything you will ever come up with in your lifetime.

Uh, what? The VFS is hardly a brilliant concept, but rather the sort of abstraction that any good designer will come up with. It's also not an original Linux invention (Sun OS has had it since 1985, I think). The kernel developers also seem to have a perpetual problem with defining an interface once and defining it well. It happened to me more than once that FUSE [sourceforge.net] broke during a supposedly minor kernel upgrade. For all its virtues, the Linux kernel is hardly a case study in good software engineering.

Re:Linus is a hacker

[PeterPumpkin (777678)]

He's a hacker with a lack of experience in projects outside his own.

Are you sure? I thought he works pretty closely with Linux-related projects, when the need arises. For example, look here. [winehq.com]

Note that Linus was the top poster for that week.

Re:Yeah

[jericho4.0 (565125)]

Bitkeeper [bitkeeper.com].

Re:Question - balls in the air?

[Abcd1234 (188840)]

The question I have is, is it still possible for these kernel gurus/hackers to effectively have the kernel and all its nuances inside their head, fully functional at a theoretical/experimental level? Or does development at this point consist of sub groups that are specialized and don't require a level of understanding to 'run the kernel in your head'?

in short, it's the latter. However, keep in mind that, in a well designed system that's properly modularized, with neatly spec'd interfaces between components, it isn't always necessary for someone to have the entire picture, with all the nitty-gritty details, in their head. Instead, one need only grasp how the system operates at a high level, from a component-oriented standpoint, where each of the components themselves conforms to a particular contract.

Put another way, while Linus may not understand how the driver for a particular digital camera works, he probably does understand the interface that driver exposes, and how that interface ties in with the rest of the system.

Re:Maintainable? Stable?

[Alan Cox (27532)]

They get to be in conflict over short time periods - that was the point of that part of the talk. If you need a fix that is provably correct, simple and immediate it tends to be the ugly bandaid type fix. Those go into -ac and expire (in theory) for the next base kernel.

Linus instead is quite happy to say "ok that is a problem but the real fix is to rewrite the logic behind [randomcomponent] to properly ensure that this cannot occur'. That might take a month and is undoubtedly the right answer but it isn't the immediate answer for people hitting the problem currently.