Linus Does Not Scale 554
EmilEifrem writes: "Seems like everybody's getting more and more frustrated by Linus' (in-) ability to handle patches. Rob Landley just wrote an "RFC on Penguin Patch Management" wherein he proposes a "Penguin Patch Lieutenant" system that he believes would scale better. The full discussion can be found on the Linux kernel mailing list. Linus seems to dislike it, as usual, source code maintenance tools/organization are for wimps!, but a lot of others find it a good idea. Anyway, it's a very good read."
first post! (Score:3, Insightful)
Linux CVS (Score:2, Informative)
Re:Linux CVS (Score:3, Interesting)
Re:Linux CVS (Score:3, Interesting)
It would also be useful if a split like Linus suggests might be done (modularizing things and then a "Linus" for each mode), as the "Linii" would each only have check-in permissions for their own module.
Re:first post! (Score:3, Interesting)
CVS does only accept patches, it does not check for quality or if it breaks anything.
This has still be done by hand. And if its not done at patch time, people will forget it.
I'm by no means a Linux fan, but bashing Linus because of that just sucks.
ONE has to check the submitted patches.
If there are 10 people checking the patches, no one knows what the other one has really done, that means even if you have people with highly communication skills (99% of the coder lack that) you still would sometimes forget to tell something and its going to break everything.
/wave
Re:first post! (Score:5, Funny)
CVS... does not check for quality or if it breaks anything.
If you've been following Linux kernel development the last year or so, you could say the same about Linus.
Re:first post! (Score:5, Insightful)
Of course you need to test patches and have others review them before you commit, but having things in CVS makes it very much easier to manage code.
who's frustrated? (Score:2, Interesting)
It's got to be a pretty small group of people who are annoyed by Linus' in-ability to handle patches... specifically those writing the code! I'd think it pretty important to keep those folks happy, since without them our beloved little OS wouldn't progress as quickly as it does.
Re:who's frustrated? (Score:3, Insightful)
Okay everybody, this is getting rediculous. Patches FROM MAINTAINERS are getting dropped on the floor on a regular basis. This is burning out
maintainers and is increasing the number of different kernel trees (not yet a major fork, but a lot of cracks and fragmentation are showing under the stress).
The maintainers should be kept happy. As linus states:
he fact is, we've had "patch penguins" pretty much forever, and they are called subsystem maintainers. They maintain their own subsystem, ie people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH (USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo Molnar (scheduler), Jeff Garzik (network drivers) etc etc.
Rik van Riel already suggested a mailing bot that says what version it is agianst (and how many times it is dropped....). But just mailing linux only helps you to be added to the spamfiler.
I gues in some time this will all stop if linus find a toy project he wants to focus on and some kernel distributer steps in with a team to take over. (Linus will say I dont care. real men....)
OK, it needs top be said... (Score:4, Funny)
*rimshot*
How about a RAIKA? A redundant array of Kernel Admins? Maybe keep them hot-swapable? That way, if one goes out to the pub, the other one can keep things going....
OK, I'll just shut up now...
Reputation (Score:3, Insightful)
I can see where Linus is coming from; Linux and Linus Torvalds are inextricably linked in the public perception, so any huge blunders that occur with the Linux kernel will sully his reputation. If it's his reputation on the line, he probably wants to keep control of potential disasters close to him.
Better to get blamed for your own mistakes than for someone elses.
Re:Reputation (Score:2, Insightful)
The only way to take Linux to the next level is to professionalize the development effort to ensure an efficient and stable environment. Without these proper methodologies in place, Linux will just end up as unreliable as Windoze.
Linux Vulnerability (Score:5, Insightful)
I'm sure Linus is healthy and a good driver, but as misfortune befell a former colleague at another job (her car was parked at a light on the off-ramp which was below the highway, a driver from the other direction suffered a cardiac arrest, crossed the median and opposite lanes and went airborne, landing on her car as she waited for the light to change. Gone, just like that,) unfortunate things happen. It would be very tragic for his wife and child to lose a father. It would be a disaster for Linux, as the unifying person would be gone and in the aftermath someone would have to take control. I imagine Linus has already considered this, but his tight grip on the kernel is a bit worrying. He should delegate more.
Jordan Hubberd was in the same position (Score:5, Informative)
This is a nice distributed system that continues to work very well when the load gets higher; also noone is indispensible, noone has to be afraid what would happen to FreeBSD if a certain person would somehow drop out.
Of course Linus has every right in the world to remain the status quo, even if it damages Linux. After all Linux is his baby and he can do with it what he likes. Whether it is a good idea to rely on an OS with this kind of a leadership structure is another matter however. But noone can force Linus to change, since he doesn't force anyone to use Linux (take it or leave it).
I know Linus doesn't like it... (Score:3, Interesting)
Re:I know Linus doesn't like it... (Score:5, Insightful)
Re:I know Linus doesn't like it... (Score:2, Interesting)
No, but CVS does allow branching of the tree. Couldn't different patches just be maintained on different branches, and merged into the trunk when it is finally accepted?
Re:I know Linus doesn't like it... (Score:3, Insightful)
Re:I know Linus doesn't like it... (Score:5, Informative)
Note that things like CVS do not help the fundamental problem at all. They allow automatic acceptance of patches, and positively _encourage_ people to "dump" their patches on other people, and not act as real maintainers.
We've seen this several times in Linux - David (Miller), for example, used to maintain his CVS tree, and he ended up being rather frustrated about having to then maintain it all and clean up the bad parts because I didn't want to apply them (and he didn't really want me to) and he couldn't make people clean up themselves because "once it was in, it was in".
I know that source control advocates say that using source control makes it easy to revert bad stuff, but that's simply not TRUE. It's not easy to revert bad stuff. The only way to handle bad stuff is to make people responsible for their own sh*t, and have them maintain it themselves.
Re:I know Linus doesn't like it... (Score:2, Insightful)
Having one person own the entire "tree" of code slows the development process to a crawl. Linux will never be able to bring functionality to the table as fast as Windows if everything has to go through one man. Do you think Bill Gates is sitting at his desk and reviewing Windows XP code? No. He has better things to do. Linus should realize this and work on implementing a development environment that will enable broader management.
I propose CVS is a part of this solution. I would also recommend code reviews by three independant parties before a branched segment of code becomes part of the main tree. Design reviews/requirements/UML should be designed before coding should ever take place. Yes, its a lot of overhead, but it allows for new people to fill your shoes when you grow weary of development. It also allows more people to have ownership of the entire project and enable faster development, with a broader level of quality control.
Re:I know Linus doesn't like it... (Score:3, Interesting)
Re:I know Linus doesn't like it.. How about this? (Score:2)
So everyone is supposed to have their own copy of the tree to make patches to? Without a 'source' CVS to sync it up to before submitting a patch, why would anyone WANT do that?
What if I submit my patch, and make changes that conflict with SamtheEagle's source tree he's workting off of?
I think there should be a source CVS tree, that Linus maintains. Before I submit my patch, I should sync my CVS to Linus', and apply the patch. Then submit the patch, referring to the current tag on the source CVS maintained by Linus. That way, Linus can apply all patches serially, and make requests to developers to resync submitted patches, when a new CVS tag is created.
No 'patch' management program, just a way to serialize patches for applying to the source.
Is Linux bigger than Linus? (Score:5, Insightful)
From its very beginning, Linus has kept a tight rein on Linux, and everybody in general let it go if for no other reason than he's a smart guy and by God, he wrote the thing! Don't get me wrong. That was a Good Thing (tm). Now, however, we are finding that it is becoming less and less practical for Linus to handle all of the patches.
Linux is becoming more and more like a cathedral and less and less like the bazaar. Whether he likes it or not, for Linux to continue to thrive (perhaps even to continue as we know it) Linus is going to have to decentralize the way patches are brought into Linux. I don't claim to have all the answers, but there must be a way to make a CVS-a-like system work and keep Linus' ability to make the final say if he wants to. Another alternative is for Linus to put more trust in his maintainers and let them accept patches for their respective subsystems.
The problem here is the star network topology that we have with respect to patches. If Linus is not willing to release his hold on the center of that star, then Linux could have a MAJOR problem!
Ben
Re:Is Linux bigger than Linus? (Score:4, Interesting)
Of course, I was spoiled with Clearcase when I was at Teradyne, but, at $1000 a seat, it isn't exactly cheap (and is dangerous in the wrong hands). Working at a startup now, and living with CVS, has tought me that it doesn't provide the kind of isolation, yet, tracking that Linus needs.
However, perhaps all is not lost. If Linus doesn't like CVS, let him not build out of CVS, but still provide an incomming CVS repository for certain trusted leutenants that filter patches. This provides the following:
1. A history of patches received.
2. A mechanism for delivering those patches to testers.
3. A source Linus can diff against his private tree.
Of course, when Linus makes changes outside the CVS repository, it will be out of sync, but the suggested Patch Penguin can be charged with syncing it up with Linus' latest changes. This way, Linus can generate his own patches from CVS and examine them at will, or on the urging of leutenants.
Another orthogonal approach to the ckernel into separately maintainable parts. There is no excuse for bundling every device driver under the sun when one has loadable modules. Device driver patches would then go to the device driver maintainer (in the same way, and for the same reason ,that Linus does not get Apache patches), offering some relief. Some core device drivers might remain Linus' responsibility, but he'd be more free to concentrate on architectural rather than implementation issues.
Yes, this would fork the kernel, in so much as there would be different bundled device driver sets. I see this no worse than the variety of existing distros, though I would like some central mirrored repository of what drivers were available.
Re:Is Linux bigger than Linus? (Score:3, Interesting)
Call me mean, but I don't respect anyone for wanting to "keep control" over what is supposed to be free software (well, except for modules), espescially when they decided to make it free. Wha? "It's free unless it becomes a hit and then I want control?" I don't think so -- I think Linus has plenty of leveragable egoboo to not need that kind of power trip.
If Linus wants to play architect, and concentrate on APIs, modularity, and scalability, great. If he wants to bundle components meeting his standard for stability, also great. But, if he can't do it all himself, he shouldn't try to prevent others from stepping in.
If he is worried about dilution of the stability associated with "his" Linux, he should excercize his trademark rights: you can't call a kernel, bundled with loadable modules, "Linux®" unless it has his blessing. Of course, meeting the lesser standard of API complience should garner a "Linux®-compatible" moniker, or "[insert favorite distro]-certified Linux® compatible", with Linus authorizing major distros to do their own testing, and applying that moniker.
Does this mean a forking of monolithic kernel bundles? Yes, but I don't think that seperate module sets are that traumatic of a fork. This is an area that the various distro-makers can move into, and an espescially fertile one for embedded kernel providers.
Re:I know Linus doesn't like it... (Score:2)
Well, no but it doesn't really solve this problem. I know BitMover's CVS like thing was designed to help though with a hierarchy of approved trees so Linus could (say) have three or four people feeding patches up to his tree for approval.
However that is still just a tool, what it really requires is for him to find a few people that sort out patches he won't like and ones he probably will... that's harder then finding a tool.
Endowments. (Score:3, Flamebait)
This attitude seems pretty common, even to me, and I don't run Linux. Linus takes a lot of flack for his methods of running the kernel development, mostly from people who think that they have a better way.
Try to remember something. It's his baby. It's his kernel. He doesn't owe you a goddamned thing. And if you've got a better way, start your own Unix style OS project.
(Reminds me of the book _The Moon is a Harsh Mistress_ -- after the Revolution on the moon is complete, then all of the armchair politicos come out of the woodwork to say how it should be. The analogy to the current situation is an exercise left for the reader.)
--saint
Re:Endowments. (Score:2, Insightful)
That's no attitude to take. What we need is cooperation and competition, each in its place, where people discuss the issues and the best ideas/implementations win. If each person who contributes to the linux kernel would have started their own kernel instead of contributing, linux would be much worse off today.
If you something to contribute, then attempt, through polite reason and good code, to do so.
Re:Endowments. (Score:2)
Re:Endowments. (Score:2)
I agree. But it seems many people attempt to criticize rather than genuinely attempting to contribute.
Again, I don't run Linux, and frankly, I really don't care all that much. But it amazes me that someone who has put in the sort of hard work and long hours that Linus has could be the subject of so much criticism from his (kernel developmentally speaking) underlings.
--saint
I agree (Score:3, Insightful)
With this proposed system he could still maintain control, yet pass on the mundane task of merging. Seems like a great idea.
The world would benefit from faster development (Score:3, Interesting)
You know, with Linus's many comments about kernel latency and pre-emptive kernels, you'd think that he'd be all for imrpoving kernel development latency!
Pre-emption (Score:5, Informative)
For example, there was a bug in the ne2000 driver that Alan Cox points out here. [iu.edu] According to Mr. Cox, "this is one tiny example of maybe thousands of other similar flaws lurking. There is no obvious automated way to find them either."
Re:Pre-emption (Score:2)
Surely finding and fixing these bugs is better than discarding improvements and hoping that the bugs will go away on their own.
Year old bug fixes... (Score:4, Insightful)
It wont be long before the MS camp picks up on this and starts to use it as a weapon - not that we havnt had to wait a year or two for bug fixes from them in the past. But Mr Torvalds isn't doing himself or us any favours on this one.
Re:Year old bug fixes... (Score:3, Interesting)
Oh, I don't know about that. After all, I got three Exchange server patches within a couple days last fall! Mind you, the first to broke more than it fixed, but they were timely....
Seriously, some bug fixes can be tricky - and sometimes the patch creates tons of other problems. I know I can not cast the first stone here when I've coded around an issue, only later to have the problem fixed and my work-a-round hork things up. I can believe some issues took a year before all the code was in alignment...
Re:Year old bug fixes... (Score:2)
One of the strenghts Linux has is that you can do this.
The real issue here is that some (not all) developers disagree with Linus' point of view. The whole discussion is one of many that have arisen since Linux was created. It isn't "news" in my book.. just gossip and soap opera material...
The underlying problem however is big enough.. if Linus can't be quick to implement a new feature or patches that are deemed nescesary by others there will be a point in time the kernel will split up... one for small systems like PC's and PC servers, one for embedded systems and one for large systems such as the zSeries etc...
Perhaps that wouldn't be all that bad though. If they can maintain compatibility on the userside for running programs i think we should not be worrying about all these new possibilities..
Trying to fit all the functionality for all those different systems in one sourcetree would eventually become the practical problem. Why do i need to download all those sourcecode when i only need a portion of it for my PC..?
Anyway, i will keep on using Linux for the foreseeable future anyway..
Re:Year old bug fixes... (Score:4, Informative)
Repeat after me: the business community doesn't run Linus kernels anyway. They run RedHat kernels, or Mandrake kernels, or maybe Debian kernels. The distributor is completely welcome to adopt these patches if they see the benefit of them and have time to assure themselves of their benefit. The state of the Linus kernel really has very little to do with what an enterprise user of Linux will experience.
Now, it will look kinda bad for Linus if we end up with a system where patches are tested out in shipped distributions before they are tested in the main kernel tree, but that's a PR problem more than anything.
I think it goes without saying... (Score:4, Redundant)
David U.
CVS/Linux fork? (Score:2, Insightful)
Re:CVS/Linux fork? (Score:2, Redundant)
Read the part about David Miller.
--
Garett
Linus' Reply (Score:5, Informative)
From Kernel Trap:
His general point is summarized with this statement, "One 'patch penguin' scales no better than I do. In fact, I will claim that most of them scale a whole lot worse". He goes on to explain this statement quite thoroughly, adding: " In short: don't try to come up with a 'patch penguin'. Instead try to help existing maintainers, or maybe help grow new ones. THAT is the way to scalability".
Say what you want about Linus' attitude - Having a kernel lieutenant probably wouldn't help either. Either way, there's only so much one person can do.
Re:Linus' Reply (Score:5, Insightful)
It doesn't matter how many maintainers there are: Linus can only work through a specific amount of code/patches per day/week/month. More maintainers result in more patches being dropped, and subsequently in more efforts to keep their patches up-to-date with the kernel.
This is really hurting development, and is frustrating a lot of people who are involved in the kernel with the risk of losing them.
Anyhow: Linus does what he likes, but it's exactly that kind of attitude that will hurt Linux in the longer run - both in technological as reputational (?) way.
Re:Linus' Reply (Score:4, Insightful)
> involved in the kernel with the risk of losing
> them.
The thing I don't get is why these people are goofing around with the kernel to begin with. Isn't one of the biggest virtues of kernel modules the ability to plug 'em in to a running system. New version? Build a new one, unuse & unload the current, and load the new one.
Why can't people who write drivers distribute them under their OWN energy? Why must the kernel be monolithic? Why not trim down the kernel to little more than a core, let linus distribute/patch THAT.
It's worked for windows for a decade now; maybe M$'s method isn't ideal either but being able to go to a vendor website, getting a driver, and dropping it into the system seems a hell of a lot more convenient than waiting for a lone man to include something in a source tree.
Seems to me the kernel is pushing into the future in all the wrong areas, when some of it's features are still living in the dark ages.
Most systems use a mere fraction of the drivers included in the kernel. This wastes bandwidth, configuration time, possibly memory, definetly disk space.
Re:Linus' Reply (Score:3, Insightful)
I agree to a certain degree that not all drivers should always be included in the kernel, but what's kept in, and what has to go? For example: filesystems - way too many of them, but you can't boot without support compiled in, or changing some real stuff on the booting process.
Re:Linus' Reply (Score:2, Funny)
So if I, in the future, teach my currently 4-week old son C programming and show him the Linux Kernel source, then......?
;)
It's called "span of control" (Score:5, Insightful)
It has been proven (through all kinds of research) that any one person has difficulty controlling much more than 10 individuals, with 8 being a practical maximum.
This is why military units are organized in strict tree structures - at no point does any one leader exceed his span of control.
So an 8-man infantry section is controlled by a Section Commander (which can be further broken down to 2 4-man fire teams, but rarely). 4 sections report to a Platoon Commander. 4 Platoons report to a Company Commander. 4 Companies report to a Battalion Commander, and so on through Regiments, Brigades, Divsions, and so forth.
In this manner, the several thousand men in a large formation like a Division can be commanded by one man.
Further aiding the General in charge of the Division is that each sub-unit is granted a certain amount of autonomy within certain boundries. As you go up the chain, orders become more general (2 Battalion is to attack objective Oscar) and as you go down the chain, orders become more specific (Bob, you're carrying the machine gun)
While Linux development does not require this level of structure (and indeed, would probably suffer if this many authority-resistant cats were forced into such a regimented structure) the basic principle of "span of control" is still applicable.
The obvious correct solution is to modularize the kernel into subsections with clearly defined areas of responsibility, with a "mini-Linus" (who Linus trusts) granted control over each module.
Not suprisingly, this is what Linus has suggested.
The trick (and the catch) is two-fold. Firstly, you have to find people who are both responsible enough to be able to act as a "mini-Linus" without dropping the ball (which means Linus has to trust them too) and secondly, the kernel has to be modular enough so that changes in Module A do not step on or otherwise negatively impact code in Module B.
That's a hard row to hoe.
What I find a little distressing is that there appears to be a whiff of revolution in the air; of people talking about overthrowing Linus (and Alan Cox seems to be among them) This will not solve the problem. It is not that **Linus** doesn't scale, but rather that any single maintainer doesn't scale. Any revolution will just be "same shit, different name"
CVS doesn't solve the problem either, as anyone with commit privilages can jsut dump anything they want in there, and it becomes canon. Linus' function as a shit filter is very very important. every patch *must* be audited by someone with very high standards before it goes in.
I hope the crew rallies around Linus - as usual, he's right.
DG
except it doesn't work (Score:3, Insightful)
It's not enough for Linus to wave his hands and say "we need a new VM", and then Joe hacker writes one.
Joe hacker, on the bottom, is fixing typos and writing documentation. Actually, Joe Hacker is probably working on apps.
The brigade commander, in your analogy, is writing the VM in bits and pieces. Linus has to go through it carefully and apply the diffs or not. Software is all about details. There are ways (e.g. modularity and interfaces) of splitting up the work, but only to a certain extent.
Imagine a war fought almost entirely by the general staff, with their subordinates serving as assistants and sopport staff -- it's the reverse of your analogy: A small group of people at the very top write most of the code.
The beginning of a major shift in Linux kernel dev (Score:5, Insightful)
This is all out the window now. The kernel is very large, and continues to grow, and Linus can no longer track the project just using his head.
In short, Linux is growing up. What it grows up to be I guess we'll see in a year. Either it starts using professional tools to manage this increasingly professional project, or the bloat (yes, the kernel is bloating) leads to unmanageable chaos, and the bloom goes off the Linux rose.
Yea, this might be it. (Score:4, Insightful)
When ideals and/or ego enter the fray you have to be careful. This could alienate the very people that are needed at this time and cause Linux to loose steam. Yes, it's his toy, yes he can take his ball and go home. But hopefully this won't cause a major split similar to what happend to Unix in the 60's/70's.
Personally, I hope I'm wrong.
Re:Yea, this might be it. (Score:3, Insightful)
I'm just not seeing the sort of personal attacks that really mark a degrading work relationship. I see people who are often critical, and sometimes rightly so, and several extremely intelligent people, who seem to generally be able to work with each other, (though lacking in some basic social skills when dealing with people they view as "lesser").
All in all, I'm not seeing that "beginning of the end" syndrome. I'm seeing a rejection of dogma, and real honest evaluations of the situation. Some people disagree and say the opposite is true. Read the mailing list, the whole thing sometime. kt.zork.net is great and all, but it leaves out certain subtelties.
No, no, no... (Score:5, Insightful)
For those who don't care to read the discussion, Linus essentially feels that this is a bad idea because no general patch manager is going to scale better than he does or get burnt out less quickly than Alan Cox.
He then goes on to say that the solution to the problem of the scalability of one maintainer is to partition the different subsystems of the kernel to such an extent that there would be precious few patches that actually require a knowledge of the entire kernel source.
Re:No, no, no... (Score:5, Interesting)
Which is the proper way to design a complex system anyway. Partition the system and then present well defined interfaces, versus making a complex, monolithic system more complex and monolithic.
Linus is correct. CVS just gives you an extremely efficient filing cabinet. Without a someone to file things properly, the filing cabinet is no better than a shoebox. Running a CVS tree makes you the equivalent of a secretary. People will just come by, throw things on your desk, and yell "File this" as they run home for the day. Linux chooses to be an engineer. He refuses to just accept what it thrown at him. He wants explanations and reasoning as to WHY a patch should be included. Not using CVS slows the development system down so that it is comprehensible by a single person. The kernel isn't just a collection of source files collected in a cabinet. It's a fine masterpiece of artwork on Linus' desk.
Every three months (Score:2, Insightful)
this same subject pops up. It's the same on Debian lists: every now and then people say how frustrated they are with the slow pace of releases and propose some new scheme which usually doesn't even differ from the old one.
Software problems are solved by coding. The core kernel people, like core Debian people, have thought about these things before, and are doing their best. All these blatherings do is waste everyone's time.
Growing pains (Score:5, Interesting)
Yeah, it's Linus' baby -- but once IBM is advertising your OS during the superbowl, it might be ok to expand the development structure a little bit.
The Star Approach (Score:2, Interesting)
Maybe we just need more Lieutennants?
Maybe we just need fewer patches!
What it will take (Score:3, Interesting)
i) A respected developer (Alan Cox, perhaps) who already has their own patch tree will open it up via some source management system
ii) Other developers, including those frustrated ones, will begin to use it, including keeping it roughly in sync with the official tree.
iii) The forked tree will develop as fast/faster than Linus' tree (assuming the pro-CVS people are right)
iv) LT will realise that its not a bad idea, and essentially adopt the forked tree as his own, complete with CVS.
In short, theoretical arguments for CVS will continue to cut little slack with LT. A demonstration of its effacacy will be very persuasive.
If (iii) is untrue, the argument will be moot anyway (why have source control if it doesn't help?)
Re:What it will take (Score:3, Informative)
The real solution, as Linus has pointed out, is that everyone has a small group of trusted people who they take patches from for a given subsystem. What you end up with is Linus at the top with his 10 or so trusted people below him each responsible for submitting patches for large subsections of the kernel, and those guys accept patches from other maintainers who again are in charge of yet smaller subsections within the previous. This type of tree can handle very complex projects and scale very well provided everything is designed modularly with -good interfaces- (another point that Linus has been trying to drive home). So what Linus wants to see happen is that people -stop sending him patches directly- and start sending them to subsection maintainers.
Now the good news. You couldn't tell it by this Slashdot post but everything I have just said has already been happening. People are moving over to the new system and things are getting worked out, the reason you still see posts like this is that some people just haven't quite worked their way into the new system yet. So yes, it's a problem and it's one that is already being solved, no need to panic.
To quote Linus: "If you can't get a patch through to me find someone else you can go through."
That statement alone will go a long way towards making the reorganization happen.
Linux needs a "centralized clearinghouse" (Score:3, Funny)
I think the folks at IBM and Oracle ought to seriously have a LONG talk with Linux Torvalds himself and convince him to create a true clearinghouse where every improvement is approved by a committee. That way, Linux improvements happen in an organized fashion, which makes things way easier for developers and IT managers.
Re:Linux needs a "centralized clearinghouse" (Score:5, Insightful)
Have you ever worked at a real job before?
Re:Linux needs a "centralized clearinghouse" (Score:4, Insightful)
It may seems so, to people coming from the corporate culture that uses language as a bludgeon (e.g. "issue" as a euphemism for "problem")--but having worked in both worlds I'm convinced that "independence" and "freedom" are better words for it than "anarchy" and "chaos."
I think the folks at IBM and Oracle ought to seriously have a LONG talk with Linux Torvalds himself and convince him to create a true clearinghouse where every improvement is approved by a committee. That way, Linux improvements happen in an organized fashion, which makes things way easier for developers and IT managers.
Convince him how? Work him over? Impress him with their org charts? Offer him the chance to work on a bigger project? Imply darkly that it won't look good on his annual review? If a variety of very smat people haven't been able to convince him with their best rational arguments, what exactly are the corporations supposed to do? And more to the point, do we really want to "convince" people of things that way?
And do you honestly believe that committees speed things up, or make them more organized, or easier to deal with? (This isn't a rhetorical question; for all I know you could be in full agreement with me, but writing in heavy sarcasm mode here and I'm just not getting it.)
Having managed a number of large projects I certainly find Linus's arguments more convincing; I'd much rather get on a ship run by one good captain than a ship where a variety of gizmos and rubber stamp committees had been put in charge for the express purpose of making the ship go faster than the captain thought prudent.
-- MarkusQ
Linus is, as is often the case, right (Score:5, Interesting)
The basic premise he makes is one that many developers seem to miss. To quote Linus:
From that, he goes on to point out that people tend to have a small group of people they work well with and trust; perhaps 5 - 10 people. So, in essence, if Linus appoints 5 - 10 maintainers of major subsystems (which he has), and each of them has 5 - 10 people the trust to maintain specific aspects of their subsystem, then there is no problem.We're not there yet. But that is the only direction I can see that will work, long term. It just takes time for the appropriate people to move into their spots; trust takes time to build, and the structure has taken time to modularize enough to allow this to work.
And, though I would prefer to see it in use just to make the maintainers' jobs easier, CVS will do nothing to solve the problem. It is a tool, not a process. We need a process. Actually we have a process, it just isn't fully implemented yet.
No, strike that. We have two processes. The first, used by the bulk of the kernel hackers, is not fully implemented yet. The second, used by a minority of the kernel hackers and a large part of the pretenders, is simple: whine alot, and, when that doesn't work, whine some more.
Linus says [alaska.edu] it much better, and in his own unique ... er ...
Idium, sir?
Yes, in his own, unique idium.
Re:Linus is, as is often the case, right (Score:2)
Except that there is a problem. Even patches from designated maintainers are being dropped on the floor.
With CVS, those maintainers would be able to check in the patches themselves.
Re:Linus is, as is often the case, right (Score:5, Insightful)
The problem is that Linus is getting lots of other patches as well. Not enough filtering is going on.
Allowing maintainers to check in their own patches will not happen as long as Linus is interested in Linux. The buck stops with him. Rightly or wrongly he prefers not to have to back out patches (roll back checkins, if you prefer). At least that is where we stand, now. I would love it if Linus would eventually trust a small group of people enough to allow a CVS system to be usefull.
What if the patch spans more subsystems? (Score:2)
From that, he goes on to point out that people tend to have a small group of people they work well with and trust; perhaps 5 - 10 people. So, in essence, if Linus appoints 5 - 10 maintainers of major subsystems (which he has), and each of them has 5 - 10 people the trust to maintain specific aspects of their subsystem, then there is no problem.
There is theoretically no problem, when there are small 'patches' (or better: fixes for glitches) which fall totally in 1 subsystem's area. However, what if a patch spans more than one subsystem? Or better: what if a patch needs changes on another subsystem? In an OS, it's placing foundation on foundation, layer on layer, they all depend on the layers/functionality blocks below that layer. You can't just say: "Hey, John, you do VFS and you, Dave, you do the VM", because the VM needs services of the VFS and vice versa.
If you have a designmanifest which has to be implemented, you can point to people which control certain parts of the project, WHICH don't have to be subsystems necessarily (vertical segmentation), but can be seen as horizontal segmentation: one group does the core layers, other groups do the layers ontop of those core layers.
Re:Linus is, as is often the case, right (Score:3, Insightful)
The "hierarchy" goes exactly one level deep. Hardly worth talking about.
The maintainers that "report" directly to Linus are the second level of that hierarchy. From there on, the "levels" begin to blur, as many of the "lower" levels of maintainers would probably eventually be in the circle of trust of more than one other maintainer.
In effect, this method becomes a web, not a tree. Well, it could be argued, I suppose, that the web looks pretty damn tree like in parts. Or that the tree has large portions that are quite web-like.
Whatever.
The main things, as s Linus originally pointed out here [alaska.edu] is to:
CVS the answer? Mebbe, mebbe not. (Score:3, Insightful)
Now, how to move forward and expand the circle without diluting the value that he brings. Something like CVS can help, but the base problem isn't technical and the solution won't be technical, either.
The real problem is factoring out that part of the Linus load that Linus does better than anybody else can do, and offload other parts to those who can do them better than a loaded-down Linus.
The hard part for everyone, Linus included, is to understand how much the "other" part feeds into Linus's ability to do the "Linus" part.
Sometimes, even mundane tasks help you to maintain the vital connection needed to do the great things well.
I'm just glad I'm not the one having to work all of this out!
Stating the Obvious (Score:2)
I'm not saying this as flamebait or to say so-and-so is wrong. I have a great deal of respect for Linus Torvalds but his brain-child has outgrown him and at some point he needs to let it go outside the house.
Why not just fork off a non-Linus version? (Score:2)
So you wouldn't be able to call it Linux? If you think you can do it better than Linus then by all means do! As long as you release the source code you are within GPL.
Copy the Linux kernel source to another location, and import it into a CVS database under a new name. Of course, then you're playing with Linux the game which BSD plays with all software projects. You'd better be sure you can organise your code to easily import features and fixes when Linus gets something you don't have.
Linus doesn't scale NOW (Score:3, Funny)
If it becomes necessary, we can always fork and branch the development tree further though that will further burden the main development branch so we'd like to avoid it if at all possible.
Not exactly accurate... (Score:2)
I think most people who read the email assumed that the "patch penguin" was a machine with out reading the email. But the proposal is that someone (Dave Jones) take everyone's patches and filter them on to Linus.
It's not that Linus doesn't like organization, it's that he likes patches to come from maintainers. If Dave Jones gets all the patches he's going to be as overloaded as Linus is.
Management (Score:3, Interesting)
Now I'm no kernel hacker, but I do think the current system could use some improvement. Wouldn't it make sense for the kernel to be split into sections (device drivers, memory/thread manager, etc) with each section having a maintainer and one "integration manager"? (Applicable areas that could be subdivided, like drivers, could divide by type again.) If the system were properly modular, the integration manager would just have to make sure that the interfaces between the systems were well designed, don't change frequently, are thourally examined between changes, and that when they must change they are reimplemented nearly simultaniously in all affected subsystems. (Hopefully, any given subsystem could also undergo rewrites without affecting other subsytem code.) Again, I'm not seriously looked into the kernel code (yet!) so I have no idea how things work right now... but the scale IMHO requires a division of the labor. (Formatting the code to match would serve to further reduce the labor involved in maintainance.)
From someone who has never hacked a kernel: (Score:3, Interesting)
I think I'm not alone when I say that, when I came to linux a few years ago, I was very impressed. Stability, speed, tons of functions, and a lot of configuration...
Well, that was all well and good. What was not all well and good is that, like so many others, I didn't actually write anything for the kernel. I assumed it was in good, and (as I have no real OS-level programming experience) more capable hands. I know that was what I was thinking when I explained to others why linux was so cool, and would always keep getting better. People would keep on sending in improvements, and these improvements, when approved for absolute stabiity (Alan Cox's old job mentioned in the leter, wasn't it?), they would be integrated into the sacred source.
Sigh. Well, that assumes the improvements get accepted or even seen in the first place, doesn't it? Linux is a huge development project (I mean, come on, it's an OS), and anything made in last I don't know how many years of development of any significant size had ought to have a tighter system of management; human or not. I'm a little worried that Linus has any opposition to these sorts of things. I'm actually amazed that linux got this far without more- I always assumed there were some definite mechanisms of the sort involved (hence my regret for never messing with the "sacred source"). I always just assumed there were.
Now I am surprised there wasn't a call to action before this. In a project of this nature, with so many people, projects, and significant economic forces putting their investments of time, money, and effort into this, you'd think that something would have been done when good patches started getting dropped. If that is the case something has to be done now.
Like I said, I never messed with the source. I never fixed anything; I never submitted patches; therefore I never had this problem. In short, I never knew. I was never involved, and never felt I had to be- I mean, better heads than mine were on the job, right (I said the source was "sacred")? Well, not all heads are better at everything, and I think that the more heads invoved, the better off this would be (at the very least, we would have reached this crisis point sooner and come to some decision on how to handle it). Maybe if some of the right people got in on it earlier, an appropriate management system would have been advocated and introduced- and these problems would have been minimized.
Possibly not. But more has to be done on this. Linus' counterpoints [alaska.edu] are well argued, but if lots of valuable code patches are really being (uneccesarily) lost due to the current system, are they good enough?
And why did mankind invent computers? (Score:3, Interesting)
Hello? What if a big company says: "To hell with computers, we do all our business-directing by hand, without decision support systems etc..", is that a clever move? No! Humans are not capable of processing a lot of complex data CORRECTLY. Computers do. Use them, whenever they can help. To avoid using computers in software versioning is IMHO one of the most stupid things you can do. What do you think, that Microsoft used Bill Gates' brain to organise the 35 million lines of sourcecode of Windows 2000?
Re:And why did mankind invent computers? (Score:3, Funny)
Apparently they do. Have you got a better explanation? (this is Slashdot, right?)
No, wait.. now this makes it sound like Linus and BillG work the same way.. mmm...conspiracy..
architectural limitation (Score:2)
What is the solution? There are lots of possibilities, but they all come down to either changing to some other programming language, or emulating what other programming languages have built in "by hand".
Something like an Objective-C runtime, where you can replace classes and methods at runtime and where most important function calls go through the runtime, would allow most of the kernel functionality to be changed and enhanced by separately developed modules without requiring patching all over the place. Such runtimes are very efficient and would probably not make a lot of difference in terms of performance. Another approach is a Mach-like microkernel. If people want to stay with plain C, another solution is to add lots of hooks for everything (kind of like you have in Emacs).
Even if this were to make the kernel run a little slower, what good is it to have a kernel that runs blindingly fast but doesn't do what I want and doesn't have drivers for the devices I want to use?
Wait a minute... (Score:4, Insightful)
Of course the advantage of this is that the mundane stuff can be done in parallel to the architectural stuff.. thus allowing more than one thing to get done at a time. Where as Linus himself can only do this serially.
The reason that this is important is that there is a very large backlog of patches that are simply not getting applied. Allegedly the whole VM issue could have been resolved if Linus had enough time to look at and apply RvR's patches. What this means is that Linux, with the inability of kernel maintainers to effectively submit patches, is turning into Minix. Remember that Linux was born out of Linus & others reaction to the fact that patches submitted to Minix couldn't get applied and redistributed (for copyright reasons). Which meant that in order to get a reasonable system, you first had to download the unpatched code, then go through a little patch-o-rama. It was ineffective. The current patch log reminds some of that situation, and has some fearing a potential kernel fork.
Of course, the biggest question that I have for the proposal is that Rob wants to just "formalize" a position that already exists and is being done, to some extent, already. If the position is already in place and already operating, then why is there still a patch backlog? How does formalizing it actually improve the situation?
CVS isn't a magic bullet. (Score:5, Insightful)
That said...
"Just use CVS" isn't anything remotely close to an answer. SCM, release control, whatever you'd like to call it is 99% about the PROCESS, not the tools. CVS is a tool. It's a great tool and one I highly recommend, but at best it's 1% of a solution. It seems quite obvious that Linus's current process is incredibly flawed. It would seem a drastically different process needs to be devised. Only once you have figured out what your process is going to be can you even start shopping for tools (CVS, whatever) to help you create it (and even then...the tools are only 1%, if you even have tools. The PROCESS is what's important).
Of course "CVS forks" of Linux failed. Anyone with even the slightest understanding of CM could have easily predicted such an end. You've got a handful of people, not even the top people, trying to constantly fold in the efforts of thousands of contributers all using a completely different process (ad-hoc patches emailed to devnull@linus.org). What did you expect?
You have two choices. Devise a workable process which Linus will agree to (with or without CVS, whatever). Or, you can devise a workable process that everyone else is happy with...and simply throw Linus off his paper throne. The first option will likely never, ever happen...at least if every single word Linus has ever uttered on the subject isn't true. The second option might actually welcome the dawn of a non-choatic Linux, one which wouldn't be so easily cast aside as a cheap toy made by a small man with a big ego.
Linux/Linus Flame (-1k Karma)
Sloppy? (Score:3, Interesting)
I wonder, how sloppy would freeBSD development be if you synced your kernel hourly with the dev kernels? How buggy would it be? What does the interdeveloper mail look like in the freebad world?
I run kernels over a year old. There's no slop. I'm (mostly) happy. In about 6 months, I'll switch to 2.4.
If you want less slop, stick with a distribution kernel. They've chosen and tested it out to be stable and reliable.
Why not just send patches to the right people? (Score:5, Insightful)
This being the case, why not just automate that? Subsystem maintainers are effectively maintainers for a known group of source files. Its not hard to figure out (automatically) from a patch which files are affected - and from that, which maintainer(s) the patch is relevant to. Given how Linus describes his 'trust network' it looks like he wants a hierarchy of subsystem managers who can merge patches for him to accept upstream. I can see that scaling a lot better than the proposal in the article.
For this to work of course, you'd have to hope the patches had reasonably local scope - if most patches affect >1 subsystem this isnt worth doing at all.
- Baz
Leadership Structure (Score:3, Insightful)
Linux doesn't need a "patch manager", it needs a House Of Commons or a Congress or whatever you want to call it. It can't do away with the president and the chiefs of staff, but it has to have a heck of a lot more checks and balances than it does now. And it definitely needs to be a lot less autocratic.
Eventually, we're going to see more and more forks as people begin to understand that the OS is no longer Alan and Linus' own private playground but a heritage of the people who use it every day. While they bicker and fight over who gets to play king on a given week, someone is going to pull the rug from under them. Royalty has always been tolerated while the populace is enamored with its trappings, but revolutions do happen once in a while.
Source code maintenance tools/organization (Score:4, Insightful)
I almost don't believe what I'm hearing. How do you manage development branches? How do you roll back your code to undo an enhancement you thought would work? How do know which bug fixes are tied to a particular instance of code? Wimp? More like "anyone who ISN'T using source code maintenance is a fool."
Re:Source code maintenance tools/organization (Score:3, Interesting)
Simply undoing the flawed submission means that you did a poor job of making sure that only quality code goes in. It does not fix anything.
Now, this is the only thing that I think is missing from Linus' vision. An SCC system can store this information efficiently. Linus' system (changelog) can only store this information at the granularity of a pre-patch.CVS can work, look at FreeBSD (Score:5, Interesting)
cjk
Toothpaste likes you
The Linux Fund? (Score:5, Funny)
How about we set up a fund so that Linus can work on Linux full-time, instead of needing a day job? That's not a permanent fix, but it would help for a while.
Re:The Linux Fund? (Score:5, Interesting)
I'm fairly positive that if he did want to he could find a Linux-related company to back him on this, despite the fact that many are in the crapper (RedHat would take him, IBM would probably pay him a full salary just to work on Linux, etc.)
Maybe he has interests outside of Linux? Ever even consider that?
Re:The Linux Fund? (Score:4, Insightful)
And we could get a gigolo so the wife stops bugging him to perform...
I got it! Then, we can force him to take ever-increasing amounts of speed, so he doesn't have to sleep!
Then he can dedicate his life to linux 24/7!!! That would be great!
</sarcasm>
Make sure he actually wants to dedicate that much of his life to writing one program.
Thank God Linus doesn't scale (Score:5, Funny)
Now let's just hope RMS doesn't scale either.
Neither did Moses (Score:5, Interesting)
Here's the Linux version:
Re:Neither did Moses (Score:3, Funny)
... you know the BSD devil is just tempting you with CVS.
bite the fruit of the tree of knowledge, c'mon.
Re:Neither did Moses (Score:3, Funny)
Rob
Monolithic kernel (Score:3, Insightful)
Such foolishness (Score:3, Interesting)
IMHO (very humble in this case as far al kernel hacking goes, but I have been writing software for a decade, using various source control systems for half of that) the fact that Linus doesn't use any kind of source control/version tracking system is immature, stupid, counterproductive and just plain wrong on his part. It's shortsighted that a project of that size and importance depends on the buddy-system (ie Alan Cox has the source as well, etc..) for backups and revision tracking. When I write a few 100's of lines of utility code that i intend to keep, you bet it's in CVS!
No, CVS or other source control is not a magic bullet, you have to use it right. But if you don't have one, you don't even have a gun.
I'm not saying he should have a CVS tree with every joe haxor and their dog allowed to write to it. But no version history and change tracking capability at all! Gibber!
/. remarks == loads of BS moded to +2 (Score:5, Informative)
Rob Landley and Linux Torwalds aren't bashing their heads in on one another nor is ANYBODY of the Kernel Team about to 'dethrone' Linus. Whatever that may be.
In fact Landley suggest a "Patch Penguin' to actually EMPOWER Linus in his actuall job as an arcitect of Linux.
Linus in turn says that officially shifting patchin jobs to somebody else (the said Patch Penguin) won't save the problem of, for instance, people being to lazy to clean up their code dependencies.
Linus wants to see a sort of 'web of maintainers' where everyone knows and works with a overseeable amount of others (just like he does) rather than a big patcheritis boiling around a main single/pair/group of developers.
It may, IMHO of a absolute non-kernel savy guy, kinda boil down to the monolitic/modular kernel discussion that comes up every know and then.
Then again, on the other hand I gather the impression that Rob Langley and Linus Torwals aren't that far apart in seeing the issue that needs to be addresses rather than seeing different ways of aproaching a solution to it.
Anyhow,
Authority comes from trust, not competence (Score:5, Insightful)
...although I concede that trust comes from competence. But what does this have to do with anything?
If you're going to overhaul (or in this case, overthrow) an established system, you need to do it with much fear & trembling. Look, I see this all the time as a manager. A new project is given to a manager. That manager hires someone competent and works with that person to build something from nothing. If the project takes off, you can end up hiring a lot of people to help support that project. It is possible for all those new hires to be more competent than the original, lead developer. But that original, lead developer now has management's trust. That lead developer, who got the damn thing off the ground in the first place, has an opinion that -- like it or not -- counts for more than the other developers. You can call that unfair, but certain people are just better rounded than others -- they have social skills, they can explain things well, they understand the market pressures, or whatever. Not to mention that it's their baby and you don't fuck around with stuff like that. As a manager, if I stomp all over the lead developer's project, that lead developer may not want to lead the next project! Sure, I can call on one of the other people brought in to help, but those other people may not be leaders.
I'm not saying that such an effort is doomed. But I am suggesting that, if you want to propose that Linus relinquish some authority, you damn well better have a value proposition for him. Shouting "this needs to be a democracy" is what you want, not what Linus wants. He's looking at that as a management nightmare, a removal of a power structure that is in place because, whatever it's faults, it was the structure that worked. If it doesn't work well anymore, you either have to convince Linus, or you have to convince everyone else to stop trusting the person in authority. That is a massive undertaking.
Web of trust vs. certification authorities (Score:4, Insightful)
The ideas about kernel patches being discussed remind me of the distinction between a "web of trust" and "certification authorities" in the world of cryptography. Certification authorities are the conventional means, which are used by the most popular web browsers. Some authority, like Verisign or Thawte, is universally accepted as trustworty, and if they sign a key, then it can be trusted. The web of trust was advocated by Phil Zimmermann and built into PGP, and AFAIK, PGP is the only context in which it's ever been applied (or even proposed). It's an attempt to reflect our everyday notion of trust -- I decide who I trust, on whatever basis I choose, and if someone I trust has signed a key, then I trust it. I can also decide if I trust people I don't know, if they are trusted by people I trust. There's no need for a central authority in a web of trust (although I might choose to include CAs in my web and trust them).
Rob Landley is suggesting that we entrust the patch penguin with the job of filtering patches towards Linus, but Linus sees the PP as a kind of central authority, and prefers to have a web of individuals around hom who he can trust.
Like Phil Zimmermann, Linus is arguing that his model is much more similar to the way that people decide to trust each other in real life. In the world of crypto, certification authorities have always been regarded with suspicion for reasons just like this -- why should I trust Verisign more than my friends? In the kernel world, Linus is saying, isn't it better and more intuitive to trust a group of people I know well, who can distribute the work amongst each other?
The analogy breaks here a bit. CAs in the crypto world are suspicious because we don't if they're corrupt -- maybe they're full of Enron managers. In the kernel world, we need to trust maintainers with their skill and above all, ability to handle the workload. Corruption is not the problem (I hope not, anyway.)
But, to take the analogy further, one should also note that PGP web of trust never really worked well as a practical matter. Most people using PGP had an island of people around them whom they trusted, with few connections outside of their immediate clique. I don't recall *ever* receiving a key from a stranger that was evaluated as trustworthy because of its position in the web topology.
So I'd say that Linus has a point in saying that the web model fits our common-sense intuitions about trust. But if Rob Landley says that it's not functioning well, the experience of PGP gives some support to his argument.
Re:Yeah, but... (Score:3, Funny)
That kid with the blanket out of Peanuts cartoons. Everyone knows that!
Re:Monolithic! Duh! (Score:2)
Re:Monolithic! Duh! (Score:2)
Re:Microsoft must be laughing (Score:4, Informative)
Well, I can't claim any insider knowledge, but I bet that all kind of crap goes on within the MS development teams that we never hear about.
When you've got a lot of bright people working hard on a very big, complex thing, something would be very wrong if you didn't get these kind of problems arising occasionally.
Personally, I think Linus is right, and all those people who are bitching should sit down and think for a moment, and perhaps think of ways they can help to make Linus's very hard task a little easier, rather than just complaining.