Is the Stable Linux Kernel Moving Too Fast? 156
darthcamaro writes "Yesterday the stable Linux 3.10 kernel was updated twice — an error was made, forcing a quick re-issue. 'What happened was that a patch that was reported to be broken during the RC [release candidate] review process, went into the release, because I mistakenly didn't pull it out in time,' Greg Kroah-Hartman said. The whole incident however is now sparking debate on the Linux Kernel Mailing List about the speed of stable Linux kernel releases. Are they moving too fast?"
No (Score:5, Insightful)
Re:No (Score:4, Interesting)
Really?
The current process resulted in us having CFQ + EXT3 as the default for a long time (some distros still have this). This basically means any sort of interactive performance is worse than horrible. The only reason we're beyond it now is because EXT3 is on its way out with EXT4 being preferred.
IIRC, wasn't CFQ one of the first major infrastructural things put into 'stable' after this 'rapid release' approach was adopted?
Also, udev.
I'm sure there are other examples... and maybe I'm projecting a bit.
Re: (Score:2)
CFQ only comes in to play when accessing new uncached data from disk when disk is at medium-high usage (at low usage any read/write queues are empty).
I'm struggling to figure out any interactive programs that fit that description. Web/email/documents/games etc... don't touch the disk in any significant way and the cache handles most disk accesses for them anyway.
Re: (Score:2)
Until you start swapping. And most users tend to run a ton of programs at once (or load up dozens of tabs in Firefox) so the kernel starts to swap
Re: (Score:2)
Ok that may be a scenario where CFQ doesn't perform adequately, however I'd say that is a pretty poor example.
You cannot reasonably expect a fluid desktop environment with moderate swapping. It is never going to happen.
Re: (Score:2)
It used to not be a problem, that's the thing. Before all the modern schedulers came to be (so, back in the 2.4 days) it was entirely possible to stream a video over the network without stuttering - while running a -j3 kernel build, updatedb, and a find on root with memory exhaustion taking place (eg. browser was loaded up). It was a matter of pride that these things could be done on Linux, because Windows would fall over under similar loads. Nowadays, Windows handles these situations better than Linux does
Re: (Score:2)
Use the deadline scheduler then if you don't want a complicated scheduler. It should be included in most prebuilt kernels.
Funnily enough deadline is recommended for server loads and CFQ is recommended for desktop use. The exact opposite of what you are suggesting.
And I have no doubt my computers can do exactly what you describe just fine without impacting desktop performance at all.
Just the other day I was doing network video streaming (1080p over SSHFS), I had 4 virtual machines running, plus Thunderbird,
Re: (Score:2)
Are you using an SSD? Was memory exhausted?
The scenarios I describe were/are disk contentious in scenarios at or near memory exhaustion, when the system dips into swap.
You can experience this as soon as the system starts dumping RAM pages to swap even today, assuming you've not got an SSD.
Re: (Score:3)
and yes, it also appears some webmasters code their websites assumiing their users have machines with quad cores and 16GB of RAM.
This. I had to throw more RAM into my wife's laptop because a significant portion of the interior design blogs she reads are coded by seriously inept people.
Re:No (Score:4, Interesting)
Linux (the kernel) is accumulating new security holes at least at the same speed as they are fixed.
Proof: Ubuntu kernel security hole fix rate has been constant for years.
(actually I have not counted the actual number of holes, only the actual number of kernel security patches - these two should correlate though).
Re: (Score:2)
I work in kernel security and I would say we have improved. You can't just tell people "don't make mistakes" and expect security to improve the only way you can improve is by improving the process.
1) We've added a few exploit prevention techniques like hiding kernel pointers.
2) Our fuzz testers have improved.
3) Our static checkers have improved.
But we're not perfect.
For example, we earlier this year we merged user namespaces [lwn.net]. Obviously this is tricky code which deals with security. People had been workin
Re: (Score:2)
Although you are more qualified than me (I gave up following security actively over ten years ago), I beg to differ. No offence.
Seems like you have invented a very complicated version of jail. Typical Linux attitude, both "NIH" and "it is a superset (i.e. can do more), therefore it is better". In security both have been proven to be bad ideas quite a few times.
[quote] Code has bugs. That's life [/quote]
I do not want to talk that much about whether some code has bugs or not, rather about attitude. That sente
Re: (Score:2)
One thing: I do not and have never claimed that some-other-OS is better (you won't get me in a OS flame war).
I claim Linux could do hugely better.
Re: (Score:2)
How does the Kernel drive what your disks are formatted to? Your disks are formatted to ext3 or 4 long before you config/compile the Kernel.
Re: (Score:2)
Re: (Score:2)
The current process didn't result in that at all. Your distro chose that default and could've changed it any time they liked.
Get involved with your distro if you care so much and help them choose sane defaults.
Re: (Score:2)
All in all, I think it's a nice problem to have. Compare that to the kernel being stagnant, it's great that being able to include submitted code safely and fast enough becomes an issue to look into. I doubt MS or any of the other big software companies have issues where features and improvements are being *produced* so fast that QA is unable to keep up. I suspect that they have more of an issue with them being *requested* faster than the company can provide.
Re: (Score:2)
Timely releases of the Linux kernel don't hurt anything anyhow, because most of the packagers and distributors don't release the updates until they've had further testing. In fact, most distros lock in at a particular kernel release, and backport patches to *that* release rather than updating the whole kernel.
So there are really three levels: dev, stable, and production.
Re: (Score:2)
Indeed. Also anybody with some experience in rolling their own kernels with sources from kernel.org knows not to use any x.y.0 releases when they want stability. Give a specific release some time to mature before using it, say 2-3 weeks. You may also have to read changelogs to find out what things were patched. There are also nice, pretty current "longterm" kernels.
That said, I have been on some current releases for some 24/7 machines and did not encounter any issues. I would say that the Linux kernel folks
Re: (Score:3)
Re: (Score:2)
That's how Linux can thrive dispite the chaos of getting patches from 1000's of independant programmers.
Re: No (Score:4, Interesting)
Re: (Score:3)
As if you can't have both. That's what branches are for! Duh.
But of course you'd have to understand branches and their use first. Why call it stable if it's just another testbed where everything is shipped regardless? If "stable" is moving so fast that bugs that are already recognized cannot be pulled back in time, something is just wrong.
Re: (Score:2)
Self flagellation is preferred over allowing the greater world to whip you mercilessly.
Compared to what? (Score:5, Insightful)
"Are they moving too fast?""
Compared to what, Windows, IOS, OSX, What?
>known bug that got by review
>caught
>fixed rapidly instead of waiting for the next release
I don't see the problem.
If this was a regular occurrence, yeah, it'd be a problem. But it's infrequent enough to be "news."
Unlike Patch Tuesdays, which aren't.
--
BMO
Re: (Score:3)
Patch Tuesdays aren't news. Patch Tuesdays that break something are.
Re: (Score:2, Funny)
Now you're being redundant.
Re: Compared to what? (Score:3)
Re: (Score:3)
Their massive install base works against them here. On Windows a one in a million bug will happen by next Tuesday.
Re: (Score:2)
Impossible bugs happen 12 times every single day on most computers (the amount of spontanious errors in RAM due to cosmic radiation and the lack of error correcting memory). Now add one extra error every week from an operating system bug. Can you spot it?
Re: (Score:2)
Re: (Score:2)
Although they are starting to become the norm.
Re:Compared to what? (Score:4)
"Are they moving too fast?""
Compared to what, Windows, IOS, OSX, What?
Compared to a speed where accidents like this are less likely to happen, if such a thing exists. It could be that OS release cycles are unsafe at any speed!
Re:Compared to what? (Score:4, Informative)
Which would be why you should use an actual distribution kernel and not the compile-it-yourself variety if you need stability and testing.
Re: (Score:2)
Perhaps the question should be "Am I adopting Linux kernels too fast?"
Well, if you were hit by this bug in a significant non-testing way then I'd say yes. Unless you are testing, don't install anything on release day.
Re: (Score:3)
What's good. (Score:4, Insightful)
Re: (Score:2)
"Are they moving too fast?""
Compared to what, Windows, IOS, OSX, What?
A long time ago, I don't remember where it was, maybe LKM, Linux Torvalds said there would never, ever be a version 3 of the Linux Colonel. I thought that was a strange thing to say, even for him. I thought to myself "things are really gomna get weird when they get to 2.99.99.99.99.99".
So now, he's changed his mind and the version numbers are zipping along. Not as fast as the absurd version number inflation of Firefox and Chrome, but still a lot faster than it used to be.
In General, I don't have any Ma
Re: (Score:2)
>Last sentence.
http://i.qkme.me/3rpqih.jpg [i.qkme.me]
--
BMO
Re: (Score:2)
>moderation "overrated"
Not like karma counts for anything, but I've always thought the "overrated" mod was chickenshit.
"I disagree with you, but I don't have the balls to reply"
--
BMO
Re: (Score:2)
problem is that most of the time it takes its sweet ass time
known bug that got by review
caught
wait half a decade for known bug to be fixed or more likley forgotten so it still remains in next version, goto 10
Re: (Score:2)
Fair point.
But the change to 3.0 was "just a number" I really don't give a shit about version numbering. Just tell me which one works.
--
BMO
Released kernels are the real testbed (Score:5, Informative)
As indicated in the debate on LKM, rc kernels get hardly any testing, although all of the tests it does get are mostly by highly motivated and astute testers
Most distros are releasing kernels at least one behind the developers tree, with not a great deal of incentive to update the kernel right away, (even if they make it available in a repository for those wanting it). So much of the real world testing on new kernels comes only after its been released, and even then it doesn't hit Joe Sixpack's machine for several months.
So at most, this was an embarrassing incident, and not a bit deal. The amazing thing is that it was caught at all. Some of us remember kernels that got into production distros with serious things broken that should have been caught much earlier.
Re: (Score:1)
So much of the real world testing on new kernels comes only after its been released, and even then it doesn't hit Joe Sixpack's machine for several months.
One of these days I am going to meet Joe, and I am going to complement him on his Abs.
Re: (Score:2)
They mean beer, not abs...you queer.
Thanks for the clarification, genius.
Oh, and woosh.
Re:Released kernels are the real testbed (Score:5, Informative)
From where I'm sitting, as someone who used to routinely build rc releases and use them, this is how things look.
Five, ten years ago you had people such as myself who would build RC (or AC, etc.) kernel trees to test things and see how they'd work. I know several people who regularly made LKML submissions, many of which turned out to contribute to fixes.
Today, using the rc releases isn't as practical because they're fairly divergent from distribution patchsets. A lot goes into a distribution kernel which isn't present in the vanilla kernel.org kernels, it seems.
More often than not, pulling everything together to build our own kernels isn't worth the extra effort: possibly due to the shortened cycle and possibly due to general code maturity, there's little benefit. Maybe our definitions of 'benefit' has changed, too - but arguably, the changes in the kernel today are nowhere near as drastic or significant as when (say) XFS was getting merged with additional kernel and disk schedulers.
That's what he said! (Score:5, Funny)
I mistakenly didn't pull it out in time.
The problem with oversight from two persons (Score:2)
... is that it would move even faster.
Really, this is a non-problem. The 'system' worked.
Thank God they're no slick sleezeballs like Ballmer,
but acked and corrected.
Enjoy your new kernel. Relax. Be happy!
Re: (Score:2)
But my heart monitor automatically downloads and installs the latest kernels within minutes of them being posted to kernel.org.
Time for an LTS Option (Score:1)
Time for an LTS option. RedHat, Canonical, Debian, should backport security fixes and maybe mature drivers to a LTS kernel for 5 years or so.
For that matter, go ahead and make a LTS gcc fork, backporting security fixes during that same time schedule.
Re:Time for an LTS Option (Score:5, Informative)
3.10 is the next LTS kernel [kroah.com] by Linux standards. The existing long term kernels [kernel.org] are 2.6.32 (as used in RHEL6, Debian Squeeze, Ubuntu LTS 10.04), 2.6.34, 3.0, 3.2.50 (used in Ubuntu 12.04 LTS), and 3.4.59.
Re: (Score:3)
2.6.32 (as used in RHEL6, Debian Squeeze, Ubuntu LTS 10.04)
Given the amount of backported features and hardware support that Red Hat continuously adds to their kernel I highly doubt that it looks anywhere close to 2.6.32 from kernel.org.
No, you want a frozen kernel (Score:5, Informative)
No, you want a frozen kernel. A stable kernel isn't one without bugs, is one where there aren't massive changes and you get dot releases with fixes
Absolutely not (Score:3)
There are plenty of older kernels being actively maintained. Stable does not equal recommended for all production needs.
No (Score:2)
Keep in mind that the stable kernel releases are not expected to be production-ready. Linus just produces the input to the actual testing and validation processes, which are performed by the distributions. That assumption is built into the kernel dev team's processes. Not that they intentionally release crap, but they don't perform the sort of in-depth testing that is required before you push a foundational piece of software onto millions of machines.
This is why we have Linux Distributions (Score:3)
I haven't needed to bypass my Linux distro and install a vanilla kernel in over ten years. I can wait. If it hasn't been packaged by the major distributions yet, it also hasn't been widely deployed. There are some highly competent kernel package maintainers out there, and they're likely to catch a lot of things themselves.
Then there's the package release and its early adopters (Debian, for example, releases the latest kernels in Experimental and/or Unstable, letting them cook for a while). I typically pick up new kernels when they're in Debian Testing or Unstable (if I really want a new feature). This minimizes the chance of running into problems.
(note, this works best for people using standard hardware. ymmv regarding embedded or obscure devices)
9X / 4.XX line will be better just don't use USB (Score:1)
USB will suck in the first 4.XX one.
If "Stable" isn't stable... (Score:3)
...then it's moving too fast.
The decline of quality of time (Score:2)
As someone who tested drivers with it:
OK through about 3.2 then it started to decline.
Faster decline around 3.4
3.7 - Who needs NFS anyway? Took until 3.9 to fix that
Rate of patch submission? (Score:2)
I follow kernel development only cursorily, looking at the kernel mailing list once in a while. But I get the distinct feeling that patch volumes have been higher over the past few months than they would be a few years ago. A version is simply something that group a set of tested patches. Generally, you don't want the sets to get too big, so it seems natural that the speed of version releases is keeping up.
It would be nice to see a plot of the number of commits and number of versions over time.
OHH EMMMM GEEEE! (Score:2)
Are we *gasp* agile?Or what!
still pissed there hasn't been a LTS since 3.4 (Score:2)
3.0, 3.2 and 3.4, released in short succession all are LTS, and since then none. I understand the devs can't maintain a zillion kernels, but could they at least space them out and/or release on a more stable cadence.
i.e. drop 3.0 or 3.2 for 3.10/11???
Re: (Score:2)
That's what he said... (Score:2, Funny)
Apologizing for mistakenly not pulling out in time...hilarious.
Re: (Score:2)
Too Fast? (Score:3, Insightful)
Let me try and catch up to it, and ask...
Seriously. Why is this even a question? Did a new stable release show up in your watch or your laptop - or your in flight entertainment system, over night?
Packagers and distribution maintainers aren't exactly up in arms about this...
Re:TDD (Score:5, Insightful)
"the fun thing about a kernel is that there is no good way to test it except to run it" [youtube.com] --Greg Kroah Hartman
I work on PostgreSQL, and nothing goes out until it's been validated on the entire buildfarm [postgresql.org]. It's hard to have such a thing for the Linux kernel though, because it's so easy for a bug to break test machines. You need to catch when machines are responding, do a hardware reset, and then rollback to a known good kernel instead. It's much harder than most software testing to automate.
Re: (Score:2)
Re:TDD (Score:4, Interesting)
The logs of the machines that break are the most important part of the test results here. You can't just throw them away when a VM dies without losing most of the testing value. If you're lucky, a busted kernel will stay alive long enough to create a crash dump when running on dedicated hardware. That's less likely to happen on VMs.
It's also possible to automate hardware resets with IPMI [wikipedia.org] or similar lights-out management code. All of that takes special hardware though, and test code like this is painful to build.
Re:TDD (Score:5, Insightful)
Re: (Score:2)
VMs create a single 'hardware' platform. You aren't testing more than a fraction of the kernel.
And your automated resets will let you boot a new kernel easily (so does 'reboot' anyway), but if it does break you can sit there resetting over and over again in to the same broken kernel. Yay!
Re: (Score:3)
You'd need VMs to emulate all sorts of hardware and architectures. We don't have all that yet. We can't emulate every mouse, network driver, etc, so there's no way to test those drivers (which are part of the kernel).
Re:TDD (Score:4, Insightful)
Which rises question of just why are they part of the kernel? Why does a mouse driver need to run at Ring 0?
Re:TDD (Score:5, Informative)
You need to do some more reading on how Linux works.
Re:TDD (Score:5, Informative)
You need to do some more reading on how Linux works.
So do you.
Linux is sort of a hybrid kernel now. Some hardware drivers are in ring 0. Quite a lot are no longer. libUSB for example allows userspace USB drivers. They work great. FUSE allows for user space filesystems which work great where absolute top performance is not necessary (eg sshfs, ftpfs etc).
A good fraction of the bluetooth stack, for example anything above L2CAP (the Bluetooth world equivalent of the IP stack's SCTP) , such as ATT and GATT is all userspace (and the non kernel side sucks donkey balls by the way). That means I could (if it didn't suck massive donkey balls) control all the various profiles with the majority of the code in userspace.
All the printer drivers are mostly in userspace (yay).
The graphics (X11) is largely in userspace for now...
Sound has a large userspace component and all the complex stuff like routing, mixing, and figuring out what to send where is in userspace (pulse or Jack).
The Linux kernel as-is is more than capable of running a mouse driver in userspace.
Re: (Score:2)
Good job on not paying attention to the last ten years of Linux development.
So the stuff that isn't a hardware driver attached to the Linux kernel isn't in the kernel ... good job on deducing that.
Printers aren't in the kernel because (get this) they aren't part of the hardware. The Bluetooth stack isn't hardware, the hardware is hardware. That stuff's in the kernel. The sound system? The hardware support is (again) in the kernel, the stuff that's at a higher level isn't.
There are some very specific edg
Re: (Score:2)
I don't follow printer drivers closely enough to know if the poster was right. I try and follow the nuance language, it is abused often.
Re: (Score:2)
Your parent's statement is not an oxymoron.
If every single print driver has components running in both ring 0 and userspace, but the preponderance of components (by number or 'size') of every single print driver is in userspace, then it is more precise to say that "all print drivers are mostly in userspace" as opposed to "printer drivers are mostly in userspace". The latter is semantically a superset of the former, as it could either mean the same thing, or also describe a situation where some printer driv
Re: (Score:3)
Not very well in this regard, according to the grandparent, which gets us back to my question: why do it this way?
But thanks for your +5 Informative comment anyway. I certainly know more having read it.
Re: (Score:2)
As though I gave myself +5 ... jeez.
The post I responded to was out of touch with how Linux is designed. If you want to know more, go do some reading. Its not hard.
Re: (Score:2)
Hmmm :)
Nothing like a good troll to read in the afternoon. I love reading Slashdot at -1 :)
What a stupid question. (Score:2)
Clearly, the author of the post has forgotten 3+ year intervals for kernel releases. Of the odd, quickly fixed pothole is the price of 6x release speed, "Hell, yes!" Is the answer.
Re: (Score:2)
Re: (Score:2)
Re:What a stupid question. (Score:5, Insightful)
Ah yeah, like kernel.org isn't trusted. Yes I know you said "distribution" but really now.
He wasn't talking about trusting that it didn't contain a trojan or something. By trust he meant vetted for quality.
It is a legitimate concern. The whole reason for having a release cycle is to have sufficient QA to prevent issues like this from happening. Distros provide that service - when Linus/Greg call a kernel done, they call it ready to start being tested. RHEL is still running 2.6 (albeit with backports).
Re: (Score:2)
The problem with RHEL is that the true version of any package (for compatibility purposes) is the part after the dash. They'll call it "2.6.18" (I think that was the RHEL5 kernel), but really it's got most of the patches (but not all!) up to a much more recent version. If you have a piece of software from outside the RH repositories, and you want to know if your libraries are sufficiently up to date, you need to go dig up the SRPM for the version of each that you have installed and peruse the changelog feat
Re: (Score:2)
Oh, I fully get that.
But that was what was meant by moving QA to the distro. In this case Red Hat basically circumvented the upstream release process entirely creating a new set of releases which met their own quality standards. Most distros can't afford to completely replicate the upstream release management for everything they distribute (or at least the important stuff).
The fact that many pay for RHEL basically (or just use it anyway via CentOS) speaks to the concerns people have about FOSS release man
Re: (Score:2)
The fact that many pay for RHEL basically (or just use it anyway via CentOS) speaks to the concerns people have about FOSS release management in general. If they trusted upstream maintainers to do the job right then Red Hat would go out of business.
Even if every upstream maintainer was perfect, there is still a lot to do to maintain a linux distro. Didn't RH do a billion$ in business last year? I assume some of that is in selling SOMETHING related to RHEL.
I do not think you have an accurate understanding of Redhat platform's business model in conjunction with the challenges of maintaining and versioning software.
In my experience, people pay for RHEL for any number of the following reasons:
1) A guarantee not to break ABI for a given x.y version of RHEL
Re: (Score:2)
Agreed on all you posted, and much of my point was directed at #1-2, and to a lesser extent #3 (the reason they can offer support is that they make sure stuff doesn't get broken all the time). Most distros do not do any of these to the same extent, but the value of any linux distro is still often in the way they add a consistent quality layer to upstream release practices.
WHAT THE FUCK! WHAT THE FUCK!!! (Score:2, Interesting)
WHAT THE FUCK!
I can't believe this. I've been reading Slashdot since 1998, and I have never seen such a stupid suggestion in all that time.
Test-driven development is not the solution to this problem. And my good gawd, behavior-driven development is even farther away from the solution than fucking test-driven development is.
Behavior-driven development is one of the biggest loads of shit to splash upon our profession in years. Customers and analysts will write tests? Riiiiiiiiiiiiight...
All that we get from B
Re:WHAT THE FUCK! WHAT THE FUCK!!! (Score:5, Funny)
WHAT THE FUCK!
I can't believe this. I've been reading Slashdot since 1998, and I have never seen such a stupid suggestion in all that time.
...
OK, one of those two must be blatantly false.
Re: (Score:2)
Nope, even that doesn't excuse it. -- voice of experience
Re:WHAT THE FUCK! WHAT THE FUCK!!! (Score:5, Funny)
Thanks Linus, but I think it's an established rule that you can't go releasing new versions of software until the ridicule surrounding your last release has died down. How else are we going to get stories for /.? Microsoft plays along. Why can't you?
Re:WHAT THE FUCK! WHAT THE FUCK!!! (Score:5, Funny)
Only said fuck four times not including subject. Not Linus.
Re: TDD (Score:2)
I've always felt that a rolling release, as exemplified by FreeBSD, naturally stablizes things. They have two branches, STABLE and HEAD (a/k/a CURRENT). New code is commited to HEAD and only once it's stabilized is it MFC'd (Merged From Current). Once in the STABLE branch it's feature frozen, and receives continuous refinement and debugging for the life of the code. There has always been a saying that "FreeBSD works like running water", and I attribute this statement to the development process.
In one of Gre
Re: (Score:2)
He is working on this one too so he was in two article in a row.
http://tech.slashdot.org/story/13/08/21/1959227/internet-infrastructure-for-everyone [slashdot.org]
Re: (Score:3)
No that's only when you break User Space.
Re: (Score:2)
If you don't know who he is abusing, then it means it is probably you.
Re: (Score:2)
Are you on RHEL 5 maybe? Then download 2.6.18 from kernel.org and do a diff; it is quite far from 2.6.18. =)
Re:Poor version control (Score:4, Informative)
"At 9 a.m. PT on Aug. 20, Linux kernel developer Greg Kroah-Hartman announced the 3.10.8 kernel update. At 3:44 p.m. PT, Kroah-Hartman announced the Linux 3.10.9 kernel."
So, it was a new release, not a re-release.