Linux Stops Reverting Most University of Minnesota Patches, Admits Good Faith (lwn.net) 83
destinyland writes: LWN has a terrific update what's happened since the discovery of University of Minnesota researchers intentionally submitting buggy code to the Linux kernel:
The writing of a paper on this research [PDF] was not the immediate cause of the recent events; instead, it was the posting of a buggy patch originating from an experimental static-analysis tool run by another developer at UMN. That led developers in the kernel community to suspect that the effort to submit intentionally malicious patches was still ongoing. Since then, it has become apparent that this is not the case, but by the time the full story became clear, the discussion was already running at full speed.
The old saying still holds true: one should not attribute to malice that which can be adequately explained by incompetence.
On April 22, a brief statement was issued by the Linux Foundation technical advisory board (TAB) stating that, among other things, the recent patches appeared to have been submitted in good faith.
Meanwhile, the Linux Foundation and the TAB sent a letter to the UMN researchers outlining how the situation should be addressed; that letter has not been publicly posted, but ZDNet apparently got a copy from somewhere. Among other things, the letter asked for a complete disclosure of the buggy patches sent as part of the UMN project and the withdrawal of the paper resulting from this work.
In response, the UMN researchers posted an open letter apologizing to the community, followed a few days later by a summary of the work they did [PDF] as part of the "hypocrite commits" project. Five patches were submitted overall from two sock-puppet accounts, but one of those was an ordinary bug fix that was sent from the wrong account by mistake. Of the remaining four, one of them was an attempt to insert a bug that was, itself, buggy, so the patch was actually valid; the other three (1, 2, 3) contained real bugs. None of those three were accepted by maintainers, though the reasons for rejection were not always the bugs in question.
The paper itself has been withdrawn and will not be presented in May as was planned...
One of the first things that happened when this whole affair exploded was the posting by Greg Kroah-Hartman of a 190-part patch series reverting as many patches from UMN as he could find... As it happens, these "easy reverts" also needed manual review; once the initial anger passed there was little desire to revert patches that were not actually buggy. That review process has been ongoing over the course of the last week and has involved the efforts of a number of developers. Most of the suspect patches have turned out to be acceptable, if not great, and have been removed from the revert list; if your editor's count is correct, 42 patches are still set to be pulled out of the kernel...
A look at the full set of UMN patches reinforces some early impressions, though. First is that almost all of them do address some sort of real (if obscure and hard to hit) problem...
The writing of a paper on this research [PDF] was not the immediate cause of the recent events; instead, it was the posting of a buggy patch originating from an experimental static-analysis tool run by another developer at UMN. That led developers in the kernel community to suspect that the effort to submit intentionally malicious patches was still ongoing. Since then, it has become apparent that this is not the case, but by the time the full story became clear, the discussion was already running at full speed.
The old saying still holds true: one should not attribute to malice that which can be adequately explained by incompetence.
On April 22, a brief statement was issued by the Linux Foundation technical advisory board (TAB) stating that, among other things, the recent patches appeared to have been submitted in good faith.
Meanwhile, the Linux Foundation and the TAB sent a letter to the UMN researchers outlining how the situation should be addressed; that letter has not been publicly posted, but ZDNet apparently got a copy from somewhere. Among other things, the letter asked for a complete disclosure of the buggy patches sent as part of the UMN project and the withdrawal of the paper resulting from this work.
In response, the UMN researchers posted an open letter apologizing to the community, followed a few days later by a summary of the work they did [PDF] as part of the "hypocrite commits" project. Five patches were submitted overall from two sock-puppet accounts, but one of those was an ordinary bug fix that was sent from the wrong account by mistake. Of the remaining four, one of them was an attempt to insert a bug that was, itself, buggy, so the patch was actually valid; the other three (1, 2, 3) contained real bugs. None of those three were accepted by maintainers, though the reasons for rejection were not always the bugs in question.
The paper itself has been withdrawn and will not be presented in May as was planned...
One of the first things that happened when this whole affair exploded was the posting by Greg Kroah-Hartman of a 190-part patch series reverting as many patches from UMN as he could find... As it happens, these "easy reverts" also needed manual review; once the initial anger passed there was little desire to revert patches that were not actually buggy. That review process has been ongoing over the course of the last week and has involved the efforts of a number of developers. Most of the suspect patches have turned out to be acceptable, if not great, and have been removed from the revert list; if your editor's count is correct, 42 patches are still set to be pulled out of the kernel...
A look at the full set of UMN patches reinforces some early impressions, though. First is that almost all of them do address some sort of real (if obscure and hard to hit) problem...
Sometimes a slap on the wrist works. (Score:3)
People make mistakes, and for most people, just letting them know they did wrong is good enough.
However I would think the Linux foundation should keep a closer eye on them.
Re: (Score:3, Insightful)
No, people screw things up and then refuse to take responsibility for their actions, making others clean up their mess. U of M should be banned for decades, to be made an example and warning for others. Let them take that time to reflect on their stupidity.
Re: (Score:1)
Not convince U of M did not do this intentionally... a ban for a period of time (year or two) and/or other restitution should be considered.
Re: (Score:1)
Ban, nay, the law should be upheld, the commited a computer crime fraudulent use of a computer network to attempt to upload bad code on purpose to disrupt the open source network. They should be able to continue their research. What happens when the police arrest you for a computer network crime, how do you get prosecuted, what was the punishment. All that research needs to be completed and a complaint should be forwarded to the police, the computer network crime reported and the individuals prosecuted.
The
Re:Sometimes a slap on the wrist works. (Score:5, Insightful)
Re: (Score:2)
What he said.
Re: (Score:1)
You don't understand University tenor and caste system. Many professors and adminstrators will still be there. Again, make them an example, so other fools will beware.
Re: (Score:1)
You don't understand University tenor and caste system.
You don't understand that those people who get *tenure* (I assume professors and administrators are not there to sing in a high voice) are not the students / PhD candidates who actually come up with and do this kind of crap, and are generally not the type of pointless busybodies that sit on ethics committees (because they think of themselves above all that).
Again, stop punishing people who have nothing to do with anything.
Re: (Score:2)
Wrong and childish of you. The school including administrators and professors are absolutely responsible for actions done for projects by students and candidate which of course they approve and oversee. Grow up.
this isn't attainder of blood (Score:1)
Re: (Score:3)
At any rate, what they did is they banned submissions from UMN email addresses -- when they malicious submissions actually game from random GMail addresses. So the ban wasn't doing anything useful in the first place except signaling displeasure to the university.
Re: (Score:2)
In ten years none of the people originally responsible for why the university was banned may even be at UMN. Instead they could be at some other university that isn't banned while the Linux foundation are still punishing people who had nothing to do with the original action that led to the punishment.
Yeah, I rented a trailer from Uhaul a little while back, and they still won't rent people with Ford Explorers due to an issue they had in the 90s. [wikipedia.org]
Re: (Score:2)
In ten years whoever failed at oversight will probably still be there. The students will be gone but the authorities supposedly keeping them in line will still be in place... And barring major life upheaval will probably still be the same person they were. That is to say, irresponsible.
Re: (Score:2)
In ten years whoever failed at oversight will probably still be there.
Not sure of U of M but most Universities I know actually have a reasonably high turnover of anything other than tenured professors.
Re: (Score:1)
Our university completed its migration to Peoplesoft in 7 years: The first two years to design and implement the new system, the next five years to retire everyone who preferred the old system.
Re:Sometimes a slap on the wrist works. (Score:4, Interesting)
Lots of reasons not to punish them:
1) It is basically unenforceable by the kernel community.
2) It punishes far more innocent people than guilty people, just by association
3) It seems that those involved get the point that they messed up
4) Why feed the fires of discontent and create bad blood.
5) It is just bad economics. The kernel team has always been about pragmatics and it costs them time to try to police this and cost them potential good developers.
6) Forgiveness is good for the soul. It feels a lot better to forgive people than obsess over how much you've been wronged.
Re: (Score:2)
False, the school could be sued for the time wasted by their stupidity, and even criminal charges brought. Harmful things done over the internet to computer systems is illegal. Making an example of fools is pragmatic, because it deters other fools.
Re: (Score:2)
That will just entourage them to "fuck up" again and just claim "it was just a prank, bro" when they are caught.
Yeah (Score:3, Funny)
blacklist all the @umn.edu email addresses. That will prevent malicious activity from ever happening again.
Re: (Score:2)
"That will prevent malicious activity from ever happening again."
Are you being sarcastic? Why wouldn't the malicious actor just submit from a non-umn email address? This is a feel good measure more than anything that will truly prevent bad activity. It's trivial for a truly malicious actor to work around it.
Re: (Score:2)
Yes I’m being sarcastic. That was the kernel team’s answer. Block UMN instead of more thoroughly vetting code.
Re: (Score:2)
Alright, sometimes it's hard to tell! Glad to hear it. I get why they did it. They were angry and wanted to make a dramatic point about how angry they are. The ban should be rescinded quickly though given that it's basically pointless other than to express their ongoing displeasure.
Re: (Score:1)
The last three decades have shown that banning particular email addresses or entire domains or countries has not successfully shut down spammers, scammers, or political dissidents. You must have seen by now that there are very profitable VPN, mail laundering, spam blocking, message sanitizing, captcha busting industries built on all sides of this game.
Re: (Score:2)
Because that's what you're advocating.
When kernel development slows down to a halt, will you then be blaming kernel developers for not doing things fast enough, because they have to reanalyze the whole kernel for every commit?
Re: Yeah (Score:3, Interesting)
At one point in time the University of Minnesota has more IP addresses assigned on campus than any other entity on the Internet. They were a major early adopter of TCP/IP. The Gopher protocol was developed there (the Gopher is the UM mascot).
The U is a huge Big 10 institution. The idea of banning the whole Univ because of one rogue experiment is a little overboard.
Re: (Score:2)
So, their grey beards excuse them from costly fraudulent behavior? One rogue experiment? Overboard?
Five years from now, you're a UMN grad competing with five other people for a Linux-based position, perhaps as a coder. Two are from Idaho, one from Indiana, another from Florida.
Which state university (gosh, Big 10!) sent fraudulent kernel code updates, and to what degree of character did their faculty respond? Will you remember the early adoption and gopher, or several twits that smugly gamed the kernel dev
Re: (Score:1)
It would be a blessing to any CS grad not to be associated with a workplace that relies on fallacies of composition or division to make important decisions.
Re: (Score:2)
Without the ban, rogue experiment or not, it would have continued to encourage shittier and shittier patches. This is not a hypothetical - that's exactly what happened. Shitty patches were sent in. They were not banned. So they continued to send shittier patches. At which point, it is not GregKH's responsibility to look at UMN. He'
Re: Yeah (Score:2)
Wrong - akin to committing crime (Score:3, Insightful)
U of M wasted, stole, hundreds of hours of time from Linux kernel. They should be banned for a decade to be made an example. Other goody two shoes fools can then take warning.
Re:Wrong - akin to committing crime (Score:5, Interesting)
I think the usual standard of accepting guilt, making restitution, and taking a punitive penalty should cover it.
If I were judge & jury, I think the university would have to set an official research policy banning similar future endeavors, cover the cost of the time lost by the kernel maintainers, plus be banned for a full academic year (synching that up with the university's schedule makes the most sense from an impact perspective).
Banned for a decade's a bit much.
Re: (Score:3)
Well, no.
There's also identifying the systemic issues which allowed it to happen, and addressing them so it can't happen again.
Wrong - akin to committing metaphor. (Score:2)
Much like in piracy cases the artist's time is "stolen" by those who chose not to pay for it.
Re: (Score:2)
No, it's the artist's right to control distribution that is "stolen".
Re: (Score:3)
wrong, wasting the artist's time filling out thousands of forms with frivolous claims would better analogy. Instead you bring up illogical unrelated thing. The thing U of M has done is legally actionable, they could face civil or criminal punishment if kernel development team chose.
Re: (Score:3)
A more appropriate punishment would be to demand payment for the time they wasted. There's probably at a minimum a 5 digit cost to this, possibly 6 digits depending on how many hours have been spent.
Bearing the financial cost of this is a far more powerful motivator at convincing organizations than a ban.
Re: (Score:2)
and a broken set of over-trusting processes allowed them to "steal" all this time.
Well, one would hope so (Score:2)
"A look at the full set of UMN patches reinforces some early impressions, though. First is that almost all of them do address some sort of real (if obscure and hard to hit) problem..."
At this point in time, one would hope this would be true of *almost all* Linux kernel patches from anyone.
Re: (Score:3)
Re: (Score:2, Interesting)
In other words, the Linux community acted like companies who yell at the messenger when a problem is found in their network. Funny how the same people who claim someone finding a flaw in a network, even when they have no business being in that network, should be held up as a good guy, but when flaws are found in their own network, suddenly it's an abominable crime which needs to be punished.
Re: I'm so confused (Score:5, Informative)
You're off on a couple important points: first, none of the three intentionally buggy patches they submitted were approved and merged - they were all rejected (but not all because the malicious code was identified). Second, and perhaps more importantly, the same group of people were simultaneously responsible for piping the output of a poorly-functioning automated code review tool directly into the patch submission process, flooding the kernel devs with a large number of bad and often nonsensical patch submissions which had not been reviewed by any humans, were not disclosed to be the output of an un-validated automated tool, and which were labeled with high priority and importance despite mostly being unhelpful or trivial. This effectively spammed the patch pipeline with a bunch of crap at the same time that these same people were submitting their intentionally buggy patches for their undisclosed penetration testing project. To me, it's the combination of these actions which is especially objectionable.
Re:I'm so confused (Score:5, Insightful)
I'm so confused on this whole thing. My understanding is that UMN submitted broken patches to see if they would be caught by the review process. They weren't.
Actually they were caught. Of the 4 "buggy" patches submitted only 3 actually turned out to be buggy and all were rejected.
They then wrote a paper on the experience.
How are they the bad guys?
Why don't you follow up with a research paper where you try shoplifting from a store without telling the owners first?
Their research would be fine if they got consent of someone on the Kernel team. The but consequences of acting the way they did were pretty serious. Even though reviewers caught all the buggy patches the initial deception means the researcher can't be trusted, meaning they needed to do a second, much more thorough investigation of the other good patches that were submitted.
And beyond that, what safeguards did the researchers have in place to make sure that if their bad patches got accepted they would later get reverted. What if they ended up in some nightly builds and burned some testers? How would you feel if you spent a couple days debugging what turned out to be a deliberate bug from some research project?
They proved that the Linux review process doesn't work. That sounds like an issue that needs addressing.
Except again, it did work.
What needs addressing is the UMW researchers. The deception involved in the project should have set off all sorts of alarm bells and the school should figure out how to ensure another group of researchers doesn't do the same in the future.
Re:I'm so confused (Score:4, Interesting)
Note: (according to the article) they also didn't tell anyone which patches were buggy. So the dev team was looking through all the patches submitted from the university to try to find them.
Re: (Score:2)
>What needs addressing is the UMW researchers.
Or their supervisors, or the organisations who were funding those researchers work....
Re: (Score:2)
So submitting bad code is like stealing? Alright whatever you say. This is more like submitting intentionally bad tasting food to a competition just to mess with the judges. People here acting like UMN slapped their mother.
Your analogy misses the critical fact that 3rd parties would have been using that buddy code / eating that spoiled food.
Re: (Score:3)
Re: (Score:3)
That's almost a fair comparison, and if nobody was dependent on the security of the linux kernel it would be true. Most software which is being used even to a fraction of Linux has a bug bounty program, people have demonstrated mitm attacks on Linux distro package distribution, and a next step on that would be to attack the source packages.
I don't think that UMN carried out this experiment in a responsible way, but Linux would be wise to consider how they're handling malicious contributions, or even how th
Re: (Score:2)
But they were caught. According to TFA, the patches that actually introduced bugs were rejected in the review process. What the kernel community is objecting to is the extra work involved in re-reviewing a bunch of patches to make sure everything has been accounted for.
The paper was objectionable both because it was the result of a big time waste for a community that didn't agree to be the subject of research and because it was counting bad patches successful when they were rejected for any stated reason ot
Quips to live by. (Score:3)
As it happens, these "easy reverts" also needed manual review; once the initial anger passed there was little desire to revert patches that were not actually buggy.
A look at the full set of UMN patches reinforces some early impressions, though. First is that almost all of them do address some sort of real (if obscure and hard to hit) problem...
Something something about being slow to anger and baby, bathwater.
I'm surprised UMN hasn't expelled the student (Score:3)
Conducting an unauthorized research project on human test subjects without proper approval including legal review and permission from appropriate stakeholders is ethically black and white. It's many steps beyond getting caught cheating on a test. It's an ethical breach that can have unmarked SUVs roll up to the campus and their occupants flashing credentials while asking for "who is in charge of this project?"
Re:I'm surprised UMN hasn't expelled the student (Score:4, Interesting)
They did submit their project for review. The review board said they didn't need to look at it, since they were not directly testing humans.
This is a UMN problem, not an individual project problem.
Re: (Score:3)
They did submit their project for review.
They did submit the project for review to the ethics board AFTER the project was completed.
It should have been submited BEFORE the project was completed.
If the project had indeed experimented on humans, or even worse, experimented on humans and HARMED them, what good does it to do submit it for an ethics review AFTER it is completed?
Re:I'm surprised UMN hasn't expelled the student (Score:5, Interesting)
It's not the student. The student had a good research idea. However, the mentoring professor AND the review board both belong to the university and screwed up.
The mentor should've reviewed the ethics of it (as a professor of CS, it may be a valid test, but you don't test something without telling someone lest you raise the ire of law enforcement. The review board didn't review the proposal - seeing that "it doesn't affect humans" isn't sufficient to conduct an ethical study.
The student only holds partial blame for coming up with the idea. The university holds the rest of the blame for blindly letting it happen.
I'm sure the university wouldn't like it if a pen tester came on campus and started to test the security of the network without anyone knowing. Heck, they'd probably call security and law enforcement for attempt to do same.
Re: (Score:2)
The mentor should've reviewed the ethics of it
You are assuming that the professor reviewed it and missed the huge ethical implications. I doubt it. They even explicitly say that unfortunately they couldn't think of another way to do this test (i.e., that patch review had to be done by uninformed people).
The review board didn't review the proposal - seeing that "it doesn't affect humans" isn't sufficient to conduct an ethical study.
I am genuinely curious how they got the proposal past the IRB. You can get an exemption for "it doesn't affect humans", but the IRB would still review the proposal to confirm your claim.
As many people pointed out, it clearly affects humans because patc
Creative punishment (Score:2, Troll)
They should have to send coeds in bikinis to wash Linus's car every weekend for a year.
Linux Weekly News is great (Score:2)
What to do? (Score:1)
The Year of GNU HURD (Score:2)
I've said it before and I'll say it again.
The Linux Kernel mantainers should ban all of UoMN for life to contribute to the Linux kernel.
That way, there will be a much needed influx of wo/man power to GNU HURD by avid researchers, hungry to make their mark, and a name for themselves, making HURD happen at last! ;-)
Make them pay for the wasted hours (Score:2)
$150 an hour?
So, perhaps 30 developers, times an average of 10 hours, times 150 is $45,000 for a simple guess. But, have the developers estimate their time. And get a count of how many worked the problem. (And make no mistake, it is / was a problem, verses simple paperwork issue.)
Make the clowns from the circus pay. People tend to remember bad behavior more, and less likely to repeat it, if they are forced to pay for it.
Of course, if the Uni does
Linux maintainers the OG Karens (Score:2, Insightful)
A simple rule for life: Not everybody is your friend and has your best interests at heart. I don't leave my doors unlocked and I keep my valuables under lock and key when not in use; the Linux Kernel & FOSS maintainers, in general, should realize that. If not, they need to find another career ASAP.
This suppression, knee-jerk bullshit, shows that we have too many thin-skinned Karens in software development these days. "OMG! you wasted our time!?!?" Fuck you, this was approved research and the researchers
Nice shuffle and dodge :] (Score:1)
That is not an old saying! (Score:2)
That is a saying that only emerged in the last 10 years!
And it is complete nonsense, used by blackeyers, to lull themselves into a comfoting delusion.
In reality, evil has used "Oh, Iook at me, I'm just sooo incompetent..." to get people to swallow their evildoing since time immemorial.
And if something is harmful, it is malice, no matter if competent or not.
No in reality, it's: Never attribute to incompetence, what you can attribute to malice!
And in any case, you can tell if it is competent or incompetent.
Co