The Fedora-Red Hat Crisis 263
jammag writes "When Linux journalist Bruce Byfield tried to dig for details about the security breach in Fedora's servers, a Red Hat publicist told him the official statement — written in non-informative corporate-speak — was all he would get. In the wake of Red Hat's tight-lipped handling of the breach, even Fedora's board was unhappy, as Byfield details. He concludes: 'If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient, then what chance do we have that other companies — especially publicly-traded ones — will act any better?'"
Re:Consider Red Hat's response vs. Debian's (Score:5, Interesting)
I pretty much agree: Fedora was obviously squelched by Red Hat corporate who was apparently afraid of the reaction of their paying customers. Despite the token board openings and motions about openness, after this nobody can pretend that Fedora is on anything but a *very* short leash held by Red Hat.
On the one hand, as a user I found myself trusting that Fedora's infrastructure crew were plugging away and probably handling things about as well as could be. On the other hand, the vague statements and lack of hard facts was (and still is) disturbing.
They should have come clean, and allowed the the community to vett their process.
Ob-FUD [just to poking Bruce for fun]: If they do come forth with details, it will be interesting to see if it was an ssh key compromised by the Debian flaw that caused this mess.
Re:Consider Red Hat's response vs. Debian's (Score:5, Interesting)
I got an email from Starfield a while back offering to re-key my SSL certificates because they had figured out that my original request was using Debian's compromised OpenSSL. I had already rekeyed by then.
Thawte is Debian based. I wonder if they had a problem.
Re:Consider Red Hat's response vs. Debian's (Score:5, Interesting)
Thawte is Debian based. I wonder if they had a problem.
I checked our Thawte keys/certs against the SSL blacklist released by Debian. I checked several from Thawte, and could not find a potential compromised key/cert.
Also, we are a Red Hat customer. I have to agree, I prefer the way Debian handled their incident, versus the way this Red Hat incident is being handled. After reading the Red Hat Security Announcement the details are so vague, I am still not sure of the scope and reach of this vulnerability.
This is an ongoing investigation (Score:5, Interesting)
This seems to be, from reading the Fedora [redhat.com] and Red Hat [redhat.com] statements, an ongoing investigation. The same way the police don't comment about investigations in progress, Red Hat is keeping mum. Keep in mind, the breach may be very complex and not something that they can confidently say "we understand" without a very detailed analysis.
They announced the issue immediately and took steps. For now, give them the benefit of the doubt that further details will be forthcoming once a proper investigation has been completed.
I don't see this as anti FOSS (Score:5, Interesting)
There are a number of possible scenarios that would recommend against being 100% candid on how far you were breached. If I was violated, I think I'd like to take a moment to do a "self-check" on all of my important bits before I started telling everyone all of the nitty-gritty details. As the article pointed out, people were told that there was a breach, and that they should not update for a few days. How is this "anti-FOSS"?
Perhaps they were on the trail of who did this? Perhaps they were comparing notes with the Ubuntu breach cited in the article, with the goal of finding the M.O? Perhaps, like any police detective, they were keeping certain clues to themselves while they investigated further? If the crimes were found to have similar approaches, keeping quiet might improve the odds of capture?
I use Fedora, and had been using Red Hat before Fedora came along. I don't think this kind of hysterical "anti-FOSS" reaction is really fits the facts as I just read them. Perhaps they have not handled this in the best possible way, but that's far from "anti-FOSS." Just because you didn't get your precious packages today, doesn't mean they've gone all corporate spin-zone on the FOSS community. Again, I'm not saying that they've handled it as well as they could have, I'm just making the point that there might be reasons for not detailing publicly the many many disgusting ways that each and every one of their private bits have been violated and penetrated numerous times, over and over again....
Give-em a break guys, I'd be more concerned if they didn't tell anyone about the break-in at all. That would really be "corporate" behavior. Simply deny the breach and lawyer-up. As it is, they're trying to fix it, and if you're so antsy to get your packages immediately, the source and diff's are there for you to check yourself. If they start getting in the habit of this, folks will start contributing to, and using other distro's.. isn't that how FOSS is supposed to work?
The Jury is Still Out (Score:5, Interesting)
IT managers now know that RH is going to go unresponsive when there's a problem.
The issue isn't even fully known, so you're jumping to conclusions.
For some reason Fedora has to re-key all their repos and, while I think that's done, it's still being mirrored. One would assume a signing key has been lost.
Redhat isn't doing that. They apparently have a signing server, and a user's credentials were apparently lost, and some packages got signed, but not put in the repos. If you run a RedHat machine and get an unsolicited contact to install some new OpenSSH packages - don't.
I think Fedora has the bigger problem at the moment. Let them work through the problem, they know how to do this. When the users are safe (still an ongoing topic of discussion on how to best ensure this) my guess is they'll be releasing more information. I further suspect we'll learn that prior disclosure would have put users at more risk. We'll see.
How can they trust Red Hat again?
Historically the Fedora guys have been trustworthy to the extreme. That's why not everybody is jumping on them right now, despite the distro-partisans who smell blood in the water. Again, we'll re-evaluate our position on that once the dust settles.
Re:gotta say, this is BAD (Score:4, Interesting)
I work for a 500 million dollar a year company, and we're a Redhat shop. We have no intent of switching because the "breach" had ZERO effect on its customers. Even though it had zero effect, they still released scripts to seek out and detect any potential vulnerabilities that were even remotely related to the "breach" (surprise surprise, our 850 RHEL4/5 installs had none). Redhat caught the "breach", made sure the damage was isolated to non production servers, and then informed its customer base and the public. The fact that they're not releasing the explicit details suurrounding the "breach" seems to suggest that they still investigating the source of the "breach" and quite possibly have law enforcement involved.
Redhat is doing the right thing, and for you to base your decision to switch on a grossly misinterpreted reaction reflects poorly on you, not them.
Re:gotta say, this is BAD (Score:4, Interesting)
Bleh. I've worked for multiple Fortune 100 companies, and for the most part, issues such as these do not make the radar of these companies. The most trouble you'll get is out of a few disgruntled users. Once a contract is signed, unless you pissed off the top brass, you typically have no problems.
OTOH, I'll disagree with you. Full disclosure means just that. At this point, they have not even said that they're going to disclose anything else, and it reflects poorly on you to go defend them.
Re:Consider Red Hat's response vs. Debian's (Score:3, Interesting)
Anybody who has been on Slashdot long enough knows that the reason UID numbers are emblazoned right up on the top of each comment was because of Bruce Perens' hissy fit when someone with a slightly misspelled copy of his name came on Slashdot and started masquerading as him (in a fashion to mock him, for the most part).
Slashdot became ever so slightly less egalitarian that day, when 'UID cred' became something touted right up on the header of each comment.
So here's a long belated: Thanks, Bruce.
Re:Consider Red Hat's response vs. Debian's (Score:2, Interesting)
If an overly open disclosure policy is perceived to affect future sales or the value of the brand (i.e. "goodwill"), legal will tell them to say nothing unless they are breaking a bigger law (i.e. gross negligence) by saying nothing.
However, The Red Hat brand is synonymous with openness and trustworthiness - if they say nothing they could be affecting the value of their brand and breaking the law. But I've never studied any of the laws governing shareholder responsibility. Anyone with knowledge of these things care to comment on how these laws could be interpreted in this case?
Re:Semantic games (Score:5, Interesting)
Exactly. It's not a breach of any FOSS licence. It's possibly a breach of FOSS project best practice, but that isn't clear either, because we don't know how the problem happened or what code had to be modified to fix it.
Even if some FOSS code was modified, there is no licence obligation to distribute the changes unless you are distributing the binaries.
As I understand it, the security breach was that someone gained remote access to their servers. It doesn't necessarily follow that any of the code served by the servers was faulty. Last time I checked, not all the code running Redhat sites was open-source.
And the breach could well have been down to a sys admin error, rather than a problem with the codebase itself. It would obviously be acutely embarassing if Redhat's in-house team turned out to have made the kind of mistake that causes people to fail their RHCE exam, but it wouldn't have anything to do with FOSS.
Also, there may not be a simple answer to the 'what does this mean for me?' question. In the Debian case, the answer was quite simple, and so was the solution. The Redhat announcements sounded to me like "We know there was a breach, we don't know exactly what happened as a result, we don't think anything serious happened, but, to be on the safe side, we are changing all the locks."
Redhat's PR department obviously misjudged the best way to handle this incident, but the expectations of the FOSS community also seem unrealistic. When a company open-sources some code, it doesn't mean that anyone in the world gets unfettered access to all the information in the company. Reading TFA, I can't help but think that it is at least partly motivated by the blogger's outrage that Redhat didn't roll out the red carpet all the way to the server room for his terribly important blog.
Which law? Quotes please. (Score:5, Interesting)
I see very often this quoted without any substantiation.
I thought that the responsibility of a company was to stick to whatever they say they will do in their chapters of incorporation, then shareholders sharing that vision would finance the venture.
If the companies' own rules mandate that openness and accountability are part of how the company functions, and shareholders used their judgement and accepted that, profit may take a second seat in the view that in the long term, the business strategy of transparency is deemed to be necessary in turn to make the enterprise profitable.
The problem with many investors is their short-sighted, quarterly short termism and companies that do not ensure ways to handle that in a way that makes sense in a longer term.
Re:Consider Red Hat's response vs. Debian's (Score:3, Interesting)
i agree that it's ridiculous. it is however true.
http://en.wikipedia.org/wiki/Dodge_v._Ford_Motor_Company [wikipedia.org]
Interesting article. However, that case was specifically about whether a company should invest in things, as opposed to paying dividends and maximising short-term income.
To generalise this to "do nothing unless it's profitable" is a gross simplification, and I believe not justifiable. In the Red Hat case, the courses of action are not so simple. They are about the best thing for the reputation of the company and either course could be chosen, depending on the judgement of the executives. I really doubt a court could rule that being more open was damaging the interests of the shareholders.
However, corporate assholes love to cite such cases as they shrug off any non-fiduciary considerations. Perhaps they should remember a later case that established that morality is not waived by being in a chain of command: Nuremberg Defence [wikipedia.org]:
"The fact that a person acted pursuant to order of his Government or of a superior does not relieve him from responsibility under international law, provided a moral choice was in fact possible to him."
Re:gotta say, this is BAD (Score:5, Interesting)
Someone else in the same situation might truthfully report: my vendor is keeping me the dark, I don't know the nature and degree of my own exposure.
That would be me - our RHEL5 system has the trojanized versions of OpenSSH mentioned in the Red Hat Security Advisory installed, and Red Hat did not provide the most crucial information for me: what harm these packages are able to cause (i.e. which passwords should I change, whether to look for secondary breaches on other - non-RHEL - systems, etc.), and how they got into my system. Also, they were pretty slow releasing the details. The packages were signed by their key on August 13, Fedora servers were taken offline a day or two later (so they definitely knew about the problem really soon), but the advisory was published on August 21. As far as I know I had the trojanized packages installed since August 15, so my system has been 0wned for 6 days, thanks to Red Hat delaying the information.
Re:Consider Red Hat's response vs. Debian's (Score:4, Interesting)
The reference to Dodge vs Ford is irrelevant. The claim was about the law. However, the problem in the Ford case was the company charter. The case was lost, because the Ford charter was about making money. The law simply said that Ford has to stick to its own charter. For instance, in Google's case the charter contains the famous "do no evil" bit, but obviously also the money-making part. Therefore, Google shareholders cannot complain if Google for moral reasons turns down a profit opportunity.
Sign. I *usually* like Red Hat (Score:3, Interesting)
I usually like Red Hat, but every once in awhile they do a really abusive something. This is another.
I was a Red Hat customer for years. Then they dropped the professional edition without ANY warning. Fedora didn't show up for over a year (or so it seemed). Well, now I use Debian, and occasionally investigate one of the other distributions. (Ubuntu, Mandrake*, one of the small ones...NOT Novell's offering. I don't trust them.)
I still want to trust Red Hat. I feel that their corporate intentions are honorable...most of the time. OTOH, I'm not about the rely on them again. They aren't trustworthy, merely well intentioned. So I want to trust them, but I know it's a bad idea.
OTOH, CentOS *seems* to have come through this without scars. Their comments indicate that they got cooperation from Red Hat in containing the problem. Perhaps companies can trust Red Hat more than individuals can...perhaps. Or maybe they were just lucky this time.
*I know they're officially Mandriva, but that's for garbage legal reasons. To me they're still Mandrake. (This isn't totally good. They've pulled some boners too.)
I for one agree with Red Hat's approach on this... (Score:2, Interesting)
one. It was pretty evident there was something being done because none of the update servers were available. By knowing this it was just a matter of minutes to realize that something fundamental was wrong. Me knee jerk reaction was to conclude that they had been compromised to some extent and that spawned an special hands-on audit on all of my systems running with any derivative of Red Hat. Less than a day after we get a post telling us that they are working on it... meaning it is deep and they haven't found the rabbit in the hole yet. For me this is enough information to go in "extra alert mode" on all my machines that would be in the realm of the same problems.
remote logins log checking being a mere fraction of the full audit.
What is the diff against Microsoft and similar? Big difference... I had the choice to compile my version of sshd (and other remote offerings) and prep it on the servers that I had that could potentially be effected by a bad transient build. I could do the diff between the updated packages if there was any, on source level. Maybe this is going too far BUT at least it gave me the option to do my stuff pre-emptively while waiting for the final dictum from Red Hat and their investigation. I call this Pro-active guarding.
Most likely... once all major customers had done something similar they were able to disclose a bit more of the problem.
Re:You can rebuild everything from a known base. (Score:2, Interesting)
Re:Press Releases... (Score:3, Interesting)
And if the original compiler was gcc, and trojaned in the way the paper describes, then the triple compilation wouldn't catch it.
But given the significant, massive changes that have been made to gcc over the years (not to mention all the other compilers that have been used to build it), the hack that the paper describes would need to involve hard-AI beyond what we have been able to achieve, and would probably take weeks to complete a single pass of compilation on the typical sort of machine used to compile gcc.
Systems were smaller, simpler, and hadn't been evolving for as many years (or had as many major components rewritten from scratch) back in the days Thompson wrote that paper.
(I built gcc with a C interpreter once, and then used the interpreted gcc to compile gcc. I did it mostly as a stress test of the interpreter, but it also served as a quick check for Thompson's trojan, which I didn't find--not that I was expecting to for the reasons cited above.)
You are correct about the purpose of the triple compilation, though. Trying to catch Thompson's hack that way would be pointless both because it wouldn't detect the hack and because the hack is no longer practical.
Diverse Double-Compiling (Score:2, Interesting)
When can you consider a compiler "clean"?
Countering "Trusting Trust" [schneier.com]
If you have any concerns with that, they should be answered in: David A. Wheeler's Page on Countering Trusting Trust through Diverse Double-Compiling (Trojan Horse attacks on Compilers) [dwheeler.com]
If you find any holes in the theory that were not discussed, then consider writing up your findings for publication.