Slashdot Log In
The Fedora-Red Hat Crisis
Posted by
Soulskill
on Wed Sep 10, 2008 12:02 AM
from the evidently-it-doesn't-start-at-the-top dept.
from the evidently-it-doesn't-start-at-the-top dept.
jammag writes "When Linux journalist Bruce Byfield tried to dig for details about the security breach in Fedora's servers, a Red Hat publicist told him the official statement — written in non-informative corporate-speak — was all he would get. In the wake of Red Hat's tight-lipped handling of the breach, even Fedora's board was unhappy, as Byfield details. He concludes: 'If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient, then what chance do we have that other companies — especially publicly-traded ones — will act any better?'"
Related Stories
[+]
Red Hat, Fedora Servers Compromised 278 comments
An anonymous reader writes "In an email sent to the fedora-announce mailing list, it has been revealed that both Fedora and Red Hat servers have been compromised. As a result Fedora is changing their package signing key. Red Hat has released a security advisory and a script to detect potentially compromised openssh packages."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Consider Red Hat's response vs. Debian's (Score:5, Insightful)
The problem with a lot of corporate Open Source is that they ignore the ethical foundation of Open Source. And eventually we find out that Open Source isn't quite as good without the ethics.
Bruce
Re:Consider Red Hat's response vs. Debian's (Score:5, Interesting)
I pretty much agree: Fedora was obviously squelched by Red Hat corporate who was apparently afraid of the reaction of their paying customers. Despite the token board openings and motions about openness, after this nobody can pretend that Fedora is on anything but a *very* short leash held by Red Hat.
On the one hand, as a user I found myself trusting that Fedora's infrastructure crew were plugging away and probably handling things about as well as could be. On the other hand, the vague statements and lack of hard facts was (and still is) disturbing.
They should have come clean, and allowed the the community to vett their process.
Ob-FUD [just to poking Bruce for fun]: If they do come forth with details, it will be interesting to see if it was an ssh key compromised by the Debian flaw that caused this mess.
Parent
Re:Consider Red Hat's response vs. Debian's (Score:5, Interesting)
I got an email from Starfield a while back offering to re-key my SSL certificates because they had figured out that my original request was using Debian's compromised OpenSSL. I had already rekeyed by then.
Thawte is Debian based. I wonder if they had a problem.
Parent
Re:Consider Red Hat's response vs. Debian's (Score:5, Interesting)
Thawte is Debian based. I wonder if they had a problem.
I checked our Thawte keys/certs against the SSL blacklist released by Debian. I checked several from Thawte, and could not find a potential compromised key/cert.
Also, we are a Red Hat customer. I have to agree, I prefer the way Debian handled their incident, versus the way this Red Hat incident is being handled. After reading the Red Hat Security Announcement the details are so vague, I am still not sure of the scope and reach of this vulnerability.
Parent
Which law? Quotes please. (Score:5, Interesting)
I see very often this quoted without any substantiation.
I thought that the responsibility of a company was to stick to whatever they say they will do in their chapters of incorporation, then shareholders sharing that vision would finance the venture.
If the companies' own rules mandate that openness and accountability are part of how the company functions, and shareholders used their judgement and accepted that, profit may take a second seat in the view that in the long term, the business strategy of transparency is deemed to be necessary in turn to make the enterprise profitable.
The problem with many investors is their short-sighted, quarterly short termism and companies that do not ensure ways to handle that in a way that makes sense in a longer term.
Parent
Re:Consider Red Hat's response vs. Debian's (Score:5, Informative)
That is ridiculous. The law does certainly not say that making money is the only thing that matters.
i agree that it's ridiculous. it is however true.
http://en.wikipedia.org/wiki/Dodge_v._Ford_Motor_Company [wikipedia.org]
Parent
Re:Consider Red Hat's response vs. Debian's (Score:5, Insightful)
Unfortunately, that uncovered something perhaps more serious at the heart of Debian. Stop hacking on stuff downstream that you don't have any real idea about and that will only affect you if it blows up. The SSL thing has been a disaster waiting to happen, and it will probably happen again.
Parent
Re:Consider Red Hat's response vs. Debian's (Score:5, Informative)
The Debian compromise lasted about two hours. The attacker had sniffed a developer password some time before then, but it wasn't until he could get root that he did anything dangerous, and he did stuff that revealed him to the site admins. The main problem was in the kernel, which had the privilege-escalation bug. Red Hat was vulnerable too.
Bruce
Parent
Does this justify the word "crisis?" (Score:5, Insightful)
Does this justify the word "crisis?" I doubt it does. In my opinion "conundrum" would be a better word.
At first read, the heading made me think that Red Hat and Fedora communities were bickering big time, threatening timely releases of software we have [all] come to rely on. Of course this is not the case.
So why the sensational heading?
This is an ongoing investigation (Score:5, Interesting)
This seems to be, from reading the Fedora [redhat.com] and Red Hat [redhat.com] statements, an ongoing investigation. The same way the police don't comment about investigations in progress, Red Hat is keeping mum. Keep in mind, the breach may be very complex and not something that they can confidently say "we understand" without a very detailed analysis.
They announced the issue immediately and took steps. For now, give them the benefit of the doubt that further details will be forthcoming once a proper investigation has been completed.
I don't see this as anti FOSS (Score:5, Interesting)
There are a number of possible scenarios that would recommend against being 100% candid on how far you were breached. If I was violated, I think I'd like to take a moment to do a "self-check" on all of my important bits before I started telling everyone all of the nitty-gritty details. As the article pointed out, people were told that there was a breach, and that they should not update for a few days. How is this "anti-FOSS"?
Perhaps they were on the trail of who did this? Perhaps they were comparing notes with the Ubuntu breach cited in the article, with the goal of finding the M.O? Perhaps, like any police detective, they were keeping certain clues to themselves while they investigated further? If the crimes were found to have similar approaches, keeping quiet might improve the odds of capture?
I use Fedora, and had been using Red Hat before Fedora came along. I don't think this kind of hysterical "anti-FOSS" reaction is really fits the facts as I just read them. Perhaps they have not handled this in the best possible way, but that's far from "anti-FOSS." Just because you didn't get your precious packages today, doesn't mean they've gone all corporate spin-zone on the FOSS community. Again, I'm not saying that they've handled it as well as they could have, I'm just making the point that there might be reasons for not detailing publicly the many many disgusting ways that each and every one of their private bits have been violated and penetrated numerous times, over and over again....
Give-em a break guys, I'd be more concerned if they didn't tell anyone about the break-in at all. That would really be "corporate" behavior. Simply deny the breach and lawyer-up. As it is, they're trying to fix it, and if you're so antsy to get your packages immediately, the source and diff's are there for you to check yourself. If they start getting in the habit of this, folks will start contributing to, and using other distro's.. isn't that how FOSS is supposed to work?
So what exactly is Red Hat hiding? (Score:5, Insightful)
OK, some servers got hacked, the attackers didn't inject rogue packages into the repository servers so no customers/users were affected. Red Hat/Fedora responded by auditing everything and releasing a statement [redhat.com], along with tools [redhat.com] to detect packages with the attackers' signature. Big deal.
Seriously, what else is there to be known about it?
Yeah, say whatever you want, but it's not as if Debian never [debian.org] had [debian.org] its servers compromised in a similar fashion, and never had to perform some PR damage control.
Unlike Debian, Red Hat is a publicly traded company with a whole bunch of customers with signed SLAs. Handling such matters without press trolls all rolling over it spreading FUD and causing unnecessary panic is _not_ an easy task, as can be beautifully shown by TFA.
I respectfully disagree with Bruce Perens. The Debian OpenSSL fiasco was so much more serious, damaging and dangerous to users all over the world, it's not even fair to compare. We're talking about millions of known networks and sessions compromised in Debian over a year and a half period, versus none in Red Hat over a week.
I appreciate how Debian acted _after_ the fact, but was there any other way to handle such a terrible mishap?
This is not about flawed Open Source policies, this is about seriously flawed journalism, where conspiracy theories are used to make a story where there is none.
New Fedora Key (Score:5, Informative)
TFA says:
However, as of September 8, the crisis continues, with Fedora users still unable to get security updates or bug-fixes.
Not true. Go here: https://fedoraproject.org/wiki/Enabling_new_signing_key [fedoraproject.org], follow the instructions and voila... updates available.
How is this "ignoring FOSS"? (Score:5, Informative)
If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient
Sorry, but I must have missed the clause in the GPL that requires full and immediate public disclosure of any security breach on your servers, or a duty to maintain 100% availability.
OTOH I do remember loads of stuff in the GPL about how there was no warranty.
There also seems to be a presumption that this "breach" represents some sort of systemic vulnerability in the Fedora/Red Hat product - TFA and several comments here reference the Debian SSL problem. What about the good old standbys of "inside job", "social engineering", "weak password" or "bugger, I knew I should have password-protected my SSH key"?
What if they're planning to fire someones ass, or even press criminal charges over the incident? That would place serious restrictions on what they could publicly announce.
Re:welcome to the world (Score:4, Insightful)
It's happened numerous times. Consider the Bruce's comment regarding Debian above.
Frankly "a real business situation" sounds a lot like a metaphor for covering your ass at other people's expense.
Parent
Re:welcome to the world (Score:5, Insightful)
Parent
The Jury is Still Out (Score:5, Interesting)
IT managers now know that RH is going to go unresponsive when there's a problem.
The issue isn't even fully known, so you're jumping to conclusions.
For some reason Fedora has to re-key all their repos and, while I think that's done, it's still being mirrored. One would assume a signing key has been lost.
Redhat isn't doing that. They apparently have a signing server, and a user's credentials were apparently lost, and some packages got signed, but not put in the repos. If you run a RedHat machine and get an unsolicited contact to install some new OpenSSH packages - don't.
I think Fedora has the bigger problem at the moment. Let them work through the problem, they know how to do this. When the users are safe (still an ongoing topic of discussion on how to best ensure this) my guess is they'll be releasing more information. I further suspect we'll learn that prior disclosure would have put users at more risk. We'll see.
How can they trust Red Hat again?
Historically the Fedora guys have been trustworthy to the extreme. That's why not everybody is jumping on them right now, despite the distro-partisans who smell blood in the water. Again, we'll re-evaluate our position on that once the dust settles.
Parent
The jury must be very patient, indeed (Score:5, Insightful)
I would have phrased it differently: The issue isn't fully known, thus there's a problem.
There's been quite a lot of time.
Parent
Re:Press Releases... (Score:5, Informative)
This goes back to the whole "trusting trust" concept. You have no way of knowing if the source you've been given reflects the binary you're using, unless you yourself compiled it (and hand-crafted the compiler you're using in assembly, and made the assembly language for your CPU, and made your CPU, but those are a different discussion.)
The point is, Red Hat signs their packages. If their signing mechanism has been compromised, it is quite conceivable that every single Red Hat package is untrustworthy. The dates on the packages are only as trustworthy as the key, so there is no beginning or end time for this: you must throw out all Red Hat packages on your system, because any could be compromised.
Source really gives you very little assurance unless you compile it.
If we want to look at this in contrast to Windows, there's not really any comparison, since we barely even begin to have a grasp of their Byzantine updating system, and couldn't even speculate as to the effects of a similar problem on their side.
Parent
Re:Press Releases... (Score:5, Insightful)
Nice try. The problem with Techies is that they don't get the larger picture. They focus on the blinking red herrings they are so used to and where they believe in.
We are talking about a serious flaw of a security model. True. But consider that most people run operating systems where executables are not signed at all.
There is no indication here at all that anyone externally found out about the problem before. It is basically that you found out that what you did over the last two years was vulnerable to potential attacks. How will it affect the future? Not at all, as the issue gets fixed.
Ah, and right now no one unauthorised actually has the key yet. It is only technically possible to crack it much easier...
Parent
Re:Press Releases... (Score:5, Informative)
But, it doesn't matter - it's all open source, you can look at the lines of code and verify for yourself that they're safe, right?
Wrong. I know this is common wisdom in the open source community, but it really isn't that simple when compilers are involved.
The reason is that the hackers COULD potentially have modified the binary of the compiler used to bootstrap the whole RedHat distribution. You can modify the compiler such that it takes harmless code and compiles backdoors into it. In particular you could modify it so that it always propagates the change when it compiles a version of itself. Since every system bootstraps from an already compiled version of the compiler, a well hidden backdoor could propagate forever, unless people actually analyze the machine code.
Read Ken Thompson's 1984(!) Turing Award lecture for the full nitty gritty details. This should be required reading for everybody in security (and all open source advocates, for that matter):
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.5728&rep=rep1&type=pdf [psu.edu]
(PDF)
Parent
Re:Press Releases... (Score:5, Informative)
Parent
Re:gotta say, this is BAD (Score:5, Insightful)
You're very trusting with all that money. Someone else in the same situation might truthfully report: my vendor is keeping me the dark, I don't know the nature and degree of my own exposure.
This would make me nervous.
Parent
Re:gotta say, this is BAD (Score:5, Interesting)
Someone else in the same situation might truthfully report: my vendor is keeping me the dark, I don't know the nature and degree of my own exposure.
That would be me - our RHEL5 system has the trojanized versions of OpenSSH mentioned in the Red Hat Security Advisory installed, and Red Hat did not provide the most crucial information for me: what harm these packages are able to cause (i.e. which passwords should I change, whether to look for secondary breaches on other - non-RHEL - systems, etc.), and how they got into my system. Also, they were pretty slow releasing the details. The packages were signed by their key on August 13, Fedora servers were taken offline a day or two later (so they definitely knew about the problem really soon), but the advisory was published on August 21. As far as I know I had the trojanized packages installed since August 15, so my system has been 0wned for 6 days, thanks to Red Hat delaying the information.
Parent
Re:Semantic games (Score:5, Interesting)
Exactly. It's not a breach of any FOSS licence. It's possibly a breach of FOSS project best practice, but that isn't clear either, because we don't know how the problem happened or what code had to be modified to fix it.
Even if some FOSS code was modified, there is no licence obligation to distribute the changes unless you are distributing the binaries.
As I understand it, the security breach was that someone gained remote access to their servers. It doesn't necessarily follow that any of the code served by the servers was faulty. Last time I checked, not all the code running Redhat sites was open-source.
And the breach could well have been down to a sys admin error, rather than a problem with the codebase itself. It would obviously be acutely embarassing if Redhat's in-house team turned out to have made the kind of mistake that causes people to fail their RHCE exam, but it wouldn't have anything to do with FOSS.
Also, there may not be a simple answer to the 'what does this mean for me?' question. In the Debian case, the answer was quite simple, and so was the solution. The Redhat announcements sounded to me like "We know there was a breach, we don't know exactly what happened as a result, we don't think anything serious happened, but, to be on the safe side, we are changing all the locks."
Redhat's PR department obviously misjudged the best way to handle this incident, but the expectations of the FOSS community also seem unrealistic. When a company open-sources some code, it doesn't mean that anyone in the world gets unfettered access to all the information in the company. Reading TFA, I can't help but think that it is at least partly motivated by the blogger's outrage that Redhat didn't roll out the red carpet all the way to the server room for his terribly important blog.
Parent