The Fedora-Red Hat Crisis 263
jammag writes "When Linux journalist Bruce Byfield tried to dig for details about the security breach in Fedora's servers, a Red Hat publicist told him the official statement — written in non-informative corporate-speak — was all he would get. In the wake of Red Hat's tight-lipped handling of the breach, even Fedora's board was unhappy, as Byfield details. He concludes: 'If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient, then what chance do we have that other companies — especially publicly-traded ones — will act any better?'"
Affecting me to effect change has a good effect. (Score:1, Informative)
Affected. The word is affected, not effected. Sorry Bruce, but I can't help it -- I'm an Anonymous Coward.
Re:Press Releases... (Score:5, Informative)
This goes back to the whole "trusting trust" concept. You have no way of knowing if the source you've been given reflects the binary you're using, unless you yourself compiled it (and hand-crafted the compiler you're using in assembly, and made the assembly language for your CPU, and made your CPU, but those are a different discussion.)
The point is, Red Hat signs their packages. If their signing mechanism has been compromised, it is quite conceivable that every single Red Hat package is untrustworthy. The dates on the packages are only as trustworthy as the key, so there is no beginning or end time for this: you must throw out all Red Hat packages on your system, because any could be compromised.
Source really gives you very little assurance unless you compile it.
If we want to look at this in contrast to Windows, there's not really any comparison, since we barely even begin to have a grasp of their Byzantine updating system, and couldn't even speculate as to the effects of a similar problem on their side.
Don't be fooled. (Score:2, Informative)
From: GM'S SATURN PROBLEM [cnn.com]
Push come to shove, Fedora's needs will never come before Red Hats interests.
Re:welcome to the world (Score:3, Informative)
Not exactly.
Fixed it for you.
(And I even did it without shredding the evidence, NDA/gag orders, DMCA take down letters or all of the other CYA tactics peculiar to the legal profession.)
Re:Press Releases... (Score:5, Informative)
But, it doesn't matter - it's all open source, you can look at the lines of code and verify for yourself that they're safe, right?
Wrong. I know this is common wisdom in the open source community, but it really isn't that simple when compilers are involved.
The reason is that the hackers COULD potentially have modified the binary of the compiler used to bootstrap the whole RedHat distribution. You can modify the compiler such that it takes harmless code and compiles backdoors into it. In particular you could modify it so that it always propagates the change when it compiles a version of itself. Since every system bootstraps from an already compiled version of the compiler, a well hidden backdoor could propagate forever, unless people actually analyze the machine code.
Read Ken Thompson's 1984(!) Turing Award lecture for the full nitty gritty details. This should be required reading for everybody in security (and all open source advocates, for that matter):
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.5728&rep=rep1&type=pdf [psu.edu]
(PDF)
Re:Consider Red Hat's response vs. Debian's (Score:5, Informative)
The Debian compromise lasted about two hours. The attacker had sniffed a developer password some time before then, but it wasn't until he could get root that he did anything dangerous, and he did stuff that revealed him to the site admins. The main problem was in the kernel, which had the privilege-escalation bug. Red Hat was vulnerable too.
Bruce
Re:Affecting me to effect change has a good effect (Score:2, Informative)
Emerson to them all!
New Fedora Key (Score:5, Informative)
TFA says:
However, as of September 8, the crisis continues, with Fedora users still unable to get security updates or bug-fixes.
Not true. Go here: https://fedoraproject.org/wiki/Enabling_new_signing_key [fedoraproject.org], follow the instructions and voila... updates available.
Re:Affecting me to effect change has a good effect (Score:3, Informative)
"Affect" DOES mean "to have an effect upon". That's not the disputed definition.
Perhaps when you switch them in your defense of your chronic switching of them, that's evidence that you're wronger than the other wrong people. :)
[Meant lightheartedly, I don't honestly care what you type.]
Re:Consider Red Hat's response vs. Debian's (Score:3, Informative)
According to reports, Debian detected one compromise because of a faulty rootkit that, props to the author, but it had many, many flaws. The other compromise was detected by a 'filesystem integrity check' -- if you think that inspires confidence in people then GTFO. Those hackers screwed up... basically Debian only discovered their systems were compromised by dumb luck and simplistic checks.
This is why Debian isn't used by anybody even moderately serious about system security.
Probably it was like the Debian situation. Someone got sloppy and had their password sniffed. Then once on the system a privilege-escalation vulnerability was used.
"Probably" meaning "you hope", because it makes Debian look better by comparison. What you are engaging in is idle speculation, but what's known is that Red Hat are very serious about security.
Re:Affecting me to effect change has a good effect (Score:1, Informative)
Indeed. Though you have attacked the wrong horn of the problem.
Affect = to have an effect on; to influence [something pre-existing]
Effect = to bring about, to implement [until finalized, hence more than an influence]; to cause to come into being [something not pre-existing]
It can't be easier for a non-native English speaker than for a native one, or can it?
Re:The jury must be very patient, indeed (Score:4, Informative)
I would have phrased it differently: The issue isn't fully known, thus there's a problem.
There's been quite a lot of time.
That's true. The issue is can you say confidently that disclosure of the problem wouldn't put users at risk?
That's the only reasonable reason for the delay that I can see. Since these guys are usually quite reasonable I'm making the assumption that's what's going on (or something I've completely missed). It may turn out my trust was misplaced - we should know shortly. Jessee Keating just announced [redhat.com] updates are going out to the mirrors since I last posted.
We do have Feodra's account of the incident (Score:2, Informative)
For some reason Fedora has to re-key all their repos and, while I think that's done, it's still being mirrored. One would assume a signing key has been lost.
Have you already read the Fedora report? Fedora did release a report about the incident [lwn.net]. Within it they say that while an attacker was able to reach a Fedora signing system they do not believe that the key's passphrase was compromised. However it states that as precaution they have decided to create a new key.
The Red Hat side of things is different and far... trickier. I point you towards this LWN article about the intrusion [lwn.net] as I think it's hard to say such simple statements about it.
Re:Consider Red Hat's response vs. Debian's (Score:5, Informative)
That is ridiculous. The law does certainly not say that making money is the only thing that matters.
i agree that it's ridiculous. it is however true.
http://en.wikipedia.org/wiki/Dodge_v._Ford_Motor_Company [wikipedia.org]
Re:Consider Red Hat's response vs. Debian's (Score:3, Informative)
The point is that Byfield raises these issues on purposes, he makes them up.
How is this "ignoring FOSS"? (Score:5, Informative)
If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient
Sorry, but I must have missed the clause in the GPL that requires full and immediate public disclosure of any security breach on your servers, or a duty to maintain 100% availability.
OTOH I do remember loads of stuff in the GPL about how there was no warranty.
There also seems to be a presumption that this "breach" represents some sort of systemic vulnerability in the Fedora/Red Hat product - TFA and several comments here reference the Debian SSL problem. What about the good old standbys of "inside job", "social engineering", "weak password" or "bugger, I knew I should have password-protected my SSH key"?
What if they're planning to fire someones ass, or even press criminal charges over the incident? That would place serious restrictions on what they could publicly announce.
Re:Press Releases... (Score:5, Informative)
Re:Consider Red Hat's response vs. Debian's (Score:1, Informative)
One noticable thing from the Debian breach is that we never knew anything about who did it. That is a failure. Possibly at the time of the incident handling, it was already a fact of life. Possibly it was a less bad failure than an alternative failure, but it's still bad. I'm hoping that RedHat is doing it's best to avoid that failure.
If RedHat still hopes to find the people who did this, then the method those people used to crack systems may be critical evidence in correctly identifying them. A correct description of the method used gives lots of extra credibility to a criminal confession. Assuming RedHat or the FBI are still hoping to catch the people responsible for the breach there's a trade off they (or more likely, the investigating FBI officers; RedHat should have only limited say at this point if things have been done correctly) have to make; where information that might help both customers and the investigation is involved. However, an already known vulnerability, such as password sniffing, isn't something which they should be giving away.
You keep repeating "breach". Let me; as a user of Ubuntu, Debian, CentOS, Fedora and RedHat (fully commercial; large install) just put this from my point of view. I have been told my packages, the ones RedHat officially distributed, were not breached. I've also found that, since I mostly followed RedHat and Fedora's security guidelines, I was reasonably able to verify that this wasn't a thing which could compromise me. If they aren't lying and are competent then, from my point of view, there is no known breach of RedHat Linux ; there was a breach of a server RedHat uses for internal development. There was a breach of the signing process. There was not a breach of the distribution (as was possible in the Debian 2003 breach) nor of the RedHat signing keys. Had this been MicroSoft, we would never have even heard that the problem happened.
The most important thing here is that it wasn't an accident that I wasn't breached. The choice by RedHat to invest in a hardware signing key kept their key safe from the intruder. The choice to use staging and controls between signing and release kept the binaries that were generated from being distributed. Neither Debian (nor for that matter Fedora, CentOS or Ubuntu) have made the choices which kept me from being compromised.
Personally, I think the RedHat comms was pretty bad. I think that there should have been a move clear initial message. When I thought that packages would be compromised I as pretty annoyed. When I found out they had good reason to believe no packages were compromised I managed to completely stop worrying. From my understanding no system following proper RedHat security procedures seems to have been compromised, so now that's clear reason to relax and slow down and wait some time.
Bruce, you have done many good things for many people; I have lots of respect for many things you have done but right now you look like you are getting the boot in against a competitor whilst they are down and not able to answer. Please give them a little space yet. If you still feel the need to kick a bit, then a targetted list of questions that they should be able to answer even if an investigation is ongoing would be a better way of going at this. Run the list past an experienced forensic computer investigator and/or lawyer with appropriate background. Put the list up on your web site and formally ask RedHat to respond.
Re:Affecting me to effect change has a good effect (Score:3, Informative)
I hope they continue to ding you, because you still have it wrong. Here's a rule that works 99% of the time:
So if you find yourself writing "effected" again, all you have to do is recognize that you want a verb and then use "affected" instead.
Fedora updates have just been re-enabled (Score:2, Informative)
https://www.redhat.com/archives/fedora-devel-list/2008-September/msg00842.html [redhat.com]
In due time you can probably expect a more complete picture of what happened. I think the "Fedora/RedHat keeps us in the dark" view is overly alarmist.
Re:Consider Red Hat's response vs. Debian's (Score:3, Informative)
Debian runs the same security tools as any high-security organization. Indeed, they made some of the ones that others are using, and are in general the prototype for process among Linux distributions. They had a cryptographic trust network before other distributions, there was an SELinux version of Debian before Red Hat picked it up, etc.
Re:I don't see this as anti FOSS (Score:3, Informative)
And most importantly, as a public company Redhat likely notified the FBI of the breach and the FBI told them they couldn't reveal any details until the investigation was complete under penalty of being charged with interfering with an investigation. This is SOP for the FBI, they don't want details out there so that if they question the actual person that did the break in and he/she reveals details that aren't public they can use it against him/her in court.
Re:Nah, you didn't understand (Score:3, Informative)
Ah I see.
Still if 5.2 is trojaned then it will take the clean 5.3 source and emit gcc53at. 53at will then emit 53bt. 53bt will then emit 53ct.
53b and 53c should be equal, and the trojan will also be equal, so 53bt == 53ct. You could comsider the trojan as part of an "optimisation pass", which will be performed by all builds of gcc.
A trojan compiler that always compiled printf with an extra check for if("crashme") exit; and always compiled the preprocessor with the printf mangler would always emit:
* A compiler that repeatably compiles printf with the trojan.
* A compiler that repeatably compiles the preprocessor with the trojanned preprocessor.
Re:Consider Red Hat's response vs. Debian's (Score:4, Informative)
Last I checked, an eToken Pro cost less than $50. I have one on my keychain. It works with OpenSSL. I can sign with it, and the private key never leaves the device. This is not rocket science. Lots of folks use smartcards rather than the eToken to do this, the eToken is just smaller and doesn't require a reader.
"Server" is the wrong word here. Debian has non-writable resources that are used to check the writable, on-net resources. These are also available for individual Debian users to create for checking their own systems, using one of the Debian security packages that is available to everyone. This again is not rocket science, it's been really cheap and easy to do since CD writing became available.
Oh yes, for more than 10 years. Debian has lots of donations of system resources, meaning entire systems and network homes for them all over the world, and more money than it needs to do this stuff.
It's more than one network, since Debian has geographical diversity, but yes various Debian systems are behind IDS, and the packages for making your own IDS are part of Debian.
You do know that you're talking about really basic security stuff, right?
Re:Consider Red Hat's response vs. Debian's (Score:3, Informative)