Forgot your password?
typodupeerror
Red Hat Software Businesses Linux Business Security

The Fedora-Red Hat Crisis 263

Posted by Soulskill
from the evidently-it-doesn't-start-at-the-top dept.
jammag writes "When Linux journalist Bruce Byfield tried to dig for details about the security breach in Fedora's servers, a Red Hat publicist told him the official statement — written in non-informative corporate-speak — was all he would get. In the wake of Red Hat's tight-lipped handling of the breach, even Fedora's board was unhappy, as Byfield details. He concludes: 'If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient, then what chance do we have that other companies — especially publicly-traded ones — will act any better?'"
This discussion has been archived. No new comments can be posted.

The Fedora-Red Hat Crisis

Comments Filter:
  • by Bruce Perens (3872) * <bruce@perens.com> on Wednesday September 10, 2008 @12:05AM (#24942425) Homepage Journal
    I liked the way that Debian handled its server breach, and the more recent SSL bug. They realized that their first responsibility was to the users. They knew that not just Debian but all Debian derivatives like Ubuntu would be effected, and that the best way to handle it was to publish the full details and what they were doing to fix them. They came out of both situations looking better than Red Hat has this time. And it's not what Fedora looks like. Red Hat obviously took control, shutting off outside reporting in a way that never would have flown with a real Open Source project rather than a company dominating an Open Source project, and thus Red Hat got the loss of credibility.

    The problem with a lot of corporate Open Source is that they ignore the ethical foundation of Open Source. And eventually we find out that Open Source isn't quite as good without the ethics.

    Bruce

    • by Anonymous Coward on Wednesday September 10, 2008 @12:22AM (#24942597)

      I pretty much agree: Fedora was obviously squelched by Red Hat corporate who was apparently afraid of the reaction of their paying customers. Despite the token board openings and motions about openness, after this nobody can pretend that Fedora is on anything but a *very* short leash held by Red Hat.

      On the one hand, as a user I found myself trusting that Fedora's infrastructure crew were plugging away and probably handling things about as well as could be. On the other hand, the vague statements and lack of hard facts was (and still is) disturbing.

      They should have come clean, and allowed the the community to vett their process.

      Ob-FUD [just to poking Bruce for fun]: If they do come forth with details, it will be interesting to see if it was an ssh key compromised by the Debian flaw that caused this mess.

      • by Bruce Perens (3872) * <bruce@perens.com> on Wednesday September 10, 2008 @12:28AM (#24942659) Homepage Journal

        Ob-FUD [just to poking Bruce for fun]: If they do come forth with details, it will be interesting to see if it was an ssh key compromised by the Debian flaw that caused this mess.

        I got an email from Starfield a while back offering to re-key my SSL certificates because they had figured out that my original request was using Debian's compromised OpenSSL. I had already rekeyed by then.

        Thawte is Debian based. I wonder if they had a problem.

        • Thawte is Debian based. I wonder if they had a problem.

          I checked our Thawte keys/certs against the SSL blacklist released by Debian. I checked several from Thawte, and could not find a potential compromised key/cert.

          Also, we are a Red Hat customer. I have to agree, I prefer the way Debian handled their incident, versus the way this Red Hat incident is being handled. After reading the Red Hat Security Announcement the details are so vague, I am still not sure of the scope and reach of this vulnerability.

          • by nimbius (983462)
            agreed. ive been a fedora/redhat user for over 10 years, and i like the fedora project. The vague disclosure and corporate shoosh time overlord mentality taken by redhat is deplorable.

            this breech is obviously a larger problem, as the update servers have gone silent for the most part (i havent seen new packages in over 3 weeks.)
            after fedora 9 neutering its xen, going production with buggy kde, and losing openGL support in Xorg, my favorite distro has become rather dull and aggravating. now theres a
      • Re: (Score:3, Insightful)

        by wumingzi (67100)

        I pretty much agree: Fedora was obviously squelched by Red Hat corporate who was apparently afraid of the reaction of their paying customers///////////// shareholders. Despite the token board openings and motions about openness, after this nobody can pretend that Fedora is on anything but a *very* short leash held by Red Hat.

        As they say on that snarky message board across town, fixed it for ya.

        As a publicly traded company, Red Hat's primary responsibility is to produce a profit for its shareholders. That i

        • by InlawBiker (1124825) on Wednesday September 10, 2008 @01:37AM (#24943123)
          That is ridiculous. The law does certainly not say that making money is the only thing that matters. Companies private and public have a responsibility to act in an ethical manner. That's what Sarbanes Oxley and ethics officers are for. Besides that it's poor public relations. It would have been in Red Hat's best interest to disclose details. If they had then maybe their credibility wouldn't be called into question.
        • Re: (Score:2, Interesting)

          by Wheat (20250)

          If an overly open disclosure policy is perceived to affect future sales or the value of the brand (i.e. "goodwill"), legal will tell them to say nothing unless they are breaking a bigger law (i.e. gross negligence) by saying nothing.

          However, The Red Hat brand is synonymous with openness and trustworthiness - if they say nothing they could be affecting the value of their brand and breaking the law. But I've never studied any of the laws governing shareholder responsibility. Anyone with knowledge of these things care to comment on how these laws could be interpreted in this case?

        • by jotaeleemeese (303437) on Wednesday September 10, 2008 @03:39AM (#24943709) Homepage Journal

          I see very often this quoted without any substantiation.

          I thought that the responsibility of a company was to stick to whatever they say they will do in their chapters of incorporation, then shareholders sharing that vision would finance the venture.

          If the companies' own rules mandate that openness and accountability are part of how the company functions, and shareholders used their judgement and accepted that, profit may take a second seat in the view that in the long term, the business strategy of transparency is deemed to be necessary in turn to make the enterprise profitable.

          The problem with many investors is their short-sighted, quarterly short termism and companies that do not ensure ways to handle that in a way that makes sense in a longer term.

        • Re: (Score:3, Insightful)

          by Alpha830RulZ (939527)

          Technically speaking, there isn't any law that says they have to maximize profits, it's a fiduciary responsibility, for which they could incur civil liability, and/or lose their jobs. They wouldn't be breaking any law to take action in favor of the users/customers/third parties, but the Board of Directors might choose to end their employment for doing so. Or not.

          Top level management has a lot of freedom in acting in the interests of the company. The main control is the Board of Directors removing them fr

    • Did you consider that Red Hat may not have legally been able to give much more information? It probably took serious effort to compromise their system, more than some random hacker.

      You're touting Debian, but what I wonder is would they even know they were hacked? Last time their servers were compromised wasn't it like several months before they even discovered it? Seriously, what good is 'oss reporting' of the problem if it has gone undetected for months? IIRC there was at least one ~2003 and one ~2006,

      • by Bruce Perens (3872) * <bruce@perens.com> on Wednesday September 10, 2008 @01:18AM (#24943023) Homepage Journal
        Red Hat has an accepted path to make vulnerability information available, through CERT. There are no super crackers or super vulnerabilities that you can't talk about. Probably it was like the Debian situation. Someone got sloppy and had their password sniffed. Then once on the system a privilege-escalation vulnerability was used.

        The Debian compromise lasted about two hours. The attacker had sniffed a developer password some time before then, but it wasn't until he could get root that he did anything dangerous, and he did stuff that revealed him to the site admins. The main problem was in the kernel, which had the privilege-escalation bug. Red Hat was vulnerable too.

        Bruce

        • Re: (Score:3, Informative)

          by cryptoluddite (658517)

          According to reports, Debian detected one compromise because of a faulty rootkit that, props to the author, but it had many, many flaws. The other compromise was detected by a 'filesystem integrity check' -- if you think that inspires confidence in people then GTFO. Those hackers screwed up... basically Debian only discovered their systems were compromised by dumb luck and simplistic checks.

          This is why Debian isn't used by anybody even moderately serious about system security.

          Probably it was like the Debian situation. Someone got sloppy and had their password sniffed. Then once on the system a privilege-escalation vulnerability was used.

          "Probably" meaning "you hope"

          • by gbjbaanb (229885)

            only discovered their systems were compromised by dumb luck and simplistic checks.

            yep, that's the way the world works. The alternative is total monitoring of everyone by erm.. robotic overloads.

            It reminds me of the old stories about terrorist atrocities prevented by the traffic cop who stopped the bad guys because their tail light was out.

            So Debian stopped the bads guys by a simple daily report from AIDE that checked /usr. Good. Job done. I'm sure RedHat have exactly the same security processes in place.

          • Re: (Score:3, Informative)

            by Bruce Perens (3872) *
            The other compromise was detected by a 'filesystem integrity check'

            Debian runs the same security tools as any high-security organization. Indeed, they made some of the ones that others are using, and are in general the prototype for process among Linux distributions. They had a cryptographic trust network before other distributions, there was an SELinux version of Debian before Red Hat picked it up, etc.

    • by Elektroschock (659467) on Wednesday September 10, 2008 @01:03AM (#24942917)

      Bruse Byfield is a troll. So why debate his accusations?

      Yes, there are many problems: patents [stopsoftwarepatents.org], open standards, dmca restrictions and so forth. But open source is still the best of all worlds.

      RedHat as a company applies the usual tactics but as a community member gives a lot. Sure corporations are vulnerable to money. Novell is a good example...

    • by rtfa-troll (1340807) on Wednesday September 10, 2008 @01:15AM (#24943007)
      Reading between the lines, it seems there's an ongoing investigation into the incident and they aren't allowed to communicate. I'll wait until I know much more about this before I make my final decision on how RedHat behaved.
    • by segedunum (883035) on Wednesday September 10, 2008 @04:12AM (#24943853)

      I liked the way that Debian handled its server breach, and the more recent SSL bug.

      Unfortunately, that uncovered something perhaps more serious at the heart of Debian. Stop hacking on stuff downstream that you don't have any real idea about and that will only affect you if it blows up. The SSL thing has been a disaster waiting to happen, and it will probably happen again.

  • A "Linux journalist" talking to a "publicist" was told to read the press release?

    I, too, without RTFA, would think most any company would be wary about talking about a recent server breach.

    But, it doesn't matter - it's all open source, you can look at the lines of code and verify for yourself that they're safe, right? Not like what you can('t) do with Windows.

    • There are still ways to handle this which cover both the need to minimize chances of a recurrence and the desire of users to know what happened and whether they are also at risk. This could include specifying whether this was due to a software bug still under investigation, a configuration error which has been fixed, or possibly an internal sabotage. Exact details could remain forthcoming until such time as complete mitigating solutions are in place, especially if a patch needs to be released to handle it

      • Re: (Score:3, Insightful)

        Sorry but

        Company has a problem with a server breach - no publicity, no comment - note even a hint

        FOSS project - We're busy go away ...

        Fedora - We have a problem we're sorting it out, we'll let you know when we know, the Server is Red Hat's and they have the same problem so they are dealing with it ....

        Looks fair enough to me ....

    • Re:Press Releases... (Score:5, Informative)

      by FlyingBishop (1293238) on Wednesday September 10, 2008 @12:41AM (#24942765)
      No, you can't...

      This goes back to the whole "trusting trust" concept. You have no way of knowing if the source you've been given reflects the binary you're using, unless you yourself compiled it (and hand-crafted the compiler you're using in assembly, and made the assembly language for your CPU, and made your CPU, but those are a different discussion.)

      The point is, Red Hat signs their packages. If their signing mechanism has been compromised, it is quite conceivable that every single Red Hat package is untrustworthy. The dates on the packages are only as trustworthy as the key, so there is no beginning or end time for this: you must throw out all Red Hat packages on your system, because any could be compromised.

      Source really gives you very little assurance unless you compile it.

      If we want to look at this in contrast to Windows, there's not really any comparison, since we barely even begin to have a grasp of their Byzantine updating system, and couldn't even speculate as to the effects of a similar problem on their side.

      • by Elektroschock (659467) on Wednesday September 10, 2008 @01:12AM (#24942977)

        Yeah, but that is the techie paranoia.

        Just because something can be done doesn't mean it actually happens. If I go to holidays and leave the door of my house open, it does not mean that something actually happens.

        The point is, Red Hat signs their packages. If their signing mechanism has been compromised, it is quite conceivable that every single Red Hat package is untrustworthy.... you must throw out all Red Hat packages on your system, because any could be compromised.

        Nonsense. Why should you "trust" RedHat Packages signed by employees?

        The whole signing shit is a troll for the privacy church. What they forget are the proportions and what is really important. We know exactly that the problem didn't affect us in the past and it won't affect us in the future now we found out. No need to panic.

      • by cgenman (325138)

        If we want to look at this in contrast to Windows, there's not really any comparison, since we barely even begin to have a grasp of their Byzantine updating system, and couldn't even speculate as to the effects of a similar problem on their side.

        Considering that signed executeables on windows has been a no-go for years, I think you're seeing the effects.

      • If you want to understand just how scary a break-in like this is, read Ken Thompson's classic Turing Award Lecture, Reflections on Trusting Trust [bell-labs.com].
    • Re:Press Releases... (Score:5, Informative)

      by eggnoglatte (1047660) on Wednesday September 10, 2008 @01:06AM (#24942937)

      But, it doesn't matter - it's all open source, you can look at the lines of code and verify for yourself that they're safe, right?

      Wrong. I know this is common wisdom in the open source community, but it really isn't that simple when compilers are involved.

      The reason is that the hackers COULD potentially have modified the binary of the compiler used to bootstrap the whole RedHat distribution. You can modify the compiler such that it takes harmless code and compiles backdoors into it. In particular you could modify it so that it always propagates the change when it compiles a version of itself. Since every system bootstraps from an already compiled version of the compiler, a well hidden backdoor could propagate forever, unless people actually analyze the machine code.

      Read Ken Thompson's 1984(!) Turing Award lecture for the full nitty gritty details. This should be required reading for everybody in security (and all open source advocates, for that matter):

      http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.5728&rep=rep1&type=pdf [psu.edu]
      (PDF)

      • At some point, you should have a compiler that is consider clean. You use that to compile, from reviewed source code, the latest and greatest compiler and generate the rest from there.

        • Re: (Score:3, Insightful)

          by eggnoglatte (1047660)

          Well, gee. Thanks for explaining the meaning of "bootstrapping" to me.

          The problem is: when can you consider a compiler "clean"? The only way to be sure is to develop it yourself in machine language (no, you can't even use an assembler, because it could generate a backdoor, too), or to fully scrutinize the machine language of an existing compiler binary.

          In practice, if you are using gcc, you have a compiler that has been recompiled by itself over and over again for at least a decade. Can you be absolutely su

      • Re:Press Releases... (Score:5, Informative)

        by TheRaven64 (641858) on Wednesday September 10, 2008 @07:09AM (#24944593) Journal
        This is why the GCC build process builds the compiler three times. First it builds it with the existing compiler. Next it builds it with the new version. Finally, it builds it with the version built with itself and compares the binaries. If the last two are different, then the old compiler is likely to have been trojaned.
        • Anyone who mentions Ken Thompson's Reflections on Trusting Trust [bell-labs.com]should also mention David A. Wheeler's "Countering Trusting Trust" [dwheeler.com]. Those who don't should be punished by having to argue both sides of the debate.

          I occasionally post the counter argument [slashdot.org] in a reply but no one sees it... Next time you see someone else with this behaviour tr, here's ammo for countering it.

          (I believe the gcc rebuilds aren't so much to remove this type of intentional bugging but rather ensure the final binary is free from things l

        • by againjj (1132651) on Wednesday September 10, 2008 @03:36PM (#24951113)

          And if the original compiler was gcc, and trojaned in the way the paper describes, then the triple compilation wouldn't catch it. Why? Step 1: the existing compiler builds binary 1 and inserts the backdoor. Step 2: Binary 1 builds binary 2 and inserts the backdoor. Step 3: Binary 2 builds binary 3 and inserts the backdoor. Step 4: binary 2 and binary 3 are compared, and if they are different, then there is an error. However, since all versions have the backdoor, there is no difference, and no error will be flagged. Try reading the linked article again.

          The triple compilation is not for detecting trojans, but "because the compiler will be tested more completely and could also have better performance." [gnu.org]

          • Re: (Score:3, Interesting)

            by Xtifr (1323)

            And if the original compiler was gcc, and trojaned in the way the paper describes, then the triple compilation wouldn't catch it.

            But given the significant, massive changes that have been made to gcc over the years (not to mention all the other compilers that have been used to build it), the hack that the paper describes would need to involve hard-AI beyond what we have been able to achieve, and would probably take weeks to complete a single pass of compilation on the typical sort of machine used to compile gcc.

            Systems were smaller, simpler, and hadn't been evolving for as many years (or had as many major components rewritten from scr

    • Well yes. (Score:2, Insightful)

      by jotaeleemeese (303437)

      But they already know what happened.

      You would expect they disclose what went wrong, that would save time and money to everybody.

      Now, how can anybody running a Red Hat system know it is safe?

      Openness is an advantage over closed systems, and it is why many of us buy from companies that are more open, in all the senses of the world.

      Losing sight of what makes them different, and thus desirable, is a recipe for financial trouble (their lawyers will be paid in any way, so they should actually use them to ensure m

  • by bogaboga (793279) on Wednesday September 10, 2008 @12:36AM (#24942719)

    Does this justify the word "crisis?" I doubt it does. In my opinion "conundrum" would be a better word.

    At first read, the heading made me think that Red Hat and Fedora communities were bickering big time, threatening timely releases of software we have [all] come to rely on. Of course this is not the case.

    So why the sensational heading?

    • by mikesd81 (518581)
      Because it is a crisis when a distro's server was penetrated and packages are possibly untrustworthy. Maybe the title would be less confusing if it was Redhat/Fedora?
    • "Crisis" is polite language for what is really meant :-)
  • Don't be fooled. (Score:2, Informative)

    by fahrbot-bot (874524)
    Fedora is independent from Red Hat as Saturn is (was) from GM. Ya, it's a little bit of a stretch, but stick with me, I think there's a parallel -- i.e., Saturn exists to benefit GM, not to build better cars.

    From: GM'S SATURN PROBLEM [cnn.com]

    Saturn wouldn't merely blossom as a division with protected status, free from the labor strife, stifling bureaucracy, and all the other dysfunctions of the mother corporation. No, it would also infect the rest of the company with its enlightened and effective management tech

  • 'If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient, then what chance do we have that other companies â" especially publicly-traded ones â" will act any better?'

    You can argue that they are ignoring the spirit that many FOSS advocates believe in; but how is not making the details of this intrusion public "ignoring FOSS"? Is there a line somewhere in the GPL that states "If you're running GPL software, and someone hacks your system, you must make all details of the hack known"?

    If solving the hack required Red Hat to modify code, they'll have to make the source available - but AFAIK that's all they're required to do.

    • by pavera (320634)

      The problem is that they are eating their own dogfood. If they were to disclose the method of attack, they would be protecting all of their customers, as their customers could take steps to prevent the same attack from succeeding against them. As it is, they are just leaving all of their customers open to a known hole.

      Further, the fact that they have been so mum about the attack leaves people with 0 ability to mitigate any damage that may have been done. They admit that the attacker was able to sign modi

    • by bursch-X (458146) on Wednesday September 10, 2008 @01:23AM (#24943049)

      "If you're running GPL software, and someone hacks your system, you must make all details of the hack known"?

      Sure you must, under the GPL even a hack would count as a derivative work, so the hackers have to make the source available, wouldn't they? ;-P

    • Re:Semantic games (Score:5, Interesting)

      by melonman (608440) on Wednesday September 10, 2008 @02:34AM (#24943441) Journal

      Exactly. It's not a breach of any FOSS licence. It's possibly a breach of FOSS project best practice, but that isn't clear either, because we don't know how the problem happened or what code had to be modified to fix it.

      Even if some FOSS code was modified, there is no licence obligation to distribute the changes unless you are distributing the binaries.

      As I understand it, the security breach was that someone gained remote access to their servers. It doesn't necessarily follow that any of the code served by the servers was faulty. Last time I checked, not all the code running Redhat sites was open-source.

      And the breach could well have been down to a sys admin error, rather than a problem with the codebase itself. It would obviously be acutely embarassing if Redhat's in-house team turned out to have made the kind of mistake that causes people to fail their RHCE exam, but it wouldn't have anything to do with FOSS.

      Also, there may not be a simple answer to the 'what does this mean for me?' question. In the Debian case, the answer was quite simple, and so was the solution. The Redhat announcements sounded to me like "We know there was a breach, we don't know exactly what happened as a result, we don't think anything serious happened, but, to be on the safe side, we are changing all the locks."

      Redhat's PR department obviously misjudged the best way to handle this incident, but the expectations of the FOSS community also seem unrealistic. When a company open-sources some code, it doesn't mean that anyone in the world gets unfettered access to all the information in the company. Reading TFA, I can't help but think that it is at least partly motivated by the blogger's outrage that Redhat didn't roll out the red carpet all the way to the server room for his terribly important blog.

  • by pavera (320634) on Wednesday September 10, 2008 @12:50AM (#24942831) Homepage Journal

    I used to be 100% redhat and fedora... Now I've moved almost all my systems to ubuntu, but I still run centos on a few servers.

    Every reputable tech company I deal with (ISP, Software, Hosting, Colo) has very clear, very open policies about outages, breaches, and security in general. If they don't I don't do business with them.

    I know the ins and outs of my ISP, Hosting, and Colo companies processes because I get emailed whenever I have an outage that says "we experienced an outage from x-y on day z, the outage was caused by our dumb admin who tripped on the power cable, we rewired our entire data center to move all of the power cables to the ceiling to prevent a similar outage in the future".

    Obviously that is a made up report, but it is extremely standard practice to let all your customers know a) when the problem happened, b) what caused the problem, c) concrete steps taken or procedures implemented to prevent similar problems in the future

    That RedHat has fallen so miserably short of this basic tenet of IT procedures is extremely scary.

    • by Jailbrekr (73837) <jailbrekr@digitaladdiction.net> on Wednesday September 10, 2008 @01:13AM (#24942981) Homepage

      I work for a 500 million dollar a year company, and we're a Redhat shop. We have no intent of switching because the "breach" had ZERO effect on its customers. Even though it had zero effect, they still released scripts to seek out and detect any potential vulnerabilities that were even remotely related to the "breach" (surprise surprise, our 850 RHEL4/5 installs had none). Redhat caught the "breach", made sure the damage was isolated to non production servers, and then informed its customer base and the public. The fact that they're not releasing the explicit details suurrounding the "breach" seems to suggest that they still investigating the source of the "breach" and quite possibly have law enforcement involved.

      Redhat is doing the right thing, and for you to base your decision to switch on a grossly misinterpreted reaction reflects poorly on you, not them.

      • by pavera (320634)

        Sorry,
        I didn't mean to imply that I switched because of this, although upon re-reading my original post, I agree it reads that way....

        I started switching after Fedora 3, which was such a colossal flop, and caused me way too many headaches... I moved most systems to centos at that time, then, since Ubuntu has released their server product, I've been using it and migrating systems from centos...

        I unfortunately don't work for a 500 mil/yr company, and so can't afford 1500-2500/yr/server for security updates.

      • by the_B0fh (208483) on Wednesday September 10, 2008 @01:35AM (#24943109) Homepage

        Bleh. I've worked for multiple Fortune 100 companies, and for the most part, issues such as these do not make the radar of these companies. The most trouble you'll get is out of a few disgruntled users. Once a contract is signed, unless you pissed off the top brass, you typically have no problems.

        OTOH, I'll disagree with you. Full disclosure means just that. At this point, they have not even said that they're going to disclose anything else, and it reflects poorly on you to go defend them.

      • by Bruce Perens (3872) * <bruce@perens.com> on Wednesday September 10, 2008 @01:40AM (#24943131) Homepage Journal

        surprise surprise, our 850 RHEL4/5 installs had none

        You're very trusting with all that money. Someone else in the same situation might truthfully report: my vendor is keeping me the dark, I don't know the nature and degree of my own exposure.

        This would make me nervous.

        • by Anonymous Coward on Wednesday September 10, 2008 @05:13AM (#24944115)

          Someone else in the same situation might truthfully report: my vendor is keeping me the dark, I don't know the nature and degree of my own exposure.

          That would be me - our RHEL5 system has the trojanized versions of OpenSSH mentioned in the Red Hat Security Advisory installed, and Red Hat did not provide the most crucial information for me: what harm these packages are able to cause (i.e. which passwords should I change, whether to look for secondary breaches on other - non-RHEL - systems, etc.), and how they got into my system. Also, they were pretty slow releasing the details. The packages were signed by their key on August 13, Fedora servers were taken offline a day or two later (so they definitely knew about the problem really soon), but the advisory was published on August 21. As far as I know I had the trojanized packages installed since August 15, so my system has been 0wned for 6 days, thanks to Red Hat delaying the information.

      • The fact that they're not releasing the explicit details suurrounding the "breach" seems to suggest that they still investigating the source of the "breach" and quite possibly have law enforcement involved.

        If that were the case, then they would have no excuse not to tell you that was the case and that full details would be forthcoming once it was no longer necessary to keep them secret.

        Since they did not do that, it seems to suggest that your hypothesis is false.

      • by QuantumG (50515) *

        Why are you putting quotes around "breach". It was a breach.. they said it was a breach. Are you debating Red Hat's press release that it was a breach?

    • I know the ins and outs of my ISP, Hosting, and Colo companies processes because I get emailed whenever I have an outage that says "we experienced an outage from x-y on day z, the outage was caused by our dumb admin who tripped on the power cable, we rewired our entire data center to move all of the power cables to the ceiling to prevent a similar outage in the future".

      Those are events with short-term fixes and you get notification after the fix is implemented, right?

      • by pavera (320634)

        not necessarily... My colo recently had a major power outage which because of the nature of the outage (a brownout, then a complete drop) fooled the generator into turning on while the power was still on (during the brownout), then because the power was still on the generator shut itself down, and put itself in a state which required manual intervention to start up.. so when the power actually failed about 2 minutes later, the whole datacenter went down (after the 2-5 minutes of battery ran out)...

        Anyway it

  • by chill (34294) on Wednesday September 10, 2008 @12:51AM (#24942847) Journal

    This seems to be, from reading the Fedora [redhat.com] and Red Hat [redhat.com] statements, an ongoing investigation. The same way the police don't comment about investigations in progress, Red Hat is keeping mum. Keep in mind, the breach may be very complex and not something that they can confidently say "we understand" without a very detailed analysis.

    They announced the issue immediately and took steps. For now, give them the benefit of the doubt that further details will be forthcoming once a proper investigation has been completed.

  • I'm not at all surprised that Redhat felt free to do whatever they felt like, fedora be damned, under the circumstances. What I don't understand, though, is why would doing what they did seem like a good idea?

    Sure, getting compromised sucks, and having to admit it sucks; but in a world of fast moving internet gossip, paranoid *nixheads, and potential leaks, oozing some smarmy nonsense, losing face, and still having to admit it sucks even more.

    I can understand why they would be tempted, if they thought t
  • I stopped using RedHat (deadrat), about the time they went "fedora" - i did not care for some of the changes.

    I do know that the govt. likes them a lot, and if you are a govt. contractor, sometimes you can only say certain things...

  • by Dr_Marvin_Monroe (550052) on Wednesday September 10, 2008 @01:07AM (#24942941)

    There are a number of possible scenarios that would recommend against being 100% candid on how far you were breached. If I was violated, I think I'd like to take a moment to do a "self-check" on all of my important bits before I started telling everyone all of the nitty-gritty details. As the article pointed out, people were told that there was a breach, and that they should not update for a few days. How is this "anti-FOSS"?

    Perhaps they were on the trail of who did this? Perhaps they were comparing notes with the Ubuntu breach cited in the article, with the goal of finding the M.O? Perhaps, like any police detective, they were keeping certain clues to themselves while they investigated further? If the crimes were found to have similar approaches, keeping quiet might improve the odds of capture?

    I use Fedora, and had been using Red Hat before Fedora came along. I don't think this kind of hysterical "anti-FOSS" reaction is really fits the facts as I just read them. Perhaps they have not handled this in the best possible way, but that's far from "anti-FOSS." Just because you didn't get your precious packages today, doesn't mean they've gone all corporate spin-zone on the FOSS community. Again, I'm not saying that they've handled it as well as they could have, I'm just making the point that there might be reasons for not detailing publicly the many many disgusting ways that each and every one of their private bits have been violated and penetrated numerous times, over and over again....

    Give-em a break guys, I'd be more concerned if they didn't tell anyone about the break-in at all. That would really be "corporate" behavior. Simply deny the breach and lawyer-up. As it is, they're trying to fix it, and if you're so antsy to get your packages immediately, the source and diff's are there for you to check yourself. If they start getting in the habit of this, folks will start contributing to, and using other distro's.. isn't that how FOSS is supposed to work?

    • by Slashcrap (869349)

      If I was violated, I think I'd like to take a moment to do a "self-check" on all of my important bits before I started telling everyone all of the nitty-gritty details.

      You're a Slashdot poster. You're unlikely to be getting violated anytime soon, no matter how hard you wish for it.

    • Re: (Score:3, Informative)

      by rahvin112 (446269)

      And most importantly, as a public company Redhat likely notified the FBI of the breach and the FBI told them they couldn't reveal any details until the investigation was complete under penalty of being charged with interfering with an investigation. This is SOP for the FBI, they don't want details out there so that if they question the actual person that did the break in and he/she reveals details that aren't public they can use it against him/her in court.

  • by Rolman (120909) on Wednesday September 10, 2008 @01:40AM (#24943129)

    OK, some servers got hacked, the attackers didn't inject rogue packages into the repository servers so no customers/users were affected. Red Hat/Fedora responded by auditing everything and releasing a statement [redhat.com], along with tools [redhat.com] to detect packages with the attackers' signature. Big deal.

    Seriously, what else is there to be known about it?

    Yeah, say whatever you want, but it's not as if Debian never [debian.org] had [debian.org] its servers compromised in a similar fashion, and never had to perform some PR damage control.

    Unlike Debian, Red Hat is a publicly traded company with a whole bunch of customers with signed SLAs. Handling such matters without press trolls all rolling over it spreading FUD and causing unnecessary panic is _not_ an easy task, as can be beautifully shown by TFA.

    I respectfully disagree with Bruce Perens. The Debian OpenSSL fiasco was so much more serious, damaging and dangerous to users all over the world, it's not even fair to compare. We're talking about millions of known networks and sessions compromised in Debian over a year and a half period, versus none in Red Hat over a week.

    I appreciate how Debian acted _after_ the fact, but was there any other way to handle such a terrible mishap?

    This is not about flawed Open Source policies, this is about seriously flawed journalism, where conspiracy theories are used to make a story where there is none.

    • by Intrinsic (74189)

      Yea I tended to agree with this. I RTFA and it just sounds like he wants to look like he his right about this whole issue. I think he is going over the top with his ranting and raving. Its reads like someone personally attacked him and now he is fighting back.

  • New Fedora Key (Score:5, Informative)

    by FrankDrebin (238464) on Wednesday September 10, 2008 @01:51AM (#24943203) Homepage

    TFA says:

    However, as of September 8, the crisis continues, with Fedora users still unable to get security updates or bug-fixes.

    Not true. Go here: https://fedoraproject.org/wiki/Enabling_new_signing_key [fedoraproject.org], follow the instructions and voila... updates available.

  • It ain't over yet (Score:3, Insightful)

    by pembo13 (770295) on Wednesday September 10, 2008 @02:34AM (#24943443) Homepage
    You can't really say they are keeping things quiet while things are still in progress. This isn't being swept under the rug, this seems to be pursued in all areas currently. If after everything, there is still no more information, then that is a story.
  • by itsdapead (734413) on Wednesday September 10, 2008 @06:43AM (#24944505)

    If Red Hat, one of the epitomes of a successful FOSS-based business, can ignore FOSS when to do so is corporately convenient

    Sorry, but I must have missed the clause in the GPL that requires full and immediate public disclosure of any security breach on your servers, or a duty to maintain 100% availability.

    OTOH I do remember loads of stuff in the GPL about how there was no warranty.

    There also seems to be a presumption that this "breach" represents some sort of systemic vulnerability in the Fedora/Red Hat product - TFA and several comments here reference the Debian SSL problem. What about the good old standbys of "inside job", "social engineering", "weak password" or "bugger, I knew I should have password-protected my SSH key"?

    What if they're planning to fire someones ass, or even press criminal charges over the incident? That would place serious restrictions on what they could publicly announce.

  • I've noticed that all updates for Fedora 9 have stopped since this happened. Have they released any information about when they will start again?

  • by MrMickS (568778) on Wednesday September 10, 2008 @08:10AM (#24944881) Homepage Journal

    I initially read the article title as The Fedora Hat Crisis. I was wondering if there was a hat shortage brought about by FOSS people wanting to Cosplay. I now realise my error but have this mental imagine of people walking into a Linux conference all bedecked in red Fedoras.

  • by HiThere (15173) <charleshixsn@ear ... t ['hli' in gap]> on Wednesday September 10, 2008 @01:50PM (#24949613)

    I usually like Red Hat, but every once in awhile they do a really abusive something. This is another.

    I was a Red Hat customer for years. Then they dropped the professional edition without ANY warning. Fedora didn't show up for over a year (or so it seemed). Well, now I use Debian, and occasionally investigate one of the other distributions. (Ubuntu, Mandrake*, one of the small ones...NOT Novell's offering. I don't trust them.)

    I still want to trust Red Hat. I feel that their corporate intentions are honorable...most of the time. OTOH, I'm not about the rely on them again. They aren't trustworthy, merely well intentioned. So I want to trust them, but I know it's a bad idea.

    OTOH, CentOS *seems* to have come through this without scars. Their comments indicate that they got cooperation from Red Hat in containing the problem. Perhaps companies can trust Red Hat more than individuals can...perhaps. Or maybe they were just lucky this time.

    *I know they're officially Mandriva, but that's for garbage legal reasons. To me they're still Mandrake. (This isn't totally good. They've pulled some boners too.)

  • by buysse (5473) on Wednesday September 10, 2008 @02:17PM (#24950033) Homepage

    Debian's a non-profit. Comparing the two isn't useful, for a couple of simple reasons. If a Debian build server is owned, what's the financial damage, and to who? How about Redhat?

    It's a lot easier for RH to show direct and indirect financial damage due to a breach, which brings in the FBI. Once the FBI is involved, your whole reply is a "No comment." It's an ongoing federal investigation. If somebody found the trojaned openssh on a DoD server, you can bet that the NSA is probably involved as well.

    Once the feds are involved, their hands are tied. If I'm right, it took a lot of work and negotiation by the lawyers to release as much info as they did.

I cannot conceive that anybody will require multiplications at the rate of 40,000 or even 4,000 per hour ... -- F. H. Wales (1936)

Working...