Forgot your password?
typodupeerror
Red Hat Software Businesses Security

Red Hat, Fedora Servers Compromised 278

Posted by kdawson
from the quick-action dept.
An anonymous reader writes "In an email sent to the fedora-announce mailing list, it has been revealed that both Fedora and Red Hat servers have been compromised. As a result Fedora is changing their package signing key. Red Hat has released a security advisory and a script to detect potentially compromised openssh packages."
This discussion has been archived. No new comments can be posted.

Red Hat, Fedora Servers Compromised

Comments Filter:
  • by Art Popp (29075) * on Friday August 22, 2008 @10:05AM (#24704945)

    These are the guys, to the annoyance of nearly everyone, who turned on SELinux on Fedora Core by default.

    These are the guys who noticed they annoyed everyone, and turned on targeted-mode by default.

    Coming from someone with many systems, completely exposed to the Internet, with thousand day uptimes, these RedHat folk are in fact sufficiently paranoid.

    They have taken all the reasonable precautions, and if their passphrase was strong, then the danger of my servers being compromised by meteor strike is a much greater worry.

    • by illumin8 (148082) on Friday August 22, 2008 @10:14AM (#24705065) Journal

      They have taken all the reasonable precautions, and if their passphrase was strong, then the danger of my servers being compromised by meteor strike is a much greater worry.

      The only thing that concerns me is this: In the Fedora announcement, they said with a high level of confidence, they don't believe the passphrase for their signing key was compromised, because they hadn't signed any packages during the period of time the box was compromised. They are going to change the signing key anyway just in case. This is a good thing.

      In the Redhat announcement, we can infer the passphrase and signing key were compromised, because the attacker signed invalid openssh packages. Even though the official RHN distribution channel was not compromised, the attacker most likely still has their private key and passphrase and can continue signing packages and attempting to distribute them. Redhat needs to step up and reissue a new signing key. There was no announcement of this.

      • Re: (Score:2, Interesting)

        by quitte (1098453)

        Or we could infer that the system was used for its purpose by the attacker. He signed those packages. Redhat looked at the logs, no other packages were signed. So the passphrase is still very likely to be save.

        • by illumin8 (148082) on Friday August 22, 2008 @11:01AM (#24705881) Journal

          Or we could infer that the system was used for its purpose by the attacker. He signed those packages. Redhat looked at the logs, no other packages were signed. So the passphrase is still very likely to be save.

          God, I seriously hope they don't have the passphrase saved so that you don't need to type it in to sign a package. If that is the case their security is very lax. Also, if it's saved, it almost certainly is compromised, because it's stored on disk somewhere. It would be trivial for the attacker to pull it out of whatever script or text file it's saved in.

          • by SETIGuy (33768) on Friday August 22, 2008 @02:18PM (#24709409) Homepage

            God, I seriously hope they don't have the passphrase saved so that you don't need to type it in to sign a package. If that is the case their security is very lax.

            I don't know about anyone else, but I am surprised that their package signing machine is connected via a network to other machines.

            Our code signing machine is locked in a cage and powered up only for purposes of code signing. Executables to be signed are written to a previously wiped USB drive which is attached to the signing machine only when packages are to be signed. The signing machine has not been connected to a network since before the keys were generated. The private key only exists on that machine and in a single separately encrypted backup.

            I've always considered that to be a minimally paranoid means of keeping private keys private. Really paranoid would be "signed on one machine, checked and signed again on another machine."

            • by darkpixel2k (623900) <aaron@heyaaron.com> on Friday August 22, 2008 @05:26PM (#24712187) Homepage

              Our code signing machine is locked in a cage and powered up only for purposes of code signing. Executables to be signed are written to a previously wiped USB drive which is attached to the signing machine only when packages are to be signed. The signing machine has not been connected to a network since before the keys were generated. The private key only exists on that machine and in a single separately encrypted backup.

              Meh!
              Well my code signing machine is more secure. We don't put USB sticks directly into the signing machine. We copy the package to a USB stick and then to the 'transfer' machine. The code signing machine is then 'connected' to the transfer machine by infared link which is unblocked by lifting a large steel slab out of the way. The transfer happens via zmodem, and it scanned on both the transfer machine and the code signing machine. Finally we sign the package and transfer it back just before the poor intern's strength gives out and the steel slab slams back down, killing the connection and the intern...(just in case he saw me type in the 42-character passphrase to the private key).

              Beat that security...

      • by Chang (2714) on Friday August 22, 2008 @10:36AM (#24705455)

        Red Hat needs to offer more info before you can make a solid judgement about this.

        If the attacker gained access to the actual system where signing takes place then Red Hat needs to change the key.

        But from the announcement wording - they are suggesting that the attacker was able to submit packages to be signed but the actual signing server was not compromised.

        They should not have been so vague about this because it is a crucial distinction to make for their customer to make a security judgement.

        As a customer I'm not happy with the vague details on what was compromised. I'm sure they did it because they have concerns about describing their package signing systems in detail but they need to open the kimono and give us the detail we need to make a decision about reloading our systems.

        Merely saying, "trust us - anything that came from the official channel is safe" doesn't fly. You let an attacker gain unauthorized access - you need to re-earn trust at this point by giving us some detailed info.

        • by calmond (1284812) on Friday August 22, 2008 @11:05AM (#24705953)
          What surprises me about this the most is that the system was connected to the network/Internet at all. I had always been of the understanding that to prevent this, the signing server was a stand-alone system accessible only by sneaker-net with physical media. You take your package on CD/DVD/USB key to the server, sign it, then take the signed package back via physical media and distribute it. One Federal Gov.t agency in my home town does this and the server is behind three locked doors too, with three different people needed to get physical access. Why didn't RedHat/Fedora do something like this? It really isn't that much of a pain in the ass when you think about it...
      • Re: (Score:3, Insightful)

        by JustKidding (591117)

        Yes, that is what surprised me, too. However, I'd think they would know what they are doing, and are acting in good faith, because they could have tried to keep the whole incident secret instead.

        I don't see why an attacker would sign the packages one that server, instead of just taking the key and signing them elsewhere. Because of this, Red Hat now has the signatures of the tampered OpenSSH packages. If the attacker had signed them elsewhere, they wouldn't, making the packages more valuable.

        Is there a tec

        • by illumin8 (148082)

          Also, I assume this means any historic packages, signed with the old key, not already in your possession at the time of the intrusion cannot be trusted. With this I mean any old versions of packages downloaded after the time the attacker got his hands on the passphrase.

          Good point. If the attacker still has the private key and passphrase, he can trivially repackage any older RPMs and sign them again.

      • by uslinux.net (152591) on Friday August 22, 2008 @11:10AM (#24706031) Homepage

        Our RedHat TAM tells us that "the signing infrastructure is completely different between fedora and RHEL" and that RHEL uses "a submit to be signed" method. So essentially, someone submitted packages and the system automatically signed them.

      • by Anonymous Coward on Friday August 22, 2008 @11:12AM (#24706093)

        In the Redhat announcement, we can infer the passphrase and signing key were compromised, because the attacker signed invalid openssh packages.

        Incorrect. The signing key used by Red Hat is inside a hardware security token.

        So even though it was possible to use the token to sign packages as soon as access to the token has been removed for the intruder, he is unable to sign any more packages.

        Mark Cox of the Red Hat security team explained this setup in a blog post some time ago at http://www.awe.com/mark/blog/200701300906.html [awe.com].

      • Re: (Score:3, Informative)

        by Trahald (698493)
        This is why they don't need new keys: http://www.awe.com/mark/blog/200701300906.html [awe.com] (keys are secure in a hardware device)
    • Re: (Score:2, Insightful)

      by Anonymous Coward

      Coming from someone with many systems, completely exposed to the Internet, with thousand day uptimes, these RedHat folk are in fact sufficiently paranoid.

      Ummm, I'm quite curious, how do you keep your system up for 3 years? Do you not update your kernel? Or is there some way to update a running kernel without rebooting that I don't know about...

      • Re: (Score:3, Funny)

        by Anonymous Coward

        Yea I guess they don't care that a kernel compromise completely negates any security benefit from SELinux.

    • Re: (Score:3, Informative)

      by pembo13 (770295)
      Targeted mode is actually the weaker of the two modes. The other mode, whose title I've forgotten, checks everything. While targeted mode only does... targeted apps.
      • by EvilRyry (1025309)

        The targeted POLICY is the weaker of the two most common policy. The other is strict which is a bit too harsh for most.

        SELinux also has modes. The only one worth using in production is enforcing which actually enforces the rules. There's also permissive which logs when rules are violated but lets them happen anyway; this is good for development but obviously won't save you from a real attack.

  • by mulvane (692631) on Friday August 22, 2008 @10:06AM (#24704959)
    They should have ran a secure OS like vista.
  • Goes to show (Score:5, Insightful)

    by BadAnalogyGuy (945258) <BadAnalogyGuy@gmail.com> on Friday August 22, 2008 @10:07AM (#24704969)

    Given enough time and energy, even Linux servers can be hacked.

    With the growing interest in Linux, I wonder if we'll see more parity of viruses between Windows and Linux.

    • Re:Goes to show (Score:4, Insightful)

      by dword (735428) on Friday August 22, 2008 @10:13AM (#24705039)
      Not unless Linux gains 50+% of the end-user market share.
      • Re: (Score:3, Insightful)

        by berwiki (989827)
        No, you are wrong, and this is the mindset that scares me in the computing world.

        If a custom box running JoeOS contains something extremely financially valuable, you can bet people will start trying to hack it.

        Security through Obscurity is not only wrong, but terrifying that people buy into the concept.
        • Re: (Score:3, Insightful)

          by blhack (921171)

          I think the parent is talking more about general viruses that are just sent out into the tubes with the intent of auto-rooting insecure boxen.

          What you're saying is true "Any system with something desirable on it is at risk of getting wHacked", but one system with important information on it is not going to spawn a breed of viri meant to just root ALL of the boxes with that OS.

      • Given that Linux has a lot of market share in the server department, I would imagine that the reward for compromising a system would be greater for linux right now than windows. After all, would you rather hack into 1000 home desktops or get a server from EBay, Slashdot, or any major to medium site that gets credit card numbers at some point?

        However, since infecting a server is lower profile than infecting 1000 home computers, people looking for notoriety won't be doing it. I imagine that if someone fin
      • by jedidiah (1196)

        The Atari ST had plenty of viruses without 50% of the end-user marketshare.

        The notion that "obscurity" protects Linux is beyond preposterous.

    • Re: (Score:3, Insightful)

      by illumin8 (148082)

      Given enough time and energy, even Linux servers can be hacked.

      With the growing interest in Linux, I wonder if we'll see more parity of viruses between Windows and Linux.

      It also goes to show that the human side is usually where compromises come in to play. Most likely some admin had a weak password that was hacked, and that admin had permission to signing packages or things he should not have had.

      I don't care how secure your OS is. If you don't follow proper security procedures, including using strong pas

      • Re: (Score:3, Insightful)

        by Shados (741919)

        Thats correct. And as much as there are many issues with Windows security that -could- be exploited, usually, even there, the human side is easier to exploit... So those "skills" are portable... Will be interesting to see how the ecosystem reacts when it starts happening more and more... technological fixes won't do...

      • I wouldn't even assume it was an admin. My guess is that a HR person of some sort had a weak password, and that from there the attacker was able to sneak into Red Hat's internal network. Within that network, the attacker would have had a much easier time getting into higher security systems, and eventually start getting those packages signed. Whoever it was probably spent several weeks working on this, especially given the sophistication of the attack (targeting the signing server to apparently compromis
    • Re: (Score:2, Insightful)

      Given enough time and energy, practically any network-connected system can be hacked. That is because security is *hard*, and there are few people who have the means to create chains that contain only strong links, and put those strong chains in the hands of a big audience.

      But given workable tools, I think security comes down more to procedures, and a competent sysadmin than anything else. I'd put more faith in a well-managed Windows server than a Linux server with an idiot as admin. With all factors equal,

    • by sm62704 (957197)

      No, Windows' popularity is only a small part of the reason it is the only OS with viruses in the wild. The biggest reason is that it uses the discredited "security through obscurity", but it, too isn't the only reason Windows is insecure.

      Mac and Linux are based on UNIX, which was developed for mainframes; mainframes have always needed security. Windows was developed before the wide popularity of the internet for stand alone computers. Stuff like Active-X is fine on a computer until you network it.

      There are

      • Re: (Score:3, Informative)

        by fatboy (6851)

        Mac and Linux are based on UNIX, which was developed for mainframes.
        No, Unix was developed for the PDP-7 and PDP-11. Those were minicomputers, not mainframes.

    • by _Sprocket_ (42527)

      With the growing interest in Linux, I wonder if we'll see more parity of viruses between Windows and Linux.

      This should sound familiar to most readers here. We've heard it before:

      http://www.simson.net/clips/2000/2000.SecurityFocus.Linux_Viruses.html [simson.net]

      http://web.archive.org/web/20000304004534/http://www.zdnet.co.uk/news/2000/3/ns-12862.html [archive.org]

      http://www.vnunet.com/vnunet/news/2120227/honeymoon-linux-users [vnunet.com]

      And the same general theme has even been fitted for the MacOS crowd:

      http://www.linuxinsider.com/story/mac%C2%AD-security/57811.html [linuxinsider.com]

      It's not that the concept is all that unlikely. Oddly enough, WinNT set a historical

    • What do you mean, "even Linux server"? How about "even every single computer in the world"?

      Whoemever told you that Linux, Windows, OS X, OpenBSD or whatever is 100% invulnerable to authorized accesses is either ignorant or lying. Nothing is perfect, and there is no reason why Linux advocates should deny that fact. Saying "haha, look, Linux IS insecure after all!!!!1111" is not any more useful than "haha, look, you're a human being and you made a mistake after all!".

      As for viruses, this compromise has got no

    • You'll sooner see phone viruses, methinks.

      The advantage (for the virus distributers) is that Windows and OSX and iPhones have is that they share a common denominator of services and applications. It is much easier to target a system if you know what services are available.

      For the most part Linux systems are custom builds with a variety of applications and services (and versions of) enabled. Someone could target a specific distribution or sets of distributions, and this has happened several times (slap

  • OpenSSH bug? (Score:4, Interesting)

    by samcan (1349105) on Friday August 22, 2008 @10:07AM (#24704979)
    Is this bug in OpenSSH related to the one that was found in Debian-related distros back about April? Maybe I'm reading the article summary incorrectly.
    • Re: (Score:3, Funny)

      by 3p1ph4ny (835701)

      In keeping with the spirit of /., I didn't read TFA.

      However, I'd say this is totally unrelated to the Debian bug. The Debian bug was caused as a result of a change a Debian package maintainer made. Since he only made the change for the Debian package and didn't push it back upstream, it's highly unlikely that they are related.

    • Re: (Score:2, Informative)

      by tuffy (10202)

      Red Hat's OpenSSH bug involves X11 forwarding. As I recall, Debian's OpenSSH bug was a "fix" that dramatically reduced the entropy available for randomizing. I don't believe the two are related.

    • by AndGodSed (968378)

      I read TFA and it seems that this is not a bug. It is rather a compromise as a result of illicit access to the servers.

      Exactly HOW or WHO did this is not mentioned in TFA.

    • Re: (Score:2, Informative)

      by xsuchy (963813)

      I'm from RH...
      No, they are not related. Offical OpenSSH from Fedora or RH repositories do not contain bug (but the low priority X11 forwarding).

      As a precautionary measure, we are releasing an updated version of these SSH packages, if you happend to install previous package from untrusted source (i.e. not RHN).

  • damn't (Score:2, Funny)

    by extirpater (132500)

    source code filching! nothing else.

  • when? (Score:2, Interesting)

    by stoolpigeon (454276) *

    Last week? Does that mean earlier this week, or the week before the week I'm in? At what point in whatever week was last week? If I did an install/update after a certain date am I covered?
     
    It would be nice if they weren't so vague about the time frame. Maybe it is to encourage people to check and not assume they will not have problems, but in a situation like this, the more accurate a picture I have of what is going on, the better I feel.

  • by Anonymous Coward

    I can confirm that roughly 30 kernel 0dayz circulate in the underground. Working for all kernelz 2.6.X up to 2.6.27-rc3 :)

    happy birthday.

    • Re: (Score:3, Funny)

      by sdsucks (1161899)

      Nice. I just compiled 2.6.27-rc4 on my notebook so I guess I'm safe for now. ;)

  • Suuure... (Score:2, Funny)

    by Anonymous Coward

    "Just run this shell script to verify you're not infected"

    No way I'm falling for that one.

    Back to work.

  • "Compromised?" (Score:5, Insightful)

    by Hyppy (74366) on Friday August 22, 2008 @10:29AM (#24705361)
    I could not RTFA (/.ed), but is there any indication of how this "compromise" occurred?

    My hats off, though, to the Red Hat folks. Full disclosure and immediate positive action speaks volumes.

    On a related note, you should not use Fedora in a production environment anyway. That's what RHEL is for. Fedora = Testing. RHEL = Stable. At least in theory.
    • Re:"Compromised?" (Score:5, Informative)

      by corbettw (214229) <corbettw@noSpAm.yahoo.com> on Friday August 22, 2008 @10:45AM (#24705603) Journal

      On a related note, you should not use Fedora in a production environment anyway. That's what RHEL is for. Fedora = Testing. RHEL = Stable. At least in theory.

      I thought it was, RHEL == RedHat Support, Fedora == Community Support. Fedora might have some bleeding edge stuff in it, if you upgrade often, but it seems about as stable as the corresponding RHEL release. The difference is the support you can count on.

      • Re: (Score:3, Informative)

        Fedora is not as stable as RHEL. If you want "community support" with RHEL's stability, you should use CentOS. In Fedora 9, we have a beta X server, a bleeding edge kernel, and the disastrous KDE 4.0.
      • Re: (Score:2, Informative)

        by dstech (807139)

        Well, RHEL also mantains a stable kABI within the entire major release, and only rebases packages when absolutely necessary (maintaining most library ABIs as well). For example, RHEL 4 ships apache 2.0.52, and has since launch. Security and bug fixes are backported, but the fundamental behavior remains the same for any instance of RHEL 4. This is also true of libraries.

        This means that a given piece of 3rd-party software is more likely to keep working after an update in RHEL than in Fedora.

      • by Phroggy (441)

        I thought it was, RHEL == RedHat Support, Fedora == Community Support. Fedora might have some bleeding edge stuff in it, if you upgrade often, but it seems about as stable as the corresponding RHEL release. The difference is the support you can count on.

        I thought that too, but I was wrong, and it bit me in the ass. Fedora is NOT appropriate for a production environment, period. Fedora drops all support 13 months after release, which means they stop issuing security patches, period. In a production environment where you're likely to have 13 month uptimes, that would mean a reinstall every time you reboot the machine.

        If you want a RedHat-based distro with long-term community support, the one you're looking for is CentOS, or so I'm told. For your desktop

    • Re: (Score:3, Insightful)

      by pembo13 (770295)
      In all fairness, and not to paint them in a bad light. The sequence was more like immediate action, and then full disclosure. But I got the feeling that the delay was due to some legal issues.
    • by MrMr (219533)
      ...At least in theory.
      Your theory fails on its first test because of the little detail that only RHEL appears to have been compromised and not Fedora.
  • by Bob9113 (14996) on Friday August 22, 2008 @10:55AM (#24705781) Homepage

    Pretty sure most of us are above this anyway, but let's avoid a distro flamewar. You can look through my past comments and see that RH is far from my preferred distro, and I love to take shots at them. But now is not the time. Anyone can get hacked, and it sucks. And they're being responsible about reporting and mitigating.

    Godspeed, gentlemen.

  • Uhm... How? (Score:5, Insightful)

    by X.25 (255792) on Friday August 22, 2008 @11:34AM (#24706437)

    I really only care to know HOW the attacker got in.

    Basically, if he used unknown 0-day and RH/Fedora have no idea what he exploited, then they should say so, so people can watch out.

    If he stole username/password from someone dumb - say so.

    If he walked into the hosting center, say so.

    I REALLY want to how know he compromised their server(s).

    I might be next v0v

  • by Anonymous Coward on Friday August 22, 2008 @04:24PM (#24711327)

    Our RHEL5/x86_64 system has been affected by this problem: I have ran the script from Red Hat openssh blacklist page [redhat.com], and found that all four openssh packages (openssh, openssh-clients, openssh-askpass, openssh-server) had their checksum on the blacklist. I took the server down, created a backup snapshot of the root disk, and I am currently reinstalling it, while checking other volumes and the root volume snapshot for any signs of intrusion.

    The most annoying thing is that Red Hat remains silent on the main problem: what the compromised packages contained, how to determine whether the possible attacker exploited the access offered by those packages or not, when exactly were the packages signed, what other precautions to do on other servers (notify users which use the same password as on a compromised server, check for other modified binaries, etc.). I have verified that I had a trojanized binaries on my system, but apart from that, it is not clear what else the possible attacker managed to do.

    Red Hat says the packages were not distributed over RHN, so I wonder how I got them. I had another repository in my yum.conf: rpmforge. Maybe this was the source of the malware. My syslog (even a copy on a syslog server) did not say anything about upgrading openssh in the last month or so. However, on Aug 15 it upgraded the YUM RHN plugin. On the same day our dovecot stopped responding, saying the time went backwards (and yes, there was time move several weeks back and then forward, according to dovecot log). Also the rpm -qi said the package was built on Aug 13 13:13:03, and signed five minutes later. However, the install time reported by rpm on my system was July 25 (which would corelate with the time slip reported by dovecot).

    Did anybody else met the trojanized openssh mentioned in the advisory? Please share your findings.

    Posting as AC for obvious reasons, sorry.

Make headway at work. Continue to let things deteriorate at home.

Working...