Stories
Slash Boxes
Comments
typodupeerror delete not in

Comments: 278 +-   Red Hat, Fedora Servers Compromised on Friday August 22 2008, @09:04AM

Posted by kdawson on Friday August 22 2008, @09:04AM
from the quick-action dept.
redhat
business
security
An anonymous reader writes "In an email sent to the fedora-announce mailing list, it has been revealed that both Fedora and Red Hat servers have been compromised. As a result Fedora is changing their package signing key. Red Hat has released a security advisory and a script to detect potentially compromised openssh packages."
story

Related Stories

This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • by Art Popp (29075) * on Friday August 22 2008, @09:05AM (#24704945)

    These are the guys, to the annoyance of nearly everyone, who turned on SELinux on Fedora Core by default.

    These are the guys who noticed they annoyed everyone, and turned on targeted-mode by default.

    Coming from someone with many systems, completely exposed to the Internet, with thousand day uptimes, these RedHat folk are in fact sufficiently paranoid.

    They have taken all the reasonable precautions, and if their passphrase was strong, then the danger of my servers being compromised by meteor strike is a much greater worry.

    • by illumin8 (148082) on Friday August 22 2008, @09:14AM (#24705065) Journal

      They have taken all the reasonable precautions, and if their passphrase was strong, then the danger of my servers being compromised by meteor strike is a much greater worry.

      The only thing that concerns me is this: In the Fedora announcement, they said with a high level of confidence, they don't believe the passphrase for their signing key was compromised, because they hadn't signed any packages during the period of time the box was compromised. They are going to change the signing key anyway just in case. This is a good thing.

      In the Redhat announcement, we can infer the passphrase and signing key were compromised, because the attacker signed invalid openssh packages. Even though the official RHN distribution channel was not compromised, the attacker most likely still has their private key and passphrase and can continue signing packages and attempting to distribute them. Redhat needs to step up and reissue a new signing key. There was no announcement of this.

      • by Chang (2714) on Friday August 22 2008, @09:36AM (#24705455)

        Red Hat needs to offer more info before you can make a solid judgement about this.

        If the attacker gained access to the actual system where signing takes place then Red Hat needs to change the key.

        But from the announcement wording - they are suggesting that the attacker was able to submit packages to be signed but the actual signing server was not compromised.

        They should not have been so vague about this because it is a crucial distinction to make for their customer to make a security judgement.

        As a customer I'm not happy with the vague details on what was compromised. I'm sure they did it because they have concerns about describing their package signing systems in detail but they need to open the kimono and give us the detail we need to make a decision about reloading our systems.

        Merely saying, "trust us - anything that came from the official channel is safe" doesn't fly. You let an attacker gain unauthorized access - you need to re-earn trust at this point by giving us some detailed info.

        • by calmond (1284812) on Friday August 22 2008, @10:05AM (#24705953)
          What surprises me about this the most is that the system was connected to the network/Internet at all. I had always been of the understanding that to prevent this, the signing server was a stand-alone system accessible only by sneaker-net with physical media. You take your package on CD/DVD/USB key to the server, sign it, then take the signed package back via physical media and distribute it. One Federal Gov.t agency in my home town does this and the server is behind three locked doors too, with three different people needed to get physical access. Why didn't RedHat/Fedora do something like this? It really isn't that much of a pain in the ass when you think about it...
          • by Timothy Brownawell (627747) <tbrownaw@prjek.net> on Friday August 22 2008, @10:30AM (#24706371) Journal
            How well does that work if you can trick someone into loading the wrong package onto that USB key?
            • by moderatorrater (1095745) on Friday August 22 2008, @11:18AM (#24707171)
              You're missing the most interesting possibility in my mind: employee sabotage. Why should open source be immune to a bad apple attempting to subvert the system for their own gain? A mid-level employee signs a package and distributes it, a customer running a rootkit checker or clamav on their system notices that the copy they have is suspicious, reports it, and suddenly you have a situation where the key itself may or may not be compromised and some checking needs to be done everywhere.
      • Re: (Score:3, Insightful)

        Yes, that is what surprised me, too. However, I'd think they would know what they are doing, and are acting in good faith, because they could have tried to keep the whole incident secret instead.

        I don't see why an attacker would sign the packages one that server, instead of just taking the key and signing them elsewhere. Because of this, Red Hat now has the signatures of the tampered OpenSSH packages. If the attacker had signed them elsewhere, they wouldn't, making the packages more valuable.

        Is there a tec

      • by uslinux.net (152591) on Friday August 22 2008, @10:10AM (#24706031) Homepage

        Our RedHat TAM tells us that "the signing infrastructure is completely different between fedora and RHEL" and that RHEL uses "a submit to be signed" method. So essentially, someone submitted packages and the system automatically signed them.

      • by Anonymous Coward on Friday August 22 2008, @10:12AM (#24706093)

        In the Redhat announcement, we can infer the passphrase and signing key were compromised, because the attacker signed invalid openssh packages.

        Incorrect. The signing key used by Red Hat is inside a hardware security token.

        So even though it was possible to use the token to sign packages as soon as access to the token has been removed for the intruder, he is unable to sign any more packages.

        Mark Cox of the Red Hat security team explained this setup in a blog post some time ago at http://www.awe.com/mark/blog/200701300906.html [awe.com].

      • Re: (Score:3, Informative)

        This is why they don't need new keys: http://www.awe.com/mark/blog/200701300906.html [awe.com] (keys are secure in a hardware device)
        • by illumin8 (148082) on Friday August 22 2008, @10:01AM (#24705881) Journal

          Or we could infer that the system was used for its purpose by the attacker. He signed those packages. Redhat looked at the logs, no other packages were signed. So the passphrase is still very likely to be save.

          God, I seriously hope they don't have the passphrase saved so that you don't need to type it in to sign a package. If that is the case their security is very lax. Also, if it's saved, it almost certainly is compromised, because it's stored on disk somewhere. It would be trivial for the attacker to pull it out of whatever script or text file it's saved in.

          • by SETIGuy (33768) on Friday August 22 2008, @01:18PM (#24709409) Homepage

            God, I seriously hope they don't have the passphrase saved so that you don't need to type it in to sign a package. If that is the case their security is very lax.

            I don't know about anyone else, but I am surprised that their package signing machine is connected via a network to other machines.

            Our code signing machine is locked in a cage and powered up only for purposes of code signing. Executables to be signed are written to a previously wiped USB drive which is attached to the signing machine only when packages are to be signed. The signing machine has not been connected to a network since before the keys were generated. The private key only exists on that machine and in a single separately encrypted backup.

            I've always considered that to be a minimally paranoid means of keeping private keys private. Really paranoid would be "signed on one machine, checked and signed again on another machine."

            • Our code signing machine is locked in a cage and powered up only for purposes of code signing. Executables to be signed are written to a previously wiped USB drive which is attached to the signing machine only when packages are to be signed. The signing machine has not been connected to a network since before the keys were generated. The private key only exists on that machine and in a single separately encrypted backup.

              Meh!
              Well my code signing machine is more secure. We don't put USB sticks directly into the signing machine. We copy the package to a USB stick and then to the 'transfer' machine. The code signing machine is then 'connected' to the transfer machine by infared link which is unblocked by lifting a large steel slab out of the way. The transfer happens via zmodem, and it scanned on both the transfer machine and the code signing machine. Finally we sign the package and transfer it back just before the poor intern's strength gives out and the steel slab slams back down, killing the connection and the intern...(just in case he saw me type in the 42-character passphrase to the private key).

              Beat that security...

    • Re: (Score:3, Informative)

      Targeted mode is actually the weaker of the two modes. The other mode, whose title I've forgotten, checks everything. While targeted mode only does... targeted apps.
      • Re: (Score:3, Funny)

        by Anonymous Coward

        Yea I guess they don't care that a kernel compromise completely negates any security benefit from SELinux.

  • by mulvane (692631) on Friday August 22 2008, @09:06AM (#24704959)
    They should have ran a secure OS like vista.
  • Goes to show (Score:5, Insightful)

    by BadAnalogyGuy (945258) <BadAnalogyGuy@gmail.com> on Friday August 22 2008, @09:07AM (#24704969)

    Given enough time and energy, even Linux servers can be hacked.

    With the growing interest in Linux, I wonder if we'll see more parity of viruses between Windows and Linux.

    • Re:Goes to show (Score:4, Insightful)

      by dword (735428) on Friday August 22 2008, @09:13AM (#24705039)
      Not unless Linux gains 50+% of the end-user market share.
      • Re: (Score:3, Insightful)

        No, you are wrong, and this is the mindset that scares me in the computing world.

        If a custom box running JoeOS contains something extremely financially valuable, you can bet people will start trying to hack it.

        Security through Obscurity is not only wrong, but terrifying that people buy into the concept.
        • Re:Goes to show (Score:4, Insightful)

          by TorKlingberg (599697) on Friday August 22 2008, @09:24AM (#24705241)
          The virus can install itself in the user home directory instead.
          • Re:Goes to show (Score:5, Insightful)

            by coryking (104614) * on Friday August 22 2008, @09:56AM (#24705795) Homepage Journal

            The virus can install itself in the user home directory instead.

            And then use one of the many local exploits to get root.

            The most scary and amusing thing is how quick some people on this site and others are to dismiss local exploits. They all think "you have to be on the console, so fuck it, this isn't important and doesn't affect me". They are wrong. These days, a remote exploit is just a human operator and a local exploit.

            • Re:Goes to show (Score:4, Informative)

              by Maznio (137785) on Friday August 22 2008, @11:24AM (#24707289) Journal

              Thankfully we have the noexec mounting option available.

              That's no good. Scripts can be run by invoking the interpreter first, like so:
              /usr/bin/perl /home/user/proggie

              and binaries by starting them like so:
              /lib/ld-linux.so.2 /home/user/proggie

        • Re:Goes to show (Score:4, Informative)

          by Goaway (82658) on Friday August 22 2008, @09:27AM (#24705299) Homepage

          There's absolutely nothing to stop anybody from installing an executable that runs automatically under a user account, without ever needing root. And that executable can do a lot of the things it may want to do without ever needing root access, either.

          • Re:Goes to show (Score:5, Interesting)

            by vadim_t (324782) on Friday August 22 2008, @10:00AM (#24705869) Homepage

            There are plenty things that can be done.

            Mounting /home with noexec
            Using the grsecurity patch, which can deny execution of files not in directories owned by root, as well as usage of network sockets.
            Using SELinux

            The tools are there. All that's needed is to use them.

            The need to download random binaries to your home directory and run them is infrequent in Linux. The most frequent case is application installers, but many of those need root access anyway (nvidia drivers for instance), and most come with the distribution. A way to fix the occasional need to do this would be a sudo-like tool that needs to be used to execute a file, but doesn't grant root privileges.

            • Re:Goes to show (Score:5, Informative)

              by Wee (17189) on Friday August 22 2008, @12:54PM (#24708937)

              If you're going to mount /home noexec, you should also mount /tmp as noexec as well. In fact, I'd wager you should do that well before you bother with /home. A lot of wormy/trojany stuff wants to write, unpack, build and execute in /tmp. In fact, while you're at it, make sure only root can run make and gcc, or get at any of the libs. All command line network tools (wget, ftp, etc) should also only be run by root. Now go through and get rid of most (all?) of the setuid root stuff. Then crank down the firewall to only allow incoming 22 and 80 (or whatever). That will take care of a wide range of automated stuff.

              -B

            • Re:Goes to show (Score:5, Insightful)

              by Goaway (82658) on Friday August 22 2008, @09:37AM (#24705467) Homepage

              The point is, there's no need to change system files or bind to privileged ports.

              Your documents contains LOTS of yummy personal information for people to steal. Identity thieves and credit card thieves will love all that stuff.

              Spammers need relays to send their spam through. You can run a relay just fine as a normal user. Same thing with the DDoS bot for exortotionists and script kiddies.

              You can mess with the internals of Firefox without root access too, through plugins. Easy to put a password stealer in there. Or you could mess with your desktop settings so that when you try to launch a browser, you get a compromised version instead.

              I'd say I've covered all the major reasons somebody would want to infect your machine here, and not a single system file or privileged port was needed for it.

              • Re: (Score:3, Interesting)

                "Spammers need relays to send their spam through. You can run a relay just fine as a normal user"

                Not if you don't have access to the firewall settings which will open the port that allows someone to connect to your relay.

                "You can mess with the internals of Firefox without root access too, through plugins. Easy to put a password stealer in there."

                Yes, but without access to the system FF folder, that plugin will go in your per-user plugin directory, and will only run for you. So only your passwords will be st

                • Re:Goes to show (Score:4, Insightful)

                  by Kjella (173770) on Friday August 22 2008, @10:37AM (#24706485) Homepage

                  Not if you don't have access to the firewall settings which will open the port that allows someone to connect to your relay.

                  Unless you happen to run one of the desktop distros which usually have a default policy of ACCEPT.

                  Of course, the "only works for one user" argument is better if presented in reverse. If your less-computer-literate kid/spouse/parent can't accidentally run code that (...)

                  Read all my documents through the world-readable home folders? Another convienience feature.

                  My experience is that people don't keep the accounts truly separate, that's just for convienience. "Hey, can I just check my webmail for a sec?" "Sure" and your email's compromised.

                  Furthermore, you'll be in a position to be able to clean their account up for them without having to wipe and reinstall the whole machine

                  in theory. In practise, I expect the malware authors to find so many ways of hiding (or just when you "rescue" his documents) that it won't practicly happen. Least not without someone more experienced than the average guy.

                • Re:Goes to show (Score:5, Insightful)

                  by Goaway (82658) on Friday August 22 2008, @10:56AM (#24706807) Homepage

                  Not if you don't have access to the firewall settings which will open the port that allows someone to connect to your relay.

                  The program can just make the initial connection to the spammer server itself. This is the same on Windows and Linux, and these programs operate just fine under Windows.

                  Yes, but without access to the system FF folder, that plugin will go in your per-user plugin directory, and will only run for you.

                  How many computers do you honestly think there are out there that have more than a single user?

                • Re:Goes to show (Score:4, Interesting)

                  by Sentry21 (8183) on Friday August 22 2008, @03:51PM (#24711749) Journal

                  Not if you don't have access to the firewall settings which will open the port that allows someone to connect to your relay.

                  They don't need your relay. If they're running on your machine, they can fetch their payload and then start sending it out through your local MTA or configured SMTP server. If you can send e-mail, so can they.

                  Yes, but without access to the system FF folder, that plugin will go in your per-user plugin directory, and will only run for you. So only your passwords will be stolen, and not those of anyone else on the computer.

                  Given that most computers running Firefox these days are single-user systems, whether running Linux, OS X, or Windows 98.

                  Then consider Linux systems. Most systems these days are set up with sudo access, as is OS X. All the bug has to do is watch to see when you run sudo yourself, and then bam, it has a (usually) five-minute window to run itself as root and infect the rest of your system.

                  It can also grab your ~/.ssh/known_hosts and then reach out to those to see which ones accept your private key; install itself there, and, again, watch for sudo access. It's not hard for someone to go from there out to infecting every machine you have access to, and root on every machine you have root on, and potentially every system that every user on that system has access to, and so on, and so on.

            • Re:Goes to show (Score:4, Insightful)

              by drsmithy (35869) <drsmithyNO@SPAMgmail.com> on Friday August 22 2008, @10:34AM (#24706449)

              Like change system files? Nope. How about bind to privileged ports? Nope.

              It can send spam, participate in DDoS attacks, act as a repository for kiddy porn, or just wait to take advantage of the next 0-day local privilege exploit.

              In short, lack of root-level access is a minor annoyance to malware, not a critical problem.

              So... it can mess up my documents? Darn.

              Yes. It can mess what are most likely the most important and least-replaceable data on your machine. This doesn't bother you ?

            • by SEMW (967629) on Friday August 22 2008, @10:57AM (#24706835)

              Like change system files? Nope. ... So... it can mess up my documents? Darn.

              Oh, good. My life's work is reconstructable in a mere few decades; wheras if it damages system files, a reinstall could take up to half an hour!

            • Re: (Score:3, Insightful)

              So cleanup is easier. But the damage may already be done, as criminals may now have your passwords, your credit card numbers, and your personal information. Plus you were probably sending spam up until the moment you noticed the infection.

    • Re: (Score:3, Insightful)

      Given enough time and energy, even Linux servers can be hacked.

      With the growing interest in Linux, I wonder if we'll see more parity of viruses between Windows and Linux.

      It also goes to show that the human side is usually where compromises come in to play. Most likely some admin had a weak password that was hacked, and that admin had permission to signing packages or things he should not have had.

      I don't care how secure your OS is. If you don't follow proper security procedures, including using strong pas

      • Re: (Score:3, Insightful)

        Thats correct. And as much as there are many issues with Windows security that -could- be exploited, usually, even there, the human side is easier to exploit... So those "skills" are portable... Will be interesting to see how the ecosystem reacts when it starts happening more and more... technological fixes won't do...

      • Re: (Score:3, Informative)

        Mac and Linux are based on UNIX, which was developed for mainframes.
        No, Unix was developed for the PDP-7 and PDP-11. Those were minicomputers, not mainframes.

  • OpenSSH bug? (Score:4, Interesting)

    by samcan (1349105) on Friday August 22 2008, @09:07AM (#24704979) Homepage
    Is this bug in OpenSSH related to the one that was found in Debian-related distros back about April? Maybe I'm reading the article summary incorrectly.
    • In keeping with the spirit of /., I didn't read TFA.

      However, I'd say this is totally unrelated to the Debian bug. The Debian bug was caused as a result of a change a Debian package maintainer made. Since he only made the change for the Debian package and didn't push it back upstream, it's highly unlikely that they are related.

  • "Compromised?" (Score:5, Insightful)

    by Hyppy (74366) on Friday August 22 2008, @09:29AM (#24705361)
    I could not RTFA (/.ed), but is there any indication of how this "compromise" occurred?

    My hats off, though, to the Red Hat folks. Full disclosure and immediate positive action speaks volumes.

    On a related note, you should not use Fedora in a production environment anyway. That's what RHEL is for. Fedora = Testing. RHEL = Stable. At least in theory.
    • Re:"Compromised?" (Score:5, Informative)

      by corbettw (214229) <corbettw@@@yahoo...com> on Friday August 22 2008, @09:45AM (#24705603) Homepage Journal

      On a related note, you should not use Fedora in a production environment anyway. That's what RHEL is for. Fedora = Testing. RHEL = Stable. At least in theory.

      I thought it was, RHEL == RedHat Support, Fedora == Community Support. Fedora might have some bleeding edge stuff in it, if you upgrade often, but it seems about as stable as the corresponding RHEL release. The difference is the support you can count on.

      • Fedora is not as stable as RHEL. If you want "community support" with RHEL's stability, you should use CentOS. In Fedora 9, we have a beta X server, a bleeding edge kernel, and the disastrous KDE 4.0.
    • Re: (Score:3, Insightful)

      In all fairness, and not to paint them in a bad light. The sequence was more like immediate action, and then full disclosure. But I got the feeling that the delay was due to some legal issues.
        • Re: (Score:3, Informative)

          I'd suggest reading both advisories again. But I'll be nice and spell it out. It seems neither OS's repositories were compromised.
          From the Fedora advisory: "Among our other analyses, we have also done numerous checks of the Fedora package collection, and a significant amount of source verification as well, and have found no discrepancies that would indicate any loss of package integrity."
          From the RHEL advisory: "Based on these efforts, we remain highly confident that our systems and processes prevented the
  • by Bob9113 (14996) on Friday August 22 2008, @09:55AM (#24705781) Homepage

    Pretty sure most of us are above this anyway, but let's avoid a distro flamewar. You can look through my past comments and see that RH is far from my preferred distro, and I love to take shots at them. But now is not the time. Anyone can get hacked, and it sucks. And they're being responsible about reporting and mitigating.

    Godspeed, gentlemen.

  • Uhm... How? (Score:5, Insightful)

    by X.25 (255792) on Friday August 22 2008, @10:34AM (#24706437)

    I really only care to know HOW the attacker got in.

    Basically, if he used unknown 0-day and RH/Fedora have no idea what he exploited, then they should say so, so people can watch out.

    If he stole username/password from someone dumb - say so.

    If he walked into the hosting center, say so.

    I REALLY want to how know he compromised their server(s).

    I might be next v0v

    • by Hatta (162192) on Friday August 22 2008, @10:35AM (#24706461) Journal

      the most likely attack was probably from those lame SSH dictionary scans on port 22. This is usually just an extreme annoyance to admins who must provide port 22 service and haven't heard of 'SSHguard'.

      Or just use SSH key authentication, this is what it's for. Anyone clever enough to use SSH on a redhat project server should be able to manage key authentication.

The trouble with the rat-race is that even if you win, you're still a rat. -- Lily Tomlin