Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Security Linux

Kernel.org Attackers Didn't Know What They Had 183

Posted by Soulskill from the you-never-appreciate-it-until-it's-gone dept.
Trailrunner7 writes "The attack that compromised some high-value servers belonging to kernel.org — but not the Linux kernel source code — may have been the work of hackers who simply got lucky and didn't realize the value of the servers that they had gotten their hands on. The attackers made a couple of mistakes that enabled the administrators at kernel.org to discover the breach and stop it before any major damage occurred. First, they used a known Linux rootkit called Phalanx that the admins were able to detect. And second, the attackers set up SSH backdoors on the compromised servers, which the admins also discovered. Had the hackers been specifically targeting the kernel.org servers, the attack probably would've looked quite different." A few blog posts in the wake of the attack have agreed with the initial announcement; while it was embarrassing, the integrity of the kernel source is not in question.
This discussion has been archived. No new comments can be posted.

Kernel.org Attackers Didn't Know What They Had More

Kernel.org Attackers Didn't Know What They Had

Comments Filter:
  • They didnt want to harm the kernel.

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      Or maybe, just maybe, the hackers wanted to appear that they didn't realize what they had gotten their hands on. I'm not trying to cause any tinfoil hats to come out, but I would still check everything.

      • Re: (Score:2)

        by Dwonis ( 52652 ) *

        Or maybe, just maybe, the hackers wanted to appear that they didn't realize what they had gotten their hands on. I'm not trying to cause any tinfoil hats to come out, but I would still check everything.

        git fsck

        Done.

    • Re:Or maybe (Score:5, Funny)

      by FatdogHaiku ( 978357 ) on Saturday September 03, 2011 @09:04PM (#37299942)

      They didnt want to harm the kernel.

      Possibly out of respect for the 11 Secret Herbs and Spices?

      • Re: (Score:2)

        by fatphil ( 181876 )
        You my laugh, but I have at least twice been approached by agencies who are looking for a colonel developer.

  • ...and didn't realize the value of the servers that they had gotten their hands on...

    ....I don't see any mention of what the phrase refers to. Is this dramatization or intentionally excluded information?

    Curious.

    • Re:and after reading the articles.... (Score:5, Informative)

      by Samantha Wright ( 1324923 ) on Saturday September 03, 2011 @02:27PM (#37297690) Homepage Journal
      Here [bell-labs.com] is what it's referring to. CS graduates are expected to recognize instances of it instinctively.

      • Here [bell-labs.com] is what it's referring to. CS graduates are expected to recognize instances of it instinctively.

        Ah, so the statement was conceptual, not based on a hidden component within these boxes. Gotcha.

        Correct me if I'm wrong.

        • Re:and after reading the articles.... (Score:5, Insightful)

          by Samantha Wright ( 1324923 ) on Saturday September 03, 2011 @02:52PM (#37297850) Homepage Journal
          That's pretty much it. Malicious control over the master copy of the kernel source means you can bake a rootkit into everything everywhere with enough clever code. All it takes is one generation of bad files to silently patch all successive copies during compilation, and you've got the stuff that cypherpunk nightmares are made of.
          • I'm not a programmer, but I'm not entirely ignorant either... which leaves me with a question... Assuming that the Kernel was compromised, and the scenario you describe came into being. Isn't it just a matter of examining the Kernel code until you find the naughty bits and expunge them? Or are you basing your nightmare on this infiltration not being detected?

            • Isn't it just a matter of examining the Kernel code until you find the naughty bits and expunge them?

              Thats right. A task that can never be proven to be completed.

              • Not really, you have to remember that the complete code as it was before these naughty bits was introduced is stored on multiple locations including Linus's own hard drive so extracting a diff between the two versions would point at all these changes immediately.

            • Re:and after reading the articles.... (Score:5, Informative)

              by jimicus ( 737525 ) on Saturday September 03, 2011 @04:35PM (#37298428)

              I'm not a programmer, but I'm not entirely ignorant either... which leaves me with a question... Assuming that the Kernel was compromised, and the scenario you describe came into being. Isn't it just a matter of examining the Kernel code until you find the naughty bits and expunge them? Or are you basing your nightmare on this infiltration not being detected?

              Not at all - the paper describes how one could write a custom C compiler which would automagically insert malicious code when it saw a particular pattern.

              With a properly crafted attack you couldn't even compile your own "known clean" compiler - the attack takes advantage of the fact that most modern C compilers can't be compiled without an existing C compiler of some sort. Provided the existing compiler is the malicious one, all it needs to do is insert its own malicious payload as part of the compiler compilation process and every subsequent compiler on that system is equally malicious even though the source code is perfectly clean.

              • That's exactly the idea. While such a move would eventually be discovered, the amount of damage that could be done before that happened could be substantial, and the amount of work required to (a) find the bad code in a copy of the source tree and, especially, then (b) detect bad binaries would be pretty staggering. It should also be stressed that this isn't just some fanciful nightmare scenario as Kahabut suspects; Ken Thompson did it to the Unix kernel long ago.

                • Ken Thompson wrote a paper on how to do it http://cm.bell-labs.com/who/ken/trust.html [bell-labs.com], but did not actually do it ...(As far as we know)

                  The point about it is that there is no source tree to examine ...

                  The point about any hack of kernel.org, is that there is no central depository of code there that is the master copy, it uses git, so there are many copies (or partial copies) scattered around many machines, compromising all of them is very unlikely, many are behind much better firewalls, examining the source

                • Re: (Score:2)

                  by mortonda ( 5175 )
                  But the point in the articles is that something exists to prevent this - Ken Thompson did not have to contend with git. To make that attack work you have to get malicious code into the tree for a while - long enough to infect the hosts that compile kernels for others... and then remove the original code.

                  Git would raise the alarm long before that, thanks to the cryptographic nature that it uses to identify the source chain.

                  It's a theoretical attack on one system, but for it to work, Linus would have to do

              • And no one has old disks lying around with old versions of the compiler from before the server was compromised?

                • Re: (Score:2)

                  by jimicus ( 737525 )

                  And no one has old disks lying around with old versions of the compiler from before the server was compromised?

                  You assume that the compiler was compromised in the most recent server compromise. For all you know it was compromised ten years ago and every compiler ever since has been compromised. Go back far enough, if you'd done that early enough in the evolution of Linux distributions it's entirely possible the C compiler on every distribution and architecture is compromised.

                  Myself, I take from that paper that for practical purposes there is no such thing as a computer you can offer a cast-iron guarantee is not comp

              • Restore the binaries directly from an earlier backup or some known-clean version. Transfer the existing and the restored binaries to a clean server via write-once media and do hash compares using multiple algorithms. Obtain several more copies of the clean versions and do the same with them.

                Recompile the source with the compromised server. Again, do the hash comparisons using a clean, unconnected, read-only machine.

                If there was some malicious binary inserted into the server or some kind of malicious hook st

            • The point with ken's trick was that it's possible to throw in changes that are not *persistent* in the source code.

          • Or we take advantage of how git works, and axe all those commints that is noncomplient against history.

          • Re: (Score:2)

            by sjames ( 1099 )

            Except for those bazillions of good systems burned onto CDROMS already, any one of which could be used to bootstrap.

            It would take a lot more than one generation of corruption to overcome that.

          • There is no one master copy, and most devs keep a local copy. Every file is checksummed, and the checksums are published, meaning it's pretty easy to tell weather a copy has been messed with.

          • Re: (Score:2)

            by smash ( 1351 )
            Even "better" would be if you were to mess with the FTP daemon, to silently serve various files from elsewhere that are trojanned. Leave the on-server source in tact, as its obvious people are going to check that...
          • So nuke the servers and every patch prior to 3.1-rc3?

  • How could the attackers... (Score:3)

    by Nutria ( 679911 ) on Saturday September 03, 2011 @02:25PM (#37297678)

    not know what they had cracked and how useful it was?

    • Maybe a semi-automated attack in search of more zombie machines for sending spam.

      • Re: (Score:2, Informative)

        by Anonymous Coward

        Any random machine with SSH enabled gets around 2-3 brute force attempts per day from automated zombies. Most likely kernel.org was breached in an automated attack. I've protected my server with Denyhosts which will add your IP address to the firewall after 5 failed passwords... indefinitely. kernel.org should either install it or switch to public key authentication. Having user/password authentication on Internet without any protection is real stupid.

    • Re: (Score:3)

      by beuges ( 613130 )

      This argument doesn't make sense. From what I've read, a kernel dev with kernel.org access had his machine trojanned, and the attackers got to kernel.org in that way. That's a far cry from script kiddies trying SSH ports on a bunch of random IP addresses. It sounds like quite a lot of effort to specifically get to the kernel.org network. Whether they managed to do some unknown damage, or access some other data whose relevance is as yet unknown, or maybe they just did it for the reputation of having hacked k

  • Nonsense! (Score:5, Funny)

    by Anonymous Coward on Saturday September 03, 2011 @02:28PM (#37297694)

    They must have gotten their hands on the kernel source code, I found it posted online!

  • sure THESE guys didn't touch the kernel... (Score:1)

    by Anonymous Coward

    but how do we know someone more sophisticated didn't already break in and mess with the code undetected?

    • Re: (Score:2)

      by migla ( 1099771 )

      >but how do we know someone more sophisticated didn't already break in and mess with the code undetected?

      Lots of people probably have the same git source repo on their own computers, so it can be compared to the kernel.org one.

    • Re: (Score:3, Informative)

      by Anonymous Coward

      Because in git, everything has got a hash checksum that is not forgeable (try to get something that compiles, does specifically something malicious, and that has the same checksum and size as before).

      And then there are thousands of copies of the entire thing around. I also have one. A simple comparison for equality will work.

  • Why? (Score:1)

    by MattGWU ( 86623 )

    Why would the attacks have to look different? Because if somebody wanted to mess with the source, they'd be more sophisticated and use more sophisticated exploits? Like Kaspersky pointed out, if they wanted to mess with the source code, a lot of what they did would have been unnecessary, but whatever initial exploit they used would have still worked! I think the real point is here 'they got in'. Better attackers just mean they wouldn't have discovered the break-in as quickly, and actual damage might have

  • The truth (Score:1, Insightful)

    by Iskorptix ( 2452916 )
    I think the truth is that failers trying to save their asses and trying to make themselves heroes here.

  • Spin (Score:5, Insightful)

    by 93 Escort Wagon ( 326346 ) on Saturday September 03, 2011 @02:49PM (#37297840)

    Given that they attackers hacked the server a minimum of 17 days before it was detected, I'm not sure I'm going to buy into a story that makes the attackers sound clueless and the server admins smart and on the ball.

  • It wasn't until they got into the machines that they realized the Kernel wasn't written in Javascript. "Dammit!"

  • How they got root access after logging in. Was it something simple like a sudo? Was it a known, unpatched kernel vulnerability? Or, was it some new vulnerability current kernels are susceptible to? Last I read, they logged in under a user account, then they got root access.

    • Re:I'd Still Like To Know... (Score:5, Informative)

      by beuges ( 613130 ) on Saturday September 03, 2011 @04:36PM (#37298434)

      From what I read, one of the guys with a kernel.org login (HPA, I believe) had his personal machine infected by a trojan. The attackers were then able to login to kernel.org impersonating him. They then used a local-only exploit to get root.

      This is why a local-only exploit is just as bad as a remote exploit. If your machine connects to a network, it has the potential to be compromised by a local-only exploit, by first exploiting a flaw in a completely unrelated program which is accessible remotely. In this case, the "flaw" was the compromised user account. It could have been a buffer overflow in an ftp or web server, which doesn't allow for privilege escalation on its own, but allows arbitrary code to be run as the current user... all the attacker has to do is make that arbitrary code trigger the local-only exploit, and your local-only exploit is now a remote one.

      It's sad that so many people on slashdot keep playing local exploits down, or keep saying things like 'well it doesn't matter if my linux mail program has a flaw - the worst that can happen if I open a dodgy attachment is they wipe out my user directory, the rest of the system is safe'. Nothing is further from the truth. It's harder, yes, but not impossible to chain a bunch of vulnerabilities together so that your local-only exploit becomes remotely accessible.

      This is why Linus doesn't like to classify bugs as security bugs vs other bugs. All bugs are potentially security bugs.

      • No, it's not just as bad (Score:4, Informative)

        by dutchwhizzman ( 817898 ) on Sunday September 04, 2011 @12:24AM (#37300838)
        It is bad, but there is a mitigation. It requires two steps in stead of just one to get root access. Given the fact that you usually try to layer your security and have logging/accounting and tripwire type of alarms set up, you have a bigger chance of catching intruders before they get access to anything really dangerous.

        If you admin thousands of systems, used by many more users, you will get compromised accounts, on a fairly regular base. Those accounts in general will be used to try and get root access. By setting up logging, accounting and various other tools, you tend to get a lot of the compromised accounts to trigger an alarm before they get root, or run their code as user. With remote root vulnerabilities, you get none.

        Any privilege escalation is something to be serious about, but crying wolf that local exploits are just as bad as remote, will make less people take you serious.

  • Feeling better (Score:5, Insightful)

    by ChatHuant ( 801522 ) on Saturday September 03, 2011 @02:57PM (#37297878)

    I was concerned about the fact that a high profile like kernel.org site was rooted, but knowing it didn't take a sophisticated and highly knowledgeable penetration team but just a group of bumbling script kiddies makes it all better.

    • Lol... bonus points for excellent use of sarcasm :)

    • Re: (Score:2)

      by drolli ( 522659 )

      And right now, somewhere... somebody of the highly knowledgeable penetration team says something like "phew i thought these stupid punks get us caught".

  • Two ways to look at this... (Score:5, Insightful)

    by CajunArson ( 465943 ) on Saturday September 03, 2011 @03:08PM (#37297924) Journal

    The first way: Haha, these skiddies didn't have what it takes to effectively hide their cracking.

    The second way: Skiddies were able to crack kernel.org using automated cracking tools just Windows, no evil genius required.

  • they should run linux on their servers, it's more secure! oh wait... whoops. ho hum, i suppose it's no good saying that they should run a FreeBSD Bastion Server, is it?

    • Now Linux is popular enough to have rootkits. This must be the year of Linux on the desktop.

    • If you have weak passwords, no OS is going to help you. The only thing that helps you there is something like DenyHosts or fail2ban. Not even OpenBSD prevents stupid.

  • This wouldn't have happened if they ran closed source Microsoft Windows Server :)

    It's a joke guys kidding :P

  • So they found the SSH Frontdoors, but did the admins find the rest?

    • Re: (Score:1)

      by Anonymous Coward

      Weather or not they found them they are gone. Nuke & reinstall. Nobody running those servers would do anything less, except perhaps restore from backup using trusted media, which is also good.

      • Disturbingly they seem to have considered not wiping and reinstalling.

        System is being verified from backups, signatures, etc. As of right
        now things look correct, however we MAY take the system down soon to do
        a full reinstall and for more invasive checking.

        (emphasis added) John 'Warthog9' Hawley
        Chief Kernel.org Administrator http://pastebin.com/BKcmMd47 [pastebin.com]

        It appears that the chief kernel.org system administrator is so naive about security that he doesn't even realize the absolute necessity of a full wipe and

        • Re: (Score:2)

          by aegl ( 1041528 )

          Perhaps considered - but rejected. www.kernel.org now says:

          "We have currently taken boxes off line to do a backup and are in the process of doing complete reinstalls."

  • Or so they think.
  • ...they were just script kiddies who knew one single method, and thought it would be cool to try it on kernel.org.
  • ...if I was in charge of damage control at kernel.org. Just sayin'.
  • I still haven't managed to figure out if the tarball you download from the main page has been compromised.. Yes GIT saved everybody and all, but they seem to not want to say anything about the front page tarball, makes me curious

    • I still haven't managed to figure out if the tarball you download from the main page has been compromised.

      It's been replaced, of course.

  • You can't conclude that from the data .. (Score:3)

    by AftanGustur ( 7715 ) on Saturday September 03, 2011 @08:47PM (#37299876) Homepage
    Usually advanced attacks are done by multiple teams. I.e. the Analysis, Compromise and exploit parts might all by done by different teams, and there may even be multiple teams for each task.

    So when you find a backdoored SSH and a Linux rootkit on your server you might only be seeing the tools from one team who got lucky.

  • Detection?? (Score:2, Interesting)

    by Anonymous Coward

    With Chkrootkit having seen its last update sometime 2009 and RK Hunter also being on the backburner, how does one even check these days for rootkits and other nasties like it? Suggestions?

  • A real risk here. (Score:3)

    by John Hasler ( 414242 ) on Sunday September 04, 2011 @10:18AM (#37302532) Homepage

    The major distributions are safe but some doofus at somewhere like Cisco or Belkin (or more likely their Chinese contractor) may have obliviously downloaded a compromised tarball and shipped it on a million routers.

Slashdot Top Deals

news: gotcha

Close