Debian Project Servers Compromised

Sean was one of many to pass along the bad news from the debian-announce mailing list: "Some Debian Project machines have been compromised. This is a very unfortunate incident to report about. Some Debian servers were found to have been compromised in the last 24 hours. The archive is not affected by this compromise! In particular the following machines have been affected: 'master' (Bug Tracking System), 'murphy' (mailing lists), 'gluck' (web, cvs), 'klecker' (security, non-us, web search, www-master). Some of these services are currently not available as the machines undergo close inspection. Some services have been moved to other machines (www.debian.org for example). The security archive will be verified from trusted sources before it will become available again." They were going to announce 3.0r2 this morning; they've checked it and it's unaffected but obviously they're still postponing that release.

Not on debian-announce archive

[Anonymous Coward]

The debian-announce archive [ http://lists.debian.org/debian-announce/debian-ann ounce-2003/threads.html ] doesn't list this message. Of course with the number of machines affected it's possible that the mailing list archive is somehow affected.

-JohnF

That explains

[jav1231]

Why my apt-get was failing from people.debian.org last nite. Not to mention why debian.org was down. :(

Re:Not on debian-announce archive

[cjwatson]

Yes, lists.debian.org runs on one of the compromised machines and is, er, not quite running on all cylinders just at the moment.

Re:Digital Signing of Packages?

[stevey]

MD5 sums are used for the contents of packages, but packages may only be uploaded and processed by the build system if they're correctly signed.

So yes it's not trivial to backdoor a package - unless you're already a Debian Developer...

Re:apt

[tfheen]

Which is why using something similar to ajt's apt-check-sigs [66.102.11.104]. (google cache, since people.d.o is down.)

Re:Not on debian-announce archive

[jamie]

As other readers have pointed out, that machine was apparently affected.

I got the email too, and I checked its Received: headers against a debian-announce message in my mail archives from about a year ago. They both came from the same source. So there's no way this is a hoax ...unless the murphy.debian.org machine that emailed it to me is compromised, in which case it's not an inaccurate hoax :/

Re:Not on debian-announce archive

[cjwatson]

murphy was compromised, but it's not a hoax (at least if you believe this random poster on slashdot ...).

Re:apt

[psamuels]

Of course this raises the whole issue of apt-get.

Indeed, that's one of the few areas where the Debian Project has lagged behind other distribution vendors technically - cryptographic signature verification for packages.

This infrastructure has been kind of long in coming, but as of a few months ago, you can now verify Debian package signatures with debsig-verify [debian.org]. Might I suggest everyone install and use that?

Re:Running Debian-Stable?

[wouterke]

Security is much much more than "just keeping your system up-to-date".

- accounts can be compromised
- unknown bugs may have been exploited (although that's unlikely in this particular case)
- crackers could have been cracking a developer's system, and using information they find on that developer's hard disk (ssh key, gpg key, ...) to log in to one of the servers
- also of importance in general is the competence of the administrators (which surely is *not* at the cause of the problem here).

Of course these systems are running debian stable; but that's most likely not the problem.

Re:How in the world...

[stevey]

Yes Debian's machines run Debian, this breakin wasn't anything to do with the software installed upon the box, as it was due to a password compromise.

If anything it's more embaressing that somebody lost their password than that the software wasn't up to date.

Signed announcement

[Anonymous Coward]

here [uni-stuttgart.de].

To verify it:

$ wget -O- http://cert.uni-stuttgart.de/files/fw/debian-secur ity-20031121.txt | gpg --verify

(drop the space, of course)

Assuming you trust the key it was signed with, of course...

Re:Where's the confirmation from debian people?

[tfheen]

At least cjwatson and myself are Debian developers. I wish I could say it's a hoax, but it's not. However, as you've already read: the archive doesn't seem to be compromised at all.

Re:apt

[mennucc1]

the distribution contain a Release.gpg file that is signed: so it is not possible, for example, to compromise a mirror, and it is more difficult for an intruder to compromise a single Debian package in the archive.
There is a script apt-check-sigs that will check the above signature: this is explained in the debian page on releases; [debian.org] unfortunately the link to download the script from there is down, here are two alternatives: google cache [216.239.51.104] my site (slow) [tonelli.sns.it]

Re:Signatures?

[Fembot]

yep, GPG signed... the public keys of all the developers are avalible on http://keyring.debian.org normaly, and it still appears to be up anyway. There is also a debian package which contains all the keys too

Re:Where's the confirmation from debian people?

[stevey]

--- snip here ---
This is a truthful report.

You may validate this message against the key for [e-mail].

Steve
--
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.2.3 (GNU/Linux)

owGbwMvMwCR44PyxzWd9eOcyns5PYrDfJ7EiJCOzWAGIEhVK ik pLMtJKcxSKUgvy
i0r0uLgi80sVchMrFcoSczJTEktSFUpAin NTi4sT01MVEtMTM/ OKS4CCqQrZqZUK
aflFCsXZFQ4pqUmZiXl6+UXpQCO4gktSy1 K5dHW5OuyZWUE27o M5QZDp9w6GBQtO
SLxI+1madnvjbIZVrZu0HcTnzGdY0LBFy+ hFp+fRBXM7HXcYc1 6Xj5A9DwA=
=xVtr
-----END PGP MESSAGE-----

Re:Has a Microsoft release ever been compromised?

[jamie]

"a Microsoft release has never been delayed because one of their servers were compromised."

I don't know if this delayed a release, but -- in October 2000, the news broke that Microsoft's internal network had been cracked for three months.

(Debian made this announcement in 24 hours.)

Read for yourself:

Microsoft Cracked [slashdot.org]

...the Wall Street Journal article which apparently broke the news - it's the most complete. What's known - the passwords were being sent to St. Petersburg, Russia. They probably had access for about three months.

"LONDON (CNNfn) - Hackers gained access to some of Microsoft Corp.'s essential product secrets, the world's most powerful technology company said Friday, acknowledging a security breach that is a major embarrassment for the software company..."

"The Wall Street Journal said security employees had discovered that passwords used to transfer the source code behind Microsoft's software were being sent from the company's computer network in Redmond, Washington, to an e-mail account in St. Petersburg, Russia. Microsoft said it was making sure hackers could not use the stolen source code to change commercial software used by businesses, governments and consumers."

Re:apt

[Anonymous Coward]

If you care about security, you're only using Debian stable.

If you're using stable, the only updates are security fixes and point releases. Both are annouced by signed emails before hitting the archive.

So just don't blindly update & upgrade on a whim. Instead, regularly check the annouce/security lists, and only upgrade when required. For the common case (security update), you'll also know exactly which packages apt should flag for update (you can also do the exercize for a point release, but it's more work ;-).

Of course, that's not 100% bullet-proof. The archive could be compromised so that just a security updated package is "trojaned" for example. But that's harder.

Practical exampl: this morning apt got errors and suggested upgrades never annouced for after an update. I smelled something bad and did not upgrade, waiting for just this kind of news ;-)

And I haven't been disappointed: the reaction has been quick & honest, and no harm on my side...

Not BitKeeper, CVS

[fmerenda]

Just in the interest of full details, BitKeeper was NOT compromised. The CVS bridge to BitKeeper was the software that was compromised. BitKeeper caught the problem and did not let the back door into the kernel source tree.

Nobody's asking you to trust the keyserver

[psamuels]

Then the next point of failure becomes the keyservers. How do you know you imported a good key, and that the keyserver hadn't been compromised when you did it?

PGP keyservers (unlike, say, Kerberos KDCs) are completely untrusted. Anyone can upload any key to a keyserver. And downloading a key from a keyserver implies nothing about that key.

To verify that you have a valid key, you have to rely on the web of trust. Basically, if a key is signed by someone whose key is signed by someone [recurse through however many levels you are comfortable with] whose key you have personally inspected, then the key can be assigned a trust metric based on how reliable you consider that chain of signatures to be. (Basically, how much you trust the integrity and acuity of the people controlling the chain of signatures.)

PGP and GnuPG have supported this infrastructure from Day 1. Asking people to trust an arbitrary third-party public keyserver was never in the plans.

Re:Sign, sign, sign, sign.

[fatbofh]

It isn't hard to create a key, upload it to the keyservers, and sign your backdoored glibc.

So unless you can trust the entity who signed the package, it's all moot.

Obviously, the debian project could sign the package using the Debian Package Signing Key, but you've just changed the problem from "how can an end user know that this key is worth trusting" to "how can debian know that this key is worth trusting". This is (probably) solvable, but still quite hard.

Note that the technology is easy, but the processess to back it up aren't.

Re:Signatures?

[Anonymous Coward]

I have no idea what the debian people are changing in their packaging system, but as long as new debian cds come with one public key that signs the rest of the keys on that server, and the corresponding private key is kept offline, there is nothing an attacker could do if they compromised the server. Any new keys they add would not be signed by the key that every debian user already has.

Re:Debian - maybe not so great

[asuffield]

How hard would it be to insert a little something something that gets updated on all the Debian boxes out there?

Precisely as hard as it would be on any other system, excluding those Debian boxes which actually verify the signatures before installing packages (where it would be impossible).

However, it would be noticed rapidly and suitable announcements made.

Re:Honestly...

[spektr]

I hate to say it, but Microsoft's haven't been compromised, and they're the bigger target.

Not true. [computerworld.com]

Everyone here knows if windowsupdate.microsoft.com had been compromised, people would be droning on about how it's some sort of illustration of Microsoft's security.

Their update server wasn't compromised, but the debian archive also wasn't compromised in this case. But, yes, we have to work harder to make our servers secure. And we will never reach the point were our systems will be unvulnerable. So what is your point? You complain that there aren't enough anti-oss-trolls here?

Re:Where's the confirmation from debian people?

[psgalbraith]

Martin Schulze is also in the Debian security team. He prepares a lot (most?) of the security fixes for stable.

Re:Honestly...

[Quixote]

I hate to say it, but Microsoft's haven't been compromised, and they're the bigger target.

I'd hate to say this too, since it is wrong.

Microsoft's internal network was compromised, as reported by the BBC [bbc.co.uk], and many other news agencies.

So, please do some research before welcoming your "secure" overlords...

Re:...not the archive.

[GammaTau]

How does this change the fact that Debian is just not good enough, and has compromised thousands of machines across the globe? Sheesh, the denial... This is just like the Mandrake frying standard PC hardware story.

As far as I understand, no machines apart from the several Debian computers have been compromised. Compromising a machine that hosts the central Debian APT repositories is a perfect opportunity for backdooring thousands of machines In this case, that didn't happen. "Thousands of machines across the globe" have not been compromised. I guess it's only good luck but Debian users were not affected by this security breach.

Re:Where's the confirmation from debian people?

[frenetic3]

Not to be pedantic, but the signature actually does contain a date:

gpg: Signature made 11/21/03 08:53:02 using DSA key ID CD4C0D9D

-fren

Re:apt

[pyros]

I don't think it was foolish. If you used the ISO images to do installs/upgrades, the GPG keys were obtained from there. And with [signed] md5 checksums available to verify the images, you know the GPG keys that RPM uses to verify the packages is trustworthy. Since you can have faith in the keys, you can have faith in the package. this has, in fact, long been one of the things people traditionally point out in the deb vs rpm holy war, in favor of rpm. From the comments I'm seeing, it looks like GPG checking is being added to apt in debian (the apt pacakges on freshrpms.net and fedora.us for Red Hat and Fedora Core already use rpm --checksig). I think it should be added to dpkg, so then apt can just relegate the verification to the actual package installation tool.

Re:I was wondering...

[Anonymous Coward]

Read up on Debian before asking such a rhetorical question. You become a debian "coder" (read: package manager) by finding a package or two to work on, and working on them. You don't directly affect Debian, and you're basically a lackey.

Skip ahead a year or two. If you're doing a good job (and I mean a _damn_ good job), someone might propose you become a Debian Developer. Here is where the danger lies, and I'm not sure how a DD let their password slip.

But certainly don't think that because you have some 2-year college course under your belt that you are going to jump into the think of Debian development right away.

Re:So what do we do to prevent this in the future?

[kayen_telva]

did you read ANY of the posts before trying to sound like a genius revolutionary ??

apt-secure
apt-check-sigs
not to mention they are already gpg signed

Re:Sign, sign, sign, sign.

[jemfinch]

Redhat, +1, Already doing it. -1, not failing to install if the packages don't verify.

Which is exactly the state in Debian, too.

Jeremy

Re:Honestly...

[hetairoi]

Their update server wasn't compromised

It has been before. when code red hit. [winnetmag.com] Although the link given in that article is no longer working there are plenty of screen shots of www.windowsupdate.com with 'hacked by chinese' on it out there somewhere.

You cannot blindly trust anything, from anyone. I don't care if Mom says her apple pie is just dandy I'm gonna run my own tests.

Re:Sign, sign, sign, sign.

[dondelelcaro]

how can debian know that this key is worth trusting". This is (probably) solvable, but still quite hard.

Before a Debian Developer enters the project the key they will use for signing has to be signed by another Debian Developer. You'll note that many Debian Developers are strongly connected [kjsl.com] on the various keysigning lists, so it is pretty hard for the key to be faked and verified by multiple people.

Finally, the NM process [debian.org] itself is the ultimate arbitrator of who enters Debian. A prospective developer gets evaluated by multiple people before he or she actually becomes a developer.

While still not foolproof, these techniques combined help reduce the lack of accountability and the lack of trust in the system. [Of course, in the end, you really need to go out and sign and get your key signed by a Debian Developer (or a couple) so you can join the web of trust and the strongly connected set too.]

Re:Digital Signing of Packages?

[vadim_t]

What "lots"? The worst thing that can happen is getting everybody's key revoked. Sure it could be quite a mess, but other than annoying everybody and forcing the developers to make new keys it wouldn't really accomplish anything.

A revokation key has little attractiveness, IMO. By most part, having your key rekoved doesn't stop you from communicating, nor it allows whoever got it pretend it's you. Nothing stops you from having more than one key either. You don't have to use the Debian one for everything.

Safekeeping is easy too. Print it on paper (it looks same as a PGP ASCII-armored key), store somewhere safe (put it in a bank for safekeeping) and then agree that when there's enough people who think the key should be revoked, go fetch the paper and type the key on the computer.

There's really no reason to keep them on a computer. Revoking your key isn't something you do often.