Docker Image Insecurity 73
An anonymous reader writes Developer Jonathan Rudenberg has discovered and pointed out a glaring security hole in Docker's system. He says, "Recently while downloading an 'official' container image with Docker I saw this line: ubuntu:14.04: The image you are pulling has been verified
I assumed this referenced Docker's heavily promoted image signing system and didn't investigate further at the time. Later, while researching the cryptographic digest system that Docker tries to secure images with, I had the opportunity to explore further. What I found was a total systemic failure of all logic related to image security.
Docker's report that a downloaded image is 'verified' is based solely on the presence of a signed manifest, and Docker never verifies the image checksum from the manifest. An attacker could provide any image alongside a signed manifest. This opens the door to a number of serious vulnerabilities." Docker's lead security engineer has responded here.
I assumed this referenced Docker's heavily promoted image signing system and didn't investigate further at the time. Later, while researching the cryptographic digest system that Docker tries to secure images with, I had the opportunity to explore further. What I found was a total systemic failure of all logic related to image security.
Docker's report that a downloaded image is 'verified' is based solely on the presence of a signed manifest, and Docker never verifies the image checksum from the manifest. An attacker could provide any image alongside a signed manifest. This opens the door to a number of serious vulnerabilities." Docker's lead security engineer has responded here.
Re: (Score:1)
That is a job for his hand.
Re: (Score:1)
Read the update (Score:5, Informative)
Read the update. Pretty much the Docker team is implementing a container verification system and working through the details of decentralized security. v1 is part of the mechanism being in place. It assumes that an upstream verification is in place which is at best-semi helpful. Everyone agrees that the current system does nothing and the message is highly misleading in that it might lead someone to believe that there is a security system in place when the plumbing isn't finished.
So there is no argument here between the parties (what nothing to fight about on /.). Worth pointing out to the /. community however not to take that message seriously yet.
Re:Read the update (Score:5, Informative)
Seems as though you're giving them a free ride for a rather poorly implemented message. And this is Slashdot, where we'll fight if we feel like it.
Docker's been pretty loose and fast, and "not taking that message seriously yet" in a supposedly production environment seems a bit sophomoric.
Re: (Score:2)
I agree it is a bad idea. And they agree it is a bad idea. Not sure what we can argue about if both sides agree they screwed up with this mesage.
Re:Read the update (Score:5, Interesting)
It's nice to try to deflate this, but the blunder and the QA mistake remain. As I like to hesitate on the side of caution, I'd change this quickly. Just agreeing that one screwed up and not halting distribution for this head-desk sort of error -- in the face of the enormous security risk endowed -- isn't quite satisfactory.
I'm here to punish no one, but in a crazy sort of way, I find this one to be a bit mind-boggling, to the tune that each and every appliance that wasn't independently MD5'd is now a freaking five star security risk. Chain of authorities are tremendously important, and reasonable people would believe, mistakenly, that all is fine, when none of it now is, because the chain of authorities chain has been broken, and for what I know, from its inception.
So you're telling me to cool down, and I'm telling you that every single Docker implementation is now reasonably suspect, because of this go-lightly screw-up.
Re: (Score:3)
No Docker implementation is any worse than it was before. They went from no security to slightly better security that in practice in most install is unlikely to be useful but with a misleadingly reassuring message.
There could very well be problems since people could be letting down their guard when they shouldn't. My point is that there isn't much debate since the Docker people explained what was going on, everyone agrees that is what is going on and the Docker people agree the message everything is OK sh
Re: (Score:2)
And my point is that the chain of authorities is broken, and needs to be fixed, and anyone not doing MD5s are at risk. I would do these anyway. Lots of others don't.
Re: (Score:3)
Yes. You have to run an out of band security system at this point for Docker same as before.
Re: (Score:2)
So it seems that simply fixing the output to indicate that the filename is in the signed manifest would solve this... when the sum in the signed manifest is checked against the actual file they can change the message back (I would assume that is the reason for such a reassuring message).
Re: (Score:3)
It is a bit more complex but yes. A much better message might be something like "plumbing 2 of 4 steps functional -- passed" or even "checksum passed: note if you don't know how the Docker checksum works you probably don't have enough auxiliary plumbing for it to be working for you, so please be cautious". which would make it clear that nothing is really being tested at this point for most users.
Fixing the message (Score:2)
It would be ok if the message said "Manifest file contains correctly formatted checksum - still need to verify."
That might also give you the hint that, if no message about "checksum verified correctly" appears later, probably no verification has been done.
Re: (Score:2)
for something that runs in a fenced network and only takes deploys and runs integration tests before being destroyed, I don't see how this is a huge problem?
You mean people use docker to run production images?
Re: (Score:3)
Yes. All the time. One of the core idea of modern PaaS systems is that from an OS standpoint what runs in dev runs in test and production no configuration during migration. Google for example last year was running over a billion containers in production.
Re: (Score:2)
Docket is a container, but not all containers are docker. I'm pretty sure Google doest down load their images from docker hub.
Re: (Score:2)
Google uses multiple container systems and a lot of their infrastructure isn't Linux but Docker is a key component of their containerization strategy. If it were even 10+% Docker you talking well in excess of 100m Docker containers in production.
Re: (Score:2)
Read Before you initiate a “docker pull” [redhat.com]. Note that it contains warnings about almost everything supposedly discovered by this researcher, and it came out before it. That's why no one in Docker land is even taking this seriously. People who are surprised by this aren't aware of what's going on with Docker by definition, because if they were they'd know this is old news.
Docker 1.X is a first release with lots of known limitations--this is one of them--plus the inevitable security issues of any
Re: (Score:2)
So step 1, collect underpants?
Re: (Score:2)
Yeah sorta of, except you can't have a single repository playing the role of the distribution. So you need a vastly more complex package manager than anything we invented before, hence the wheel spinning.
Re:Read the update (Score:5, Insightful)
Upstream verification won't help. The client has to verify that the image it received is the same one the server verified, otherwise someone can hack a router to silently redirect the client to a malicious server and serve up whatever image they want alongside a copy of the signed manifest for the official image and you're fsckd. What they need is:
Step 4 is apparently what's missing from the client.
Re: (Score:2)
DevOps style production has less bugs, better reliability and much higher satisfaction than classic approaches. Turns out, ADD works. You may not like it but the research is unequivocal for complex and changing systems.
Re: (Score:2)
IBM / Rational, Forrester research, GE. I could keep going.
What is this new madness? (Score:3, Insightful)
I'm about to leave for Sears, inseam and waist measurements in hand. And here I read that my image security is at risk. I better find a new brand of pants I guess.
re (Score:2)
just another example of the "bleep'ed ed bleep" that passes for a good idea
it REALLY is time for a X30+ solar flare to kill the electricity for 10 years
then MAYBE we will have had time to well THINK FIRST!!!
and change the priories from
new and "Bleeped up"
to stable and SECURE
Re: (Score:2)
just another example of the "bleep'ed ed bleep" that passes for a good idea
it REALLY is time for a X30+ solar flare to kill the electricity for 10 years
then MAYBE we will have had time to well THINK FIRST!!!
and change the priories from
new and "Bleeped up"
to stable and SECURE
If you're interested in stable and secure, you're already not using Docker, so problem solved.
The problem with insisting that everything has to be well thought out and planned first is that gets in the way of innovation, and things slow way down while you do your planning. But while you're spending a couple years trying to plan out the project and account for every use case and vulnerability, by the time you've written the code, it's already out of date and not useful, so the planning has to start over agai
Re: (Score:2)
Hmm, why not? If you're using docker in a production environment I'd hope you're building your own images, so where is the security concern? If you're just testing out software and testing builds then sure pull away from the docker hub, but if you're deploying to production you better have your own artifact repository storing your personal generated images.
What? (Score:5, Insightful)
Don't tell us what the fuck a docker is or anything...
Dockers is a brand of khaki garments (Score:5, Funny)
Dockers is a brand of khaki garments and other accessories from Levi Strauss & Co.
It's obviously a single pant-leg or shoe, hence, Docker, not Dockers.
You're welcome.
Re: (Score:3)
And I'd be pretty insecure if mine had a hole in them.
Not even considering how my image would suffer.
Re: (Score:3)
Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications. [docker.com]
Can it be enabled? (Score:3)
Docker's report that a downloaded image is 'verified' is based solely on the presence of a signed manifest, and Docker never verifies the image checksum from the manifest.
Can it be enabled? If yes, then I do not see a problem.
Otherwise, the signing crap is just that: crap.
It takes needlessly long time to verify the signature. (Because they are not slow! - they are so secure, so very much OMG secure.)
It is a huge risk to reconfigure a production system to use unsigned data if emergency arises. (Think recovery from a local backup.)
Developers forget to renew their certificates and suddenly, in the middle of a production, whole system goes down, because OMG the certificate has expired and data may be not secure!!!
And then, in the end, the signing keys get leaked or stolen...
Love that response (Score:5, Insightful)
A summary of that wall-of-text "response" from the Docker "lead security engineer":
"Bullshit, bullshit v1 bullshit. Bullshit discussions about bullshit CVE bullshit. (yes we know its broken) Bullshit v2 bullshit, next version bullshit Bullshit."
If you can't dazzle them with your intelligence, baffle them with your bullshit.
THESE AREN'T THE DROIDS YOU'RE LOOKING FOR (Score:2)
I'm glad I'm not the only one who had that reaction. Unintelligible drivel!
Here are my favorite excerpts:
-- "There is nothing particularly new in Jonathan's post and I thank him for facilitating a conversation [about nothing particularly new apparently]."
-- "Image security is of the upmost importance to us. For these reasons, we've [reached] many of the same conclusions [that there is no image security]."
-- "v1 is not v2. v1 has a flawed design. we have a draft for v2. v2 will be better. v2 will be much mor
Re: (Score:3)
If you can't dazzle them with your intelligence, baffle them with your bullshit.
1) They have complete lack of image validation
2) Docker prints "Image validated" as part of the pull.
This is just flat out embarrassing, If I was the "lead security engineer" of that, I would actually worry about job stability.
There is not a whole lot he could say that would save face.
And yet, the response is truly terrible. He essentially says "yeap, we knew about it, shipped it anyway, and talked about fixing it"
It does not take a PR genius to figure out a good response. For example: "Thank you for
They must love xkcd (Score:2, Funny)
Docker is dead (Score:1)
Re: (Score:3)
Err. Rocket is from the guys that brought you CoreOS. CoreOS uses systemd. Not the same thing.
Summary is scaremongering (Score:2, Insightful)
Read the article, summary makes it sound as if Docker doesn't verify the checksums and it does. What his complaint is, that it verifies the checksum AFTER decompress, de-tar'ing from a HTTPS source, and only does a cursory check on the TAR file.
He complains that the check on the TAR file is imperfect, which is true, and that the act of unpacking might reveal a vulnerability in the unpacker which could compromise the machine.
So, to be clear, his proposed attack is "intercept the https source" (which is possi
Re: (Score:2)
"Read the article, summary makes it sound as if Docker doesn't verify the checksums and it does. What his complaint is, that it verifies the checksum AFTER decompress, de-tar'ing from a HTTPS source, and only does a cursory check on the TAR file."
Are you sure about that? From the description from the docker guy, it sounds like they don't verify it at all.
Docker is fucking awful (Score:1)
It's one of those stupid buzzword masturbatory things that every quarter-assed IT person thinks is going to SAVE THE WORLD but isn't really any better than a bunch of scripts except that half of it (cf. this article) isn't really implemented properly yet and lack of understanding is excused with "well, it wasn't written in-house".
C is not a memory-safe language (rubbish) (Score:2)
One fallacy means that the entire work might just be a continuous set of fallacies.
(C is only memory-unsafe if not used safely - which, given that there's very few barriers to a programmer from shooting themselves in the foot - is always a risk)
Docker images are bullshit (Score:2)
Build a base tar.gz with debootstrap and use Dockerfiles. Downloading images is insecure by design. What makes you trust the docker verification? How are they supposed to spot a backdoor in a whole big image? Build it yourself, use Dockerfiles you read first.