Forgot your password?
typodupeerror
Open Source The Courts Linux Your Rights Online

Free Software, a Matter of Life and Death 197

Posted by kdawson
from the serious-as-it-gets dept.
ChiefMonkeyGrinder writes "Software on medical implants is not open to scrutiny by regulatory bodies. Glyn Moody writes: 'Software with the ability to harm as well as help us in the physical world needs to be open to scrutiny to minimise safety issues. Medical devices may be the most extreme manifestation of this, but with the move of embedded software into planes, cars and other large and not-so-large devices with potentially lethal side-effects, the need to inspect software there too becomes increasingly urgent.' A new report 'Killed by Code: Software Transparency in Implantable Medical Devices' from the Software Freedom Law Center points out that, as patients grow more reliant on computerized devices, the dependability of software is a life-or-death issue. 'The need to address software vulnerability is especially pressing for Implantable Medical Devices, which are commonly used by millions of patients to treat chronic heart conditions, epilepsy, diabetes, obesity, and even depression.' Will making the source code free to scrutiny address the issue of faulty devices?"
This discussion has been archived. No new comments can be posted.

Free Software, a Matter of Life and Death

Comments Filter:
  • by fuzzyfuzzyfungus (1223518) on Tuesday July 27, 2010 @01:10PM (#33048442) Journal
    That the Pacemaker Genuine Advantage warning I got last week was a bit of a shock...
  • by guruevi (827432) <evi AT smokingcube DOT be> on Tuesday July 27, 2010 @01:11PM (#33048460) Homepage

    Dupe! This was covered a couple of days ago.

    • Re: (Score:3, Informative)

      And as people pointed out the first time around, medical devices are tested extensively before being deployed. I am an ardent free software supporter, but the safety/reliability issue is simply the wrong argument. I would say the more important argument when it comes to medical software is control -- do you really want to have a corporation that you have absolutely no control over to be in control of a device that sustains your very life? What happens if that company goes bankrupt, and the source code di
      • by fuzzyfuzzyfungus (1223518) on Tuesday July 27, 2010 @01:19PM (#33048604) Journal
        Not to worry. Authentication dongles will be available in a variety of sizes, to make insertion endurable for all our users.
      • Re: (Score:2, Insightful)

        by Anonymous Coward

        That's not specific to software-controlled devices though. If you're dependent on taking a pill every week to keep you alive and/or healthy, you're in trouble if the supply chain gets disrupted in any way.

        • Re: (Score:3, Interesting)

          by shentino (1139071)

          Running out of pills is one thing.

          Having your pace-maker shut down due to non-compliance with an EULA is quite another.

          Sure, corporations can make a killing...but it will come with a murder conviction.

          I seriously doubt it would EVER be legal to remotely disable a pace-maker.

          • by jgagnon (1663075)

            "We didn't sell you that pacemaker, we leased it to you. You read the EULA before it was inserted, right?"

          • I seriously doubt it would EVER be legal to remotely disable a pace-maker.

            Why do you doubt that? "Our licensing is reasonable and non-discriminatory."

            Do you also doubt that a pacemaker manufacturer would refuse to provide a critical software update unless each pacemaker user pays them for it?

            • Re: (Score:3, Insightful)

              by bberens (965711)

              Do you also doubt that a pacemaker manufacturer would refuse to provide a critical software update unless each pacemaker user pays them for it?

              Wait, do you mean before or after I talked to Action 9 news?

          • by vlm (69642)

            Sure, corporations can make a killing...but it will come with a murder conviction.

            I seriously doubt it would EVER be legal to remotely disable a pace-maker.

            Kind of like it would be illegal for an insurance company not to cover all prescription medicines, or for a pharm company to ever stop manufacturing a pill?

      • by kipd (1593207) on Tuesday July 27, 2010 @01:47PM (#33048932)
        Yes... No bugs, thoroughly tested: http://www.ccnr.org/fatal_dose.html [ccnr.org]
        • by mea37 (1201159)

          Hmm... so by posting a ilnk to a single instance of a medical device on which testing was not sufficient, the point to which you're drawing attention is that no bug has ever escaped the scrutiny that goes with an open source philosophy?

          No significant system can be guaranteed bug free, regardless of the number of eyes on the source code. There are defined standards for safety of medical equipment. Would the overall safety be greater if you added "many eyeballs" to the testing already required? Maybe, mayb

          • And if OSS is so immune to bugs, how come Firefox has so many problems? How come they patch tons of security updates with each release? This isn't just an OSS project, it is a popular, well funded one. And it has bugs left and right.

            So, if OSS is so good, what's going on?

            Or maybe, just maybe, is the real trick the quality of the review and the methods for writing code, which can be done closed or open?

      • by digitig (1056110)

        And as people pointed out the first time around, medical devices are tested extensively before being deployed.

        Not just tested. Safety-of-life systems are usually subject to extensive checks on the development process and in the most significant cases to extensive static analysis. Even if you opened the code up to inspection it would be pretty much useless without all of the design documentation and all the other documentation that explains why the code is sufficiently safe. I don't see how the free software model gives any significant safety advantage; all that would happen is the developers would get bogged down b

      • by Draek (916851)

        Hell, what if some third-party finds your pacemaker infringes on their patent and demand monthly royalties from you? replacing, say, a pacemaker is a *bit* harder than uninstalling your copy of ffmpeg.

      • by loshwomp (468955)

        And as people pointed out the first time around, medical devices are tested extensively before being deployed. I am an ardent free software supporter, but the safety/reliability issue is simply the wrong argument. I would say the more important argument when it comes to medical software is control -- do you really want to have a corporation that you have absolutely no control over to be in control of a device that sustains your very life? What happens if that company goes bankrupt, and the source code dies with the company? What if they decide they want to start charging people a yearly fee for using their pacemakers (a situation that does not seem too far fetched, given what I have seen proprietary software companies do in the past)?

        I'd say those are all safety and reliability issues.

      • by ceoyoyo (59147)

        If any of those scenarios are possible they are possible with hardware just as easily as with software.

      • Re: (Score:3, Interesting)

        by mcgrew (92797) *

        do you really want to have a corporation that you have absolutely no control over to be in control of a device that sustains your very life?

        I'm glad my eye implant [slashdot.org] is mechanical and doesn't have any software.

        medical devices are tested extensively before being deployed

        I'm damned glad of that, too, but as it was only approved three years before I had the surgery, I do worry that the struts might break twenty or thirty years down the road. OTOH, I wouldn't have wanted to wait until I was 70 to have it; it's an

      • Re: (Score:3, Insightful)

        by westlake (615356)

        Do you really want to have a corporation that you have absolutely no control over to be in control of a device that sustains your very life?

        It can't be any other way.

        The development, testing, and licensing of the device could cost ten million dollars, a hundred million. There is no upper limit - and any company taking over the production and distribution of the device is going to see costs on the same scale.

        There simply aren't very many companies with the strength and experience to do that.

      • Re: (Score:3, Insightful)

        by DerekLyons (302214)

        I would say the more important argument when it comes to medical software is control -- do you really want to have a corporation that you have absolutely no control over to be in control of a device that sustains your very life?

        And how is that worse having a group of random self appointed individuals, over whom I have absolutely no control, in control of a device that sustains my very life?

        What happens if that company goes bankrupt, and the source code dies with the company?

        From the number of FOSS p

  • Makes sense (Score:5, Insightful)

    by MBGMorden (803437) on Tuesday July 27, 2010 @01:14PM (#33048496)

    To me, this is just common sense. This code doesn't necessarily have to be FL/OSS in my mind - let them keep the copyright, but it most definitely should have code available for public review. Would you be willing to take a new wonderdrug where the drug company won't tell anyone what's actually in it, but assures you that it'll work? If they must disclose the formula to their drugs, then they ought to be required to disclose the code to their software. Let existing laws like copyright ensure that no one else uses it.

    • Re: (Score:3, Insightful)

      Except that the mechanisms behind many of the drugs we use are not fully understood by the companies that make those drugs. They only disclose the chemical formula behind the drugs, not the logic of why that particular chemical works the way it does.
      • by MBGMorden (803437)

        While that presents some difficulty, it's still not that bad. IMHO, that's akin to releasing the code with the comments stripped out. Sure, it makes testing more difficult, but it does ensure that you can reproduce their pill/executable to run tests on yourself.

        • by digitig (1056110)
          The number of people in this discussion who think that safety is a matter of testing seems to me to be a pretty good indicator of why the idea won't work.
    • by ceoyoyo (59147)

      In that case the engineering drawings for a 777 (or anything else) should also be open to public scrutiny. Is that reasonable?

      • Re: (Score:3, Insightful)

        by MBGMorden (803437)

        Some level of documentation for such things will be available. How do you think A&P's all over the country work on them? Just pop the hood and figure it out as they go along?

        And yes, I think that anything on which the safety of a life depends should be open to scrutiny. Alarm clocks and keyboards? Not so much.

        • Re: (Score:3, Interesting)

          by Blakey Rat (99501)

          The cheap Chinese copies made from the "open to scrutiny" plans are probably not going to be as safe as the Boeing originals.

        • Re: (Score:3, Insightful)

          by JWSmythe (446288)

          It's the same argument that an automobile manufacturer doesn't release the detailed specs of a vehicle, because the owners manual doesn't show a breakdown of the engine. They are available (for a price, of course) to the people that need the information.

          Here's the list of manuals for a Boeing 777 [boeing.com].

          But for both aircraft and auto manufacturers, I don't believe they release detailed specs of say the software that makes their vehicles work. I doubt A&P mechanic

          • by ceoyoyo (59147)

            Mechanics also don't fix the cruise control hardware. They throw out the unit and buy a new one. Ditto with pacemakers and 777s. If the part doesn't work, you get a new one. The provided information usually only describes the system down to the level of replaceable components, not below. The software in each of those replaceable components is part of the black box.

            Did your computer come with the circuit diagram for the motherboard (the Apple II did)? Suppose something goes wrong and you need to fix it

            • Re: (Score:3, Interesting)

              by JWSmythe (446288)

              I almost totally agree. Most parts are made to be replaced as a module. When's the last time you replaced the bearings in a hard drive, or soldered a pin on a memory stick? But, sometimes we do, and that's without the assistance of technical schematics.

              A few months ago, I was given a hard drive to fix. The machine had been hit by a power surge, and a chip on the control board (the one on the bottom of the hard drive) melted. I replaced the board, and the drive started work

        • by ceoyoyo (59147)

          I very much doubt the full engineering documentation is available, and it is almost never available publicly. Many critical systems manufacturers will already make some parts of the software specifications, testing methodology, troubleshooting guides, etc. available to people authorized to make repairs or do troubleshooting.

      • by trout007 (975317)
        Here is one of those "when I was a kid" stories. Back in the old days when you wanted to safely control a machine you used pneumatic logic. You had the same logic operators like and/or delays,ect. So your sensor would be a little valve that when pressed would cause air to flow and activate the circuit. Then relay logic took over which was much faster and allowed the addition of electronic sensors like micro switches. Then we had Programmable Logic Controllers. These were computers but not general purpose. T
    • by LWATCDR (28044)

      Maybe but this problem has already been solved.
      The aerospace industry has been flying code that can kill for decades. They have a procedures to develop and test mission critical code for everything from navigation to flight control systems.
      Is it prefect? No but the system does seem to work well. Just base the certification process for medical software on the certification process for aerospace software and you have a good working solution.

    • by Snowmit (704081)

      What you're describing is patents on software.

      • by MBGMorden (803437)

        No, I said copyrights. Patents are a completely different animal. I personally see no need to protect the manufacturer's algorithm on detection of a particular trait, even if it could be reimplemented in different code. And yes, I feel that that algorithm should be publicly reviewable if someone's life depends on it.

    • Re: (Score:3, Informative)

      by Hatta (162192)

      This code doesn't necessarily have to be FL/OSS in my mind - let them keep the copyright

      Authors of open source software retain their copyright.

      • by MBGMorden (803437)

        True. Poor choice of wording. Authors of open source code keep their copyright, but it really only applies to the opportunity to close the code again. Any version that has been legitimately released under the GPL and makes it into the wild can't be retracted. Others could still re-release the code again with modifications. That ability doesn't need to be there at in in this case. The point here isn't to promote or enable derivative works, but rather to ensure safety and security of the code.

    • This is what the FDA (in the USA) is for. There are similar organizations in other countries.

      I do quality assurance for a medical device manufacturer. Anybody who has dealt with medical devices and the FDA knows that you don't mess with the FDA. Period. They take public safety very, very seriously.

      Opening code to the public is not necessarily the best way to do this. I can't say it's definitively a BAD thing, but I can see it causing more issues than it's worth. If everybody knows how the device works

  • The thing about this is that this is not really a question about badly written software. I think the current regulatory system provides a high enough level of protection against badly written software that making the software open source would not add a significant amount of increased security. However, a greater concern is the possibility that someone could insert code with specific triggers which could be used for malicious purposes. It is not that I believe that they would, it is that the implications fo
    • by Fallon (33975)
      <quote>The thing about this is that this is not really a question about badly written software. I think the current regulatory system provides a high enough level of protection against badly written software that making the software open source would not add a significant amount of increased security. However, a greater concern is the possibility that someone could insert code with specific triggers which could be used for malicious purposes. It is not that I believe that they would, it is that the im
      • by bws111 (1216812)

        This is exactly why the code should not be 'open'. You just wind up with hysterics like yours, and very little in the way of actually useful information. Did it occur to you that security may be way down on the list of concerns when making such a device? Actual real-world things are somewhat more important. Like battery life. Trying to maintain communications under all circumstances. Heat generated. Reliability.

        Yes, theoretically someone could kill your friends by hacking their insulin pumps. Or, th

        • by Yetihehe (971185)
          Yeah, but typically not by passing by in a place with many people with a radio in pocket. Plus no one would notice it immediately and the user would just collapse some time after.
        • You are right that there are other things that are more important than security in the design and development of these devices and those things were rightly solved first. Now that those things have mostly been solved it is time to start thinking about security and to start fixing the security issues that were left open while the more important issues were resolved.
          Most of the other ways that someone could kill another person leave significant evidence as to who committed the crime. Right now, there is ver
          • by bws111 (1216812)

            The lack of security is not a flaw, it is a design choice. The company (and regulators) know very well that the device is insecure. What could possibly be gained by exposing that situation to the world?

            Secondly, you can't magically add security and not affect one of the other things. Encryption is compute-intensive, which means more heat, less battery life, and more complexity. Now, maybe if new longer life batteries come along you could add security and not sacrifice battery life. Or you could say 'wh

    • by Darinbob (1142669)
      I've worked many years on medical devices (not implantable). The FDA has a strong regulatory oversight, as long as many other regulatory bodies (international as well). This includes scouring through bug lists, reviewing QA procedures, etc. And these were relatively safe and benign devices.
      • And in what way would any of the various regulatory agencies catch an intentional backdoor left in one of these devices? Or a kill switch?
        • by bws111 (1216812)

          What would be the point of putting an intentional back door or kill switch in a medical device?

          Once you have reached your level of paranoia you should know that just because you are looking at SOME source code does not mean it exactly matches the binary in your device. There could always be something in the tool chain that inserts backdoors and kill switches independent of the source code. So unless you are planning on building your own tool chain from scratch, inspecting, compiling, and actually installi

  • open to allow analysis

    and maybe free as in reusable to improve it,

    but not necessarily free as in beer!

  • People will die from shortcomings of technology, whether the tech is FOSS or not. Capitalism being the state religion in most of the developed world, I think it's safe to say that proprietary software will be with us forever. Fortunately, idealism is still a common enough human trait we can say the same of FOSS. One isn't going to "win" over the other, ever. We will simply continue to have both. And that's as it should be.
    • Personally, I would not be very comfortable if a company I have absolutely no control over has control over the software that runs the pacemaker in my chest. What if they decide to start charging a yearly fee for their pacemaker software? What if they refuse to provide critical software updates unless I fork over more money to them?

      Medical software should be libre, there is just no arguing that. When it comes to software that sustains a person's life, no single corporation or entity of any sort should
      • by bws111 (1216812)

        Their software makes you 'uncomfortable'? Well, don't use it - that should ease your discomfort. OK, you may be dead, but hey, at least those greedy bastards aren't getting any of YOUR money.

      • by digitig (1056110)

        Personally, I would not be very comfortable if a company I have absolutely no control over has control over the software that runs the pacemaker in my chest.

        If you needed one, you'd be a damn site less comfortable without the pacemaker. Briefly.

        What if they decide to start charging a yearly fee for their pacemaker software?

        How would they enforce it? The things are not connected to the internet, you know.

        What if they refuse to provide critical software updates unless I fork over more money to them?

        Er -- do you actually know what a pacemaker is? What would comprise a "critical software update"? The whole thing gets replaced every few years, anyway, because of battery life.

    • by digitig (1056110)

      People will die from shortcomings of technology, whether the tech is FOSS or not.

      And since we're talking about medical implants here, even more people will die without the technology. That's no reason not to try to improve the safety of the implants, but we need to keep it in perspective.

  • by stagg (1606187) on Tuesday July 27, 2010 @01:23PM (#33048664)
    I'm a big fan of FOSS, but I've got my QA methodology hat on right now. Opening up the source code isn't providing better Quality Assurance coverage, it's just getting more eyes on the code, and at best a bunch of User Acceptance Testing. (Though clearly not with pace makers, I doubt people are going to line up for that Beta test. Unless maybe Blizzard claims it's part of their next big MMO.) This could be as much an argument for higher standards of quality assurance as for open source software. In face, hell, I can see companies opening up the source to reduce their liability and cut the costs of their own QA.
    • From what I've seen, most FOSS projects seem to skip the QA step unless it's a dual licensed product with a company behind it.

      • by stagg (1606187)
        Although that's a legitimate concern, and relates to my suspicion that Open Source is a bit of a red herring here, I don't think it's always true. I work for an institution that has both outsourced and done in house QA on open source projects before implementing them. Some of the larger Open Source projects have had extremely extensive QA work done and have great community support for it. And of course, some do not.
  • Operator Error (Score:3, Insightful)

    by Darkness404 (1287218) on Tuesday July 27, 2010 @01:27PM (#33048710)
    Even the best software can go completely wrong with the wrong person operating it.
  • Double-edged sword (Score:3, Interesting)

    by gmueckl (950314) on Tuesday July 27, 2010 @01:37PM (#33048820)

    Making the code available for scrutiny is a double-edged sword. Sure, it might help to catch critical bugs in the software faster. But with some devices, it really raises a host of new issues. Some pacemakers out there are configurable wirelessly once they are in the patient's body. This is actually a very critical feature. But do you want to risk everyone being able to reverse-engineer the protocol used for adjusting the settings for such a device? A wrongly configured pacemaker can be deadly. Right now these things are fairly secure because they are rather rare and not interesting enough as targets for hacking.

    Besides, proving that some piece of code works as intended (or at least fails gracefully in all circumstances) in an essentially uncontrolled environment is quite a feat. Embedded equipment is hard to service, has to have a longer hardware lifespan than normal hardware, is often custom designed for a single application and thus may have subtle hardware defects not reproducible on similar test systems... oh, and who says that the compiler or even the CPU is bug free? This is all common knowledge around here, but I just want to give the full list. What this means is that just opening the source code might not cut it. The validation would have to be performed on the hardware it was designed to run. So, where's the call to open up the hardware design and documentation to public scrutiny as well?

    • by Hatta (162192) on Tuesday July 27, 2010 @02:15PM (#33049272) Journal

      But do you want to risk everyone being able to reverse-engineer the protocol used for adjusting the settings for such a device?

      Yes. Security through obscurity is essentially no security at all. The only thing that should be secret is the private encryption key that is uniquely associated with the remote control, which should be under strict physical security at all times.

      What you say? There's no encryption implemented in these devices? That's a big problem whether the code is open or not.

      • by ceoyoyo (59147)

        Security through obscurity by itself is to be avoided. Obscurity does add security to an otherwise secure system. Ask a secure facility sometime if you can have a copy of their guard patrol schedule.

        Giving up the extra security requires something in return. It's far from clear that the extra bug-finding ability (if there actually is any) that is associated with open code is enough to be worth it.

    • by steelfood (895457)

      You know, that's just security by obscurity, and you know what they say about that...

  • by Ihlosi (895663) on Tuesday July 27, 2010 @01:41PM (#33048864)
    ... if you don't know the hardware it runs on and the external circuitry.

    Really. Finding security holes in software that runs on a plain vanilla PC is one thing, finding the cause of glitches in the nanosecond range on an embedded system is another thing entirely.

    • by gknoy (899301)

      No, but it makes it easier for an auditing body to do so, or for competitors to point out (and prove) that their device is safer.

      • by Ihlosi (895663)
        No, but it makes it easier for an auditing body to do so,

        Official auditing bodies could have the source code any time they ask for. They don't.

        , or for competitors to point out (and prove) that their device is safer.

        ... which still doesn't help you a lot if you don't know _their_ hardware. Your software might malfunction in one out of 2^21 cases due to some obscure bug, but their hardware could go up in flames the second you look at it the wrong way.

    • Re: (Score:2, Insightful)

      by Dribbitz (239455)

      ^THIS

      Implantable pacing devices, cardioverters, and pumps (life-sustaining devices) depend on complex custom hardware designs as their platform, and that hardware is *highly* interactive with the software. Many of these devices can only achieve their miraculous longevities on a primary cell by deferring functions to hardware. If you don't have access to the information re: the hardware, the code itself might as well be inscriptions in Atlantean glyphs. You'd have to bust trade-secret protection to get a

  • i will overclock it. Live fast. Die young.
  • by ChipMonk (711367) on Tuesday July 27, 2010 @01:58PM (#33049070) Journal
    The Therac-20 radiation therapy device worked reasonably well. Despite the software flaws, the hardware safeties in place prevented any deadly accidents. Problem is, because of the hardware safeties, nobody knew just how bad the software was. It had never been formally verified.

    Then some numbskull decided, "Hey, let's let the software handle the safety interlocking, and we can cut down on hardware manufacturing costs!" The result was the Therac-25 [wikipedia.org], which maimed and killed people.

    After the machine was recalled, someone finally sat down and did a real analysis of the code, and found a whole raft of problems and bad assumptions. Nancy Leveson wrote the definitive report [mit.edu] (PDF) on the failures in the R&D processes that made the Therac-25 so deadly.

    Yet, armed with this warning (among many others), both manufacturers and purchasers keep human lives as transactions on a double-entry ledger. It simply comes down to, how many deaths per thousand uses are "acceptable"? Manufacturers and medical facilities already have so many costs. Is it worth it to add on the cost of formal code analysis?

    But nobody will ask the Therac-25 victims and their families.

    I decided early on in my I.T. career, that I didn't want the stress of people's lives depending on my correct code. I hadn't had any training in formal verification. In hindsight, I see my worries would have come from incompetent management, more than from myself.
    • by ceoyoyo (59147)

      So what? Engineering hardware is just the same. The Pinto is the classic example.

  • Some mechanical devices and most bridges and buildings require licensed engineers or architects to put their stamp of approval on the designs. They do not require publication of the engineering or architectural drawings though.

    I for one would welcome professional licensing for certain "it can kill you if it goes wrong" software, particularly in isolated devices whose software can't be tampered with undetectably.

    If a licensed Professional Software Engineer puts his seal on a pacemaker or airplane, and the s

  • This is simply one more instance of a larger problem. I once heard a brilliant observation; "If uncertain, you can always discover the game you're actually playing (vs. the one you think you're play) by whichever game you happen to be winning." Our society is predicated on a game called "PROFIT". Period, end of discussion. There are a lot of other games... satisfaction, fulfillment, sustainability, quality of life, or even personal freedom. All of these other games we say we're playing. All these other game

The ideal voice for radio may be defined as showing no substance, no sex, no owner, and a message of importance for every housewife. -- Harry V. Wade

Working...