Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Red Hat Software Bug IT

Red Hat Security Update Renders Systems Unbootable (redhat.com) 88

Posted by msmash from the closer-look dept.
PAjamian writes: A recently released Red Hat update for the BootHole Vulnerability (firehose link) is causing systems to become unbootable. It is widely reported that updates to the shim, grub2 and kernel packages in RHEL and CentOS 7 and 8 are leaving various systems that use secure boot unbootable. Current recommendations are to avoid updating your system until the issue is resolved, or at least avoid updating the shim, grub2 and kernel packages. Update, shared by PAjamian: Red Hat is now recommending that users do not apply grub2, fwupd, fwupdate or shim updates until new packages are available.
This discussion has been archived. No new comments can be posted.

Red Hat Security Update Renders Systems Unbootable More

Red Hat Security Update Renders Systems Unbootable

Comments Filter:

  • Update (Score:5, Informative)

    by PAjamian ( 679137 ) on Friday July 31, 2020 @10:20AM (#60351369)

    Red Hat is now recommending that users do not apply apply grub2, fwupd, fwupdate or shim updates until new packages are available.
    https://access.redhat.com/solu... [redhat.com]

    • They have no system in place to pull bad updates?

      • Re:Update (Score:5, Interesting)

        by habig ( 12787 ) on Friday July 31, 2020 @10:40AM (#60351457) Homepage
        They can pull the bad updates from their download servers. Presumably this works for mirrors too. For systems that have already downloaded them (my auto-yum did so this morning, apparently) that's harder. What I'd do is to push the old, ok version with a bumped version number to the update servers, so such systems would get the "reverted" package on the next update. That'd be the most straightforward way to do it.

        • They can pull the bad updates from their download servers. Presumably this works for mirrors too. For systems that have already downloaded them (my auto-yum did so this morning, apparently) that's harder. What I'd do is to push the old, ok version with a bumped version number to the update servers, so such systems would get the "reverted" package on the next update. That'd be the most straightforward way to do it.

          Did you test this? I thought they pushed the new DBX update with this and in that case you can't boot the old shims. If you have secure boot disabled you're not at all affected by this.

          • Re: (Score:2)

            by habig ( 12787 )
            I'm not RH, so can't test this. Was just proposing a way RH might go about it.

      • Re: (Score:2)

        by gweihir ( 88907 )

        It is probably just for the day or so that the mirrors need to sync.

    • I think my desktop was affected by this too. (Linux Mint 20) I had auto update enabled and it applied a kernal and grub2 update. It prompted me to reboot and I got a Grub error... Luckily I had timeshift and just rolled back. I can just image how bad this would suck if someone just let the upgrade run on a server and with no easy way to rollback. :(

    • grub2 (2.02~beta2-36ubuntu3.27) [launchpad.net] xenial; urgency=medium * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc package, since we cannot be certain that it will install to the correct disk and a grub-install failure will render the system unbootable. LP: #1889556. -- Steve Langasek Thu, 30 Jul 2020 21:27:00 -0700

    • Re: (Score:3)

      by Rob Y. ( 110975 )

      I just got and installed a grub2 update on my Ubuntu desktop yesterday - and the system booted up fine today. I have secure boot disabled, though. But still - is this specific to Red Hat, or did I just get lucky?

      And, do different distros configure grub2 differently enough that one would have a problem like this and others would not be affected. Don't they all use essentially the same open source code? Or are there different versions of grub floating around that each get their own security patches?

    • Also this link:
      https://access.redhat.com/secu... [redhat.com]

  • I just updated all servers yesterday.

  • Secure boot! (Score:5, Insightful)

    by Anne Thwacks ( 531696 ) on Friday July 31, 2020 @10:30AM (#60351411)
    Well, if you can't boot - it is secure!

    • Well, if you can't boot - it is secure!

      I think the STIGs from the NSA state systems are only "secure" while they're still in the box -- inside a storage closet, inside a vault, inside a volcano (next to the recipe for Starburst candy).

      • > inside a storage closet, inside a vault, inside a volcano

        Who needs a volcano? Just put it in a disused lavatory. And don't forget the note on the door: "Beware of the leopard"

    • Also, since when do Linux systems even NEED to reboot...

      • Re: (Score:1)

        by Anonymous Coward

        How else do you test that you can recover from a failure?

      • Re: (Score:2)

        by Megane ( 129182 )
        When you trip over the power cord? Or in my case yesterday, a loose power cord rocks its way out of the socket, once at the power strip, and again at the other power strip that was plugged into, while I was cleaning up stuff around my MythTV server. One of the drives failed to come up, but it was an ancient one that I had decommissioned last year and only left on in case I needed stuff from it. I had to comment it out of fstab to get the thing running again because the poettering-ware was unhappy about it.

  • First, of course because it is not actually secure, it is DRM. Second, because it is badly designed. As everything that MS is involved in.

    • Yes, that's the common view if you have no idea about how things reall work. Apparently you have never checked out how to remove MS keys and add your. Sad...

      • Re: (Score:2)

        by gweihir ( 88907 )

        You are an idiot. I happen to be a security expert. Your opinion is no match for my expertise.

        Hint: The whole thing stays badly designed, whether I add my own keys or not. For simplicity and absolutely no real-world decrease in security, I just chose to disable it instead.

        • Experts like you are unfortunately also with my customers. After there were security problems, I can straighten them out again.

  • Ah yes, Quality Control (Score:4, Insightful)

    by bobstreo ( 1320787 ) on Friday July 31, 2020 @10:48AM (#60351485)

    Always the first casualty in the war on corporate expense.

    Seriously. How does Red Hat (IBM) get away with releasing crap like this?

    Even in my personal use, kernel updates and so always get pushed off a few days to a week because I'm kinda paranoid in general about everything.

    • I recall strongly the old RH (before they went public, before systemd) shit like this did not happen. I switched to FreeBSD a few yrs ago just to get away from the steaming pile that most distros have become.

    • Re:Ah yes, Quality Control (Score:4, Insightful)

      by bobbied ( 2522392 ) on Friday July 31, 2020 @11:08AM (#60351531)

      To be absolutely fair, Red Hat doesn't write this code you know. They just supposedly do validation of code built by others (for the most part).

      So, the Grub boot loader and kernel releases involved where likely just not validated on specific kinds of systems. It works fine for *some* systems, and apparently that includes the fleet of systems Red Hat verifies their releases on, it just doesn't work on a subset of the world's system types. You cannot test *every* possible situation.

      I don't exactly feel that Red Hat shouldn't be embarrassed about this (they should be, and apparently are), However, I also don't think it's somehow because they are being a bad corporate partner for the GNU software packages they use. I think they generally do a good job with their vetting process and I consider Red Hat to be a pretty good standard for Linux distributions. I'm guessing that Red Hat will step up their surveillance and improve their vetting process to catch such issues in the future.

      • Red Hat does in fact contribute a great deal of code to various open/free software projects including the kernel, systemd, LibreOffice, and many others. I don't know that any of that code is related to this particular issue, but Red Hat, being among the biggest Linux distributors, is far more than a mere packager of other people's work. It is a fairly solid and prolific contributor to FOSS generally.

        • Re: (Score:2)

          by DeVilla ( 4563 )
          I guess I'm not certain if Red Hat was involved with the code in questions, but when I heard about the problems with secure boot, the first name I shouted at the sky was "Matthew Garrett" who I believed worked at Red Hat when he was on LWN trying to convince the world that playing along with Microsoft's secure boot plans was a good idea for the Linux world.

          • It's easy to think in retrospect that that was the wrong choice. Maybe it was, but a case can be made that it was the better of the two bad choices available to us at the time.

            The future of hardware support for Linux wasn't nearly as clear then as it is now. The Linux community was between the proverbial rock and hard place. Play with Microsoft, or get literally locked out of most of the new systems made going forward. It was by no means clear at that time that most servers would continue to ship with t

            • Re: (Score:2)

              by DeVilla ( 4563 )

              Back then, then cloud still ran primarily on Linux. Locking out Linux would not have worked on enterprise or server grade software. And the commodity, home market wouldn't be worth the effort to engineer a special lock-in. For Linux, not supporting the Microsoft key would have been a game of Chicken that Microsoft and the vendors who implemented locking out non-MS software would have lost. Instead we validated the plan.

              Secure boot should always allow the user a way to select the keys they (dis)Trust.

              • A game of chicken for sure; I'm just not as confident that we would have won.

                At any rate, given that we are where we are, is there a good path forward from here?

    • How did they get away with releasing crap like this?

      Just following decades of leads of low quality control in software. You're paranoid about installing updates because there's a whole long history of updates being potentially very shitty.

      I even remember a time when updates came with warnings like "do not install if you are not experiencing the errors listed in the document." Which I always followed, but felt weird about -- so it's broken, it's just not broken for me and the fix might actually break somet

    • Well Red Hat business is around consulting and support agreements. Which is the dark side of the OSS business model.

      With OSS software business model, which restricts selling licenced copies, which as the vendor you can stop people from sharing the software. Also means you need to find ways to make OSS Profitable.
      Distributions: The old way and how Red Hat started. Back in the good old days where downloading a distribution even over fast connections took a long time. You were better off buying a CD prepack

      • Re: (Score:2)

        by DeVilla ( 4563 )
        I get were you are going here, and there some disincentives for making easy to use software and then leaving it alone when it work with a support business model. But by far, I believe the true "Dark Side" of OSS business is "Open Core". There are too many technologies like Docker and the like that I would rather see being run by eager college students (or grad students) than businesses who are just trying to build in hooks & dependencies for their own servers services.

    • Re: (Score:2)

      by gweihir ( 88907 )

      Seriously. How does Red Hat (IBM) get away with releasing crap like this?

      Seriously? Just look at the abysmal 3rd rated crap Microsoft has been pushing for decades now. And the incompetent masses are sucking it up and even defending it. Red Hat probably just has found out that they can make an even larger pile of money with a far worse product because their customers are stupid.

    • I expect this sort of screw-up from a company like Microsoft. But I have always held Red Hat to a higher standard.

    • Seriously. How does Red Hat (IBM) get away with releasing crap like this?

      The same way that every other imperfect human run company does, only even more so because Red Hat screw it up so rarely. Take a chill pill man. Be happy your servers are all down. Unemployment is high and people need jobs.

  • Spent quite a few hours yesterday trying to get my CentOS 8 home server back up after it would not boot after overnight auto-updates. Fortunately it's not a critical thing for me, I can imagine some admins out there probably had a really bad day though.

    Finally found the (temporary) solution to my problem on the Redhat Bug Tracker linked above. I'll be reevaluating the value of automatic updating going forward.

    • Re: (Score:2)

      by gweihir ( 88907 )

      Just do not use "secure" boot and the problems basically go away. "Secure" boot does not really secure anything anyways. It is just part of a a DRM approach to make it harder for people to do copyright infringement.

      • Just do not use "secure" boot and the problems basically go away. "Secure" boot does not really secure anything anyways. It is just part of a a DRM approach to make it harder for people to do copyright infringement.

        I'm not using Secureboot. The bug seems to affect systems with it both enabled or disabled. In my case the shimx64.efi seems to be the culprit.

        • Re: (Score:2)

          by gweihir ( 88907 )

          Then better use "classical" mode in addition. UEFI is too complex to be reliable.

          • Then better use "classical" mode in addition. UEFI is too complex to be reliable.

            This is the first issue I've had with it. At first I suspected a hardware failure, as on power on it would just sit at the HP logo. I could still enter setup though, so the first thing I did was run the UEFI diagnostic programs from an HP 4-in-1 USB key (https://www8.hp.com/us/en/campaigns/hpsupportassistant/pc-diags.html). That is the sort of thing makes UEFI worthwhile.

            Once the hardware checked out, I automatically assumed bootloader issue. I was able to boot my system using the Grub2 rescue disk ( ht [supergrubdisk.org]

    • We're having to face this right now. In the general case, only a few choices exist if you need to manage any system, but especially one directly facing the Internet. All are problematic.

      Auto-update. Then things might randomly break due to bad updates. This is true even if you only auto-update with security patches. This was a security patch.

      Don't auto-update. Then things might randomly break due to not having needed security patches.

      Download, evaluate, and manually/individually install updates as need

      • Re: (Score:2)

        by Anonymous Coward
        You missed a good method: cascade auto-update. Autoupdate a few canary servers constantly, but set cron to autoupdate all the other servers two or three days after the others, and only in the late afternoon Monday-Thursday. Critial errors like this are caught. Insidious long term errors like weekly logs going into /dev/null won't get caught, but at least you don't have to buy a duplicate for every server and five more sysadmins.

        • That's a great suggestion. We don't really have budget for "canary servers" where I'm at, but we do have plenty of VMs, and no reason we couldn't set a few of them aside for this purpose.

          Perfect world: we'd be running automated and continuous testing, including testing of logging, occasional disaster recovery, and such. But right now I'll just be happy if we can keep our customer servers running without much hassle.

    • There is no problems with auto-updating, it just lets you know which updates are available. I have all auto-upgrades disabled although. I just restrict access to the applications to sysadmins so clients can't enter new data into the system, take a snapshot or backup of the image if snapshots are not available, manually upgrade, test and put everything back online. Just rollback if anything goes wrong. This is a pretty standard scheme.

      • Yeah I'm running on bare metal at home but I'm going to build a box for VMs soon. No question snapshots are a great thing.

        • Have a look at proxmox ve, up to par with VmWare ESX or whatever it is called nowadays I would say although some functionality is only available through the command line but this shouldn't be a problem if you are used to running Linux:
          https://www.proxmox.com/ [proxmox.com]

          Free as in free beer, just change your repo for the dev one if you want to be able to update for free without any subscription. Of course, subscribe and support if you can.

    • Re: (Score:2)

      by Wolfrider ( 856 )

      --Super Grub Disk is your friend... Having one already burned and ready to boot can be a lifesaver.

      https://distrowatch.com/table.... [distrowatch.com]

  • Hey if no one can boot the systems, they can’t be hacked. Working smarter not harder. Thanks, I’ll be here all week. Tip your waitstaff

  • did centos pick the bad rpm's as well?

    • did centos pick the bad rpm's as well?

      Hmm, it's right there in the linked articles and the summary.

      No, the affected RPM's did not make it into any CentOS repos. Update away to your hearts content. Tell your boss it's totaliy OK because a random drunk, smelly idiot on the Internet told you so.

      • The CentOS repository did have new kernels, grub2 and shim rpms. FWIW, we don't do secure boot and virtuals without it are so far booting OK as we'd done a few virtuals and one physical server before the notice went out. Will hold off if new kernels are coming out again.

        • Re: (Score:2)

          by kwalker ( 1383 )

          The problem isn't the kernel (package), it's in grub2 and/or shim (more likely shim). Reverting the updates for grub2, mokutil, and shim-x64 allowed my systems to boot normally (with the new kernel).

        • The CentOS repository did have new kernels, grub2 and shim rpms. FWIW, we don't do secure boot and virtuals without it are so far booting OK as we'd done a few virtuals and one physical server before the notice went out.

          Then you already have the updates referenced above. If they didn't brick your systems then you're fine and can carry on.

          Will hold off if new kernels are coming out again.

          The new packages will be to fix this issue but may fix other issues as well. You should upgrade when they come out.

  • Not another bad update! They should switch to Linux since Microsoft keeps...oh, wait.
  • Where is the original poster getting the issues with fwupdate and fwupd? These packages are not listed anywhere in the RedHat advisory...

  • I love stories like this... (Score:5, Funny)

    by russbutton ( 675993 ) <russ@NoSPaM.russbutton.com> on Friday July 31, 2020 @01:41PM (#60352263) Homepage
    ...because I retired last year. Not my problem anymore!

    Yeeeeee-hawwww!

    • ...because I retired last year. Not my problem anymore!

      Yeeeeee-hawwww!

      I can relate. I'm retired too, but I was a network guy, so I worked mostly with Cisco, F5 and such. Still feel bad for my former co-workers though, endless bugs still keeping them busy as well.

    • Sing it, brutha!

      Congrats on getting out alive, enjoy your freedom!

  • I have two machines (desktop and laptop) that I updated regularly. I always do one at a time. If one becomes unbootable (and it happens occasioanlly), I use the other machine to download a fresh distro to install. Only once the machine is bootable, do I repeat the upgrade process on the other machine.

Slashdot Top Deals

Fundamentally, there may be no basis for anything.

Close