Red Hat Reverts Spectre Patches to Address Boot Issues (bleepingcomputer.com) 78
An anonymous reader quotes BleepingComputer:
Red Hat is releasing updates for reverting previous patches for the Spectre vulnerability (Variant 2, aka CVE-2017-5715) after customers complained that some systems were failing to boot. "Red Hat is no longer providing microcode to address Spectre, variant 2, due to instabilities introduced that are causing customer systems to not boot," the company said yesterday. "The latest microcode_ctl and linux-firmware packages are reverting these unstable microprocessor firmware changes to versions that were known to be stable and well tested, released prior to the Spectre/Meltdown embargo lift date on Jan 3rd," Red Had added.
Instead, Red Hat is recommending that each customer contact their OEM hardware provider and inquire about mitigations for CVE-2017-5715 on a per-system basis. Besides Red Hat Enterprise Linux, other RHEL-based distros like CentOS and Scientific Linux are also expected to be affected by Red Hat's decision to revert previous Spectre Variant 2 updates, so these users will also have to contact CPU/OEM vendors.
At least one site "characterized the move as Red Hat washing its hands of the responsibility to provide customers with firmware patches," writes Data Center Knowledge, arguing instead that Red Hat "isn't actually involved in writing the firmware updates. It passes the microcode created by chipmakers to its users 'as a customer convenience.'" "What I would have said if they'd asked us ahead of time is that microcode is something that CPU vendors develop," Jon Masters, chief ARM architect at Red Hat, told Data Center Knowledge in a phone interview Thursday. "It's actually an encrypted, signed binary image, so we don't have the capability, even if we wanted to produce microcode. It's a binary blob that we cannot generate. The only people who can actually generate that are the CPU vendors."
Instead, Red Hat is recommending that each customer contact their OEM hardware provider and inquire about mitigations for CVE-2017-5715 on a per-system basis. Besides Red Hat Enterprise Linux, other RHEL-based distros like CentOS and Scientific Linux are also expected to be affected by Red Hat's decision to revert previous Spectre Variant 2 updates, so these users will also have to contact CPU/OEM vendors.
At least one site "characterized the move as Red Hat washing its hands of the responsibility to provide customers with firmware patches," writes Data Center Knowledge, arguing instead that Red Hat "isn't actually involved in writing the firmware updates. It passes the microcode created by chipmakers to its users 'as a customer convenience.'" "What I would have said if they'd asked us ahead of time is that microcode is something that CPU vendors develop," Jon Masters, chief ARM architect at Red Hat, told Data Center Knowledge in a phone interview Thursday. "It's actually an encrypted, signed binary image, so we don't have the capability, even if we wanted to produce microcode. It's a binary blob that we cannot generate. The only people who can actually generate that are the CPU vendors."
Hey (Score:2)
I didn't knew it had the reboot feature!
Re: (Score:1)
The microcode is loaded on boot, so you can still boot from a rescue disk (flash drive, CD, whatever) and remove the offending microcode from the directory that it's installed in and even install the "good" microcode while you're there, then reboot and all is well once again.
Bullshit advice by RH (Score:5, Informative)
As written in the summary, the CPU maker is the only entity who can create a fixed microcode, be it for inclusion into firmware (BIOS/UEFI) files or in Linux kernel microcode files. So asking the OEM vendors won't help one bit cause they can simply do nothing at all except use a file create by Intel. If RH is too small to force Intel to create working microcode without boot up bugs, then others aren't big enough either.
Then next thing is: what will RH do in the future? Never apply the Spectre microcodes due to instability? If so, what happens in the future when other microcode updates are needed, like before? Intel has afaik only a single microcode file for Linux for pretty much all CPUs together. There is no mix and match, no way for Linux to selectively choose what to load. That is why RH had to go back to an older version without the breakage for Spectre. they couldn't just disable the new buggy part of the microcode.
This file you can see and download here: https://downloadcenter.intel.c... [intel.com]
Normally it lives in your initrd.
Sooner or later RH has to include a current microcode file from Intel in RHEL again. Would have been nice if they had clearly communicated this to their customers. Not "wash their hands" but "we will continually work with Intel until this issue is resolved."
Re:Bullshit advice by RH (Score:4, Interesting)
Actually, here is the approach we have adopted with regards to this problem; Don't apply the patches until this is sorted out, I give it about 3 months before we can make up our minds but any guess is as good as mine.
Something that critically linked to the hardware as to be tested with all available hardware and configurations which is pretty hard to do. So, as usual, let the users do the testing and the testing cycle isn't completed yet! ;-)
Re: (Score:2)
red hat didn't cause this bug, they are just an OS. why can't intel put the microcode file up for download on their website? Oh wait they did [intel.com]
Re: (Score:2)
Each file is a microcode update for one CPU.family-model-stepping, not different versions for the same CPU.
If you set MICROCODE_SIGNAURES="-S" in make.conf before you emerge sys-firmware/intel-microcode, it only installs the one for your CPU (or should have - it's slightly broken and will install all the steppings for the same family/model)
But anyhow, you can only load one file. They're signed and the CPU will refuse to load anything else, whether you use the full microcode.dat or the ff-mm-ss file.
To av
Lots of microcode files. (Score:2)
Under Oracle Linux 7.4, the command "ls /lib/firmware/intel-ucode/ | wc -l" reports 95 files in that location.
You can run "rpm -qi microcode_ctl" and "rpm -ql microcode_ctl" to get more information about what these are.
Re: (Score:1)
This looks like a case of Red Hat telling its customers to go tell their OEMs to go tell Intel they broke stuff with their microcode patch and need to fix it. It's a common strategy to take if you don't think you have enough leverage telling them yourself.
Re: (Score:2)
I imagine they will if/when Intel gives them a blob that doesn't make systems crash. I can understand their frustration, Intel hands them a blob of microcode but provides no way to debug or even understand the contents. It fails miserably and RH is left holding the bag with it's customers. They get to deal with the angry calls and the hit to reputation while Intel skates.
If they punt to the OEMs, then the OEMs get to deal with the angry calls and perhaps re-thinks Intel as a CPU source. More likely, they a
I hope that RedHat start distributing it again (Score:5, Insightful)
I understand why they stopped distributing the microcode. It is something that they cannot test fully, cannot fix and which might brick customers' hardware. If something goes wrong I expect someone to try to sue them. Distributing the microcode bring little benefit (to RedHat) but brings potential risk/cost.
The onus is really on AMD/Intel/... to test the new microcode on *every* CPU that they have produced and with a large selection of releases & configurations of operating systems (MS Windows, Linux, macOS, ...) with different support chips (RAM, USB, ...). Yes: a lot of work, but they are the ones who screwed up here. They have also known about these problems for some 6 months - so it is a complete disgrace that they have not already produced the microcode updates AND tested them.
Hopefully in a month or two, when the testing has been done properly: RedHat, Microsoft, ... will push out the microcode updates. We need these distributors, or vendors, to do so because otherwise very few machines will end up patched. Large organisations might contact the CPU vendors, most SME or home users probably don't know what the machines contain and won't think to worry about the update... which would just leave them vulnerable for when someone works out how to produce a real Spectre cracker program. It is ''when'' not ''if''.
Re: (Score:1)
The did not "stop distributing the microcode", they "rolled back to the last known stable release".
What CPU microcode is available via the microcode_ctl package to mitigate CVE-2017-5715 (variant 2)? [redhat.com]
Re: (Score:1)
>Which might brick customers' hardware
Microcode is non-persistent. It is loaded by either the motherboard or the OS at boot time. OS-loaded microcode can't brick your hardware; worst case, reinstall. A bad BIOS flash may be more problematic
Re: (Score:2)
Microcode is non-persistent. It is loaded by either the motherboard or the OS at boot time.
Most commonly either through a initramfs that loads it directly, or a udev rule that loads it when the devices are set up early in boot. /sys/devices/system/cpu/microcode/reload if there is a /lib/firmware/intel-ucode/ file that matches the family/model/stepping, or by writing the extracted firmware directly into /dev/cpu/microcode
But also after boot, by calling micocode_ctl or iucode_tool specifying what microcode update should be loaded, or signaling the kernel by writing 1 to
I prefer to do it with microc
Brick? I think not. (Score:2)
The microcode update "should not" brick hardware. The output of the command "rpm -qi microcode_ctl | tail -3" tells you why:
The microcode update is volatile and needs to be uploaded on each system boot i.e. it doesn't reflash your cpu permanently, reboot and it reverts back to the old microcode.
Re: (Score:1)
Spectre Patches Anyone? (Score:5, Funny)
Seems like there is more damage being done by the Meltdown / Spectre hysteria and attendant patch frenzy than by exploits of the vulnerabilities themselves.
Friends don't let friends Spectre patch their computers!
Re:Spectre Patches Anyone? (Score:4, Insightful)
You've got a point. The question is, how would you know if you were penetrated by Meltdown? Spectre is something that needs to be dealt with carefully. Meltdown? That's a bit more dangerous.
My personal suggestion is to start by blocking javascript execution. That's not enough, but it buys you some time. But beyond that, I don't know. I'm presuming that it will get sorted out, so buying yourself some time is valuable.
Re: (Score:2)
Re: (Score:3)
That's because the most serious vulnerabilities affect virtualized systems more, and given the popularity of the cloud, they are extremely affected because there is no control over what software is run.
These attacks may seem theoretical, and for a desktop user, they mostly are. Even server users with dedicated machines are generally safe as they own the
Re: (Score:2)
That's because the most serious vulnerabilities affect virtualized systems more, and given the popularity of the cloud, they are extremely affected because there is no control over what software is run.
These attacks may seem theoretical, and for a desktop user, they mostly are. Even server users with dedicated machines are generally safe as they own the entire machine and thus control the software on it. Virtualized hosts, not so much, and this ranges from simple VPS hosting companies, to cloud service companies like Amazon and Google. In this case, accessing not-your-memory can be beneficial, like trying to find the encryption keys used by someone sharing your machine. Cloud providers are especially scared because during high peak retail times, encryption keys may be spun to many instances to handle the load, and crafty attackers may simply spin up instances to extract those keys.
Very good points. I agree that it is the shared environments that have to take the Meltdown / Spectre threats seriously.
Translation (Score:1)
Translation: each customer of Intel better think twice next time they consider and trust said company.
Microcode updates from OSes? (Score:3)
So Linux provides the ability to update the CPU microcode during boot. Does Windows do something similar? I know that Microsoft has control over their Surface devices and often rolls out UEFI / BIOS updates via Windows updates, but does the OS itself have this functionality too?
It would seem there's a pretty big exposure if Linux is able to push out updates directly from Intel, but Microsoft says: Apply a BIOS update from your PC OEM. Last BIOS issued to my motherboard was in 2014 and it is listed as beta.
Re: (Score:3)
I did this this summer when there was a bug regarding threads for the Skylake CPUs until I had the BIOS update available.
Re: (Score:3)
Re: (Score:2)
Cheers!
I've said it before... (Score:5, Insightful)
Re: (Score:2)
Wrong, patches only released January 9 for meltdown and spectre, and it remains to be seen if all possibilities of spectre attack are even addressed
https://www.vmware.com/securit... [vmware.com]
Re: (Score:2)
Re: (Score:2)
hahaha, and VMWare has now reverted ALL the SPECTRE patches
https://kb.vmware.com/s/articl... [vmware.com]
"ESXi was already patched last fall"....BWAHAHAHA you are really a naive shill aren't you?
Re: (Score:2)
Your lazy and careless attitude is typical of a PC owner who thinks what is true for his little home devices are true for virtualized systems
Re: (Score:2)
Meanwhile, your checks, credit card transactions, banking accounts, insurance, merchant payments pass through virtualized systems. Everyone here depends on virtualized systems. 100% of the people here depend on virtualized systems, and it isn't just Intel and AMD that have speculative execution bugs, the POWER chips in IBM Z Mainframes and AIX boxes do too...
Re: (Score:2)
Even worse, IBM has only been releasing half-patches, the complete fix is due in mid-February. With the speed that Intel and AMD have released fixes, a
Re: (Score:2)
Except the actual fixes prevent booting many machines, so patches are being reverted and only partial fixes are available in Intel/AMD land.
Actually, it was realized in the mid 1990s intel's optimization had the security holes we are now calling MELTDOWN and SPECTRE, The NSA sponsored paper is "The Intel 80x86 Processor Architecture: Pitfalls for Secure Systems". No one took it seriously, is all.
Re: (Score:2)
It's not that mainstream.
Fuck me are you in for a surprise when you hear of this thing called "the cloud".
Re: (Score:2)
Re: (Score:2)
False, you are the ignorant one fearing bad press for vmware, maybe your stock will take a hit? The truth is VMware has reverted several SPECTRE and MELTDOWON patches for causing instability. All intel and AMD issues are NOT currently patched.
Re: (Score:2)
VMWare has already issued patches to isolate any infection
The VMWare "patches" are little more than the same patches that everyone else is rolling out, and requires the same microcode update. That same update that VMware has pulled [vmware.com] just like the rest of the industry has.
But go ahead, crash the delicately balanced ecosystem encompassing everything evolved over the last 40 years or so so that you can be "secure" (while phishers, etc wreak havoc through regular, working methods).
There's different risk profiles and different attack vectors for different systems. They all should be addressed as needed. Oh and for making that statement: You're a complete moron.
Re: (Score:2)
Your lazy and careless attitude is typical of a PC owner who thinks what is true for his little home devices are true for virtualized systems
Anyone who puts his data and his software on _someone else's computer_ didn't really care about its security in the first place. Otherwise he would have invested in a computer of his own, and kept everything in-house.
Re: (Score:2)
Re: (Score:2)
Dude, talk about missing the point. If your data is already encrypted anyway it's no problem if it gets sniffed by Spectre and/or Meltdown, now is it? The discussion was about when your data is potentially available for sniffing, i.e. when it is not encrypted.
Intel (Score:5, Insightful)
Wow, few paragraphs about Intel's fuckup and the culprit name isn't mentioned even once. It's Intel. Red Hat merely retracted updates developed and provided by Intel.
Re: (Score:2)
Ah, so that's why they said to contact the OEM. The OEM will know which patches (from Intel) are safe to use on their system.
Thanks god my Sgi Octane is finally stable (Score:2)
Re: (Score:2)
The R10000, 12 and 14K had speculative execution, you may be vulnerable to spectre-like attacks
Signed microcode update (Score:3)