Ubuntu 17.10 Temporarily Pulled Due To A BIOS Corrupting Problem (phoronix.com) 167
An anonymous reader writes: Canonical has temporarily pulled the download links for Ubuntu 17.10 "Artful Aardvark" from the Ubuntu website due to ongoing reports of some laptops finding their BIOS corrupted after installing this latest Ubuntu release. The issue is appearing most frequently with Lenovo laptops but there are also reports of issues with other laptop vendors as well. This issue appears to stem from the Intel SPI driver in the 17.10's Linux 4.13 kernel corrupting the BIOS for a select number of laptop motherboards. Canonical is aware of this issue and is planning to disable the Intel SPI drivers in their kernel builds. Canonical's hardware enablement team has already verified this works around the problem, but doesn't provide any benefit if your BIOS is already corrupted.
That ain't right! (Score:4, Insightful)
How the hell have we let what's supposed to be ROM get so fucked up by a simple software upgrade? That, Ain't, Right!
Re: (Score:2)
It's been EEPROM for decades. And if you expose an interface for updates, that interface is an exposed risk. Especially in the driver for the SPI bus that actually does the writing.
Re: (Score:2, Insightful)
You don't need to overwrite the EEPROM to cause problems. In this case, since it's caused by the SPI driver, it looks like corrupt data is being written to the volatile memory used by the BIOS to save settings. This is battery backed memory that can be reset, but it's a bit trickier in laptops.
Re: (Score:3)
In this case, at least one person involved was unable to resolve it by replacing the CMOS battery:
https://bugs.launchpad.net/ubu... [launchpad.net]
Re: (Score:2)
I wonder if they did a half-hour+ "thorough reset", or just a quick power removal. I.e. did they just get unlucky that a "soft" reset left some bad data behind in volatile memory, or if the EEPROM itself was modified.
I miss the old days when you had to physically move a motherboard jumper if you wanted to modify the EEPROM. Almost as much as I miss the existence of hardware-enforced write protection toggles on removable media. (My first USB drive had one - I don't think I've even seen one since, though I'
Re: (Score:2)
I miss the old days when you had to physically move a motherboard jumper if you wanted to modify the EEPROM.
There are still PC motherboards with a write enable jumper, but they are pretty much all server boards. Remember server cases with keylock switches? They're still out there...
Re: (Score:2)
A half-hour? Just drain the caps with a few presses of the power button.
Re: (Score:2)
That almost always works. Unfortunately, charges can linger in the chips themselves as well. Rare, but can cause no end of headaches when the wrong bit(s) remain set and you don't realize that's possible.
Re: (Score:2)
I miss the old days when you had to physically move a motherboard jumper if you wanted to modify the EEPROM.
Just as I miss the days of setting master/slave on IDE drives or bus positions on SCSI disks, or IRQ numbers of IO/SB cards.
Those days.
Re: (Score:2)
Really? Was there an advantage to those that I'm missing compared to modern solutions?
Writing to EEPROM is typically a very rare activity - to the point that any particular write has a good chance of having been unintentional/unauthorized by the owner. Seriously - how many people do you know who actually, for example, update their motherboard's BIOS? And of those who do - how many do so for reasons that haven't already cost them enough problem-solving effort that the added effort of changing a jumper is tr
Re: (Score:2)
Really? Was there an advantage to those that I'm missing compared to modern solutions?
Job security.
Re: (Score:2)
Did you mean to reply to my comment with this? I didn't mention anything about BIOS. Just that I missed the hardware that needed IRC and bus position jumpers, the same way that I miss Turbo C and Turbo Pascal. Those were good days.
Re: (Score:2)
Yeah, I thought you were being a smartass, comparing the headaches around the hardware you mentioned to the nuisance of an EEPROM protection jumper.
My apologies. Those where good days.
Re: (Score:2)
I've looked at those before, and it does look like a nice drive, but has tons of features I don't have any use for and is ridiculously expensive ($45 for an 8GB drive?). Meanwhile a $5 16GB drive does everything I need, except for lacking a $0.10 write protect switch, and is cheap enough that I don't care about losing it.
Re: (Score:1)
The problem is likely that USB drives with hardware write locks are made in much smaller production runs so the price is higher. It just comes with the territory for niche use cases.
Re: (Score:2)
But why is write-lock a niche use case? In the floppy days everybody I knew used the lock tabs on a regular basis - and if anything the reasons for doing so have increased dramatically. Why should you risk getting infected by whatever's on your friend/coworkers/etc. computer just to give them a copy of some file or other? Or risk your backups when you want to retrieve something? Same thing with early USB drives - using the write-lock was just common sense. Then they just sort of disappeared.
I also suspect a
Re: (Score:1)
That doesn't sound possible. Write protection on a USB is still done in software, and that particular product is a *black box*.
Re: (Score:2)
Don't forget disconnecting from power and leaving it switched ON for an extended period, to make sure to drain any lingering charge. That's the bit many people often forget, since it usually doesn't matter, except when the ghost of random chance decides to $#@! with your day.
Re: (Score:2)
Don't forget disconnecting from power and leaving it switched ON for an extended period
Or, you know, push the power button once and see the fans consuming what little remains in the capacitors so you don't have to wait for an extended period
Re: (Score:2)
That drains the power capacitors - but not necessarily charge stored in the electronics themselves. In DRAM for example, every bit is implemented as an independent capacitor. Other devices, even those that aren't intended to act as capacitors may also be capable of storing an internal charge with unpredictable lingering effects.
Re: Short terminals after battery removal (Score:1)
Re: (Score:2)
No, SRAM uses a bistable flip-flop which is stable so long as power is supplied, an - DRAM actually needs an external memory refresh circuit to regularly read/write the stored data before the capacitors discharge so much that 1s and 0s can't be reliably distinguished. https://en.wikipedia.org/wiki/... [wikipedia.org]
Re: (Score:2)
Sadly, I recently discovered that those only offer software-based protection - i.e. they tell the computer "don't write to me", but rely on the software to honor that request, and don't actually do anything to protect your data from a malfunctioning or compromised computer. Unlike write-protect tabs on floppy disks for example, where the drive would simply return an error to the OS if you tried to write to a protected disk.
Re: (Score:2)
Re: That ain't right! (Score:1)
Re: (Score:1)
Re: (Score:1)
I can only read when proper paragraphs are used, you insensitive clod!
Re: That ain't right! (Score:2)
Nice wallotext!
which is to blame Linux or Lenovo? (Score:2)
Re:which is to blame Linux or Lenovo? (Score:5, Informative)
From the Canonical bug report [launchpad.net]:
At least on Lenovo Thinkpad Yoga, the BIOS seems to monitor the SPI-NOR
write protection bit and if it is flipped to read/write it assumes the
BIOS configuration was changed on next reboot. It then, for unknown
reasons, resets the BIOS settings back to default.
Re:which is to blame Linux or Lenovo? (Score:5, Interesting)
Lenovo need to stop people writing the Bios because otherwise they'd able to remove the crapware Lenovo put in the Bios to stop people removing the crapware they put in the Windows by installing a fresh Windows image.
With an unmodified Lenovo Bios the crapware will be re-installed via Windows Platform Binary Table
https://www.howtogeek.com/2263... [howtogeek.com]
Beginning with Windows 8, a PC manufacturer can embed a program - a Windows .exe file, essentially - in the PC's UEFI firmware. This is stored in the "Windows Platform Binary Table" (WPBT) section of the UEFI firmware. Whenever Windows boots, it looks at the UEFI firmware for this program, copies it from the firmware to the operating system drive, and runs it. Windows itself provides no way to stop this from happening. If the manufacturer's UEFI firmware offers it up, Windows will run it without question.
Were it not for this Bios resetting feature - a ludicrously determined user could do the following
1) Remove Windows
2) Use some other OS to dump the Bios out
3) Hack said dump to mess up the Windows Platform Binary Table and reflash it
4) Reinstall Windows from an image
And then they'd have a copy of Windows with no Lenovo Service Engine installed! The horror! Instead it seems like Lenovo have had the Bios reset itself to stop step 3), so the determined user would still have LSE installed.
Re:which is to blame Linux or Lenovo? (Score:4, Interesting)
Re: (Score:2)
Probably some UEFI configuration utility that sets write and then some sort of staged config change update area.
Presumably if the system does not have a new bios, it then goes to process what turns out to be an empty changeset, which was not a case they anticipated, and hitting some weird fallback to recover from invalid utility behavior.
Interestingly enough Win10 1709 did this (Score:5, Interesting)
I have a Lenovo Yoga which I dual boot with Linux Mint Sarah and just after I installed 1709 Fall Creators Update the fingerprint reader stopped working. I gave up trying to remedy it and reset Windows, but that didn't fix it.
I then realised that it wasn't shutting down properly either. ACPI shutdown in both OSs would leave it halted but powered on, so the only way to restart was to hold the power button to kill it, and then switch on.
I finally checked the warranty and saw it was 14 months old so took it apart, removed the battery and motherboard battery, left it for an hour, powered it on, flattened the partition table an reinstalled. Works perfectly again, but after a huge amount of time wasted.
So, it's either a coincidence, or there's something modern OSs are trying to do which screw up BIOSs.
Re: Interestingly enough Win10 1709 did this (Score:2)
Maybe, but I was pretty lax about keeping Linux up to date, and hadn't used it a great deal so the configuration was unlikely to have changed recently.
Re: (Score:2)
It's probably the way the fingerprint reader integrated with Windows logon. The upgrade from 7 to 10 outright broke several laptops in a way that required a format/reinstall due to completely corrupting the login system to the point where neither using a password nor a password reset solved anything.
Re: Interestingly enough Win10 1709 did this (Score:2)
Oh yes, and along with it not powering down, it would also take about 35 seconds from lights on to STARTING the POST check. Multiple bios reload defaults also had no effect. Just the "Kill the power" option.
That's not the same problem (Score:4, Informative)
I finally checked the warranty and saw it was 14 months old so took it apart, removed the battery and motherboard battery, left it for an hour, powered it on, flattened the partition table an reinstalled. Works perfectly again, but after a huge amount of time wasted.
If removing the batteries solved the problem, then your problem wasn't BIOS. It was CMOS. The "CMOS RAM" is the volatile memory which stores settings used by the BIOS. The BIOS itself is stored in non-volatile memory. This was originally a PROM in the IBM PC, but today is pretty much always Flash ROM. Sometimes it's a flat quad package, sometimes it's a serial 8-pin DIP and it has to be shadowed into RAM just to function, but it's pretty much always flash. (Those ones are cool because they're usually socketed, and SPI-interfaced, which means they're useful for other projects...)
Re: (Score:1)
At least it wasn’t an HP laptop. It can always be worse...
The timing. (Score:1)
Bad luck for this to happen during the year of the linux desktop.
OS-level Updates (Score:3)
These days, "BIOS-level" upgrades ARE "OS-level" upgrades. UEFI even has its own shell.
Case in point:
I recently purchased a Dell PowerEdge server to run VMware's ESXi hypervisor. Before installing ESXi, I wanted to update the BIOS to the most current version. On Dell's site, I could not a "BIOS-level" update package, only "OS-level" ones. I talked to Dell, and to my surprise, the answer was to run their Windows executable from the BIOS shell.
Re: (Score:2)
UEFI even has its own shell.
Well, it's an *option* to use some of the space to host a UEFI shell, though not all vendors bother to do so. It's basically like installing dos in your BIOS rom. UEFI shell is basically the same thing as command.com: an extremely thin layer on top of firmware calls. Of course that USEFI can have network drivers and filesystem drivers whereas BIOS only ever had the concept of block device access and access MBR as a program baked in.
When it comes to servers, increasingly the expectation is that you use th
Re: (Score:1)
Dell's BIOS updates are hybrid executables that work from Windows, FreeDOS and UEFI Shell/UEFI Setup on desktops. They can also be used to directly update from Linux using the UEFI Capsule Update specification.
Re: (Score:2)
I wonder how they do that?
FreeDos and Windows would be easy - you'd have a Win32 PE file where the DOS exe stub exe was the FreeDos update exe. Same with FreeDos and UEFI because UEFI uses PE files. So you have a PE file for the UEFI code with the FreeDos code as the stub file.
UEFI uses PE files but the API is completely different to Win32. They must have worked out some way to mix MZ EXE, Win32 x86 PE and UEFI PE into one file.
Re: (Score:3)
Oops. the link to how PE files contain a Dos exe stub didn't work
https://upload.wikimedia.org/w... [wikimedia.org]
Originally this just printed "This program requires Microsoft Windows". Of course if you have a Dos version of something you can make that the stub.
The problem is that there's no documented way to combine two PE files for different APIs - Win32 and UEFI. It's also not really clear to me what the UEFI app should be built for - you can have native x86, native x64 or EBC - a byte code format.
Even though most Windo
Re: (Score:2)
There may not be an official way, but if your possible systems are all from a limited set and you're really only having to differentiate between Windows and UEFI or DOS and Windows (the other poster pointed out that there's an official way of having both DOS and Windows entry points), then it should be possible to find a system call that just provides information on both systems and has a return value that lets you tell them apart. You do this early and then jump to the relevant entry point.
Even more fun,
Re: (Score:2)
There may not be an official way, but if your possible systems are all from a limited set and you're really only having to differentiate between Windows and UEFI or DOS and Windows (the other poster pointed out that there's an official way of having both DOS and Windows entry points),
That was me
then it should be possible to find a system call that just provides information on both systems and has a return value that lets you tell them apart. You do this early and then jump to the relevant entry point.
The problem is that even though Windows and UEFI both use PE files, there are significant differences.
Windows applications use subsystem=2 for GUI and 3 for console
https://msdn.microsoft.com/en-... [microsoft.com]
UEFI applications uses subsystem=10
http://wiki.osdev.org/UEFI#Bin... [osdev.org]
Also windows executables are .exe and UEFI applications are .efi
I.e. it's hard to build a single executable that would pass the subsystem and file extension checks to run on both Windows and UEFI. Which of course is by design - otherwis
Re: (Score:2)
That's a Dell thing, it's not part of the UEFI core spec. Some laptops from a few years back had a Linux OS in there to play DVDs and browse the web without booting Windows.
Re: (Score:2)
Isn't wonderful (Score:5, Insightful)
That we have moved from simple reliable BIOS systems that provided a little boot code to get the system going on a ROM, to an advanced re-programable system so that software BUGs can now brick your PC!
Progress!
Re: (Score:2)
That happened over 20 years ago. Time to get over it.
Re: (Score:1)
Or, time to fix it?
Re: (Score:1)
It happened in 2011 actually. That was only 6 years ago. Your advice to "get over it" is unlikely to make the problems caused by it to go away. Making a bit of a fuss every now and then is actually less effort, and is at least more likely to improve the situation.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Updateable BIOS's have been around since at least the mid 90s. BIOS has not been on a ROM since then.
Re: (Score:2)
The difference is that formerly, the OS didn't dare include capabilities to screw with the BIOS ROM as a matter of course (it was always possible). There was too much concern that there wasn't enough consitency between vendors.
Here you have Intel thinking the entire world is using Tiano core and Intel reference platform, so Intel feels more confident putting code mainline to screw with the SPI roms.
Re: (Score:2)
It's the other way around. The BIOS was complex and did a huge amount of initialisation work that was then ignored by the OS anyway. It ran arbitrary code from option ROMs. And it was x86 only.
UEFI isn't perfect, but at least it ditches most of that crap. My 2012 era laptop can boot to the login screen in under 4 seconds, faster than the old BIOS can POST.
Re: (Score:2)
That we have moved from simple reliable BIOS systems
WTF? We never had that. The BIOS has for the past 30 years been a hacked together crapshoot for other systems to work around.
Re: (Score:2)
Re: (Score:2)
I remember a problem with a modem I bought back in the day, that was solved in a firmware update.
The firmware update was a package in the mail with a new chip and a chip puller.
Note having one of these devices was pretty neat. If I had a few thousand of them however....
Re: (Score:2)
Now, there's zero opportunity cost for some cracker to try and mess up your stuff.
If you want to, say, steal my tools, you have to show up - and I might shoot you - there's
Re: I remember (Score:2)
But how can the vendors be Agile(tm) if they can't release bug-riddled crapware and let their customers serve as free QA?
Re: (Score:2)
Re: (Score:2)
The utterly pathetic thing is that most SPI flash actually directly supports such a jumper. The mainboard manufacturers just likely have found out that people are too stupid to handle jumpers and left it out.
Another kernel bug did something similar (Score:2)
I was hit by the bug when booting a live cd on my brother's pc but it was recoverable and so I managed to write a correct firmware and got the drive working again.
Re: (Score:2)
Re: (Score:1)
I'd argue that it was both.
Sending junk commands isn't exactly something I'd think the kernel should do.
We need to stop (Score:1)
Recurring problem with them. (Score:2)
Separation of concerns (Score:2)
Why has the world forgotten it?
I updated to 17.10 on my Yoga 2 pro, am I fucked? (Score:1)
Re: (Score:2)
Just never reboot your machine ever again. :)
DUAL BIOS Motherboards (Score:3)
Good thing I have a motherboard with dual BIOS so if one gets screwed up due to a bad flash I can flip the switch to the back up BIOS and then copy itself over to the corrupted BIOS.
So is there any test or list of affected models? (Score:2)
I don't want to file this story as disaster porn, but so far it hasn't been anything I could describe as helpful. Ditto the comments. Double ditto the link and the comments there.
Right now I have 17.10 running on a Lenovo and a Toshiba, and so far I haven't noticed any problems. Lack of evidence is no proof of the negative...
Seems like my easy "response" is to hope that the next updates from Ubuntu take care of the problem (for extremely low values of care?). Unless it's already too late, in which case...
No
Re: (Score:2)
You do understand, don't you, that Ubuntu != Linux. I run Fedora, and there's been no mention of this particular issue affecting Fedora, even though it's always very big on pushing out new kernels as quickly as possible. I don't know if either distro modifies the kernel before making it available, or if this isn't in the kernel itself but some support module, but AFAICT it's distro-specific. I do hope, of course, that it gets cleared up quickly.
Re: (Score:2)
I certainly agree with you that there are many distributions, and I think that is one of the best aspects of Linux. Monolithic thinking is anti-freedom. Check my sig.
However whenever a major distro looks bad, all of Linux catches some shade. I still blame the financial models, but been there, done that, no one's interested in such crazy ideas as alternative possibly even better financial models.
What even is that? (Score:2)
Re: (Score:2)
Update the BIOS on managed systems. Set things in NVRAM (boot order, secure boot keys, etc).
Re: (Score:2)
Is there a man page for the Linux program that sets things in NVRAM? I don't recall seeing that anywhere.
Re: (Score:2)
Don't know about a man page, but the /sys/firmware/efi/efivars filesystem contains the efi nvram. Read and write.
Re: (Score:2)
To be clear, any program that can write files can set things in nvram through that filesystem (the data does need a 4 byte attribute prepended to it). The only program I can think of that specifically changes nvram is efibootmgr (there is a man page for that), but even that is just using the efivars filesystem.
Re: (Score:2)
confidence game (Score:2)
I've lost confidence in Ubuntu. Canonical seems increasingly user-hostile. And in my subjective opinion - reinforced by this story - their technical quality has been declining for a while.
So what's the alternative?
My Docker containers are already based on Alpine, and running atop ECS, which is based on Amazon Linux. But I still need cloud VMs and bare metal servers. One thing I really appreciate about Ubuntu it's the ease of running the same OS on my servers and on my Thinkpad laptop.
*BSD is out because af
Re: (Score:2)
HP Laptops with Insyde BIOS, affected or not? (Score:1)
Re: (Score:2)
Re: (Score:2)
127.0.0.1 www.ubuntu.com
Re: (Score:2)
::1 www.ubuntu.com
FTFY.
Re: (Score:2)
You ought to fix that for Ubuntu.com first; their nameservers don't seem to offer an IPv6 (AAAA) RR.
Obligatory :-) included at no extra charge.
Re: (Score:2)