Closure On the Linux Lockup Bug 115
jones_supa writes: Dave Jones from Red Hat has written a wrap-up of the strange bug that has made some machines running Linux to freeze. (Previous discussion.) Right down to his final week at Red Hat before Dave gave all his hardware back, Linus Torvalds managed to reproduce similar symptoms, by scribbling directly to the HPET timer. He came up with a hack that at least made the kernel survive for him. When Dave tried the same patch, the machine ran for three days before he interrupted it, which was a promising result. The question remains, what was scribbling over the HPET in his case? The only two plausible scenarios Dave could think of were that Trinity generated 0xFED000F0 as a random address and passed that to a syscall which wrote to it, or a hardware bug. That's where the story ends for now. Linus' hacky workaround didn't get committed, but him and John Stultz continue to back and forth on hardening the clock management code in the face of screwed up hardware, so maybe soon we'll see something real get committed on that area.
does not sound like closure to me (Score:4, Informative)
"probably a hardware bug" is code for "well, we bought new hardware and threw out all the old stuff, sorry"
Re:does not sound like closure to me (Score:5, Informative)
Re-read the summary. They know what is causing the lockup, they don't know what is making the system call which is triggering the bug. Once you know what is causing the lockup it can be fixed, and the hack that was written made the lock-ups stop. At no point did anyone throw out or try new hardware, though one thought is everything is originating from a hardware bug.
Re: (Score:1)
"probably firmware SMM code messing with the HPET counter behind our back" != "probably a hardware bug"
Re: (Score:1)
Re: (Score:2)
Someone with the right equipment should be able to do a hardware trace and catch the culprit.
Re:does not sound like closure to me (Score:5, Interesting)
RTFA, they have good reason to point at the hardware. Then there's the bazillions of servers running on different hardware that have never seen the bug.
Many teams would have written it off as a hardware bug a long time ago, but the linux kernel team was willing to consider and investigate the possibility that it was a rarely triggered bug in the software before they passed the buck.
Sometimes it really is a hardware bug.
plus don't crash on bad hardware. Hotplugged CPU (Score:3)
>. Many teams would have written it off as a hardware bug a long time ago, but the linux kernel team was willing to consider and investigate the possibility that it was a rarely triggered bug in the software before they passed the buck.
And try to avoid crashing due to hardware bugs, if possible.
A contractor once hotplugged one of the CPUs in one of my servers. That's right, they took the processor out and replaced it with the machine running. The box did not crash. It kept running at least for the f
Re: (Score:2)
Hot swapping the CPU without an immediate crash had to be a million to one shot!
But yes, resilient software is always a good thing.
I do hope Linus's patch goes in in some form to at least make it clear what the problem is if someone with similarly borked hardware sees the problem.
Re:plus don't crash on bad hardware. Hotplugged CP (Score:4, Informative)
>Hot swapping the CPU without an immediate crash had to be a million to one shot!
With QPI interconnect and the voltage and temp supervisory circuits on chip, it's not such a long shot these days, especially on Xeons with failover support that is explicitly intended to cope with a neighbor CPU going down.
Re: (Score:3)
Yes it's great to support hotplugged CPUs! 1969 called and they want to let you know they supported online reconfiguration back then too: http://en.wikipedia.org/wiki/M... [wikipedia.org]
Re: (Score:2)
That's interesting. Apparently it was supported well enough that they actually did hotplug CPUs regularly, as standard practice. I wonder if they "unmounted" the components before removal and "mounted" them upon insertion. That's a much easier approach, especially for CPUs, than handling a CPU suddenly going AWOL.
Re: (Score:2)
USB has slightly longer contacts on the power pins for much the same reason.
Re: (Score:2)
Yes. Exactly this. Pulling the latches on the card generates an interrupt. In the systems I designed (for a mainframe raid disk system in this case), a little green light would light up when it was ready. So pull the latches out, wait for green light, pull the card out. The light generally lit up in a few milliseconds, so you could just rip the card out.
I presume this is how it worked for all products from this (very large, well known) manufacturer, because that's what the spec required.
Linux CPU hotplug support link (Score:4, Informative)
Replying to myself, but I figured someone reading this might be interested. Linux does support CPU hotplug where you disable the CPU before removing it. Your motherboard might get mad about it if it's not supported by the board, though.
http://www.cyberciti.biz/faq/d... [cyberciti.biz]
Re: (Score:3)
Yes. It's mostly used for reconfiguring VMs, but it is possible to do it with real hardware if the board supports it.
It's interesting how as time goes on, PC hardware is slowly coming to resemble an affordable version of the mainframes they replaced.
Re: (Score:2)
Was not one reason why mainframes was so highly valued that one could hotswap virtually anything without interrupting workflow?
Re: (Score:2)
Yes, I can see that would limit the damage, but it still leaves the OS surprised to have running tasks just go away.
It would likely work less well with AMD processors since a chunk of memory would also go away.
Re:plus don't crash on bad hardware. Hotplugged CP (Score:5, Funny)
Sometimes it
Sometimes it -- what? Did someone attempt to hot-swap your CPU again? (-:
Re: (Score:2)
Sometimes it screws up the post, where "it" is the Android browser.
Re: (Score:2)
Solaris supported hot pluggable CPUs in the last century!
Re: (Score:2)
It's still not a given that it's the hardware. It's likely that something is scribbling over the HPET timer. As to whether that's due to faulty hardware or a software bug is still undetermined.
Random memory corruption is oh so painful. :(
Re:does not sound like closure to me (Score:4, Funny)
One of the more memorable quotes I heard while developing embedded systems: if you can fix it in software, it isn't a hardware bug
Annoying as hell to the software team when it is clearly a bug in the hardware, but very true at a practical level for the engineering team trying to get product out the door.
Re: (Score:2)
I'm famioliar with that one. Same thing happens in boot ROMs.
Re: (Score:1)
if you can fix it in software, it isn't a hardware bug
I'm a hardware and software guy, and I can tell you that is entirely bullshit. While I understand it may seem this way because sometimes software guys can't write a driver to save their lives, there are many bugs in hardware which are actual hardware bugs (race conditions, dropped interrupts, whatever) that have workarounds in software.
I've seen buggy hardware NAND flash ECC units "fixed" by doing ECC entirely in software, leaving the hardware unit unused, and taking a bit throughput hit.
I also seem to reca
Re: does not sound like closure to me (Score:1)
My windows servers have an uptime of 49 years, 31 days, 22 hrs, 15 mins and 4539 ms. No Linux server can beat that
Re: (Score:2)
Re: does not sound like closure to me (Score:1)
Re: (Score:1)
Everyone else? Like all hardware is OSX certified? Try putting any old HDD or SSD into a macbook and see how that works.
Re: (Score:2)
bwahahahahahaha, come on, we need sarcasm font here!!
Re: (Score:1)
In other words.. (Score:2, Funny)
Closed NOTABUG?
Editors, edit! (Score:3)
Really?
Re: (Score:3)
The second sentence isn't much better:
Right down to his final week at Red Hat before Dave gave all his hardware back, Linus Torvalds managed to reproduce similar symptoms, by scribbling directly to the HPET timer.
Was Linus at Dave's place working on the issue? Is the first part a sentence fragment and Dave did something before he gave his hardware back we aren't being told? Or is the first part really a continuation of the first sentence, and Dave was working on his writeup all the way until the deadline for returning his hardware?
him? (Score:1)
Hey youse editors, you want I should take the mug out?
hardening is NOT blaming the hardware (Score:5, Interesting)
Too many clueless comments already that don't understand the difference between "blaming the hardware" and hardening to deal with demonstrably-broken hardware (and/or firmware for devices). I've spent years writing drivers for various OS', including Windows and Linux. It is rare for any complex device to be bug-free at the hardware level (look how many patches are BIOS-applied to CPUs, for example). Sometimes, under NDA, of course, the Windows driver writers are apprised of the deficiencies, or, at least, get better response from the vendor when an anomaly appears. Linux rarely gets that same assistance.
My favorite example, though, is all-IBM. We were porting AIX to the PS/2s and 370s. We consistently had problems with the diskette interface under AIX and the response from Boca Raton was always "it works in MS-DOS, so it's your code, not our hardware". When OS-2 came around, they ran into exactly the same problem in the hardware. By then, we had a work-around (slower, more locks, but no more glitches) which was how OS-2 got around it, as well.
Re: (Score:2)
Too many clueless comments already
Not bad given you were the ~4th poster and 2 of them didn't mention the hardware.
Re: (Score:3, Funny)
What you posted about his being the 4th post struck me as wrong, given how far it was down the page. I'm bored, so I took a moment to look at how many posts have an earlier timestamp than the one you are slamming (at least 8), and 2 make dismissive statements about hardware, including the first comment of article at 8:12, and another at 8:19 seemingly dismissing hardware as a possibility.
So your snide comment is not based in fact. It's like you are reading a different page. Maybe you need glasses. An attitu
Re: (Score:2)
The other posts were, in fact, made later, but someone was messing around with the HPET timer and, well, bugs.
"friend" and "foe", but no "neckbeard" (Score:1)
I wish Slashdot would allow me to mark users not just as "friend" or "foe", but as "neckbeard". :) That must have been 1986 or 1987?
Re: (Score:3)
0: I do shave my neck. :) In fact, the beard has been gone for more than a year.
1: a bit later, early 1990; we all got a big laugh out of the 486SX/487 when those came out. https://en.wikipedia.org/wiki/Intel_80486SX [wikipedia.org]
meant in the best possible way. Gray beard. (Score:2)
PS I meant that in the best possible way. I didn't really think through the connotations of "neck beard" before posting. I was really thinking more "gray beard" , including wizardly connotations.
Re: (Score:2, Funny)
AC here, no longer posting as myself since I've long lost my SO account, can't be bothered to find the password for the ancient yahoo email address, and after working on the inside in finance will probably never post an opinion (as my own) again. (Yes, that was a run on sentence.)
If 1986 qualifies as a "neckbeard" you missed the mark, unless he's a Berkley neckbeard. The 80's were a magical time when power ties, very bad print shirts, and driving your overpriced car with women and blow was available to an
Re: (Score:2)
Re: (Score:2)
No, I think he's implying that coding has gone out of fashion (or at least no longer guarantees a high-paying job.)
Coding going out if fashion? Have you been living in a cave these last few years?
Re: (Score:2)
Folds in space time continuum (Score:1)
Obviously, it's folds in the space time continuum that is causing HPET (the high precision hardware timer) to jump backwards, causing negative deltas and lockups.
Perhaps a future version of ourselves has transcended space-time and is trying to contact us to help us with our bad harvests? Did Linus try to determine any kind of co-ordinates from the glitch?
Has NASA seen any kind of weird portholes near Jupiter?
Re: (Score:2)
Re: (Score:2)
In the mean time... (Score:1)
No it doesn't (Score:2)
Re: (Score:2)
Whether or not you see a blue screen with a lot of text on it is beside the point. Every OS can potentially panic. Even if it's configured to paper over the problem by doing it quietly and rebooting, the system has gone tits up.
Re: (Score:2)
How much would it cost to have a computer which could leave a trace of the cause of a lockup, even if the machine exploded?
You would have to have double your main memory, basically. Not really that expensive.
Re: No it doesn't (Score:2)
The problem is that when the kernel panics, everything grinds to a stand still. More specifically: hard drive controller/driver. How are you going to write the data if you don't have access to the disks?
This is by design, as the disk controller could br the reason for the lockup, and you would potentially corrupt your entire disk by trying to write to it.
I'm sure its been thought of before, but my first thought is to include a very small chunk of memory on the motherboard, with a stupidly simple api that is
Re: (Score:2)
hell I cant recall the last time I saw XP BSOD
"closure" (Score:1)
About as much as this year being the year of the linux desktop... no really, it's gonna be THIS year... promise.
"him and John Stultz continue ..." (Score:3)
"... him and John Stultz continue to back and forth ..."
What in the world is happening, editors?
Re: (Score:2)
"... him and John Stultz continue to back and forth ..."
What in the world is happening, editors?
The only editors on slashdot are some vi's, some pines, and a couple of notepads and textedit. Certainly, no human editors....
Re: (Score:1)
They have obviously outsourced the editing to India.
Re: (Score:2)
They have obviously outsourced the editing to India.
Or New Jersey
Call me crazy (Score:5, Interesting)
Sorry if I've found the wrong stuff. I'm doing this via a quick googling...
Is this really the code for reading and writing the HPET?
http://www.cs.fsu.edu/~baker/d... [fsu.edu]
I've been a powerpc programmer in aviation for a while. If you need to read the time base register (also a 64 bit up counter) you have to be aware that your read might coincide with the lower 32 bits incrementing and carrying into the upper 32 bits. So you read the upper 32 bits, read the lower 32 bits, then re-read the upper bits and make sure the upper bits didn't change. If they did repeat this process. But if they are the same then you combine the 32 bit halves into a 64 bit time and call it good.
Re: (Score:1)
Re: (Score:1)
Is this really the code for reading and writing the HPET?
Yup.
I've been a powerpc programmer in aviation for a while. If you need to read the time base register (also a 64 bit up counter) you have to be aware that your read might coincide with the lower 32 bits incrementing and carrying into the upper 32 bits. So you read the upper 32 bits, read the lower 32 bits, then re-read the upper bits and make sure the upper bits didn't change. If they did repeat this process. But if they are the same then you combine the 32 bit halves into a 64 bit time and call it good.
That would be entirely wrong here.
The upper 32 bits of the current timer value are latched into the register at the upper address when the lower 32 bits are read from the lower address.
Re: (Score:2)
OK then. Where in this return statement are the lower 32 bits read first? I don't believe the bitwise or operator is a sequence point. (The logical one is)
return readl(addr) | (((unsigned long long)readl(addr + 4)) http://www.intel.com/hardwared [intel.com]...
but I did find the following, which documents the race condition I explained above.
http://www.intel.com/content/d... [intel.com]
I will search for newer documentation than a 1.0a.
Re: (Score:2)
Might want to check your first link.
Re: (Score:2)
Sorry for the bad post. Yes, the first link does not work, but it is what is documented in hpet.c as the reference. A sentence went missing somewhere saying that I couldn't find it. The second link, which does work, is what I've found so far. I have yet to find something newer which documents the latching behavior that was claimed.
Sorry again for the bad post.
-Nyall
Re: (Score:2)
Well in that case, you'd need to ensure the lower 32 bits are read first so you're reading the upper 32 bits that you latched this time through, not last time through. And if that's the case, the code is still wrong because there's nothing to force a sequence point between the two reads. The compiler is free to re-order the two reads in that expression.
Re: (Score:3)
readq is only defined in that code if undefined elsewhere, and is only used to read counters on 64-bit architectures.
on 32-bit architectures, that code uses readl to read the counter.
readq is undefined in some 32-bit architectures, so is defined there- but only used there to read the configuration register (not likely to roll over
Also, the actual reading of the counter is done indirectly: it's returned from the IRQ handler for the HPET.
Re: (Score:1)
http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/arch/x86/include/asm/io.h#L49
Line 49 looks like where readq is defined for x64 architecture.
Freezes on Mac under Parallels (Score:1)
I had the freeze bug in a VM system on a Mac running Parallels. I downloaded Ubuntu 14.04 from Parallels and could not get around it. Then I downloaded directly from Canonical and it worked just find. I assumed it was a bad download from Parallels, but perhaps it is more subtle. The virtual machine has the same vulnerabilities - is that a clue?
How to Follow this Bug (Score:1)
Re: (Score:2)
Was it caused by Monkeeing around?