The Many Paths To Data Corruption 121
Runnin'Scared writes "Linux guru Alan Cox has a writeup on KernelTrap in which he talks about all the possible ways for data to get corrupted when being written to or read from a hard disk drive. This includes much of the information applicable to all operating systems. He prefaces his comments noting that the details are entirely device specific, then dives right into a fascinating and somewhat disturbing path tracing data from the drive, through the cable, into the bus, main memory and CPU cache. He also discusses the transfer of data via TCP and cautions, 'unfortunately lots of high performance people use checksum offload which removes much of the end to end protection and leads to problems with iffy cards and the like. This is well studied and known to be very problematic but in the market speed sells not correctness.'"
Keep your porn on separate physical drives! (Score:5, Funny)
In addition, you should never access young data and pornographic data in the same session, as the young impressionable data may get corrupted by the pornographic data if they exist in RAM at the same time.
Data corruption is a serious problem in computing today, and it is imperative that we take steps to stop our young innocent data from being corrupted.
Re: (Score:3, Funny)
indeed, young pornographic data is disturbing. Fortunately there is a legal social firewall of 18.
Re: (Score:2)
Unfortunately, as to your "Fortunately there is a legal social firewall of 18.", it depends on which block in which city you are cruising as to whether or not they may be at least/over 14, much less 18.
At least that's what a traveling friend of mine told me....honest!
Re: (Score:1)
Re: (Score:2)
Who's "Boy", how you know Boy is an expert, and what makes Boy the poster's?
Re: (Score:1, Insightful)
Re: (Score:2)
Whatever you do, don't dilute the data 200 times by zeroing 9/10 of the data each time, otherwise your drive will be full of porn
Paul Cylon (Score:3, Funny)
Re: (Score:2, Insightful)
Re: (Score:2)
Glitch on the bus, Gus! (Score:3, Funny)
You can make up some more yourself....
benchmarks (Score:5, Insightful)
Re:benchmarks (Score:5, Interesting)
I've concluded that nobody cares about data integrity. That's sad, I know, but I have yet to see product manufacturers sued into oblivion for building fundamentally defective devices, and that's really what it would take to improve things, IMHO.
My favorite piece of hardware was a chip that was used in a bunch of 5-in-1 and 7-in-1 media card readers about four years ago. It was complete garbage, and only worked correctly on Windows. Mac OS X would use transfer sizes that the chip claimed to support, but the chip returned copies of block 0 instead of the first block in every transaction over a certain size. Linux supposedly also had problems with it. This was while reading, so no data was lost, but a lot of people who checked the "erase pictures after import" button in iPhoto were very unhappy.
Unfortunately, there was nothing the computer could do to catch the problem, as the data was in fact copied in from the device exactly as it presented it, and no amount of verification could determine that there was a problem because it would consistently report the same wrong data.... Fortunately, there are unerase tools available for recovering photos from flash cards. Anyway, I made it a point to periodically look for people posting about that device on message boards and tell them how to work around it by imaging the entire flash card with dd bs=512 until they could buy a new flash card reader.
In the end, I moved to a FireWire reader and I no longer trust USB for anything unless there's no other alternative (iPod, iPhone, and disks attached to an Airport Base Station). While that makes me somewhat more comfortable than dealing with USB, there have been a few nasty issues even with FireWire devices. For example, there was an Oxford 922 firmware bug about three years back that wiped hard drives if a read or write attempt was made after a spindown request timed out or something. I'm not sure about the precise details.
And then, there is the Seagate hard drive that mysteriously will only boot my TiVo about one time out of every twenty (but works flawlessly when attached to a FW/ATA bridge chipset). I don't have an ATA bus analyzer to see what's going on, but it makes me very uncomfortable to see such compatibility problems with supposedly standardized modern drives. And don't get me started on the number of dead hard drives I have lying around....
If my life has taught me anything about technology, it is this: if you really care about data, back it up regularly and frequently, store your backups in another city, ensure that those backups are never all simultaneously in the same place or on the same electrical grid as the original, and never throw away any of the old backups. If it isn't worth the effort to do that, then the data must not really be important.
Re: (Score:2)
And then, there is the Seagate hard drive that mysteriously will only boot my TiVo about one time out of every twenty (but works flawlessly when attached to a FW/ATA bridge chipset).
And then there's my 80 Gig Western Digital that was very flakey (as soon as the warranty was up) in BX chipset (or equivalent) motherboard PCs, but I used it to replace the original drive in a Series 1 stand alone Philips Tivo and it's been working flawlessly in it for about a year now. Before you blame WD, I'm writing this on a BX chipset PC that's been running another WD 80 Gig that's almost identical (came off the assembly line a few months earlier) and it's been working fine since before I got the ne
Re: (Score:2)
In the end, I moved to a FireWire reader and I no longer trust USB for anything unless there's no other alternative (iPod, iPhone, and disks attached to an Airport Base Station). While that makes me somewhat more comfortable than dealing with USB, there have been a few nasty issues even with FireWire devices.
I don't recall seeing anything with regards to FireWire vs USB that would give FireWire an advantage in data integrity (though may be missing some finer points about the respective specs). OTOH, I have seen specs (one of the LaCie RAID in a box drives) that give a 10 to 20% performance advantage to FW despite the 'lower' peak speeds - one reason is that FW uses separate pairs for xmit and rcv.
Re: (Score:2)
There's no technical reason for FW drives to be more reliable. The limited number of FireWire silicon vendors, however, does mean that each one is likely to get more scrutiny than the much larger number of USB silicon vendors, IMHO.
Re: (Score:1)
Re: (Score:2)
Close. The FireWire controller is smarter and allow it to do a lot more work without the CPU being involved. That's the reason FireWire performs faster than USB 2.0 despite being a slower bus.
Of course, the new USB 3.0 will bump USB speed way up. I'm not holding my breath, though. Considering what USB 2.0 does to the CPU load when the bus is under heavy use, I'd expect that Intel had better increase the number of CPU cores eightfold right now to try to get ahead of the game.... :-)
this group of com [slashdot.org]
Re: (Score:1)
Yep, that's it: loads of useless data, produced by a society barely able to perform some relatively weak techno tricks while completly failing to solve basic issues. Something is wrong in this biometric cash flow production model.
Re: (Score:1)
End-to-end (Score:5, Informative)
Hello ZFS (Score:5, Informative)
I am looking forward to the day when all RAM has ECC and all filesystems have checksums.
No (Score:4, Interesting)
as long as ZFS itself, the CPU, and RAM are working correctly, no other errors can corrupt ZFS data.
Sorry, but that is absurd. Nothing can absolutely protect against data errors (even if they only happen in the hard disk). For example, errors can corrupt ZFS data in a way that turns out to have the same checksum. Or errors can corrupt both the data and the checksum so they match each other.
This is ECC 101 really.
Re: (Score:2)
You can use SHA as the checksum algorithm; the chance of undetected corruption is infinitesimal.
Re:No (Score:5, Funny)
This is about as likely as simultaneously winning every current national and regional lottery on the planet. And then doing it again next week.
And if we're talking about a 512 bit hash then it's possible that a new planet full of lotteries will spontaneously emerge from the quantum vacuum. And you'll win all those too.
Re: (Score:2, Funny)
If this happens, be sure to keep the money from the quantum vacuum lotteries in a separate account, or it will annihilate with your real money.
Re: (Score:2)
Re: (Score:3, Funny)
Real-life proof of ZFS detecting problems (Score:4, Informative)
http://blogs.sun.com/elowe/entry/zfs_saves_the_day_ta [sun.com]
And you'll understand
Re:Hello ZFS (Score:4, Informative)
But, ECC is available. If it is important to you, pay for it.
Re: (Score:2)
Re: (Score:2)
Not gonna happen. The problem is that ECC memory costs more, simply because there is 12.5% more memory. Most people are going to go for as cheap as possible.
It'll happen for the same reason RAID5 on certain types of arrays will be obselete in 4 - 5 years. Eventually memory sizes are going to get so big that the statistical probability of a memory error will effectively guarantee they happen too frequently to ignore.
Re: (Score:2)
Re: (Score:3, Interesting)
Obsolete? What would you replace it with then?
RAID6. Then a while after that, "RAID7" (or whatever they call triple-parity).
In ca. 4-5 years[0], the combination of big drives (2TB+) and raw read error rates (roughly once every 12TB or so) will mean that during a rebuild of 6+ disk RAID5 arrays after a single drive failure, a second "drive failure" (probably just a single bad sector, but the end result is basically the same) will be - statistically speaking - pretty much guaranteed. RAID5 will be obsel
Re: (Score:2)
The big issue is that ECC memory doesn't cost only 12.5% more than regular memory, otherwise you'd see that lots of knowledgeable (or correctly guided) people would buy ECC.
Re: (Score:1)
ECC may be available
( I always build systems with http://www.crucial.com/ [crucial.com] ECC RAM,
and no I'm nae affilliated,
but they're the ONLY brand who've never once proven flaky,
in my experience. . . )
but the problem is that almost no motherboards support ECC.
Gigabyte's GC-RAMDISK and GO-RAMDISK ( up-to 4GB ) hardware-ram-drive without ECC *support*,
is typical of this idiocy:
The only way to make the things trustworthy is to run a RAID5 or RAID6 array of 'em,
and that gets bloody expensive
( though the s
Re: (Score:2)
But the reason I have it set up is not to
Other schemes (Score:2)
Re: (Score:2)
CRCs are only used for detecting errors. Once you've detected a bad disk block, you can use replication (RAID 1), parity (RAID 4/5/Z), or some more advanced FEC (RAID 3/6/Z2) to correct the error. The benefit of CRCs is that you can read only th
I think this has happened to me (Score:5, Interesting)
Of course, no one believes me. But maybe this presentation is on to something. Or perhaps I did something in a bonehead fashion totally unrelated.
Re: (Score:2)
Re: (Score:1)
Perhaps the FAT filesystem is interpreted differently on the player to how Linux expects it to be? (Or VV)
Re: (Score:1)
Re: (Score:2)
MySQL? (Score:5, Funny)
I was expecting an article on using MySQL in production.
Re: (Score:2)
Just wait till the banks data gets munged (Score:2)
Hah (Score:2)
Re:Hah (Score:4, Insightful)
Re: (Score:2)
...but in the market speed sells not correctness. (Score:4, Interesting)
Ah - this is the bane of computer technology.
One time I remember writing some code and it was very fast and almost always correct. The guy I was working with exclaimed "I can give you the wrong answer in zero seconds" and I shut up and did it the slower way that was right every time.
This mentality of speed at the cost of correctness is prevalent, for example I can't understand why people don't spend the extra money on ECC memory *ALL THE TIME*. One failure over the lifetime of the computer and you have paid for your RAM. I have assembled many computers and unfortunately there have been a number of times where ECC memory was not an option. In almost every case where I have used ECC memory, the computer was noticably more stable. Case in point, the most recent machine that I built has never (as far as I know) crashed and I've thrown same really nasty workloads it's way. On the other hand, a couple of notebooks I have have crashed more often than I care to remember and there is no ECC option. Not to mention the ridicule I get for suggesting that people invest the extra $30 for a "non server" machine. Go figure. Suggesting that stability is the realm of "server" machines and infer end user machines should be relegated to a realm of lowered standards of reliability makes very little sense to me especially when the investment of $30 to $40 is absolutely minuscule if it prevents a single failure. What I think (see lawsuit coming on) is that memory manufacturers will sell quality marginal products to the non ECC crowd because there is no way of validating memory quality.
I think there needs to be a significant change in the marketing of products to ensure that metrics of data integrity play a more significant role in decision making. It won't happen until the consumer demands it and I can't see that happening any time soon. Maybe, hopefully, I am wrong.
Re: ...but in the market speed sells not correctne (Score:1, Insightful)
I do understand it. They live in the real world, where computers are fallible, no matter how much you spend on data integrity. It's a matter of diminishing return. Computers without ECC are mostly stable and when they're not, they typically exhibit problems on a higher level. I've had faulty RAM once. Only one bit was unstable and only one test
Re: (Score:2)
Computers are machines and don't need to be designed to be fallible. ECC is a small insurance policy to avoid problems exactly like the one you described. How much time did you spend on burning CD's that were no good, or running various memtests, not to mention the possible corrupted data you ended up saving and other unknown consequences ? Had you bought ECC RAM, your problem would have been corrected or more than likely detected not to menti
Re: (Score:1, Interesting)
That's beside the point. Computers ARE fallible, with or without ECC RAM. That you think they could be perfect (infallible) is testament to the already low rate of hardware defects which harm data integrity. It's good enough. I've experienced and located infrequent d
Re: (Score:2)
Yes. They are, but considerably less fallible with ECC. Remember, "I can give you the wrong answer in zero seconds." There's no point in computing at all unless there is a very high degree of confidence in computational results. Software is just as fallible as hardware but again, I can, and do, make considerable effort in making it less fallible.
I am the default sysadmin for a family member's business. There was a time where the system was fraught with
Re: ...but in the market speed sells not correctne (Score:2)
I use to sell firewalls. People always wanted to know how fast it would work (most were good up to around 100Mbps, when most people had at most 2Mbps pipes at most), very few people asked detailed questions about what security policies it could enforce, or the correctness and security of the firewall device itself.
Everyone knew they needed something, very few had a clue about selecting a good product, speed they understood, network securi
Re: (Score:2)
Sarcasm right ?
Missing option. (Score:1)
Re: (Score:2)
http://www.srim.org/SER/SERTrends.htm [srim.org]
what we have lost (Score:5, Insightful)
Large mainframe systems have had data integrity problems solved for a long, long time. It is today unthinkable that any hardware issues or OS issues could corrupt data on IBM mainframe systems and operating systems.
Personal computers, on the other hand, have none of the protections that have been present since the 1970s on mainframes. Yes, corruption can occur anywhere in the path from the CPU to the physical disk itself or during a read operation. There is no checking, period. And not only are failures unlikely to be quickly detected but they cannot be diagnosed to isolate the problem. All you can do is try throwing parts at the problem, replacing functional units like the disk drive or controller. These days, there is no separate controller - its on the motherboard - so your "functional unit" can almost be considered to be the computer.
How often is data corrupted on a personal computer? It is clear it doesn't happen all that often, but in the last fourty years or so we have actually gone backwards in our ability to detect and diagnose such problems. Nearly all businesses today are using personal computers to at least display information if not actually maintain and process it. What assurance do you have that corruption is not taking place? None, really.
A lot of businesses have few, if any, checks that would point out problems that could cost thousands of dollars because of a changed digit. In the right place, such changes could lead to penalties, interest and possible loss of a key customer.
Why have we gone backwards in this area when compared to a mainframe system of fourty years ago? Certainly software has gotten more complex but basic issues of data integrity have fallen by the wayside. Much of this was done in hardware previously. It could be done cheaply in firmware and software today with minimal cost and minimal overhead. But it is not done.
Re: (Score:2)
A lot of businesses have few, if any, checks that would point out problems that could cost thousands of dollars because of a changed digit.
I would think it's extremely unlikely that such random corruption would happen on some byte somewhere which actually gets interpreted as a meaningful digit; much more likely to either corrupt some format or produce some noticeable garbage somewhere (not "wrong-yet-meaningful" data).
Re:what we have lost (Score:4, Interesting)
For the same reason why experienced car drivers crash in ridiculous situations: they are too sure of themselves.
The industry is so huge, that the separate areas of our computers just accept the rest is a magic box that should magically operate as is written in the spec. Screwups don't happen too often, and when they happen they are not detectable, hence no one woke up to it.
That said don't feel bad, we're not going downwards. It just so happened speed and flashy graphics will play important role for another couple of years. Then after we max this out, the industry will seek to improve another parameter of their products, and sooner or later we'll hit back the data integrity issue
Look at hard disks: does the casual consumer need more than 500 GB? So now we see the advent of faster hybrid (flash+disk) storage devices, or pure flash memory devices.
So we've tackled storage size, we're about to tackle storage speed. And when it's fast enough, what's next, encryption and additional integrity checks. Something for the bullet list of features...
Re: (Score:2)
There is no checking, period.
I'm sorry, but for even the crappiest PC clones, you've been wrong since 1998. Even the worst commodity disk interfaces have had checking since then: UDMA uses a 16 bit CRC for data; SATA uses 32 bit CRC for commands as well. Most servers and workstations have had ECC memory for even longer. Furthermore, if you cared at all about your data, you already had end-to-end CRC/ECC.
Yeah, mainframes are neat, but they don't save you from end-to-end integrity checking unless you really don't give a damn about it
Re: (Score:1)
Re: (Score:2)
Are you sure? The actual cost is minuscule and significantly less than the cost of potential errors.
Comment removed (Score:4, Interesting)
Re: (Score:2)
We do have weak link with desktops regarding RAM however. While modern workstations and server are generally installed with ECC RAM, our desktops are not.
The major failing of the original Apple Xserve 'supercomputer' cluster was the lack of ECC - ISTR an estimate of a memory error every few hours (estimate made by Del Cecchi on comp.arch), which would severely limit the kinds of problems that could be solved on the system. I also remember the original systems being replaced a few months later with Xserves that had ECC.
And yes, you would take a 1 to 2% performance hit, but so what; Data integrity is more impor[t]ant.
A 1 to 2% performance hit is less costly than having to do multiple runs to make sure the data didn't get munged.
Note: The newer Intel P965 chipset does not support ECC memory while their older 965x does. Crying shame too given the P965 has been designed for Core 2 Due and Quad Core CPUs.
You're right about the
Re: (Score:2, Insightful)
The nice thing about AMD is that with the integrated memory controller, you don't need support in the chipset. I'm not sure about Semprons, but all of the Athlons support ECC memory. The thing you have to watch out for is BIOS/motherboard support. If the vendor doesn't include the necessary traces on the board or the configuration settings in the BIOS, it won't
Re: (Score:2)
Re: (Score:1, Informative)
What's worse? It IS free!
Motherboard chips (e.g. south bridge, north bridge) are generally limited in size NOT by the transistors inside but by the number of IO connections. There's silicon to burn, so to speak, and therefore plenty of room to add features like this.
How do I know this? Oh wait, my company made them.... We never had to worry about state-of-the-art process technology because it wasn't worth it. We could afford to be several genera
Re: (Score:1, Informative)
You meant 975x, not 965x. The successor of 975x is X38 (Bearlake-X) chipset supporting ECC DRAM. It should debut this month.
Re: (Score:1)
checksum offload (Score:1)
offloaded to another device like say an expensive NIC ? and thus removes that overhead from the CPU
Re: (Score:1, Interesting)
Also, these don't protect against errors on the card or PCI bus. (If the data was corrupted on the card or on the bus after the checksum validation but before it got to system RAM for any reason, this corr
Re: (Score:2)
How much does scrubbing cost? (Score:2, Interesting)
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
Timely article ... (Score:3, Interesting)
Re: (Score:1)
To be honest, most of the comments about ECC RAM here have convinced me that it's worth it just for more peace of mind
HEY. (Score:3, Funny)
Re: (Score:2)
I've seen it happen (Score:2)
Re: (Score:1)
Much Ado About Very Little (Score:1)
Today, with these "unreliable" hard drives, and (apparently, if we believe the post) less hardware checking being done, I very, very seldom receive data, or retrieve data from storage, that is detectably corrupted. My CRCs almost inva
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
I have seen this many times, unfortunately. :-( (Score:3, Interesting)
One master, all the others are copies.
Starting maybe 15 years ago, when these directory structures were in the single-digit GB range, we started noticing strange errors, and after running full block-by-block compares between the master and several slave servers we determined that we had end-to-end error rates of about 1 in 10 GB.
Initially we solved this by doubling the network load, i.e. always doing a full verify after every copy, but later on we found that keeping the same hw, but using sw packet checksums, was sufficient to stop this particular error mechanism.
One of the errors we saw was a data block where a single byte was repeated, overwriting the real data byte that should have followed it. This is almost certainly caused by a timing glitch which over-/under-runs a hardware FIFO. Having 32-bit CRCs on all Ethernet packets as well as 16-bit TCP checksums doesn't help if the path across the PCI bus is unprotected and the TCP checksum has been verified on the network card itself.
Since then our largest volume sizes have increased into the 100 TB range, and I do expect that we now have other silent failure mechanisms: Basically, any time/location when data isn't explicitly covered by end-to-end verification is a silent failure waiting to happen. On disk volumes we try to protect against this by using file systems which can protect against lost writes as well as miss-placed writes (i.e. the disk reports writing block 1000, but in reality it wrote to block 1064 on the next cylinder).
NetApp's WAFL is good, but I expect Sun's ZFS to an equally good job a significantly lower cost.
Terje
Re: (Score:1)
Hard to say for certain, as comparing WAFL and ZFS ignores the often overlooked but additional integration that you get with a Filer as opposed to a more general purpose system running ZFS, which is still pretty green when it comes to this sort of stuff:
http://mail.opensolaris.org/pipermail/zfs-discuss/2006-November/036124.html [opensolaris.org].
It's been my experience that data corruption typically occurs in RAM (ECC), at HBA, ca
Faulty offloading (Score:2)
The only way I found out was with an Ethereal trace at each end - I could see that every 0x2000 bytes there was a corruption. We turned TCP segment offloading off, and it worked fine because the maximum packet size was 1536 bytes - before 0x2000.
Even Microsoft tried to push for ECC RAM (Score:2)
Earlier this decade I was living with strange, random crashes when booting Linux that would only seemingly only occur when booting from cold (but not every time!). It was only years later when running a memtest on someone else's sticks (which turned out to
data corruption happens often and easily (Score:2)
The filesystem should provide this.
Linux people have been denying for years that hardware will cause data corruption. Therefore they can deny their own responsibility in detecting and correcting it.
It is everyone's responsibility to make OS people aware of how often hardware causes data corruption.
http://www.storagetruth.org/index.php/2006/data-corruption-happens-easily/ [storagetruth.org]
Re: (Score:2)
Re: (Score:2)