Oracle Adds Data-integrity Code To Linux Kernel 53
jazir1979 writes "ZDNet is reporting that Oracle has added code to the Linux kernel for ensuring data integrity. The code has been developed in partnership with Emulex and was recently accepted into the 2.6.27 kernel release." According to the article, "The code adds metadata to data at rest or in transit, to monitor whether that data has been corrupted. It helps make sure that I/O operations are valid by looking at that metadata — which acts as verification information — exchanged during data transmissions."
Re: (Score:3, Insightful)
The Year of Linux on the Database? Nah, that happened a long time ago.
Dumb question... (Score:4, Insightful)
How badly does this affect performance?
Re:Security??? (Score:5, Insightful)
Integrity is a security principle, and that is the sense that they're using the word in the summary. It's pretty much the only definition of the word that makes sense in a computing context. More precisely, we're talking about confidence that the data stored in the system is the same as the data retrieved at a later time. The only difference between this and a more cryptographic sense of the word is that this doesn't attempt to guard against malicious attacks if an adversary had offline access to the disk. (Or so I presume, having not RTFA'd).
Re:Dumb question... (Score:4, Informative)
Re: (Score:2)
CRC is somewhat expensive. 200-300 MB/s on a modern CPU looking into ways to optimize
SSE4 will have a CRC instruction (any poly)
linky(pdf) [usenix.org]
Not sure what he means by CPU there, if he means core then it's not so bad (other than freaking expensive per core Oracle licensing), if that's per processor then that's very bad. Guess I might only be able to implement this on the new Shanghai beast we got last week or a new Corei7 Xeon system
Oracle submitted a 2nd patch (Score:4, Funny)
It adds a 2nd layer of metadata that is used to verify the first layer of metadata wasn't corrupted so you can be EXTRA confident that your original data was actually handled correctly.
Then Oracle submitted a 3rd patch (Score:3, Funny)
Re: (Score:2)
I have been running it since well, when it became available for Linux and I had to hand patch and compile my own kernel. Not once have I had any file system corruption in the intervening years. Well apart from when a disk developed bad sectors, but that is hardly the fault of XFS...
Some time ago (I forget when) I did have a few files truncated to zero on a kernel panic usually a failed restore, and usually my bookmarks. Not had that in six or seven years now though.
They have even fixed the issue where you n
Terribly old news (Score:4, Informative)
Block integrity patches were discussed in excellent article on LWN [lwn.net] in July 2008. Kernel 2.6.27 was released in October 2008. This is old news.
Re:Terribly old news (Score:5, Insightful)
Oh my god! Not news from October. That was going on two months ago. Everyone knows everything that happened two months ago. What were the editors thinking? Fire them immediately. Let's all go to digg or reddit or myspace where they don't do things like post things that are almost two months old. PANIC PANIC PANIC!!!
Wow, just let other people read it and go on about your business not caring.
Re:Terribly old news (Score:5, Informative)
That LWN writeup is far better too though, TFA is terrible. LWN makes it clear that this adds device checksum support, i.e. if your SATA drive supports adding checksum data to blocks this patch will enable that functionality.
Re: (Score:2)
Thanks for that summary. It's been many years now since I kept up with the kernel changelogs and articles... the kernel now Just Works.
Congratulations... Oracle (Score:3, Insightful)
On a more serious note (yes I did RTFA), somebody please explain where this fits. Other than network or disk errors (which generally already have error detection schemes), I'm not sure what the target problem is that this is supposed to fix. The article says "the code helps maintain integrity as data moves from application to database, and from Linux operating system to disk storage", that it checks I/O operations, and that "code contribution includes generic support for data integrity at the block and file-system layers". That's still not clear what they think the problem is. Don't most of the modern file systems already check data operations?
Re: (Score:3, Informative)
I don't know where it fits either, but ZFS and eventually BTRFS actually have checksums at the block level, and can heal over corrupted blocks using redundant copies whose checksums do work. That alone is enough reason to use ZFS for a file server, but similar improvements could be made inside the Linux stack without a new filesystem on top. However ZFS' reliability also comes from copy-on-write updates which is not trivially installed into an existing filesystem.
Re: (Score:2)
Below the filesystem. ZFS can export zvols, which are just block devices whose storage maps on to ZFS blocks. They get checksumming and copy-on-write semantics, even snapshotting, and yet still support any filesystem on top, a whole partition table, a virtual machine disk, or whatever. They're just block devices, but they still get most of the reliability advantages of ZFS itself.
It's not as efficient as storing a ZFS filesystem which can track its used/free space and feed that in to the reliability systems
Re: (Score:3)
I'm not certain but it appears to be checksumming data while it is moving around the kernel after a write or read call is made.
Seems like something that should be handled in hardware with ECC, but what do I know.
Re:Congratulations... Oracle (Score:4, Informative)
I'm not certain but it appears to be checksumming data while it is moving around the kernel after a write or read call is made.
Seems like something that should be handled in hardware with ECC, but what do I know.
Kernel bugs can cause data to get corrupted and hardware ECC won't correct that. Likewise with transfers from memory to disk. Ultimately it'll need to be a hardware/software thing but the software portion is needed as well.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
SAN vendor (Xiotech) has the first system to support the standard (Emprise 5000/7000
Which standard?
Has Xiotech added Emulex HBAs to the compatibility matrix for those systems yet? ROFL.
Re: (Score:2)
RAM errors.
Spontaneous bit flips change data in transit.
It also helps against errors in kernel code or malicious data injection attacks
Re: (Score:3, Informative)
Re: (Score:2)
You might not understand why we need it, but trust me, it is needed. Not all storage device drivers a
Re: (Score:2)
Re: (Score:2)
It may not be the same implementation, but Solaris on Sparc has built in ECC across all data paths, including CPU -> memory, CPU -
erasure codes (Score:1)
Re: (Score:2)
vs a journaled fs? (Score:2)
Re: (Score:3, Informative)
The purpose of a journal is to make sure that operations either happen or they don't happen - i.e. you don't leave the filesystem in some half way state if the power goes out.
It doesn't verify the actual data written or anything.
Re: (Score:2)
Looks like it is T10 SCSI (Score:2)
Info [wwpi.com]
Re: (Score:2)
Maybe linux will break .8% of the market with this groundbreaking advance.. *snicker*
Windows 37.4% (268), Linux 34.6% (248), Unknown 19.2% (138), Macintosh 7.6% (55), FreeBSD 0.5% (4), Solaris 0.4% (3).
I think it depends on which market you're talking about.