Forgot your password?
typodupeerror
Bug Data Storage Linux

EXT4 Data Corruption Bug Hits Linux Kernel 249

Posted by Soulskill
from the plenty-of-time-to-fix dept.
An anonymous reader writes "An EXT4 file-system data corruption issue has reached the stable Linux kernel. The latest Linux 3.4, 3.5, 3.6 stable kernels have an EXT4 file-system bug described as an apparent serious progressive ext4 data corruption bug. Kernel developers have found and bisected the kernel issue but are still working on a proper fix for the stable Linux kernel. The EXT4 file-system can experience data loss if the file-system is remounted (or the system rebooted) too often."
This discussion has been archived. No new comments can be posted.

EXT4 Data Corruption Bug Hits Linux Kernel

Comments Filter:
  • Re:Bisected? (Score:5, Informative)

    by Slayne (10400) on Wednesday October 24, 2012 @03:30PM (#41755891) Homepage

    Nope - bisection is a common technique for tracking down the cause of a bug by doing a binary search through the code history.
    https://en.wikipedia.org/wiki/Code_Bisection

  • by dacut (243842) on Wednesday October 24, 2012 @03:38PM (#41756001)

    From Ted Ts'o's commentary, it's an optimization ("jbd2: don't write superblock when if its empty") gone awry:

    The reason why the problem happens rarely is that the effect of the buggy commit is that if the journal's starting block is zero, we fail to truncate the journal when we unmount the file system. This can happen if we mount and then unmount the file system fairly quickly, before the log has a chance to wrap.

    Basically, this optimization has the side effect of not updating the transaction log in this rare case. You can end up replaying old transactions after new ones, which will scramble metadata blocks. Given the rather unique conditions needed to hit this one, I'm not going to lose any sleep over any servers running without Ted's fix (though I'll certainly apply it once RedHat releases the patch).

  • Re:Bisected? (Score:5, Informative)

    by petermgreen (876956) <plugwashNO@SPAMp10link.net> on Wednesday October 24, 2012 @03:51PM (#41756179) Homepage

    What they actually split in half is a sequence of changesets (also known as commits).

    The idea is you have a seqence of changesets that take you from the last known good revision to the first known bad revision. By splitting that sequence in half and determining if the revsion in the middle is good or bad you can in principle halve the number of revisions between last known good and first known bad until you find the revision that introduced the bug. Reality is messier because of nonlinear history, because some revisions may be "broken" such that it is not possible to determine if they are "good" or "bad" and because some bugs may be difficult to test for but still bisection is a useful tool for finding problem revisions among a long history relatively easill.

  • by Tough Love (215404) on Wednesday October 24, 2012 @03:58PM (#41756273)

    It means you could get an incorrect replay after a crash and end up needing to do a fsck. Good thing Ext2/3/4 fsck is awesome. Of course, having no replay bug will be much better. Note: the bug was introduced this October 8th. You are not running this kernel on your server or workstation unless you are a dev, it hasn't filtered through to distros yet.

  • Summary is wrong (Score:5, Informative)

    by DrJimbo (594231) on Wednesday October 24, 2012 @04:05PM (#41756397)

    The EXT4 file-system can experience data loss if the file-system is remounted (or the system rebooted) too often.

    This is wrong. The problem occurs when the fs is unmounted too *soon*. Twice in a row. The bug only appears if the journal buffer does not wrap. You only get catastrophic results if this happens twice in a row.

  • Re:Low impact (Score:5, Informative)

    by h4rr4r (612664) on Wednesday October 24, 2012 @04:16PM (#41756543)

    http://answers.microsoft.com/en-us/windows/forum/windows_cp-files/bug-report-serious-filesystem-corruption-and-data/17f69e19-92ca-4e1e-b9d5-f78f1ac4e963 [microsoft.com]

    Bugs happen. The difference here is that Linux development is done in the open so people find out about them.

  • by Anonymous Coward on Wednesday October 24, 2012 @05:34PM (#41757633)

    Ubuntu users are at risk.

    http://www.ubuntuupdates.org/package/core/quantal/main/proposed/linux-image-3.5.0-18-generic

    Look for " jbd2: don't write superblock when if its empty
            - LP: #1066176"

    If any Ubuntu users have proposed repo enabled and they've updated to 3.5.0-18, they're vulnerable.

  • Re:Low impact (Score:5, Informative)

    by sk999 (846068) on Wednesday October 24, 2012 @06:17PM (#41758291)

    Still, for all of the shit that Linux users talk about Windows, WINDOWS has NEVER had anything as serious as a FILE system CORRUPTION bug.

    Finally, someone talking sense ... oh wait.

    http://www.computerworld.com/s/article/9054178/Microsoft_s_Windows_Home_Server_corrupts_files [computerworld.com]

    "Microsoft's Windows Home Server CORRUPTS FILES"
    "'Don't edit' list includes photos, as well as Quicken and QuickBooks files, warns Microsoft; no word on patch"

    Never mind ...

  • by fatphil (181876) on Wednesday October 24, 2012 @06:29PM (#41758439) Homepage
    $ git show eeecef0af5e
    commit eeecef0af5ea4efd763c9554cf2bd80fc4a0efd3
    Author: Eric Sandeen <sandeen@redhat.com>
    Date: Sat Aug 18 22:29:40 2012 -0400

            jbd2: don't write superblock when if its empty
  • Re:Part of the game (Score:4, Informative)

    by fatphil (181876) on Wednesday October 24, 2012 @06:33PM (#41758489) Homepage
    It is *not* 10 days old.

    linux-stable$ git show 14b4ed22a6
    commit 14b4ed22a6b5fc1549504336131be4f5f6ba1bf4
    Author: Eric Sandeen <sandeen@redhat.com>
    Date: Sat Aug 18 22:29:40 2012 -0400

            jbd2: don't write superblock when if its empty

            commit eeecef0af5ea4efd763c9554cf2bd80fc4a0efd3 upstream.

            This sequence:

            # truncate --size=1g fsfile
            # mkfs.ext4 -F fsfile
            # mount -o loop,ro fsfile /mnt
            # umount /mnt
            # dmesg | tail

            results in an IO error when unmounting the RO filesystem:

            [ 318.020828] Buffer I/O error on device loop1, logical block 196608
            [ 318.027024] lost page write due to I/O error on loop1
            [ 318.032088] JBD2: Error -5 detected when updating journal superblock for loop1-8.

            This was a regression introduced by commit 24bcc89c7e7c: "jbd2: split
            updating of journal superblock and marking journal empty".

            Signed-off-by: Eric Sandeen <sandeen@redhat.com>
            Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
            Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

    diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c
    index e149b99..484b8d1 100644
    --- a/fs/jbd2/journal.c
    +++ b/fs/jbd2/journal.c
    @@ -1354,6 +1354,11 @@ static void jbd2_mark_journal_empty(journal_t *journal)

                    BUG_ON(!mutex_is_locked(&journal->j_checkpoint_mutex));
                    read_lock(&journal->j_state_lock);
    + /* Is it already empty? */
    + if (sb->s_start == 0) {
    + read_unlock(&journal->j_state_lock);
    + return;
    + }
                    jbd_debug(1, "JBD2: Marking journal as empty (seq %d)\n",
                                        journal->j_tail_sequence);
  • Re:Bisected? (Score:5, Informative)

    by Just Some Guy (3352) <kirk+slashdot@strauser.com> on Wednesday October 24, 2012 @06:56PM (#41758765) Homepage Journal

    The summary should say "bisected and found" not "found and bisected". Bisecting is a way of finding bugs.

    No. They found the bug, then bisected the commits between "last known working" and HEAD to discover what patch caused it.

  • Re:Low impact (Score:5, Informative)

    by sk999 (846068) on Wednesday October 24, 2012 @07:36PM (#41759199)

    Nice try, but fail. That wasn't a bug in Windows, it was a bug in applications.

    Really? Not according to Microsoft.

    http://support.microsoft.com/kb/946676 [microsoft.com]

    "A BUG has been discovered in the way that the initial release of Windows Home SERVER manages FILE transfer and balancing across multiple hard drives. In certain cases, depending on application use patterns, timing, and the workload that is placed on the Windows Home Server-based computer, certain FILES could become CORRUPTED."

    "... For distributing data across the different hard drives that are MANAGED by WINDOWS Home Server, the WINDOWS Home Server mini-filter driver REDIRECTS I/O ... A BUG has been discovered in the REDIRECTION mechanism which, in certain cases, depending on application use patterns, timing, and workload, may cause interactions between NTFS, the Memory Manager, and the Cache Manager to get out of sync. This causes CORRUPTED data to be written to FILES."

  • Re:Bisected? (Score:4, Informative)

    by Tough Love (215404) on Wednesday October 24, 2012 @09:08PM (#41759941)

    Ah I see, we have ambiguity about what "find a bug" means. From the user's perspective, "finding a bug" means producing the buggy behavior. But from the developer's perspective, "finding a bug" means finding the erroneous code. And we are talking about developers here. From my perspective, until the bug was "found" by bisecting it was only "known to exist", not found. See?

    By the way, I've actually bisected bugs, have you? No? OK.

  • by Anonymous Coward on Wednesday October 24, 2012 @09:37PM (#41760151)

    I have a Google+ post where I've posted my latest updates to this still-developing story:

    https://plus.google.com/117091380454742934025/posts/Wcc5tMiCgq7

    Also, I will note that before I send any pull request to Linus, I have run a very extensive set of file system regression tests, using the standard xfstests suite of tests (originally developed by SGI to test xfs, and now used by all of the major file system authors). So for example, my development laptop, which I am currently using to post this note, is currently running v3.6.3 with the ext4 patches which I have pushed to Linus for the 3.7 kernel. Why am I willing to do this? Specifically because I've run a very large set of automated regression tests on a very regular basis, and certainly before pushing the latest set of patches to Linus. So while it is no guarantee of 100% perfection, I and many other kernel developers *are* willing to eat our own dogfood.

  • by tytso (63275) on Wednesday October 24, 2012 @09:42PM (#41760179) Homepage

    I have a Google+ post where I've posted my latest updates to this still-developing story:

    https://plus.google.com/117091380454742934025/posts/Wcc5tMiCgq7 [google.com]

    Also, I will note that before I send any pull request to Linus, I have run a very extensive set of file system regression tests, using the standard xfstests suite of tests (originally developed by SGI to test xfs, and now used by all of the major file system authors). So for example, my development laptop, which I am currently using to post this note, is currently running v3.6.3 with the ext4 patches which I have pushed to Linus for the 3.7 kernel. Why am I willing to do this? Specifically because I've run a very large set of automated regression tests on a very regular basis, and certainly before pushing the latest set of patches to Linus. So while it is no guarantee of 100% perfection, I and many other kernel developers *are* willing to eat our own dogfood.

Pound for pound, the amoeba is the most vicious animal on earth.

Working...