This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Latest kernel from GIT destroys my NAND flash

Hello,

the latest version of Texas Instruments' fork of the linux kernel (from git://arago-project.org/git/projects/linux-omap3.git, branch ti81xx-master) is killing my NAND flash. 

We've reproduced this behavior both with the current ti81xx-master branches' HEAD (commit 52752d53adda8c345650c5c7000856cffc638208) as well as with the source tagged with v2.6.37_TI814XPSP_04.01.00.07.

The symptoms: We're relying on UBI and UBIFS and after incorporating the latest GIT patches UBIFS keeps reporting bad NAND blocks. At every reboot new bad blocks were reported by UBI; often these blocks have just been erased, so they really should not be broken. A lot of blocks have been marked bad this way.

We tried with multiple factory-new NAND flash chips - all had an increasing number of bad blocks reported after every boot - so the NAND is most likely not broken.

The behavior went away after we rolled back to an old version (commit 3228b1d8e5270ee56002e7f59d839ed8b6b85606) which does not have this problem (but unfortunately lacks of BCH8 support).

Here's a snapshot of UBI reporting bad blocks:

[ 6.350000] UBI error: ubi_io_read: error -74 (ECC error) while reading 131072 bytes from PEB 525:0, read 131s 
[ 6.370000] UBI error: torture_peb: read problems on freshly erased PEB 525, must be bad 
[ 6.370000] UBI error: erase_worker: failed to erase PEB 525, error -5 
[ 6.680000] UBI error: ubi_io_read: error -74 (ECC error) while reading 126976 bytes from PEB 13:4096, read 1s 
[ 6.840000] UBI error: ubi_io_read: error -74 (ECC error) while reading 131072 bytes from PEB 523:0, read 131s 
[ 6.850000] UBI error: torture_peb: read problems on freshly erased PEB 523, must be bad 
[ 6.860000] UBI error: erase_worker: failed to erase PEB 523, error -5



Block addresses (and PEB values) vary of course, since new blocks are reported broken after every boot.
  • Hello?

    Hello there?

    Is anybody home?

  • Hi,

    Is your rootfs in NAND?

    If yes, you may need to erase nand (flash_erase) and write your rootfs to nand using the new release.

    The problem you were facing is probably caused by mismatching the existing ECC schema with the new ECC schema (bch8) which will cause bad blocks.

    Regards,

    Yan

  • Hi,

    you seem to miss the point. I reformatted the whole device exactly because you switched to BCH8 ECC. Otherwise I would not even be able to boot, not a single NAND access would have worked, init would not have been executed by the kernel, etc. This is not what is happening. I am losing flash blocks on freshly formatted NAND, even on factory-new NAND (see my original post for this) because the driver reports random ECC errors occasionally for blocks which are perfectly fine.

    Here is how you can reproduce the problem:

    - Fetch the Texas Instruments' Linux kernel fork 

    git clone git://arago-project.org/git/projects/linux-omap3.git

    - Switch to the 81xx branch: 

    git checkout ti81xx-master

    - Now switch to the revision tag I specified:

    git checkout 52752d53adda8c345650c5c7000856cffc638208

    - Build the kernel

    - Flash the kernel to a device

    - Reformat the whole device

    => You will see new bad blocks reported for your NAND flash after every boot.

    This does not happen e.g.  with (older) revision 3228b1d8e5270ee56002e7f59d839ed8b6b85606.

  • Thilo,

    I have flashed UBIFS on the latest code several times and have never encountered this issue. The UBIFS mount and read/write seem to take place perfectly for me. Can you make sure that the UBIFS image you have created is as per the documentation mentioned here <http://processors.wiki.ti.com/index.php/UBIFS_Support>?

  • Parth,

    did you use the kernel from arago GIT with revision 52752d53adda8c345650c5c7000856cffc638208 like I specified?

  • Thilo,

    Yes, I am using the same code base as you and UBIFS is working fine with me. That is why I wanted to cross-check if your UBIFS image is fine or not.

    Regards,

    Parth

  • I don't use a pre-generated UBIFS image. We use a read-only squashfs with transparent writeable AUFS overlays which store data on a UBI/UBIFS back-end partition. See https://github.com/DFE/HidaV/wiki/File-system-layout for documentation.

    Our UBIFS is completely empty initially. It fills up when modifying something on the root fs (overlay) since AUFS only tracks changes to the underlying file system. Even attaching and mounting a completely empty UBIFS will report bad blocks.

    I can reproduce the problem even without UBI/UBIFS just by accessing the NAND flash (using e.g. "dd"). I get bad blocks reported even when the NAND is freshly formatted.

    Additional info: I'm on a DM8148 EVM dev board.

  • Thilo,

    Have you solved this problem?  I'm encountering the same issue on a DM8168 board.

    Thanks,

    Jason

  • No, we were unable to solve the issue, and Texas Instruments is ignoring this (as they, in our experience, seem to ignore the ti81xx platform and its massive Linux problems  as a whole).

    We're still using the old kernel release tag "v2.6.37_TI814XPSP_04.01.00.06.patch2". This tag corresponds to commit "745a06cb913ad7a9ab44855aa36d8c012cd5793b" which is referenced in Texas Instrument's original kernel build recipe in their official "meta-ti" OpenEmbedded board support layer. Arago, Yocto, Poky and others use this version to build their firmware.  No-one seems to use more recent versions than that. I think the current GIT head of TI's kernel fork is completely broken and nobody noticed because nobody uses it.


  • Thilo,

    I think the NAND flash on the 8148 EVM is an SLC part.  1-bit HW ECC is sufficient for this.  I was able to change the ECC mode in the latest TI GIT kernel by modifying the following files:

    - kernel/arch/arm/mach-omap2/board-flash.c

    (board_nand_init function - "board_nand_data.ecc_opt = OMAP_ECC_HAMMING_CODE_HW;")

    -kernel/drivers/mtd/nand/omap2.c

    (probe function - "pdata->ecc_opt = OMAP_ECC_HAMMING_CODE_HW;")

    This change eliminated the problem.

    I'm not a NAND expert by any means, but I'm not sure if using BCH8 HW ECC with an SLC part is actually a valid thing to do.  I suspect that the 8-bit ECC is using too much of the OOB space and that UBIFS tries to access some of this space and thus corrupts the ECC.  This might be fixed by modifying the OOB layout, but I did not go down that road.

  • Jason,

    thanks for your reply and also thanks for the workaround you proposed.

    We're using Hamming ECC on our systems except for the NAND area we write the bootloader to. Considering your concerns BCH8 would be a valid algorithm to use in NAND, though. I think it's reasonable to use it by default as the current TI kernel code does. Well, it _would_ be reasonable if the driver was not buggy. In fact you are required to use BCH8 ECC if you want to boot from NAND since the ROM loader can only run NAND bootloaders which have been written in BCH8 ECC mode.

    We do boot from NAND so your workaround would unfortunately only solve a part of our problem. Also the NAND driver has a buggy sub-page support which breaks UBI and YAFFS2 for some people.

  • Thilo,

    I have almost solved the issue with BCH8. Could you please mail me @ renjith.thomas@pathpartnertech.com