AM335x NAND data intermittent problems

Hi Rod,

I will forward this to the SW team. Can you post what Linux version you are using?

Hi,

Using ECC is declared for SDK 7 in a .dts file and for SDK 6 in a board file. In nandtest application the ECC is active.  Please write what SDK do you using SDK 6 or 7. And can you check if this nand work well in a u-boot on any of your set. One good site for MTD utils is:
http://processors.wiki.ti.com/index.php/Mtdutils

BR
Ivan

We are using buildroot to manage the kernel and rootfs build.  Here's the linux version info from the boot log:

Linux version 3.2.0 (rodc@rodc-linux-lap) (gcc version 4.7.3 2013
0328 (prerelease) (crosstool-NG linaro-1.13.1-4.7-2013.04-20130415 - Linaro GCC
2013.04) ) #24 Mon Sep 15 15:35:36 EDT 2014

We are using BCH8.  In the board setup code this is ecc_opt = OMAP_ECC_BCH8_CODE_HW. 

The Micron datasheet for our flash says that 4-bit ECC is the minimum required ECC.  I am pretty certain that we did not change this from what TI uses in the original EVM code, and the datasheet for the flash they use says the same thing as ours regarding ECC.

We are using SDK am335x-evm-05.06.00

I have not "torture tested" the "nand erase.chip" and "nand write" in u-boot, but I have used them several times and have not noticed any issues.  One thing that I did notice is that when I have the bootup problem with the rootfs, and I then re-boot and stop at u-boot, if I do "nand bad" to check for bad blocks, there are never any bad blocks detected. 

The NAND configuration is done in board-am335xevm.c:

static void evm_nand_init(int evm_id, int profile)
{
    struct omap_nand_platform_data *pdata;
    struct gpmc_devices_info gpmc_device[2] = {
        { NULL, 0 },
        { NULL, 0 },
    };

    setup_pin_mux(nand_pin_mux);
    pdata = omap_nand_init(am335x_nand_partitions,
        ARRAY_SIZE(am335x_nand_partitions), 0, 0,
        &am335x_nand_timings);
    if (!pdata)
        return;
    pdata->ecc_opt =OMAP_ECC_BCH8_CODE_HW;
    pdata->elm_used = true;
    gpmc_device[0].pdata = pdata;
    gpmc_device[0].flag = GPMC_DEVICE_NAND;

    omap_init_gpmc(gpmc_device, sizeof(gpmc_device));
    omap_init_elm();
}

Hi,

Did you change the timing parameters in a am335x_nand_timings structure.

BR
Ivan

No, the parameters were left unchanged.  And I re-compared the datasheets for the NAND that TI used on the EVM and the NAND that we use on our boards.  These are both Micron NANDs and the timing, DC and AC electrical characteristics are identical.

I have discovered that the file system that we picked for our rootfs - squashfs, may have problems when used with NAND.  This is a read-only fs, but it apparently does not do NAND bad-block detection.  So it could apparently exhibit intermittent problems even though it never modifies the NAND.  Squashfs is running on top of MTD.

However, my device (which currently is exhibiting the problem I reported), does not have a bad block problem.  I checked the NAND with u-boot's bad block checker.  From linux I also ran "nanddump -f /dev/null /dev/mtd7" to have a nand-aware utility read over the partition (/dev/mtd7).  Here is the result:

[root@MT2000 sbin]# nanddump -f /dev/null /dev/mtd7
ECC failed: 0
ECC corrected: 5
Number of bad blocks: 0
Number of bbt blocks: 0
Block size 131072, page size 2048, OOB size 64
Dumping data starting at 0x00000000 and ending at 0x08000000...
ECC: 1 corrected bitflip(s) at offset 0x00537800
[root@MT2000 sbin]#

So it found some ECCs and corrected a bitflip, but no bad block found.  I don't know if the "bitflip" incident or the ECC correction is something that squashfs would not detect and therefore cause it to read incorrect data and report the errors I showed earlier.

Does anyone know if this could explain the issues I reported?

Does someone have an idea about what read-only rootfs to use with NAND?

Thanks.

Hi Rod,

Usually squashfs works with a journal, are you sure that in your squashfs you switched off the journal. Which version of Squashfs filesystem do you use.

BR
Ivan

Hi Ivan,

We are using squashfs 4.0.  I don't see anything about a journal either in the make menuconfig for the linux squashfs options or in the doc for squashfs.  squashfs is is a read-only filesystem, so wouldn't a journal be unnecessary and unwanted?

When I dig into the web info, there are definitely some warnings about not using squashfs with NAND unless you have an underlying way to manage the bad blocks (like using UBIFS underneath).  Here are a few:

Dealing with NAND bad blocks in production programming 12/2013
http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/100/t/251129.aspx

NAND flash bad blocks combined with squashfs could be a dangerous combination 5/2010
http://www.lg-hack.info/cgi-bin/sn_forumr.cgi?cid=2675&fid=2679&tid=2721


Discussion about FS.  There is a link inside warning about using squashfs with nand
http://wiki.openwrt.org/doc/techref/filesystems#squashfs

Some of these links are older, but there does not seem to be any update to squashfs that I can find about being able to use it with NAND by itself.  Also, I'm not sure if the AM335x on-board NAND assist (the ELM) changes any of this.

Thanks,

Rod

Wolfgang,

Thanks for your input. I contacted the guys that work on MTD and UBI at linux-mtd@lists.infradead.org.  They replied that UBI is a good answer just as you did.  The link at the bottom of the the following email snippet points to some doc about how to set up UBI for just this usage:

--------------------------------

On Mon, Nov 24, 2014 at 10:21:16PM +0100, Richard Weinberger wrote:

> On Mon, Nov 24, 2014 at 9:41 PM, Rod Campbell <rod.campbell@telosalliance.com> wrote:

> > Hello,

> >

> > Is there an update about implementing a read-only file system on

> > NAND?  The most recent discussion I could find was from May 2012:

> >

> > http://lists.infradead.org/pipermail/linux-mtd/2012-May/041200.html

> >

> > The conclusion seemed to be to use ubi underneath something like

> > squashfs to provide bad-block awareness.  Is this still the recommended solution?

>

> Yes. Using the new UBI block driver this is trivial.

To expand on that:

 http://www.linux-mtd.infradead.org/doc/ubi.html#L_ubiblock

 Brian

--------------------------------

So I think using UBI underneath squashfs is how we will proceed.  The main thing I'm wondering about is increased boot time.  But data integrity comes first.

One gotcha I experienced tracking this down is that the reports I read talked about squashfs not handling "bad blocks".  And as far as I know (after running nand-aware utilities like nanddump and nandtest) this NAND MTD partition does NOT have bad blocks.  But it DOES have a "bitflip" reported when I run nanddump.  So I guess there are other issues that cause squashfs to read bad data besides an actual "bad block" situation.

I say this not because I want to try to use squashfs without UBI underneath, but because I thought that there may be some additional issue causing the intermittent problems we are experiencing - such as ECC not set up right, other NAND parameters not right, difference between NAND parameters in u-boot compared to linux and compared to the NAND chip itself, etc.

Thanks,

Rod