AM335x EVM rootfs on NAND - trouble

Niko Mauno

I've been familiarizing myself with TI's am335x EVM for over a week now.

Successes:
- boot EVM using the included Linux/Android SD cards.
- built 2nd and 3rd stage bootloaders from source code "u-boot-2011.09-psp04.06.00.03"
- built linux kernel from source code "linux-3.1.0-psp04.06.00.03.sdk"
- flashed self-built bootloaders to EVM's NAND and booted to them
- booted with TFTP/NFS using self-built kernel and rootfs
- flashed linux kernel to EVM

PROBLEMS:
- I've flashed root filesystem to NAND - I've tried both UBI and JFFS2 - these both work to some extent but there's obvious trouble.
* When booting (first time after flashing) with JFFS2 rootfs, at boot/mount time, following errors reported:
...
[   22.734349] JFFS2 notice: (626) read_dnode: wrong data CRC in data node at 0x0005b7f8: read 0x28d41a51, calculated 0x850c5a22.
[   30.493848] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x0212076c: read 0x945940f9, calculated 0x3e86588a.
[   32.516459] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x021f8bf4: read 0xf65ecbd9, calculated 0x25e5f2b1.
[   34.071718] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x00487738: read 0xaf467a9a, calculated 0xd5496e91.
[   38.625075] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x01e67598: read 0xae8ce98c, calculated 0xff96c45a.
[   40.805661] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x016bccbc: read 0xa1936e3, calculated 0x658c5886.
[   41.198161] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x0155e780: read 0x7046e4be, calculated 0xd315ce96.
[   42.010097] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x01286afc: read 0x7001a90e, calculated 0xdf7c3004.
[   47.529271] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x02d1860c: read 0xcc59fc33, calculated 0x3f6c56d6.
[   48.924694] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x02c1e9dc: read 0x631dbdb4, calculated 0xce23febe.
[   49.640718] JFFS2 notice: (1362) read_dnode: wrong data CRC in data node at 0x02cd3b40: read 0xeacf8a7e, calculated 0xc9c20e0c.
[   52.418622] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x02fad6b4: read 0x82947468, calculated 0xb9f3c207.
[   55.128544] JFFS2 notice: (1362) check_node_data: wrong data CRC in data node at 0x033c42d0: read 0xe20134e9, calculated 0x127a544.
[   59.288111] JFFS2 notice: (1362) read_direntry: header CRC failed on dirent node at 0x39e4bf0: read 0x8541237f, calculated 0xf3d461c
...
* When booting (first time after flashing) with UBI rootfs, at boot/mount time, following errors are reported:
...
[   91.819221] UBI: run torture test for PEB 1507
[   92.185899] UBI: PEB 1507 passed torture test, do not mark it as bad
[   92.332445] UBI: run torture test for PEB 1507
[   92.708611] UBI: PEB 1507 passed torture test, do not mark it as bad
[   92.847639] UBI: run torture test for PEB 1507
[   93.245844] UBI: PEB 1507 passed torture test, do not mark it as bad
[   93.401034] UBI: run torture test for PEB 1507
[   93.766578] UBI: PEB 1507 passed torture test, do not mark it as bad
[   93.942744] UBI: run torture test for PEB 1507
[   94.305433] UBI: PEB 1507 passed torture test, do not mark it as bad
[   94.462394] UBI: run torture test for PEB 1507
[   94.825511] UBI: PEB 1507 passed torture test, do not mark it as bad
[   94.972736] UBI: run torture test for PEB 1507
[   95.336975] UBI: PEB 1507 passed torture test, do not mark it as bad
[   95.492495] UBI: run torture test for PEB 1507
[   95.859982] UBI: PEB 1507 passed torture test, do not mark it as bad
...

In both JFFS2 and UBI cases, I've been (eventually) able to get to the shell prompt, but there is obvious trouble.

The rootfs NAND flashing has occurred from U-Boot and has been succesful with setting 'nandecc hw 2' - 'nandecc hw 0' didn't seem to work.

I've also tried flashing the NAND from tftp/nfs booted environment; it has been succesful for MLO and U-Boot, but when executing 'nandtest -m' for /dev/mtd6 or /dev/mtd7, the following kind of errors are always reported:
Marking bad blocks
ECC corrections: 0
ECC failures   : 0
Bad blocks     : 0
BBT blocks     : 0
00540000: reading...
   1 bit(s) ECC corrected at 00540000
00540000: checking...
compare failed. seed 986930424
Byte 0x5e01 is f6 should be f2
(nandtest binary is from package mtd-utils version 1.4.9)

I am currently uncertain where the badness occurs. My suspicion is that perhaps the ECC scheme of the provided linux kernel is somehow incompatible or inoperative. I was hoping that you could provide me an explanation and perhaps some pointers how to get over this problem.

My root filesystem images have been created with the command:
mkfs.jffs2 -x lzo --root=/path/to/crossroot --faketime \
--output=/path/to/image.jffs2 --pad --little-endian \
--eraseblock=0x20000 -n
for JFFS2, and:
mkfs.ubifs -r /path/to/crossroot -o /path/to/ubifs.img -F -m 2048 -e 126976 -c 1580
echo "
\[ubifs\]
mode=ubi
image=/path/to/ubifs.img
vol_id=0
vol_type=dynamic
vol_name=rootfs
vol_flags=autoresize" > ubinize.cfg
ubinize -o /path/to/image.ubi -m 2048 -p 128KiB -s 512 -O 2048 ubinize.cfg
for UBI.

over 13 years ago

0 Niko Mauno over 13 years ago

Prodigy 40 points

Narrowing down the potential problem causes;
I've programmed the kernel and rootfs images to NAND from both U-Boot and from NFS-mounted userspace.
Then in both cases, I dumped the images from NAND to RAM and did a CRC32 check (using U-Boot crc32 command).
All tests passed, i.e. the images are written correctly to NAND.

So the problem still is;
- 'nandtest -m' from mtd-utils 1.4.9 fails for /dev/mtd6 and /dev/mtd7
- during first (and consequtive) boots, rootfs reports dozens of
'UBI: run torture test for PEB XXXX
UBI: PEB XXX passed torture test, do not mark it as bad'
or
'JFFS2 notice: (XXXX) check_node_data: wrong data CRC in data node at 0xYYYY: read 0xZZZZ, calculated 0xWWWW'
(depending if rootfs is UBI or JFFS2).

Anybody?

0 Niko Mauno over 13 years ago in reply to Niko Mauno

Prodigy 40 points

Bumping thread -- also mentioning that I'm starting to get more serious errors during boot:

[   43.004487] UBI: PEB 1072 passed torture test, do not mark it as bad
[   43.142257] UBI: run torture test for PEB 1072
[   43.454632] UBI: PEB 1072 passed torture test, do not mark it as bad
[   44.012584] UBIFS error (pid 1223): make_reservation: cannot reserve 160 bytes in jhead 1, error -30
[   44.022122] UBIFS error (pid 1223): ubifs_write_inode: can't write inode 6783, error -30
[   45.680774] UBI: run torture test for PEB 1072
[   45.999190] UBI: PEB 1072 passed torture test, do not mark it as bad
[   46.122502] UBI: run torture test for PEB 1072
[   46.434107] UBI: PEB 1072 passed torture test, do not mark it as bad
[   46.552691] UBI: run torture test for PEB 1072
[   46.864128] UBI: PEB 1072 passed torture test, do not mark it as bad
[   47.002624] UBI: run torture test for PEB 1072
[   47.315389] UBI: PEB 1072 passed torture test, do not mark it as bad
[   47.452401] UBI: run torture test for PEB 1072
[   47.764120] UBI: PEB 1072 passed torture test, do not mark it as bad
[   47.902810] UBI: run torture test for PEB 1072
[   48.214670] UBI: PEB 1072 passed torture test, do not mark it as bad
[   48.352688] UBI: run torture test for PEB 1072
[   48.664767] UBI: PEB 1072 passed torture test, do not mark it as bad
[   48.792410] UBI: run torture test for PEB 1072
[   49.105245] UBI: PEB 1072 passed torture test, do not mark it as bad
[   49.252829] UBI: run torture test for PEB 1072
[   49.564686] UBI: PEB 1072 passed torture test, do not mark it as bad
[   49.702192] UBI: run torture test for PEB 1072
[   50.013877] UBI: PEB 1072 passed torture test, do not mark it as bad
[   50.152206] UBI: run torture test for PEB 1072
[   50.463859] UBI: PEB 1072 passed torture test, do not mark it as bad
[   50.602159] UBI: run torture test for PEB 1072
[   50.913744] UBI: PEB 1072 passed torture test, do not mark it as bad
[   51.062418] UBI: run torture test for PEB 1072
[   51.375003] UBI: PEB 1072 passed torture test, do not mark it as bad
[   51.512199] UBI: run torture test for PEB 1072
[   51.824034] UBI: PEB 1072 passed torture test, do not mark it as bad
[   51.972212] UBI: run torture test for PEB 1072
[   52.283882] UBI: PEB 1072 passed torture test, do not mark it as bad
[   52.422407] UBI: run torture test for PEB 1072
[   52.734081] UBI: PEB 1072 passed torture test, do not mark it as bad
[   52.872471] UBI: run torture test for PEB 1072
[   53.185359] UBI: PEB 1072 passed torture test, do not mark it as bad
[   53.311707] UBI: run torture test for PEB 1072
[   53.623468] UBI: PEB 1072 passed torture test, do not mark it as bad
[   53.752174] UBI: run torture test for PEB 1072
[   54.063534] UBI: PEB 1072 passed torture test, do not mark it as bad
[   54.070313] UBIFS error (pid 1223): make_reservation: cannot reserve 160 bytes in jhead 1, error -30
[   54.079969] UBIFS error (pid 1223): ubifs_write_inode: can't write inode 6789, error -30
[   54.098242] UBIFS error (pid 1223): make_reservation: cannot reserve 160 bytes in jhead 1, error -30
[   54.107870] UBIFS error (pid 1223): ubifs_write_inode: can't write inode 6563, error -30
[   54.137409] UBIFS error (pid 1223): make_reservation: cannot reserve 160 bytes in jhead 1, error -30
[   54.146952] UBIFS error (pid 1223): ubifs_write_inode: can't write inode 6557, error -30
[   54.155416] UBIFS error (pid 1223): make_reservation: cannot reserve 160 bytes in jhead 1, error -30
[   54.164986] UBIFS error (pid 1223): ubifs_write_inode: can't write inode 6562, error -30
[   54.184347] UBIFS error (pid 1223): make_reservation: cannot reserve 160 bytes in jhead 1, error -30
[   54.193969] UBIFS error (pid 1223): ubifs_write_inode: can't write inode 6552, error -30
[   54.212491] UBIFS error (pid 1223): make_reservation: cannot reserve 160 bytes in jhead 1, error -30
[   54.222017] UBIFS error (pid 1223): ubifs_write_inode: can't write inode 6790, error -30
[   54.282322] UBI: run torture test for PEB 1072
[   54.593967] UBI: PEB 1072 passed torture test, do not mark it as bad

sometimes the rootfs goes to unbootable state, after which reflashing the rootfs is about the only option.

Furthermore, another developer with another am335x EVM (he works for the same project as me) verified told that he's also encountering similar problems with his board.

0 Avinash Philip over 13 years ago in reply to Niko Mauno

TI__Intellectual 950 points

Hi,

By default AM335x doesn’t support JFFS2.

Since AM335x uses BCH8 as ECC scheme, UBI-FS is the recommended NAND Filesystem.

However, if you want to use JFFS2, kindly follow steps mentioned in

http://processors.wiki.ti.com/index.php/AM335x_JFFS2_Support_Guide#AM335x_JFFS2_support

Related to UBIFS torture test, the fix will be released in next release.

Kindly test with the attached patch and let me know.

Regards,

Avinash

gpmc_timings.txt

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
diff --git a/arch/arm/mach-omap2/board-flash.c b/arch/arm/mach-omap2/board-flash.c
index 9c1a358..aaaf351 100644
--- a/arch/arm/mach-omap2/board-flash.c
+++ b/arch/arm/mach-omap2/board-flash.c
@@ -131,6 +131,29 @@ static struct gpmc_timings nand_timings = {
    .wr_data_mux_bus = 0,
 };
 
+static struct gpmc_timings am335x_nand_timings = {
+
+   .sync_clk = 0,
+
+   .cs_on = 0,
+   .cs_rd_off = 44,
+   .cs_wr_off = 44,
+
+   .adv_on = 6,
+   .adv_rd_off = 34,
+   .adv_wr_off = 44,
+
+   .we_off = 40,
+   .oe_off = 54,
+
+   .access = 64,
+   .rd_cycle = 82,
+   .wr_cycle = 82,
+
+   .wr_access = 40,
+   .wr_data_mux_bus = 0,
+};
+
 static struct omap_nand_platform_data board_nand_data = {
    .gpmc_t     = &nand_timings,
 };
@@ -148,8 +171,9 @@ __init board_nand_init(struct mtd_partition *nand_parts,
    board_nand_data.gpmc_irq = OMAP_GPMC_IRQ_BASE + cs;
 
    if (cpu_is_am335x()) {
-       board_nand_data.ecc_opt  = OMAP_ECC_BCH8_CODE_HW;
-       board_nand_data.xfer_type = NAND_OMAP_PREFETCH_POLLED;
+       board_nand_data.ecc_opt     = OMAP_ECC_BCH8_CODE_HW;
+       board_nand_data.xfer_type   = NAND_OMAP_PREFETCH_POLLED;
+       board_nand_data.gpmc_t      = &am335x_nand_timings;
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

diff --git a/arch/arm/mach-omap2/board-flash.c b/arch/arm/mach-omap2/board-flash.c
index 9c1a358..aaaf351 100644
--- a/arch/arm/mach-omap2/board-flash.c
+++ b/arch/arm/mach-omap2/board-flash.c
@@ -131,6 +131,29 @@ static struct gpmc_timings nand_timings = {
 	.wr_data_mux_bus = 0,
 };
 
+static struct gpmc_timings am335x_nand_timings = {
+
+	.sync_clk = 0,
+
+	.cs_on = 0,
+	.cs_rd_off = 44,
+	.cs_wr_off = 44,
+
+	.adv_on = 6,
+	.adv_rd_off = 34,
+	.adv_wr_off = 44,
+
+	.we_off = 40,
+	.oe_off = 54,
+
+	.access = 64,
+	.rd_cycle = 82,
+	.wr_cycle = 82,
+
+	.wr_access = 40,
+	.wr_data_mux_bus = 0,
+};
+
 static struct omap_nand_platform_data board_nand_data = {
 	.gpmc_t		= &nand_timings,
 };
@@ -148,8 +171,9 @@ __init board_nand_init(struct mtd_partition *nand_parts,
 	board_nand_data.gpmc_irq = OMAP_GPMC_IRQ_BASE + cs;
 
 	if (cpu_is_am335x()) {
-		board_nand_data.ecc_opt  = OMAP_ECC_BCH8_CODE_HW;
-		board_nand_data.xfer_type = NAND_OMAP_PREFETCH_POLLED;
+		board_nand_data.ecc_opt		= OMAP_ECC_BCH8_CODE_HW;
+		board_nand_data.xfer_type	= NAND_OMAP_PREFETCH_POLLED;
+		board_nand_data.gpmc_t		= &am335x_nand_timings;
 	}
 
 	gpmc_nand_init(&board_nand_data);

0 Niko Mauno over 13 years ago in reply to Avinash Philip

Prodigy 40 points

Hello Avinash, and Thank You for the response.

> By default AM335x doesn’t support JFFS2.
> Since AM335x uses BCH8 as ECC scheme, UBI-FS is the recommended NAND Filesystem.

I recalled this was the case, but I wanted to include my JFFS2 test results, along UBI, for completion.

> Related to UBIFS torture test, the fix will be released in next release.
> Kindly test with the attached patch and let me know.

The patch You provided - adjusting the NAND timings - seems, according to my brief tests, to fix this issue:

First, I booted the EVM with patched kernel and executed 'nandtest -m -p 10 /dev/mtd7'.
Total of 49 ECC corrections were made.
At least one or more corrections were made during each pass.
After the nandtest command finished, it shows:
ECC corrections: 49
ECC failures   : 0
Bad blocks     : 0
BBT blocks     : 0

Next, I erased and programmed the UBI rootfs to /dev/mtd7 (no problems whatsoever), and booted w/ patched kernel and NAND rootfs.
No ECC or other NAND/UBI related errors are reported and the root filesystem mount time seems optimal.
Furthermore, the rootfs mount time remained optimal even after couple times powering off the board during nand write syncs.

BR,

Niko Mauno

0 sumiko over 13 years ago in reply to Niko Mauno

TI__Intellectual 2060 points

Avinash,

I have downloaded PSP 04.06.006.07 which is included in SDK 5.04.01.00 ,released Apr/11th.

But it doesn't seem the indicated patch is applied in the latest PSP.

Can I expect this would be applied in the next PSP ?

Thanks and Regards,

Sumiko Yamaji

0 TimHarron over 13 years ago in reply to sumiko

TI__Mastermind 18130 points

Sumiko,

I just ran through this same confusion myself, but found that there was another patch reverting the contents of the one Avinash posted. What happened is that Avinash's patch was applied in PSP04.06.00.06, and then reverted in PSP04.06.00.07.

The reason behind that is that instead of adding the GPMC timing configuration to board-flash.c they moved it into the board config file board-am335xevm.c found in arch/arm/mach-omap2. If you look at this file in the current SDK/PSP release you should see the am335x_nand_timings structure at line 1160 or so.

Hope this helps.

0 sumiko over 13 years ago in reply to TimHarron

TI__Intellectual 2060 points

Thanks, Tim,

I have checked the PSP source code and then, actually tested to run the new kernel with UBIFS.

Without applying the patch , it successfully runs now.

Thanks and Regards,

Sumiko Yamaji

P.S. sorry not for checking "verify answer". I can't make it "verify" because I'm not an initiator of this topics.

Processors

Processors forum

AM335x EVM rootfs on NAND - trouble