SD card boot problem

Graeme Parker

We have just moved from our prototype stage and into production. During production test we have noticed a problem affecting a single board.

During the Linux boot phase the Kernel panics and it is almost certainly due to this:

Waiting for root device /dev/mmcblk0p2...
mmc0: new high speed SDHC card at address e624
mmcblk0: mmc0:e624 SU08G 7.40 GiB (ro) <=========== READ ONLY
mmcblk0: p1 p2
VFS: Cannot open root device "mmcblk0p2" or unknown-block(179,2)

We are using a microSD card which does not have the write protect pin. The WP pin on the DM8168 is left unconnected so should thus have a logic low due to the IPD which is enabled. There is no errata for this problem so I'm not sure what to conclude.

If I use CCS and change the PINCTRL for the WP pin to GPIO, then I can see that it is at a logic low. I can do the same for the CD (card detect) pin and can observe that this changes state when the card is removed.

I have also Googled the error "cannot open root device mmcblk0p2 or unknown-block(179 2)" and Google knows this phrase and points to these forums where others have had the same problem. The solution, as far as I can tell, seems to be to replace the EVM in those cases. That would imply a "dead" DM8168?!

What are your thoughts on this? Is this indicative of a faulty device?

Thanks.

over 11 years ago

0 Pavel Botev over 11 years ago

TI__Guru**** 170625 points

Hi Graeme,

Are you on EZSDK? If yes, what version? What is the silicon revision of your DM816x device?

This looks to me like your root file system is damaged. Can you confirm that your /media/ROOTFS partition is fine (can be mounted on the Ubuntu PC, and contain all the needed files and folders, like /bin, /boot, /etc, linuxrc, etc.)

Also post your boot arguments.

Regards,

Pavel

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

Hi Pavel

Yes, we're using EZSDK version 5..05.

I'm not sure it does show a corrupted file system as the same microSD card works fine in all of our other boards. In fact, I've just tried another (working) microSD card with a different build of Linux and that also shows exactly the same problem (but also works ok in our other boards).

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

Sorry, it's silicon version 1.1

0 Pavel Botev over 11 years ago in reply to Graeme Parker

TI__Guru**** 170625 points

I made a try with my own micro SD card and DM816x/AM389x EVM.

1. When I try with the locked micro SD card, I have the same as you:

Waiting for root device /dev/mmcblk0p2...
mmc0: new high speed SDHC card at address 0002
mmcblk0: mmc0:0002 00000 3.70 GiB (ro)
mmcblk0: p1 p2
VFS: Cannot open root device "mmcblk0p2" or unknown-block(179,2)
Please append a correct "root=" boot option; here are the available partitions:
1f00             256 mtdblock0 (driver?)
1f01               8 mtdblock1 (driver?)
1f02            2560 mtdblock2 (driver?)
1f03            1272 mtdblock3 (driver?)
1f04            2432 mtdblock4 (driver?)
1f05             128 mtdblock5 (driver?)
1f06            4352 mtdblock6 (driver?)
1f07          204928 mtdblock7 (driver?)
1f08           50304 mtdblock8 (driver?)
b300         3880960 mmcblk0 driver: mmcblk
b301           40131 mmcblk0p1 00000000-0000-0000-0000-000000000mmcblk0p1
b302         3839535 mmcblk0p2 00000000-0000-0000-0000-000000000mmcblk0p2
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)
Backtrace:
[<c004abb8>] (dump_backtrace+0x0/0x110) from [<c039d918>] (dump_stack+0x18/0x1c)
r7:d6414000 r6:00000000 r5:c002ec44 r4:c0523750
[<c039d900>] (dump_stack+0x0/0x1c) from [<c039d97c>] (panic+0x60/0x17c)
[<c039d91c>] (panic+0x0/0x17c) from [<c00092b4>] (mount_block_root+0x1e0/0x220)
r3:00000000 r2:00000000 r1:d643bf58 r0:c045c03c
[<c00090d4>] (mount_block_root+0x0/0x220) from [<c00093a0>] (mount_root+0xac/0xcc)
[<c00092f4>] (mount_root+0x0/0xcc) from [<c0009530>] (prepare_namespace+0x170/0x1d4)
r4:c0522f64
[<c00093c0>] (prepare_namespace+0x0/0x1d4) from [<c00087c4>] (kernel_init+0x114/0x154)
r5:c00086b0 r4:c0522f00
[<c00086b0>] (kernel_init+0x0/0x154) from [<c0072df4>] (do_exit+0x0/0x5e4)
r5:c00086b0 r4:00000000

2. When I try with not locked micro SD card, everything works fine:

Waiting for root device /dev/mmcblk0p2...
mmc0: new high speed SDHC card at address 0002
mmcblk0: mmc0:0002 00000 3.70 GiB
mmcblk0: p1 p2
EXT3-fs: barriers not enabled
kjournald starting. Commit interval 5 seconds
EXT3-fs (mmcblk0p2): warning: mounting fs with errors, running e2fsck is recommended
EXT3-fs (mmcblk0p2): using internal journal
EXT3-fs (mmcblk0p2): recovery complete
EXT3-fs (mmcblk0p2): mounted filesystem with writeback data mode
VFS: Mounted root (ext3 filesystem) on device 179:2.

So you need to double check that the micro SD card is not locked.

When I locked my own micro SD card, I can not mount the ROOTFS on my Desktop Ubuntu PC, I have "Unable to mount ROOTFS". Can you try if you can mount your root file system? If you are using the EZSDK script, it should has the name ROOTFS and available under /media/ROOTFS when you plug the micro SD in your card reader.

Best Regards,

Pavel

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

Thanks.That's good news - sort of!

Yes, when I plug the card into my PC the rootfs can be mounted.

How do you lock your microSD card?

0 Pavel Botev over 11 years ago in reply to Graeme Parker

TI__Guru**** 170625 points

I plug my micro SD card into a micro SD adapter. And in this adapter, I have a switch, named "LOCK", and I use this switch to lock and unlock the micro SD card. See the picture: http://www.iltelefoninoofferte.it/index.php?main_page=popup_image&pID=703&zenid=eac7175d323ac3c4877a10c9520b0ca3

Can you try to use root file system not from SD card, but from NFS or Flash (NAND, NOR, SPI). Thus we will be sure that the problem is in the MMC/SD controller peripheral module.

Best Regards,

Pavel

0 Pavel Botev over 11 years ago in reply to Pavel Botev

TI__Guru**** 170625 points

I have made one more experiment.

1 case - when SD card is not locked. In this case, when I interrupt the boot process in the u-boot stage and then I read the SD_SYSTEST (0x48060128) and SD_PSTATE (0x48060224) I have the following values:

TI8168_EVM#md 0x48060128
48060128: 00000000 00000000 00000000 00000000 ................

TI8168_EVM#md 0x48060224
48060224: 01f70000 00000d02 000e0007 20008002 ...............

2 case - when SD card is locked:

TI8168_EVM#md 0x48060128
48060128: 00004000 00000000 00000000 00000000 .@..............

TI8168_EVM#md 0x48060224
48060224: 01ff0000 00000d02 000e0007 20008002 ...............

Could you please provide your SD_SYSTEST and SD_PSTATE values in u-boot?

Regards,

Pavel

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

Thanks. Here are the results:

TI8168_EVM#md 0x48060128
48060128: 00000000 00000000 00000000 00000000 ................

TI8168_EVM#md 0x48060224
48060224: 01f70000 00000d02 000e0007 20008002 ...............

0 Pavel Botev over 11 years ago in reply to Graeme Parker

TI__Guru**** 170625 points

It seams there are differences between micro SD card and standard SD card regarding the WP pin:

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/717/p/249400/873354.aspx#873354

Regards,

Pavel

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

Thanks for the link.

I've read through that thread but honestly I'm not sure what their problem was.

It looks like their issue might be due to the WP pin not being used or driven to the wrong state when using a microSD card.

I'm not sure that applies in my case. We are leaving the IPD of the WP pin to enable write access to the card. We have 13 boards built now and two suffer from the microSD (ro) problem.

I'm going to look at the Linux SD card driver and see if I can remove the WP link.

0 Graeme Parker over 11 years ago in reply to Graeme Parker

Expert 1265 points

Not being a seasoned Linux s/w developer, where would you look to find the sources of the SD card driver?

0 Graeme Parker over 11 years ago in reply to Graeme Parker

Expert 1265 points

One more thing. I noticed on these forums that another person has had the same fault:

http://e2e.ti.com/support/embedded/linux/f/354/t/177933.aspx

("Getting error "VFS: Cannot open root device "mmcblk0p2" or unknown-block(179,2)" while booting from SD card, DVSDK 4.X, platform: DM36X")

The solution here was to replace the EVM. Do you have a record of this replacement and if the faulty EVM was investigated further?

0 Pavel Botev over 11 years ago in reply to Graeme Parker

TI__Guru**** 170625 points

Hello,

Graeme Parker said:
where would you look to find the sources of the SD card driver?

For the MMC/SD driver check:

http://processors.wiki.ti.com/index.php/TI81XX_PSP_MMC_SD_Driver_User_Guide

ti-ezsdk_dm816x-evm_5_05_02_00/board-support/linux-2.6.37-psp04.04.00.01/drivers/mmc

BR,

Pavel

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

Thanks. Of course now I have another question.

I downloaded ezsdk_dm816x-evm_5_05_02_00_setuplinux and installed this on my Ubuntu PC.

After installation is complete I do not have this location ti-ezsdk_dm816x-evm_5_05_02_00/board-support/linux-2.6.37-psp04.04.00.01/drivers/mmc

The only folder in board-support is u-boot-2010.06-psp04.04.00.01

Is there something else I should download?

0 Pavel Botev over 11 years ago in reply to Graeme Parker

TI__Guru**** 170625 points

You should go to this page: http://software-dl.ti.com/dsps/dsps_public_sw/ezsdk/latest/index_FDS.html

Then you should download the ezsdk_dm816x-evm_5_05_02_00_setuplinux bin file and place it in your HOME directory on your Ubuntu 10.04 LTS 32-bit or Ubuntu 11.10 32-bit PC.

You should install the CodeSourcery tool chain, as per instructions in this web page: ./arm-2009q1-203-arm-none-linux-gnueabi.bin

If necessary make the installer executable manually by executing:

    chmod +x ezsdk_dm816x-evm_5_05_02_00_setuplinux

Execute the installer on the host and follow the instructions:

    ./ezsdk_dm816x-evm_5_05_02_00_setuplinux

At the end of the installation, you should have the ti-ezsdk_dm816x-evm_5_05_02_00/board-support/linux-2.6.37-psp04.04.00.01 folder in your HOME directory.

Regards,
Pavel

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

Thanks.

I now have my Ubuntu PC setup and am able to run: make linux, make linux_install, make u-boot and make u-boot_install. And I can run the mksdboot.sh script too.

I have found the section in the SDcard driver source that I believe could provide a workaround to the read-only problem.

But, from what I can tell, mksdboot.sh is trying to get files from \usr\local\ezsdk\board-support\prebuilt-images, but this is not where the results of the make are. Is there a proper guide to rebuilding the Linux kernel, rebuilding u-boot, and making an SDcard?

To be honest, it's a complicated makefile and I'm not sure where the results are made. Where are MLO, u-boot.bin and uImage? I do know that a copy ends up in \home\<user-name>\targetfs\home\root\dm816x-evm\boot. Should I just use these for the SD-card?

0 Pavel Botev over 11 years ago in reply to Graeme Parker

TI__Guru**** 170625 points

Graeme,

You can create SD card with the script. Then you can replace the pre-built MLO, u-boot.bin and uImage with the new generated ones.

For how to create/make/build new MLO and u-boot.bin, check the U-Boot User Guide: http://processors.wiki.ti.com/index.php/TI81XX_PSP_U-Boot

For how to create/make/build new uImage, check the PSP User Guide: http://processors.wiki.ti.com/index.php/TI81XX_PSP_User_Guide#Linux_Kernel

The new generated MLO and u-boot.bin are in ti-ezsdk_dm816x-evm_5_05_02_00/board-support/u-boot-2010.06-psp04.04.00.01

The new generated uImage is in ti-ezsdk_dm816x-evm_5_05_02_00/board-support/linux-2.6.37-psp04.04.00.01/arch/arm/boot

Regards,

Pavel

0 Graeme Parker over 11 years ago in reply to Pavel Botev

Expert 1265 points

I've modified the sd.c file to always return FALSE for the read-only check. Now the board works perfectly!

Putting the original read-only check back into the kernel causes the Linux boot procedure to fail as before saying that the file system is (ro). So I'm pretty sure I'm on the right track.

I should investigate further within the sd.c (and associated) files to find exactly where the fault lies. To me, at this point, it would appear to be faulty silicon.

0 Mike Kahn over 10 years ago in reply to Graeme Parker

Prodigy 10 points

Graeme,

We have a board based on the DM8168 that has been in production for about a year. We have a configuration that sounds identical to yours: microSD card with the WP pin on the 8168 left unconnected. Recently, the microSD card that we had been using went EOL and in the process of qualifying a new microSD card, we stumbled across this problem with the read only filesystem. At this point we are going to try to repeat your experiment with the changes to sd.c to force the return to always be FALSE. That said, you indicated in this thread that you were going to do some further research to try to get to the root cause of the failure. Did you ever complete that research?

Thank you in advance for your help,

Mike Kahn

0 user4922635 over 7 years ago in reply to Graeme Parker

Prodigy 70 points

Hi,

I am also facing the same issue with my custom board using DM8168 and EZSDK 5.5.2.
Im able to do nfs booting, But stuck on the same place as mentioned in the post when using sd card booting. Anybody able to resolve this issue.

0 Graeme Parker over 7 years ago in reply to Mike Kahn

Expert 1265 points

Sorry for such a very late response.

No, I never made any further progress as the change I made worked - so we shipped it!

0 Graeme Parker over 7 years ago in reply to user4922635

Expert 1265 points

Hi,
No, not resolved. I just continued with the fix I found by modifying sd.c

0 user4922635 over 7 years ago in reply to Graeme Parker

Prodigy 70 points

Hi Graeme Parker,

Thanks for the responce.

What fix? That fix will work only for the affected board right. For rest of the board you are having another build images

0 Graeme Parker over 7 years ago in reply to user4922635

Expert 1265 points

The problem I had was that the filesystem was ro.
I modified the sd.c file (in drivers/mmc/core/sd.c) to always return FALSE for the read-only check. Now the board works perfectly!
Around line 500:
/*
commented out to ignore ro bug GP120313
if (!reinit) {
int ro = -1;

if (host->ops->get_ro)
ro = host->ops->get_ro(host);

if (ro < 0) {
printk(KERN_WARNING "%s: host does not "
"support reading read-only "
"switch. assuming write-enable.\n",
mmc_hostname(host));
} else if (ro > 0) {
mmc_card_set_readonly(card);
}
}
*/
It was 4 years ago.

0 user4922635 over 7 years ago in reply to Graeme Parker

Prodigy 70 points

Hi Graeme Parker,

This code change worked with our board also. Only one board was having this issue and we kept it for development purpose.

Processors

Processors forum

SD card boot problem