Linux/AM5728: When booting, eMMC not mounted.

Hi Henry,

henry kim said:

Boot mode is QSPI. (u-boot, kernel)

Root filesystem is in eMMC.

From what I understand, your final goal is to have u-boot, kernel in QSPI and rootfs in eMMC, is that correct?

henry kim said:

After booting normally,(kernel, rootfilesystem is normally mounted) insert the SD card.

What do you mean regarding "after booting normally"? From where you load/boot u-boot/kernel/rootfs? QSPI, eMMC, SD card?

And why SD card is involved here? Are you planning to flash the rootfs in eMMC from SD card?

See also the below wiki pages:

http://processors.wiki.ti.com/index.php/Linux_Core_U-Boot_User's_Guide#Using_SD.2C_eMMC_or_USB_storage

Regards,
Pavel

Hi, Pavel

Thanks for your reply.

From what I understand, your final goal is to have u-boot, kernel in QSPI and rootfs in eMMC, is that correct?

- Yes,

What do you mean regarding "after booting normally"? From where you load/boot u-boot/kernel/rootfs? QSPI, eMMC, SD card? And why SD card is involved here? Are you planning to flash the rootfs in eMMC from SD card?

- It means that booting is completed. not failed. u-boot and kernel from QSPI, rootfs from eMMC.

- It is not for flashing the rootfs, just copy a file to the eMMC.

There was no problem before copying file from the SD card to the eMMC.

Once copy a file, the problem always occurs at power off/on. (power cable disconnect/connect)

However, when the reset button is pressed, the boot completes normally.

This symptom is also reproduced in the TMDXIDK5728.

Thanks and best regards.

Henry.

Henry,

henry kim said:

- It is not for flashing the rootfs, just copy a file to the eMMC.

There was no problem before copying file from the SD card to the eMMC.

Once copy a file, the problem always occurs at power off/on. (power cable disconnect/connect)

Where exactly in rootfs you copy that file? Can you provide full console log when copy that file?

henry kim said:

However, when the reset button is pressed, the boot completes normally.

Boot completes normally and you see your file in eMMMC rootfs?

Regards,
Pavel

Henry,

We have a special script for transferring files from SD card to eMMC in PSDK Linux automotive, this is mk-eMMC-boot.sh

In this script we use ext4 for eMMC rootfs and we have these steps:

- mount emmc_rootfs

- cp -rvf  mmc_rootfs/*  emmc_rootfs/.

- sync

- unmount emmc_rootfs

We use "cp -rvf  {SD card}  {eMMC}" to transfer files from Sd card to eMMC.

I am attaching the mk-eMMC-boot.sh script for your reference.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/354/6254.mk_2D00_eMMC_2D00_boot.sh

Hi Pavel,

I just did the following :

mkdir /temp

cp {SD card}/temp.txt  /temp/ 

sync   

Boot completes normally, I can see the file in eMMMC rootfs.

I want to know what causes the problem after copying a file, rather than copying the rootfs.

Henry,

henry kim said:

cp {SD card}/temp.txt  /temp/ 

Can you try with the below command, is there any difference?

cp -rvf  {SD card}/temp.txt  {eMMC}/temp/

Can you try also to unmount eMMC after "sync" and before power off, is there any difference in that case?

Regards,
Pavel

Also I see on the failing case you have the below bootargs and messages:

Kernel command line: console=ttyO1,115200n8 root=PARTUUID=b20c611f-28c7-a549-b1b0-142b607587bf rw rootfstype=ext4 rootwait

Waiting for root device PARTUUID=b20c611f-28c7-a549-b1b0-142b607587bf...

mmc1: error -110 whilst initialising MMC card

omap_hwmod: mmc2: _wait_target_disable failed

 

It seems to me that linux kernel is trying to mount rootfs not from eMMC, but from SD card.

Can you try to unmount and remove SD card after "sync" and before power off/on, is there any difference?

Can you also provide me the console log output when you have successful load of the eMMC rootfs?

Regards,
Pavel

Hi Pavel,

Here is successful log : 6864.AM572x_Normally_Boot_Log.txt

Regards,

Henry

Henry,

You boot arguments seems to be correct.

Can you provide your sysboot[5:0] settings in both cases (normal boot and fail boot)?

Please also provide answers on the other open questions in my previous replies.

Regards,
Pavel

Can you try with the below command, is there any difference?

cp -rvf  {SD card}/temp.txt  {eMMC}/temp/

Can you try also to unmount eMMC after "sync" and before power off, is there any difference in that case?

- There is the same problem.

Regards,

Henry

henry kim said:

cp -rvf  {SD card}/temp.txt  {eMMC}/temp/

Can you try also to unmount eMMC after "sync" and before power off, is there any difference in that case?

- There is the same problem.

Could you please provide full console log which includes these commands?

Regards,
Pavel

Hi Pavel,

sysboot[5:0] settings as follow.

both cases(normal boot and fail boot) are in use as a red square box. (1st: QSPI, 2nd: SD)

Regards,

Henry

Henry,

Your sysboot[5:0] pins are configured for QSPI boot, which seems to be correct in your case.

Could you please answer on the other open questions, like for example do you remove the SD card from the board after copy operation and before power off/on the board?

Regards,
Pavel

Pavel,

Can you try to unmount and remove SD card after "sync" and before power off/on, is there any difference?

- There is the same problem. (Unmount and remove SD card, and repeat power off/on)

- However, when the reset button is pressed, the boot completes successful.

Regards,

Henry

Henry,

henry kim said:

- There is the same problem. (Unmount and remove SD card, and repeat power off/on)

When you make power off, you actually unplug power cable, is that correct?

Yes stated that you can reproduce the issue with TMDXIDK5728. Can you try instead to unplug the power cable, to reset the TMDXIDK5728 board through SW1 (PORz) and SW2 (RESETn). Will you have this issue, when using SW1 and/or SW2 buttons?

Can you try before unplug the power cable, to power off the board through user space command, what is the result in this case? Make sure you wait the board for several seconds to properly power off/down before unplug the cable.

Note that you must follow power off/down sequence, otherwise you might damage your board and/or decrease POH.

Regards,
Pavel

You can use "poweroff" command from user space. See also the below e2e threads:

e2e.ti.com/.../501908
e2e.ti.com/.../572173

Regards,
Pavel

Hi Pavel,

When you make power off, you actually unplug power cable, is that correct?

- Yes.

Yes stated that you can reproduce the issue with TMDXIDK5728. Can you try instead to unplug the power cable, to reset the TMDXIDK5728 board through SW1 (PORz) and SW2 (RESETn). Will you have this issue, when using SW1 and/or SW2 buttons?

- If you press the SW2 button, boot will be completed successfully.

Can you try before unplug the power cable, to power off the board through user space command, what is the result in this case? Make sure you wait the board for several seconds to properly power off/down before unplug the cable.

- When I run the power-off command, the boot complete successfully.

- The flow is as follows.

Run the power-off command -> "systemd-shutdown[1]: All filesystems unmounted" message output (about 12sec) -> Power cable disconnect -> Power cable connect

Stopping User Manager for UID 0... [  OK  ] Stopped target Timers. [  OK  ] Stopped Daily Cleanup of Temporary Directories.          Stopping Load/Save Random Seed...          Stopping Session c1 of user root. [  OK  ] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch. [  OK  ] Stopped Setup Virtual Console. [  OK  ] Stopped target System Time Synchronized. [  OK  ] Stopped Update UTMP about System Runlevel Changes.          Stopping Update UTMP about System Boot/Shutdown... [  OK  ] Stopped target Multi-User System.          Stopping System Logging Service...          Stopping Avahi mDNS/DNS-SD Stack...          Stopping Lightning Fast Webserver With Light System Requirements...          Stopping Kernel Logging Service...          Stopping Network Name Resolution...          Stopping Telephony service...          Stopping thermal-zone-init.service... [  OK  ] Stopped target Login Prompts.          Stopping Getty on tty1...          Stopping Serial Getty on ttyS1... [  OK  ] Stopped Serial Getty on ttyO2.          Stopping Serial Getty on ttyS2...          Stopping strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf... [  OK  ] Stopped target Swap. [  OK  ] Stopped strongSwan IPsec IKEv1/IKEv2 daemon using ipsec.conf. [  OK  ] Stopped Kernel Logging Service. [  OK  ] Stopped System Logging Service. [  OK  ] Stopped Telephony service. [  OK  ] Stopped Avahi mDNS/DNS-SD Stack. [  OK  ] Stopped Getty on tty1. [  OK  ] Stopped Serial Getty on ttyS1. [  OK  ] Stopped Serial Getty on ttyS2. [  OK  ] Stopped Network Name Resolution. [  OK  ] Stopped Lightning Fast Webserver With Light System Requirements. [  OK  ] Stopped User Manager for UID 0. [  OK  ] Stopped Load/Save Random Seed. [  OK  ] Stopped Session c1 of user root. [  OK  ] Stopped thermal-zone-init.service. [  OK  ] Stopped Update UTMP about System Boot/Shutdown. [  OK  ] Stopped User Manager for UID 0. [  OK  ] Stopped Create Volatile Files and Directories. [  OK  ] Stopped target Local File Systems.          Unmounting /media/ram...          Unmounting /var/volatile...          Unmounting /run/user/0...          Stopping gdbserverproxy.service... [  OK  ] Removed slice User Slice of root.          Stopping Login Service... [  OK  ] Stopped target Network.          Stopping Network Service... [  OK  ] Removed slice system-serial\x2dgetty.slice.          Stopping Permit User Sessions... [  OK  ] Removed slice system-getty.slice. [  OK  ] Stopped Network Service. [  OK  ] Stopped Login Service. [  OK  ] Unmounted /media/ram. [  OK  ] Unmounted /var/volatile. [  OK  ] Unmounted /run/user/0. [  OK  ] Stopped gdbserverproxy.service. [  OK  ] Stopped Permit User Sessions. [  OK  ] Stopped target Remote File Systems.          Stopping rng-tools.service... [  OK  ] Reached target Unmount All Filesystems. [  OK  ] Stopped target Local File Systems (Pre). [  OK  ] Stopped Create Static Device Nodes in /dev. [  OK  ] Stopped Remount Root and Kernel File Systems.          Stopping D-Bus System Message Bus... [  OK  ] Stopped Apply Kernel Variables. [  OK  ] Stopped Load Kernel Modules. [  OK  ] Stopped D-Bus System Message Bus. [  OK  ] Stopped rng-tools.service.          Stopping thttpd.service... [  OK  ] Stopped thttpd.service.          Stopping tiipclad-daemon.service...          Stopping telnetd.service... [  OK  ] Stopped tiipclad-daemon.service. [  OK  ] Stopped telnetd.service.          Stopping uim-sysfs.service... [  OK  ] Stopped uim-sysfs.service. [  OK  ] Stopped target Basic System. [  OK  ] Stopped target Slices. [  OK  ] Removed slice User and Session Slice. [  OK  ] Stopped target Paths. [  OK  ] Stopped Dispatch Password Requests to Console Directory Watch. [  OK  ] Stopped Forward Password Requests to Wall Directory Watch. [  OK  ] Stopped target Sockets. [  OK  ] Closed RPCbind Server Activation Socket. [  OK  ] Closed Syslog Socket. [  OK  ] Closed dropbear.socket. [  OK  ] Closed Avahi mDNS/DNS-SD Stack Activation Socket. [  OK  ] Closed D-Bus System Message Bus Socket. [  OK  ] Reached target Shutdown. [  OK  ] Reached target Final Step.          Starting Power-Off... systemd-shutdown[1]: Sending SIGTERM to remaining processes... systemd-journald[209]: Received SIGTERM from PID 1 (systemd-shutdow). systemd-shutdown[1]: Sending SIGKILL to remaining processes... systemd-shutdown[1]: Sending SIGKILL to PID 1127 (rngd). systemd-shutdown[1]: Unmounting file systems. systemd-shutdown[1]: Remounting '/tmp' read-only with options ''. systemd-shutdown[1]: Unmounting /tmp. systemd-shutdown[1]: Remounting '/' read-only with options 'data=ordered'. EXT4-fs (mmcblk0p1): re-mounted. Opts: data=ordered systemd-shutdown[1]: Remounting '/' read-only with options 'data=ordered'. EXT4-fs (mmcblk0p1): re-mounted. Opts: data=ordered systemd-shutdown[1]: Remounting '/' read-only with options 'data=ordered'. EXT4-fs (mmcblk0p1): re-mounted. Opts: data=ordered systemd-shutdown[1]: All filesystems unmounted.

This application is IED.(Intelligent Electronic Device)

Most IED power off by disconnecting the power cable. IED can not use command to power-off.

If a power outage occurs,  it is impossible to respond.

So, it's hard to apply that command.

Is there another way to solve this problem?

Regards,

Henry

And, I want to know the cause of this problem.
Please let me know.

Regards,
Henry

Henry,

Journaling filesystem is less prone to corruption on power loss/failure, but you are using such (EXT4) and you still have filesystem corruption. First, confirm that you're not mounting with barrier=0/nobarrier. That often improves performance, at the cost of corruption if a proper shutdown isn't performed. Also check your kernel logs to make sure barriers aren't being disabled by ext4 because something in the stack doesn't support them.

Warning: Disabling barriers when disks cannot guarantee caches are properly written in case of power failure can lead to severe file system corruption and data loss.

See the below pointers for more info:

wiki.archlinux.org/.../ext4
www.embeddedarm.com/.../preventing-filesystem-corruption-in-embedded-linux


You can also check the option of using SafeFAT:
www.ti.com/.../embeddedsoftwarefulldetails.tsp


You can also check for HW detection of power loss, see if the below pointers will be in help:
e2e.ti.com/.../197885
e2e.ti.com/.../341182
e2e.ti.com/.../1062998
processors.wiki.ti.com/.../Battery_Backup




Regards,
Pavel

Henry,

For proper power loss/failure handling, you should have backup power supply which should supply the AM57x board during power down/off sequence. Unplugging the power cable should be the event for starting power down the board.

Regards,
Pavel

Hi, Pavel

Thanks for your reply.

The solution you proposed is not suitable for us.

We want to implement a solution that does not use backup power.

We already know the instability of the system when disconnecting the power cable.

But, I can not understand that the memory is not broken and it can not boot.

I want to know the exact cause.

Boot completed successfully when warm reset on IDK board, it seems that data of the eMMC is not broken or lost.

Therefore, it does not seem to be the damage caused by power off.

When I tried writing data to eMMC at initail u-boot, and power-off/on. There were no problem of eMMC breaking.

The problem appear when I wrote the data on the SD card to eMMC.

I think, the problem is the linux driver when I write data from the SD card to the eMMC.

Best Regards,

Henry

Henry,

I do not think MMC/eMMC linux driver is the problem. When you are in u-boot, you can unplug the power cable and boot ok, as in u-boot we have much less functionality enabled in the eMMC. Consider u-boot as a bare metal programming, where we have basic registers settings.

While in linux kernel and user space, we have MMC layers, file system, etc. Here the thinks are much more complex than u-boot, here we have much more processes, operations and interactions.
There might be some background process running in eMMC while you unplug the power cable, see the below wiki:
processors.wiki.ti.com/.../SD_User's_Guide

Note that even if you manage to workaround this eMMC filesystem corruption (when unplug the power cable in user space), there is still the issue with not handling the power down/off sequence properly, which will reduce the lifetime (POH) of your device, it might also damage your device. Then corruption of the eMMC filesystem will be your smallest problem.

Regards,
Pavel