This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM335x eMMC boot

Other Parts Discussed in Thread: AM3352

Hello

A customer has been having some other issues with booting from the eMMC on their board board based on Beaglebone HW. Occasionally it seems like u-boot can't talk to the eMMC chip and it fails to load and we get output in the console like this (for reference, mmc 1 is the eMMC):

=> boot
Card did not respond to voltage select!
Card did not respond to voltage select!
switch to partitions #0, OK
mmc1(part 0) is current device
SD/MMC found on device 1
switch to partitions #0, OK
mmc1(part 0) is current device
SD/MMC found on device 1
reading boot.scr
** Unable to read file boot.scr **
reading uEnv.txt
528 bytes read in 5 ms (102.5 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
32364 bytes read in 59 ms (535.2 KiB/s)
Loaded am335x-boneblack.dtb
mmc_read_data: timedout waiting for status!
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
mmc fail to send stop cmd
 ** ext4fs_devread read error - block
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
** Can't read partition table on 1:0 **
** Invalid partition 2 **
starting USB...
USB0:   Port not available.
BOOTP broadcast 1
BOOTP broadcast 2
BOOTP broadcast 3
BOOTP broadcast 4
BOOTP broadcast 5
BOOTP broadcast 6
BOOTP broadcast 7
BOOTP broadcast 8
or this:
mmc_read_data: timedout waiting for status!
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
SD/MMC found on device 1
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
** ext4fs_devread read error - block
Failed to mount ext2 filesystem...
** Unrecognized filesystem type **
omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
** Unrecognized filesystem type **
** No partition table - mmc 1 **

This happens maybe one out of every 10-15 times the board is rebooted. Rebooting it normally seems to fix the issue (temporarily) and it will boot into linux just fine.

The voltages on the data/clock/command lines all seem to be at 3.3V, and we're using the October 2016 release of u-boot so I don't think it's some kind of long standing bug in that. According to the schematics we're using 'MTFC2GMDEA-0M WT' for the eMMC - I don't know if this is non standard or would affect it in any way.

They are not even sure whether this is a hardware or software issue - do you have any ideas on what we can look at to see what's causing this problem to happen?

Regards Steve

  • The software team have been notified. They will respond here.
  • Hi Steve,

    Apologies for the late reply.

    Regarding the "Card did not respond to voltage select!" message, have you tested the mmc commands in u-boot? Does the emmc respond to mmc info, mmc part, etc..? If you don't see the "Card did not respond to voltage select!", then it seems the root cause is:
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
    Which usually indicates a problem with the fat write command.

    What it comes to my mind is:
    1. Try repartitioning your eMMC from u-boot:
    processors.wiki.ti.com/.../Linux_Core_U-Boot_User's_Guide
    2. On the occasion you boot your board:
    Rebooting it normally seems to fix the issue (temporarily) and it will boot into linux just fine

    try to run fsck on the eMMC.

    Best Regards,
    Yordan
  • Hello Yordan, thanks for the response. Please see comments from the customer below in response to your suggestions.




    I've run fsck and badblocks (not the very longest test) on the emmc and there seemed to be nothing wrong with it.

    I rebooted it a few times until it gave me the error and ran "mmc part" after it dropped into the console. At this point it could recognise the partitions, and typing "boot" made it boot successfully.

    I then rebooted it a few times more until it failed to boot again, then ran "mmc part" and it just hung for a couple of seconds then said:
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
    ** Can't read partition table on 1:0 **
    I waited a few seconds and typed "mmc part" again and it now recognised the partitions, and again I could successfully boot into linux.

    I then rebooted it a few more times and it gave me this error which I don't remember seeing before:
    Failed to iterate over directory boot
    ** File not found /boot/am335x-boneblack.dtb **
    After it dropped into the console I could then successfully boot into linux manually.

    It's very strange behaviour, it will fail to recognise the mmc one second then a few seconds later it will work perfectly fine. There is also the strange issue which happened the third time where it seemed to be able to successfully mount the partition but not read the files on it.
  • Hi,

    I am not really sure what is actually going on... The u-boot hangs on reading the mmc state, see drivers/mmc/omap_hsmmc.c:
    while ((readl(&mmc_base->pstate) & (DATI_MASK | CMDI_MASK)) != 0)

    The read sometimes takes more than 1 sec and the driver times out...

    Just for the test, have they tried increasing MAX_RETRY_MS, let's say define it as 2 secs?

    Best Regards,
    Yordan
  • Hi Yordan, so the customer has tested your suggestion and still seeing some errors, please see below. In particular the error relating to pxelinux.cfg. Any further suggestions?

    Kind regards Steve

    I Increased the MAX_RETRY_MS to 2 ms and it's still occasionally failing to boot. There's some weird errors about pxelinux.cfg as well...?

    Card did not respond to voltage select!
    Card did not respond to voltage select!
    Card did not respond to voltage select!
    switch to partitions #0, OK
    mmc1(part 0) is current device
    Scanning mmc 1:1...
    reading /am335x-boneblack.dtb
    32364 bytes read in 13 ms (2.4 MiB/s)
    switch to partitions #0, OK
    mmc1(part 0) is current device
    SD/MMC found on device 1
    switch to partitions #0, OK
    mmc1(part 0) is current device
    mmc_read_data: timedout waiting for status!
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
    SD/MMC found on device 1
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
     ** ext4fs_devread read error - block
    Failed to mount ext2 filesystem...
    ** Unrecognized filesystem type **
    omap_hsmmc_send_cmd: timedout waiting on cmd inhibit to clear
    ** Unrecognized filesystem type **
    ** No partition table - mmc 1 **
    ## Error: "bootcmd_nand0" not defined
    starting USB...
    USB0:   Port not available.
    BOOTP broadcast 1
    BOOTP broadcast 2
    BOOTP broadcast 3
    BOOTP broadcast 4
    BOOTP broadcast 5
    BOOTP broadcast 6
    BOOTP broadcast 7
    BOOTP broadcast 8
    BOOTP broadcast 9
    BOOTP broadcast 10
    BOOTP broadcast 11
    BOOTP broadcast 12
    BOOTP broadcast 13
    BOOTP broadcast 14
    BOOTP broadcast 15
    BOOTP broadcast 16
    BOOTP broadcast 17
    BOOTP broadcast 18
    BOOTP broadcast 19
    BOOTP broadcast 20
    BOOTP broadcast 21
    BOOTP broadcast 22
    BOOTP broadcast 23
    BOOTP broadcast 24
    BOOTP broadcast 25
    BOOTP broadcast 26
    BOOTP broadcast 27
    BOOTP broadcast 28
    BOOTP broadcast 29

    Retry time exceeded; starting again
    missing environment variable: pxeuuid
    Retrieving file: pxelinux.cfg/01-a8-1b-6a-43-00-5a
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/00000000
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/0000000
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/000000
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/00000
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/0000
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/000
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/00
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/0
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/default-arm-am33xx
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/default-arm
    *** ERROR: `serverip' not set
    Retrieving file: pxelinux.cfg/default
    *** ERROR: `serverip' not set
    Config file not found
    starting USB...
    USB0:   Port not available.
    BOOTP broadcast 1

    etc.

  • Hi,

    Sorry for the delayed.

    I suggested increasing the MAX_RETRY_MS in order to test if a longer period for getting response from readl(&mmc_base->pstate) will help.

    The
    " Retrieving file: pxelinux.cfg/01-a8-1b-6a-43-00-5a
    *** ERROR: `serverip' not set"
    in my oppinion, is happening because the u-boot gives up the emmc boot & jumps to the next boot device.

    Anyway, have they tried with the latest u-boot (2016.05)? Also can they share their am335x_evm.h file as well as the other related configuration headers: ti_am335x_common.h & ti_armv7_common.h?

    Also I see, the following in the bootlog:
    reading uEnv.txt
    528 bytes read in 5 ms (102.5 KiB/s)
    Loaded env from uEnv.txt

    What is the content of their uEnv.txt file?

    Are they seeing this problem on more than one of their boards (if they have more than one EVM produced)? If it's just that one, they could revisit its design/assembly..

    Best Regards,
    Yordan
  • Yordan / Ron,  please see below from the customer.

    Kind regards Steve

    We're actually using uboot 2016.11 at the moment - we were having this boot issue with 2016.05 and thought upgrading might fix it. This boot issue is still happening but we haven't had any new issues with this version, so we're probably going to stick with it.

    Instead of giving the config headers I've just attached the patches we're using (which are probably easier to read). As mentioned, these patches apply to version 2016.11 of u-boot. The only header we're changing is the am335x_evm.h file, the other configuration headers are just left as they are. For reference, the patches are because:

    1. We're storing the uboot env in a file in the boot partition so we just want to enable that by setting FAT_ENV_DEVICE_AND_PART

    2. We have a simple script (checkfailure) that just makes sure the board booted last time and switches partitions if not. We have tried skipping this on a board and it still has the boot issue.

    I had a look in the uEnv.txt and realised that it just had most of the same options in the default environment (consoleblank=0, bootpart=0:2, ...) so I've removed it (as in 0003-....patch), but we're still having the boot problem.

    We are seeing this issue on multiple boards (and across a couple of very minor hardware revisions)

     

  • Hi Steve,

    Another suggestion to test:
    Increase the timeout for mmc_reset_controller_fsm(). That is modify omap_hsmmc.c as follows:
    /* If we fail after 1 second wait, something is really bad */
    #define MAX_RETRY_MS 1000
    - #define MMC_TIMEOUT_MS 20
    +#define MMC_TIMEOUT_MS 40
    #define OMAP_HSMMC_SUPPORTS_DUAL_VOLT BIT(0)

    Hope this helps.

    Best Regards,
    Yordan
  • Hi Steve,

    Following up on the offline discussion, they can try to lower the MMC clock:
    1. by adding max-frequency parameter to their dts node.
    2. If the above doesn't work, they could limit the clock by modifying the defines in omap_hsmmc.c driver:
    #define OMAP_MMC_MIN_CLOCK 400000
    #define OMAP_MMC_MAX_CLOCK 52000000

    Best Regards,
    Yordan
  • Hi Steve,

    I have the same problem as what your customer is facing. I have tried all the suggestions in this thread, and multiple versions of u-boot to no avail. Have you been able to solve this problem? Any hints or advice would be greatly appreciated.

    Some more details about our board: Its a custom design based on the beaglebone black, but with a TPS65910A31 PMIC. It uses the same micron eMMC memory as the beagle bone black. ti-u-boot was pulled from git. For testing I modified the the board.h and board.c file a bit to skip the board ID check., and to probe our PMIC.

    Thanks in advance for any help.
  • Did you find a solution? I am seeing a similar problem with a custom board based on beaglebone black with a TPS65910A31 PMIC.
  • Hello Aaron, no this problem was never resolved and still remains open. My customer decided to rely on the WDT to reboot the AM3352 and retry a boot and managed to put the problem on hold as they needed to focus on getting the product to market. Any feedback from your side would be appreciated.

    Steve 

  • Hi Steve,

    I think I might have solved the problem for the custom board I'm working on. I made changes to the pinmux for the eMMC in mux.c, I also added updated the u-boot device tree pinmux for the eMMC pins.

    In /board/ti/am335x/mux.c, I added a new struct for the eMMC connections (which are the same as the Beaglebone black connections)

    static struct module_pin_mux emmc_mmc1_pin_mux[] = {
    {OFFSET(gpmc_ad7), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT7 */
    {OFFSET(gpmc_ad6), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT6 */
    {OFFSET(gpmc_ad5), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT5 */
    {OFFSET(gpmc_ad4), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT4 */
    {OFFSET(gpmc_ad3), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT3 */
    {OFFSET(gpmc_ad2), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT2 */
    {OFFSET(gpmc_ad1), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT1 */
    {OFFSET(gpmc_ad0), (MODE(1) | RXACTIVE | PULLUP_EN)}, /* MMC1_DAT0 */
    {OFFSET(gpmc_csn1), (MODE(2) | RXACTIVE | PULLUP_EN)}, /* MMC1_CLK */
    {OFFSET(gpmc_csn2), (MODE(2) | RXACTIVE | PULLUP_EN)}, /* MMC1_CMD */
    {OFFSET(gpmc_a4), (MODE(7) | RXACTIVE | PULLDOWN_EN)}, /* gpio1_20 */
    {-1},
    };

    I added a line in enable_board_pin_mux(void) to use the the mux for my board

    } else if (board_is_my_custom_board()) {
    configure_module_pin_mux(emmc_mmc1_pin_mux);
    } else if ...

    I also changed the pin mux in my equivalent to am335x-bone-common.dtsi

    emmc_pins: pinmux_emmc_pins {
    pinctrl-single,pins = <
    0x80 (PIN_INPUT_PULLUP | MUX_MODE2) /* gpmc_csn1.mmc1_clk */
    0x84 (PIN_INPUT_PULLUP | MUX_MODE2) /* gpmc_csn2.mmc1_cmd */
    0x00 (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad0.mmc1_dat0 */
    0x04 (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad1.mmc1_dat1 */
    0x08 (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad2.mmc1_dat2 */
    0x0c (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad3.mmc1_dat3 */
    0x10 (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad4.mmc1_dat4 */
    0x14 (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad5.mmc1_dat5 */
    0x18 (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad6.mmc1_dat6 */
    0x1c (PIN_INPUT_PULLUP | MUX_MODE1) /* gpmc_ad7.mmc1_dat7 */
    0x50 (PIN_OUTPUT_PULLDOWN | MUX_MODE7 ) /* gpmc_a4.gpio1_20 */
    >;

    I am using u-boot 2017.01 from the TI SDK. The board has a micron 4GB eMMC and the AM3358BZCZ as well as the TPS65910A31 . I'm not sure why the beaglebone black wouldn't have this issue too.