TMDX654GPEVM: U-boot error

Hi,

I got a bit more information. Booting with the older version of U-Boot in the SD card shipped with the EVM, I could get to the command line and get some more information.

"mmc list" successfully lists both devices. Selecting the SD card works fine, and I can even run "mmc info". But trying to select the eMMC causes the same failure I was seeing with the newer u-boot, and hangs:

=> mmc list
sdhci@0: 0
sdhci@0: 1 (SD)
=> mmc dev 1
switch to partitions #0, OK
mmc1 is current device
=> mmc dev 0
arasan_sdhci_execute_tuning:Tuning failed
Select HS400 failed -1

I could then work around this issue by disabling all access to the eMMC from U-Boot, ie:

configs/am65x_evm_a53_defconfig:
CONFIG_ENV_IS_IN_MMC=n
CONFIG_FASTBOOT_FLASH_MMC_DEV=1

include/environment/ti/mmc.h:
#define DEFAULT_MMC_TI_ARGS \
+ "mmcdev=1\0" \

However I am still surprised as to what may have caused this issue. The board was not abused or mistreated, and I am not aware of having done anything that could have caused this. As mentioned earlier, I was just running Linux applications and hadn't even customized u-boot etc. yet.

Can anybody offer any insight as to what may have caused this problem, and how to resolve it? (assuming that it is not a HW issue)

Thanks!

Best regards,

Jose

Hi Jose,

there were some changes related to AM654x MMC tuning over the course of the U-Boot development. Can you please try the latest top-of-tree ti-u-boot-2019.01 branch available below and see if this fixes your issues and report back here (without your defconfig changes and other modifications you had introduced to work around the issues).

https://git.ti.com/gitweb?p=ti-u-boot/ti-u-boot.git;a=shortlog;h=refs/heads/ti-u-boot-2019.01

Thanks and Regards,
Andreas

Hi Andreas,

Thank you very much for your reply.

To make sure I had everything clean, I flashed again a new SD card from am65xx-evm-linux-06.01.00.08.img (the latest image I could find in the webpage); I cloned the u-boot repository from the ti-u-boot-2019.01 branch (particularly, commit 2584ef34f905320fbc36b68639f75787f7ba36b4), re-built it, and used it to replace the u-boot.img, tispl.bin and tiboot3.bin files from the new clean SD card.

Unfortunately I am still getting the same result:

U-Boot SPL 2019.01-00002-g2584ef34f9 (Dec 19 2019 - 07:26:23 +0000)
SYSFW ABI: 2.6 (firmware rev 0x0013 '19.7.1-v2019.07a (Terrific Llam')
Trying to boot from MMC2
Starting ATF on ARM64 core...

NOTICE: BL31: v2.1(release):ti2019.02-rc4
NOTICE: BL31: Built : 16:44:58, Oct 19 2019
I/TC:
I/TC: OP-TEE version: ti2019.02-89-ge5a8779-dev (gcc version 8.3.0 (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36))) #1 Sat Oct 19 16:44:08 UTC 2019 aarch64
I/TC: Initialized

U-Boot SPL 2019.01-00002-g2584ef34f9 (Dec 19 2019 - 07:25:31 +0000)
Detected: SER-PCIEUSBEVM rev E3
Trying to boot from MMC2
cannot find image node 'k3-am654-pcie-usb3.dtbo': -1

U-Boot 2019.01-00002-g2584ef34f9 (Dec 19 2019 - 07:25:31 +0000)

SoC: AM654 PG 1.0
Model: Texas Instruments AM654 Base Board
Board: AM6-COMPROCEVM rev E3
DRAM: 4 GiB
MMC: sdhci@4f80000: 0, sdhci@04FA0000: 1
Loading Environment from MMC... am654_sdhci_execute_tuning:Tuning failed
sdhci_send_command: Timeout for status update!
sdhci_send_command: Timeout for status update!
sdhci_send_command: Timeout for status update!
[...]

Is there any information that I could fetch from the device to help diagnose the issue? I am afraid I am out of ideas...

I am starting to suspect that the eMMC might have gone wrong, but I have no idea how. I would suspect ESD but I wasn't even handling the board when it started failing (after flipping the switch in the morning I usually just interact with it over serial/network).

Any ideas to try or to narrow down the root cause will be most welcome...

Thanks,

Jose

Jose,

Jose Fernandez2 said:

Any ideas to try or to narrow down the root cause will be most welcome...

I'm checking with some folks here internally, will let you know what I find.

Regards, Andreas

Jose,

just to double-check, your initial post said the old SD card image worked, but a subsequent post then says it now also shows "sdhci_send_command: Timeout for status update!" errors once you type "mmc dev 0". Basically, as of now you have no U-Boot image at all that functions completely error free as it comes to eMMC, correct?

Also what happens with the current SDK 6.01 image booting all the way to Kernel? Does mmc0 a.k.a. eMMC come up properly and can be used/accessed?

Then, for U-Boot can you try disable the HS200 mode by adding /delete-property/ mmc-hs200-1_8v to the &sdhci0 node in the k3-am654-base-board-u-boot.dtsi file?

Regards, Andreas

Hi Andreas,

Happy new year!

Apologies if I caused any confusion; it was a mixture of my lack of understanding in my first email and lack of clarity in my last one... The SD card shipped with the EVM does boot all the way to the kernel, but only later I realized that it does so because it doesn't try to use the eMMC. The SD card image that comes with the latest downloadable SDK seems to get stuck when trying to initialize the eMMC to store/retrieve the uBoot env data, so they never get to load the kernel. If I wait long enough (about a minute or so), the eMMC initialization bails out and I get dropped to an uBoot prompt, but if I try to continue booting it gets stuck again; since the failure I have not been able to get to the kernel with those images.

So, as you say, I don't have any uBoot image that can successfully initialize the eMMC.

I disassembled my set-up for the Christmas break, but I will get an answer to your other questions as soon as I get it back online, hopefully in the next day or so.

Thanks,

Jose

Hi again,

I did the following (I did it all by hand so this "script" might need some adjustments, but I hope it is clear enough):

cd board-support/u-boot-2019.01+gitAUTOINC+029e4c009a-g029e4c009a

cat > no_hs200.patch << _EOF_
diff --git a/arch/arm/dts/k3-am654-base-board-u-boot.dtsi b/arch/arm/dts/k3-am654-base-board-u-boot.dtsi
index 29ba8fc4ef..188b892c3e 100644
--- a/arch/arm/dts/k3-am654-base-board-u-boot.dtsi
+++ b/arch/arm/dts/k3-am654-base-board-u-boot.dtsi
@@ -429,6 +429,7 @@
 
 &sdhci0 {
        u-boot,dm-spl;
+       /delete-property/ mmc-hs200-1_8v;
 };
 
 &sdhci1 {
_EOF_

patch -p1 < no_hs200.patch

cd ../..
make u-boot-spl_clean
make u-boot-spl

scp board-support/u-boot_build/r5/tiboot3.bin root@ip:/run/media/mmcblk1p1/
scp board-support/u-boot_build/a53/tispl.bin root@ip:/run/media/mmcblk1p1/
scp board-support/u-boot_build/a53/u-boot.img root@ip:/run/media/mmcblk1p1/u-boot-am65xx-evm.img
ssh root@ip sync

but it didn't help - still getting the same "sdhci_send_command: Timeout for status update!" errors.

When booting with my modified u-boot (disabling eMMC0 altogether), Linux does not list it at all:

root@am65xx-evm:~# ls /dev/mm*
/dev/mmcblk1    /dev/mmcblk1p1  /dev/mmcblk1p2

The more I look at it, the more I feel this is a HW failure. I just cannot understand what might have caused it, especially given that I didn't handle the EVM between the last successful boot and the first failed one.

Any ideas or anything else I can do to find more information?

Thanks,

Jose

Hi Jose,

Jose Fernandez2 said:

The SD card shipped with the EVM does boot all the way to the kernel, but only later I realized that it does so because it doesn't try to use the eMMC.

For this specific case, does the eMMC show up under /dev? Also what is the output of dmesg | grep -i mmc once you reach Kernel prompt?

Regards, Andreas

Hi Andreas,

No, the eMMC never gets to show in /dev; however dmesg does show something interesting (I should have checked it earlier, sorry!):

[    2.961189] mmc0: Unknown controller version (4). You may experience problems.
[    2.968545] mmc0: CQHCI version 5.10
[    3.003217] mmc0: SDHCI controller on 4f80000.sdhci [4f80000.sdhci] using ADMA 64-bit
[    3.011869] mmc1: Unknown controller version (4). You may experience problems.
[    3.019225] mmc1: CQHCI version 5.10
[    3.053807] mmc1: SDHCI controller on 4fa0000.sdhci [4fa0000.sdhci] using ADMA 64-bit
[    3.065660] mmc0: switch to bus width 8 failed
[    3.074092] mmc0: switch to bus width 4 failed
[    3.089004] mmc0: mmc_select_hs200 failed, error -84
[    3.093978] mmc0: error -84 whilst initialising MMC card
[    3.250113] mmc0: switch to bus width 8 failed
[    3.264247] mmc0: switch to bus width 4 failed
[    3.277319] mmc0: mmc_select_hs200 failed, error -84
[    3.282296] mmc0: error -84 whilst initialising MMC card
[    3.409338] mmc0: switch to bus width 8 failed
[    3.425272] mmc0: switch to bus width 4 failed
[    3.435242] mmc0: mmc_select_hs200 failed, error -84
[    3.440222] mmc0: error -84 whilst initialising MMC card
[    3.580323] mmc0: switch to bus width 8 failed
[    3.599513] mmc0: switch to bus width 4 failed
[    3.609480] mmc0: mmc_select_hs200 failed, error -84
[    3.614454] mmc0: error -84 whilst initialising MMC card
[    4.252510] mmc1: new high speed SDHC card at address e624
[    4.259259] mmcblk1: mmc1:e624 SA08G 7.40 GiB 
[    4.268829]  mmcblk1: p1 p2
[    4.290238] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Opts: (null)
[    5.764476] EXT4-fs (mmcblk1p2): re-mounted. Opts: (null)

So it seems that Linux can detect it, bat it fails when trying to configure it (with similar symptoms as uboot).

At least this shows that the eMMC is not completely bust; still not sure whether this may be just a SW/configuration issue, or that the device is partially damaged. Any ideas?

Thanks,

Jose

Jose,

you may or may not have seen it but the AM654x device has an errata (see errata sheet in the product folder http://www.ti.com/product/AM6548) named i2026 (“MMCSD: Negative Current from UHS-I PHY May Create an Over-Voltage Condition on VDDS6 and VDDS7 Which Exposes the Device to a Significant Reliability Risk”). What you are experiencing may or may not be related to this. Talking to some folks here I was also made aware we had early prototype boards exhibiting some variant of this concern but HW fixes were applied to prevent/work around such issues. Given you have a revision E3 board you should not be affected by this. Will keep you updated as this issue is being discussed further internally.

What serial # does your board have?

Since your board no longer works correctly and if the broken eMMC access is an issue for you your board should get replaced.

Regards, Andreas

Hi Andreas,

Thank you for all your support on this.

This is all the information I can read on the board:

TMDX654GPEVM
SL# 39184P810418

WUX MLN32
94V-0 1
3418-L663
E59746

AM654x EVM PROCESSOR BOARD PROC062
ASSM #:4P0081 REV:1.0

I have taken a look at the errata but I don't know enough to tell whether it may be related. From what I read there and from your response I understand that my board revision is likely powering VDDSH6 and VDDSH7 at 1.8V and therefore this shouldn't happen, did I get that correctly? My other suspect since the beginning was ESD, even though I wasn't handling the board at the time of the failure. Have you seen or been made aware of this kind of failure caused by ESD?

Thanks,

Jose

Hi Andreas,

I bought the board from Digikey. I can initiate an RMA in a few days as soon as I complete the immediate task. Should I do it with Digikey or directly with TI?

Thanks again for all the support!

Best regards,
Jose

Hi Andreas,

Thank you very much for all the help in reaching the bottom of this issue; it was my first time using TI's customer support and I am very pleasantly surprised with the quality.

I will start an RMA with Digikey as you suggested.

Thanks,
Jose