TDA4VM: Boot Issue with TDA4-based SOM board.

John H. Gardner

Part Number: TDA4VM

Customer reports the following TDA4 Boot issue on a custom-made board, but one that is modeled directly on the TI SOM from the TDA4 EVM.

The issue is that 10% TDA4 SOM gets hung up on boot and has the error message “Trying to boot from MMC2”.

Of the SOMs that hang up they hang up 80% of the time on boot.

Again, the customer copied the schematic parts and layout that TI did for their EVM SOM.

Here is the log from a SOM failing to boot.

U-Boot SPL 2019.01-g66126341c8 (Oct 24 2019 - 03:52:25 +0000)

SYSFW ABI: 2.6 (firmware rev 0x0013 '19.8.0-v2019.08 (Terrific Llama')

Trying to boot from MMC2

Loading Environment from MMC... *** Warning - No MMC card found, using default environment

Remoteproc 2 started successfully

** File not found /lib/firmware/j7-mcu-r5f0_0-fw **

Starting ATF on ARM64 core...

It was also noted by an TI FAE late last year that the log noted above was also seen in a J7 EVM /SOM. At the time, it was suggested that a couple things can help:

[] Applying power cycle without too much time between turning off and then on

[] Pushing the MCU POR button twice without too much time in between

He went on to further note that "This feels like we have a power rail timing or other sequencing issue, but [he had] not dug deeper on this one yet."

over 5 years ago

0 John H. Gardner over 5 years ago

TI__Mastermind 44561 points

Some initial feedback from the apps to be followed up by the customer:

Can you try MCU_BOOTMODE[9:8] to 11b to enable POST bypass?
Can you confirm that you have MMC1 wired to the SD Card and not MMC2? (matching the EVM PROC078E7 schematic)
Can you provide a log file or snippet for a passing vs failing case? Does the boot sequence recognize MMC2 in both passing and failing cases?
1. U-Boot SPL 2019.01-g66126341c8 (Oct 24 2019 - 03:52:25 +0000)
2. SYSFW ABI: 2.6 (firmware rev 0x0013 '19.8.0-v2019.08 (Terrific Llama')
3. Trying to boot from MMC2
4. Loading Environment from MMC... *** Warning - No MMC card found, using default environment
What is the pass/fail rate? Is a given board more or less likely to fail? Can all boards fail?
1. Same question for EVM? Vs custom-made SOM?

0 Bryan Lilly1 over 5 years ago in reply to John H. Gardner

Prodigy 120 points

3. Failing case:

U-Boot SPL 2019.01-g66126341c8 (Oct 24 2019 - 03:52:25 +0000)

SYSFW ABI: 2.6 (firmware rev 0x0013 '19.8.0-v2019.08 (Terrific Llama')

Trying to boot from MMC2

Loading Environment from MMC... *** Warning - No MMC card found, using default environment

Remoteproc 2 started successfully

** File not found /lib/firmware/j7-mcu-r5f0_0-fw **

Starting ATF on ARM64 core...

END OF LOG

Passing case:

U-Boot SPL 2019.01-g66126341c8 (Oct 24 2019 - 03:52:25 +0000)
SYSFW ABI: 2.6 (firmware rev 0x0013 '19.8.0-v2019.08 (Terrific Llama')
Reading on-board EEPROM at 0x50 failed -1
Trying to boot from MMC2
Loading Environment from MMC... *** Warning - No MMC card found, using default environment

Remoteproc 2 started successfully
** File not found /lib/firmware/j7-mcu-r5f0_0-fw **
Starting ATF on ARM64 core...

NOTICE: BL31: v2.1(release):ti2019.02-rc4
NOTICE: BL31: Built : 03:52:00, Oct 24 2019
I/TC:
I/TC: OP-TEE version: ti2019.02-89-ge5a8779-dev (gcc version 8.3.0 (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36))) #1 Thu Oct 24 03:52:22 UTC 2019 aarch64
I/TC: Initialized

U-Boot SPL 2019.01-g66126341c8 (Oct 24 2019 - 03:52:45 +0000)
Reading on-board EEPROM at 0x50 failed -1
Trying to boot from MMC2

U-Boot 2019.01-g66126341c8 (Oct 24 2019 - 03:52:45 +0000)

SoC: J721E PG 1.0
Model: Texas Instruments K3 J721E SoC
Reading on-board EEPROM at 0x50 failed -1
Board: J721EX-PM1-SOM rev E2
DRAM: 4 GiB
Flash: 0 Bytes
MMC: sdhci@4f80000: 0, sdhci@4fb0000: 1
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In: serial@2800000
Out: serial@2800000
Err: serial@2800000
Reading on-board EEPROM at 0x50 failed -1
Net: Could not get PHY for ethernet@046000000: addr 0
phy_connect() failed
eth-1: ethernet@046000000
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
** Unable to read file boot.scr **
81 bytes read in 0 ms
Loaded env from uEnv.txt
Importing environment from mmc1 ...
12319700 bytes read in 517 ms (22.7 MiB/s)
Load Remote Processor 3 with data@addr=0x80080000 12319700 bytes: Success!
** File not found /lib/firmware/j7-main-r5f1_0-fw **
9024636 bytes read in 373 ms (23.1 MiB/s)
Load Remote Processor 6 with data@addr=0x80080000 9024636 bytes: Success!
9023540 bytes read in 374 ms (23 MiB/s)
Load Remote Processor 7 with data@addr=0x80080000 9023540 bytes: Success!
11915956 bytes read in 1704 ms (6.7 MiB/s)
Load Remote Processor 8 with data@addr=0x80080000 11915956 bytes: Success!
13338632 bytes read in 559 ms (22.8 MiB/s)
98343 bytes read in 5 ms (18.8 MiB/s)
3653 bytes read in 1 ms (3.5 MiB/s)
3742 bytes read in 3 ms (1.2 MiB/s)
## Flattened Device Tree blob at 82000000
Booting using the fdt blob at 0x82000000
Loading Device Tree to 00000000fdda6000, end 00000000fdec1fff ... OK

Starting kernel ...

LOG CONTINUES TO LOGIN SCREEN

0 Bryan Lilly1 over 5 years ago in reply to Bryan Lilly1

Prodigy 120 points

Response to question #1:

We set MCU_BOOTMODE[9:8] to 11b on one of our SOM/Baseboard pairs. The baseboard is a known working one and the SOM consistently fails to get to the login screen.

Log of failing SOM:

Remoteproc 2 started successfully
** File not found /lib/firmware/j7-mcu-r5f0_0-fw **
Starting ATF on ARM64 core...

END OF LOG

Log of passing SOM (same baseboard):

Remoteproc 2 started successfully
** File not found /lib/firmware/j7-mcu-r5f0_0-fw **
Starting ATF on ARM64 core...

U-Boot SPL 2019.01-g66126341c8 (Oct 24 2019 - 03:52:45 +0000)
Reading on-board EEPROM at 0x50 failed -1
Trying to boot from MMC2

U-Boot 2019.01-g66126341c8 (Oct 24 2019 - 03:52:45 +0000)

In: serial@2800000
Out: serial@2800000
Err: serial@2800000
Reading on-board EEPROM at 0x50 failed -1
Net:
Warning: ethernet@046000000 using MAC address from ROM
eth0: ethernet@046000000
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
** Unable to read file boot.scr **
81 bytes read in 0 ms
Loaded env from uEnv.txt
Importing environment from mmc1 ...
12319700 bytes read in 516 ms (22.8 MiB/s)
Load Remote Processor 3 with data@addr=0x80080000 12319700 bytes: Success!
** File not found /lib/firmware/j7-main-r5f1_0-fw **
9024636 bytes read in 373 ms (23.1 MiB/s)
Load Remote Processor 6 with data@addr=0x80080000 9024636 bytes: Success!
9023540 bytes read in 375 ms (22.9 MiB/s)
Load Remote Processor 7 with data@addr=0x80080000 9023540 bytes: Success!
11915956 bytes read in 1706 ms (6.7 MiB/s)
Load Remote Processor 8 with data@addr=0x80080000 11915956 bytes: Success!
13338632 bytes read in 560 ms (22.7 MiB/s)
98343 bytes read in 6 ms (15.6 MiB/s)
3653 bytes read in 1 ms (3.5 MiB/s)
3742 bytes read in 3 ms (1.2 MiB/s)
## Flattened Device Tree blob at 82000000
Booting using the fdt blob at 0x82000000
Loading Device Tree to 00000000fdda6000, end 00000000fdec1fff ... OK

Starting kernel ...

Log continues to login screen.

0 Bryan Lilly1 over 5 years ago in reply to John H. Gardner

Prodigy 120 points

In response to question #2:

MMC0 goes to an eMMC flash device on the baseboard. MMC1 goes to the SD card. MMC2 is used for I2C3, UART, and GPIO.

0 John H. Gardner over 5 years ago in reply to John H. Gardner

TI__Mastermind 44561 points

Bryan,

Thanks for all of these details. I will append the answer to question 4:

The answer to # 4 is of 3 TI SOMs , 1 intermittently and frequently fails to boot.

Of the 15 custom-made SOMs you built, 10 were tested, 9 of those make it to the login screen and 1 consistently fails.

John

0 kcastille over 5 years ago in reply to John H. Gardner

TI__Guru 52002 points

Bryan, Can you ship the 1 failing TI SOM back to us for debug? John will give shipping instructions off line.

Thanks,
Kyle

0 John H. Gardner over 5 years ago in reply to Bryan Lilly1

TI__Mastermind 44561 points

Bryan,

Upon receipt of board - we tested SD card boot and every time it was able to boot and able to run additional tests (memory, etc).

Couple questions:

1) What was the fail rate for D3 with this SOM? You mention intermittent and frequently before - can you quantify in any capacity?

2) What version of the SDK are you using for the testing on your board?

John

0 Bryan Lilly1 over 5 years ago in reply to John H. Gardner

Prodigy 120 points

John,

1) We were seeing in the neighborhood of 40-60% boot failure. The software guys said that could appear as once in a while or several in a row.

2) I have been using an old version of software for testing our custom hardware. I do not know the exact version. Here is the output from "uname -a" if that helps:

Linux j7-evm 4.19.73-g0cabba2b47 #1 SMP PREEMPT Thu Oct 24 03:54:33 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

I tested our hardware with 6.02 today and the problem appears to have been fixed in that version.

We were likely using 6.01 with the TI SOM that we sent back to you.

Bryan

0 John H. Gardner over 5 years ago in reply to Bryan Lilly1

TI__Mastermind 44561 points

As noted, we discussed with our J7 firmware team on the issue above. They had root caused it after my initial reports of the issue and found something in the SYSFW. The SYSFW runs on the DMSC controller core. The updated sysfw with the fix is in SDK 6.02.

Finally SYSFW team found a bug in the sequence the A72 was brought up:
powering on the PD_A72_0 before PD_A72_cluster_0 (with all LPSCs in SWRSTDisable), even though the LPSC sequencing is handled properly.

That is precisely why we see a hang as soon as we hand over the control from R5 to A72(ATF).

I believe the sysfw update is present on 6.02(2019.12) has the fix and hence we no longer see the issue.

This should close the issue.

Processors

Processors forum

TDA4VM: Boot Issue with TDA4-based SOM board.