TMDX654IDKEVM: TMDX654IDKEVM: Linux RT SDK 8.2 sometimes fails to boot with FDT_ERR_BADOFFSET

Hello,
Your issue description looks like intermittant issue, somehow related with FDT overlay.
Have you observed an way to trigger or reproduce the issue consistently on your board?
Best,
-Hong

No, I don't know a way how to trigger this every time. It happens for about 10% of boots.

(I am posting here so that this thread is not auto closed. We are still interested in the fix.)

(I am posting here so that this thread is not auto closed. We are still interested in the fix.)

(I am posting here so that this thread is not auto closed. We are still interested in the fix.)

(I am posting here so that this thread is not auto closed. We are still interested in the fix.)

Hi!

can you provide an entire dump of your U-Boot environment (stop boot, and enter 'env print', and copy & paste the result here via Insert->Code).

If the issue is related to loading from the SD card media often a valuable experiment can be to reduce the SD card access speed/capabilities, this making it less likely for any signal integrity issues to manifest themselves. To do this, add the following to the appropriate MMC node such as &sdhci1 in the dts file and re-build / re-deploy U-Boot:

$ git diff
diff --git a/arch/arm/dts/k3-am654-base-board.dts b/arch/arm/dts/k3-am654-base-board.dts
index 56b6477b4a..93e0764e2c 100644
--- a/arch/arm/dts/k3-am654-base-board.dts
+++ b/arch/arm/dts/k3-am654-base-board.dts
@@ -249,6 +249,8 @@
        ti,driver-strength-ohm = <50>;
        sdhci-caps-mask = <0x7 0x0>;
        disable-wp;
+       no-1-8-v;
+       max-frequency = <24000000>;
 };

 &wkup_i2c0 {

The sdhci-caps-mask = <0x7 0x0>; property that is already there should disable all UHS modes, but is usually used in conjunction with the no-1-8-v; property, which may be missing. The max-frequency = <24000000>; property further relaxes timing/signal integrity requirements. If any of those modifications help eliminate the issue that would be good to know.

Regards, Andreas

I can't reproduce FDT_ERR_BADOFFSET boot fail in TI RT SDK 8.6

I used this image:

ti-processor-sdk-linux-rt-am65xx-evm-08.06.00.47/filesystem/tisdk-default-image-am65xx-evm.wic.xz

U-boot envinronment for TI RT SDK 8.2 where i can reproduce it with the same IDK and SD card:

U-Boot SPL 2021.01-g44a87e3ab8 (Mar 24 2022 - 06:00:43 +0000)                   
SYSFW ABI: 3.1 (firmware rev 0x0016 '22.1.1--v2022.01 (Terrific Llam')          
SPL initial stack usage: 1416 bytes                                             
Trying to boot from MMC2                                                        
spl_load_fit_image: Skip load 'dm': image size is 0!                            
Starting ATF on ARM64 core...                                                   
                                                                                
NOTICE:  BL31: v2.5(release):08.01.00.006-dirty                                 
NOTICE:  BL31: Built : 05:29:41, Mar 24 2022                                    
                                                                                
U-Boot SPL 2021.01-g44a87e3ab8 (Mar 24 2022 - 05:30:44 +0000)                   
SYSFW ABI: 3.1 (firmware rev 0x0016 '22.1.1--v2022.01 (Terrific Llam')          
Detected: Silicon SR2.0                                                         
i2c_omap i2c@42120000: pinctrl_select_state_full: uclass_get_device_by_phandle_9
Detected: AM6-IDKAPPEVM rev A                                                   
Detected: SER-PCIE2LEVM rev A                                                   
Trying to boot from MMC2                                                        
                                                                                
                                                                                
U-Boot 2021.01-g44a87e3ab8 (Mar 24 2022 - 05:30:44 +0000)                       
                                                                                
SoC:   AM65X SR2.0                                                              
Model: Texas Instruments AM654 Base Board                                       
Board: AM6-COMPROCEVM rev A                                                     
DRAM:  4 GiB                                                                    
MMC:   sdhci@4f80000: 0, sdhci@4fa0000: 1                                       
Loading Environment from MMC... OK                                              
In:    serial                                                                   
Out:   serial                                                                   
Err:   serial                                                                   
am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA00102 cpsw_ver: 0x6BA0
Detected: Silicon SR2.0                                                         
Detected: AM6-IDKAPPEVM rev A                                                   
Detected: SER-PCIE2LEVM rev A                                                   
Net:   eth1: icssg2-eth, eth0: ethernet@46000000port@1                          
Hit any key to stop autoboot:  0                                                
=>                                                                              
=>                                                                              
=> env print                                                                    
addr_fit=0x90000000                                                             
arch=arm                                                                        
args_all=setenv optargs ${optargs} earlycon=ns16550a,mmio32,0x02800000 ${mtdpar}
args_mmc=run finduuid;setenv bootargs console=${console} ${optargs} root=PARTUU}
args_ubi=setenv bootargs console=${console} ${optargs} rootfstype=ubifs root=ubs
baudrate=115200                                                                 
board=am65x                                                                     
board_name=am65x                                                                
board_rev=A                                                                     
board_serial=0010                                                               
board_software_revision=01                                                      
boot=mmc                                                                        
boot_fdt=try                                                                    
boot_fit=0                                                                      
boot_rprocs=if test ${dorprocboot} -eq 1 && test ${boot} = mmc; then rproc init;
boot_rprocs_mmc=env set rproc_id;env set rproc_fw;env set secure_suffix;if teste
bootcmd=run findfdt; run envboot; run init_${boot}; run boot_rprocs; run get_ken
bootdelay=2                                                                     
bootdir=/boot                                                                   
bootenvfile=uEnv.txt                                                            
bootm_size=0x10000000                                                           
bootpart=1:2                                                                    
bootscript=echo Running bootscript from mmc${mmcdev} ...; source ${loadaddr}    
console=ttyS2,115200n8                                                          
cpu=armv8                                                                       
dfu_alt_info_emmc=rawemmc raw 0 0x800000 mmcpart 1;rootfs part 0 1 mmcpart 0;ti1
dfu_alt_info_mmc=boot part 1 1;rootfs part 1 2;tiboot3.bin fat 1 1;tispl.bin fa1
dfu_alt_info_ospi=tiboot3.bin raw 0x0 0x080000;tispl.bin raw 0x080000 0x200000;0
dfu_alt_info_ram=tispl.bin ram 0x80080000 0x200000;u-boot.img ram 0x81000000 0x0
dorprocboot=0                                                                   
dtboaddr=0x89000000                                                             
envboot=mmc dev ${mmcdev}; if mmc rescan; then echo SD/MMC found on device ${mm;
eth1addr=70:ff:76:1d:5f:d0                                                      
eth2addr=70:ff:76:1d:5f:d1                                                      
eth3addr=70:ff:76:1d:60:bc                                                      
eth4addr=70:ff:76:1d:60:bd                                                      
eth5addr=70:ff:76:1d:60:be                                                      
eth6addr=70:ff:76:1d:60:bf                                                      
ethaddr=10:08:2c:6a:7d:ab                                                       
fdt_addr_r=0x88000000                                                           
fdtaddr=0x88000000                                                              
fdtcontroladdr=fdec70a0                                                         
fdtoverlay_addr_r=0x80200000                                                    
findfdt=setenv name_fdt k3-am654-base-board.dtb;setenv fdtfile ${name_fdt}      
finduuid=part uuid ${boot} ${bootpart} uuid                                     
get_fdt_mmc=load mmc ${bootpart} ${fdtaddr} ${bootdir}/${name_fdt}              
get_fdt_ubi=ubifsload ${fdtaddr} ${bootdir}/${name_fdt}                         
get_fit_mmc=load mmc ${bootpart} ${addr_fit} ${bootdir}/${name_fit}             
get_kern_mmc=load mmc ${bootpart} ${loadaddr} ${bootdir}/${name_kern}           
get_kern_ubi=ubifsload ${loadaddr} ${bootdir}/${name_kern}                      
get_overlay_mmc=fdt address ${fdtaddr};fdt resize 0x100000;for overlay in $name;
get_overlaystring=for overlay in $name_overlays;do;setenv overlaystring ${overl;
importbootenv=echo Importing environment from mmc${mmcdev} ...; env import -t $}
init_mmc=run args_all args_mmc                                                  
init_ubi=run args_all args_ubi; sf probe; ubi part ospi.rootfs; ubifsmount ubi:;
kernel_addr_r=0x82000000                                                        
loadaddr=0x82000000                                                             
loadbootenv=fatload mmc ${mmcdev} ${loadaddr} ${bootenvfile}                    
loadbootscript=load mmc ${mmcdev} ${loadaddr} boot.scr                          
loadfdt=load ${devtype} ${bootpart} ${fdtaddr} ${bootdir}/${fdtfile}            
loadimage=load ${devtype} ${bootpart} ${loadaddr} ${bootdir}/${bootfile}        
mmcboot=mmc dev ${mmcdev}; devnum=${mmcdev}; devtype=mmc; if mmc rescan; then e;
mmcdev=1                                                                        
mmcloados=if test ${boot_fdt} = yes || test ${boot_fdt} = try; then if run load;
mmcrootfstype=ext4 rootwait                                                     
mtdids=nor0=47040000.spi.0                                                      
mtdparts=mtdparts=47040000.spi.0:512k(ospi.tiboot3),2m(ospi.tispl),4m(ospi.u-bo)
name_fit=fitImage                                                               
name_kern=Image                                                                 
name_overlays=k3-am654-idk.dtbo k3-am654-pcie-usb2.dtbo                         
partitions=name=rootfs,start=0,size=-,uuid=${uuid_gpt_rootfs}                   
pxefile_addr_r=0x80100000                                                       
ramdisk_addr_r=0x88080000                                                       
rd_spec=-                                                                       
rdaddr=0x88080000                                                               
rproc_fw_binaries=0 /lib/firmware/am65x-mcu-r5f0_0-fw 1 /lib/firmware/am65x-mcu 
rproc_load_and_boot_one=if load mmc ${bootpart} $loadaddr ${rproc_fw}; then if i
run_fit=bootm ${addr_fit}#${fdtfile}${overlaystring}                            
run_kern=booti ${loadaddr} ${rd_spec} ${fdtaddr}                                
scriptaddr=0x80000000                                                           
soc=k3                                                                          
stdin=serial,usbkbd                                                             
update_to_fit=setenv loadaddr ${addr_fit}; setenv bootfile ${name_fit}          
vendor=ti                                                                       
                                                                                
Environment size: 5232/131067 bytes                                             
=>

I will check if your U-Boot change helps in SDK 8.2 later.

Zdenek Bouska said:

I can't reproduce FDT_ERR_BADOFFSET boot fail in TI RT SDK 8.

Good to see that you seem to have established a working-good reference point. I assume everything else was the same for your test, right? (same board, same SD card, that all failed with v8.2)

From this we should be able to boil this down to a specific change that makes the test NG -> OK.

Regards, Andreas

Yes, everything else was same.

Reproducibility could be different for each SDK, I am not sure it is fixed SDK 8.6, I just can't reproduce it there.

SDK 8.0 - reproduced

SDK 8.1 - not reproduced

SDK 8.2 - reproduced

SDK 8.6 - not reproduced

Thanks for the detailed results, that looks a bit strange indeed. Agreed with your doubts whether it is fixed or not based on this track record.

Have you tried the experiments I suggested of applying the no-1-8-v; and/or max-frequency = <24000000>; DTS properties to the mmc node? This is important to determine where the issue is, and where to look for potential source changes that could result in the observed behavior.

Regards, Andreas

I applied the no-1-8-v; and max-frequency = <24000000>; DTS properties on SDK 8.2 and it did not help, I can still reproduce FDT_ERR_BADOFFSET boot issue.

Thank you for the additional experiment. If that this doesn't help there may be some command/protocol incompatibility issue of sorts, not necessarily a signal integrity or electrical issue. But of course at this stage we should not exclude anything quite yet. Let me discuss with my colleagues regarding next steps and get back to you by early next week.

I was able to reproduce this issue on SDK 8.6 when I recompile u-boot with one config change:

replacing CONFIG_ENV_IS_IN_MMC with CONFIG_ENV_IS_NOWHERE in configs/am65x_evm_a53_defconfig

I tried it 100 times:

49 times it succesfully booted to Linux

40 times i got this error:

failed on fdt_overlay_apply(): FDT_ERR_BADOFFSET
failed to apply DT overlay fdt-3
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###

11 times i got this error:

failed on fdt_overlay_apply(): FDT_ERR_BADSTRUCTURE
failed to apply DT overlay fdt-3
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###

Steps to reproduce on SDK 8.6

1) Install ti-processor-sdk-linux-rt-am65xx-evm-08.06.00.47-Linux-x86-Install.bin

2) cd ~/ti-processor-sdk-linux-rt-am65xx-evm-08.06.00.47/

2) flash filesystem/tisdk-default-image-am65xx-evm.wic.xz to SD card

3) mount first partition of SD card to /mnt/boot

4) replace CONFIG_ENV_IS_IN_MMC with CONFIG_ENV_IS_NOWHERE in board-support/u-boot-2021.01+gitAUTOINC+dce3935756-gdce3935756/configs/am65x_evm_a53_defconfig

5) make u-boot

6) sudo cp board-support/u-boot_build/r5/tiboot3.bin board-support/u-boot_build/a53/tispl.bin board-support/u-boot_build/a53/u-boot.img /mnt/boot

7) sync

8) sudo umount /mnt/boot

9) connect microUSB cable to UART

10) start IDK

Console log of all 100 tries is attached:

SDK_8_6_A53_ENV_IS_NOWHERE_FTD_ERR.log.txt

(I am posting here so that this thread is not auto closed. We are still interested.)

Zdenek,

thanks for sharing your test logs of the additional experiment you have done. We should check next if the issue is really due to some memory alignment issue during the FDT overlay application (as the error message suggests) or not. Re-building your U-Boot with different ENV setting as you did will result in a different memory layout, which could be indeed be impacting the observed behavior.

Can you please add the below log code to your U-Boot code, and then share the resulting logs of a "good" and "failed" boot attempt? Any FDT operation usually requires boot original and overlay FTD to be aligned at word (4-byte) boundaries, so I wonder what you will find here.

a0797059@dasso:~/git/u-boot (ti-u-boot-2021.01)
$ git  diff
diff --git a/scripts/dtc/libfdt/fdt_overlay.c b/scripts/dtc/libfdt/fdt_overlay.c
index 7a65c35af6..8cd1c83a82 100644
--- a/scripts/dtc/libfdt/fdt_overlay.c
+++ b/scripts/dtc/libfdt/fdt_overlay.c
@@ -831,6 +831,10 @@ int fdt_overlay_apply(void *fdt, void *fdto)
        uint32_t delta;
        int ret;

+#ifdef CONFIG_TARGET_AM654_A53_EVM
+       printf("%s: fdt=0x%p fdto=0x%p\n", __func__, fdt, fdto);
+#endif
+
        FDT_RO_PROBE(fdt);
        FDT_RO_PROBE(fdto);

Regards, Andreas

Hi Zdenek,

also one more thing, I thought we had checked this already but it doesn't seem the case reviewing the records. Have you tried different (brand) SD cards to boot from to see if there is any impact on the behavior? If not, can you please try it? We need to separate possible HW (SD card IF) from SW failures.

Thanks, Andreas

Hello Zdenek

Were you able to try the suggestion for trying SD card per Andreas's previous recommendation? Any update ?

Hello,

sorry for a late answer.

If I add the printf, than I can't reproduce it. But if it is valuable it prints this on the OK case:

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Sep 06 2023 - 11:01:30 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
SPL initial stack usage: 1416 bytes
Trying to boot from MMC2
spl_load_fit_image: Skip load 'dm': image size is 0!
Starting ATF on ARM64 core...

NOTICE: BL31: v2.8(release):v2.8-226-g2fcd408bb3-dirty
NOTICE: BL31: Built : 15:00:50, Mar 25 2023
I/TC:
I/TC: OP-TEE version: 3.20.0 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Sat Mar 25 15:17:32 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check optee.readthedocs.io/.../porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Fixing SA2UL firewall owner for GP device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Sep 06 2023 - 11:01:55 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
Detected: Silicon SR2.0
i2c_omap i2c@42120000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
Detected: AM6-IDKAPPEVM rev A
Detected: SER-PCIE2LEVM rev A
Trying to boot from MMC2
fdt_overlay_apply: fdt=0x808ea318 fdto=0x81f482c0
fdt_overlay_apply: fdt=0x808ea318 fdto=0x81f482c0

U-Boot 2021.01-00001-g18e219dc1f-dirty (Sep 06 2023 - 11:01:55 +0200)

SoC: AM65X SR2.0 GP
Model: Texas Instruments AM654 Base Board
Board: AM6-COMPROCEVM rev B
DRAM: 4 GiB
MMC: sdhci@4f80000: 0, sdhci@4fa0000: 1
In: serial
Out: serial
Err: serial
am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA00102 cpsw_ver: 0x6BA80102 ale_ver: 0x00293904 Ports:1 mdio_freq:1000000
Detected: Silicon SR2.0
Detected: AM6-IDKAPPEVM rev A
Detected: SER-PCIE2LEVM rev A
Net: eth1: icssg2-eth, eth0: ethernet@46000000port@1
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
Failed to load 'boot.scr'
1490 bytes read in 2 ms (727.5 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
1 bytes read in 2 ms (0 Bytes/s)
Already setup.
18862592 bytes read in 747 ms (24.1 MiB/s)
64327 bytes read in 6 ms (10.2 MiB/s)
11268 bytes read in 3 ms (3.6 MiB/s)
fdt_overlay_apply: fdt=0x0000000088000000 fdto=0x0000000089000000
2333 bytes read in 3 ms (758.8 KiB/s)
fdt_overlay_apply: fdt=0x0000000088000000 fdto=0x0000000089000000
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Device Tree to 000000008feed000, end 000000008fffffff ... OK

Starting kernel ...

[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[ 0.000000] Linux version 5.10.168-rt83-gd14fbab5cc (oe-user@oe-host) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 9.2.1 20191025, GNU ld (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10)) 2.33.1.20191209) #1 SMP PREEMPT_RT Sat Mar 25 15:39:18 UTC 2023
[ 0.000000] Machine model: Texas Instruments AM654 Base Board

Yes, I tried multiple SD cards:

8GB SanDisk which was included with IDK

16GB SanDisk

16GB Kingston

I can reproduce it on all of them.

Zdenek Bouska said:

If I add the printf, than I can't reproduce it. But if it is valuable it prints this on the OK case:

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Sep 06 2023 - 11:01:30 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
SPL initial stack usage: 1416 bytes
Trying to boot from MMC2

Thanks for the additional inputs. Looks like you since moved to SDK v8.6 as well, correct?

Can you confirm that the same SDK v8.6 code base also "breaks" when you don't have the printf() in there, and this being the only change?

If that's the case then this may indeed be some memory alignment issue, probably related to dynamic memory allocation in the context of FDT handling that we need to track down.

Zdenek Bouska said:

8GB SanDisk which was included with IDK

16GB SanDisk

16GB Kingston

I can reproduce it on all of them.

Thanks for those tests, that's good to know.

Regards, Andreas

> Can you confirm that the same SDK v8.6 code base also "breaks" when you don't have the printf() in there, and this being the only change?

It "breaks" on Linux RT SDK 8.6 with configuration change to U-Boot: replacing CONFIG_ENV_IS_IN_MMC with CONFIG_ENV_IS_NOWHERE in configs/am65x_evm_a53_defconfig:

~/ti-processor-sdk-linux-rt-am65xx-evm-08.06.00.47/board-support/u-boot-2021.01+gitAUTOINC+dce3935756-gdce3935756 processor-sdk-local$ git diff
diff --git a/configs/am65x_evm_a53_defconfig b/configs/am65x_evm_a53_defconfig
index 5ec0114675..7c4e444993 100644
--- a/configs/am65x_evm_a53_defconfig
+++ b/configs/am65x_evm_a53_defconfig
@@ -78,7 +78,7 @@ CONFIG_SPL_OF_CONTROL=y
CONFIG_SPL_MULTI_DTB_FIT=y
CONFIG_SPL_MULTI_DTB_FIT_NO_COMPRESSION=y
CONFIG_ENV_OVERWRITE=y
-CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_IS_NOWHERE=y
CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
CONFIG_SYS_RELOC_GD_ENV_ADDR=y
CONFIG_SYS_MMC_ENV_PART=1

This is a script which I used to recompile u-boot:

#!/bin/bash
set -x
set -e
sudo mount /dev/sdd1 /mnt/boot
rm -rf ../r5/
rm -rf ../a53/
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- am65x_evm_r5_defconfig O=../r5
make -j 8 ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- O=../r5
make ARCH=arm CROSS_COMPILE=aarch64-none-linux-gnu- am65x_evm_a53_defconfig O=../a53
make -j 8 ARCH=arm CROSS_COMPILE=aarch64-none-linux-gnu- ATF=../prebuilt-images/bl31.bin TEE=../prebuilt-images/bl32.bin O=../a53
sudo cp ../r5/tiboot3.bin ../a53/tispl.bin ../a53/u-boot.img /mnt/boot/
sync
sudo umount /mnt/boot

Zdenek Bouska said:

It "breaks" on Linux RT SDK 8.6 with configuration change to U-Boot: replacing CONFIG_ENV_IS_IN_MMC with CONFIG_ENV_IS_NOWHERE in configs/am65x_evm_a53_defconfig:

Seems to confirms the theory, moving things around in memory seems to affect things, likely alignment, making it fail or work.

Zdenek Bouska said:

This is a script which I used to recompile u-boot:

Steps look good I'd say.

Zdenek Bouska said:

If I add the printf, than I can't reproduce it. But if it is valuable it prints this on the OK case:

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Sep 06 2023 - 11:01:30 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
SPL initial stack usage: 1416 bytes
Trying to boot from MMC2
spl_load_fit_image: Skip load 'dm': image size is 0!
Starting ATF on ARM64 core...

NOTICE: BL31: v2.8(release):v2.8-226-g2fcd408bb3-dirty
NOTICE: BL31: Built : 15:00:50, Mar 25 2023
I/TC:
I/TC: OP-TEE version: 3.20.0 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Sat Mar 25 15:17:32 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check optee.readthedocs.io/.../porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Fixing SA2UL firewall owner for GP device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Sep 06 2023 - 11:01:55 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
Detected: Silicon SR2.0
i2c_omap i2c@42120000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
Detected: AM6-IDKAPPEVM rev A
Detected: SER-PCIE2LEVM rev A
Trying to boot from MMC2
fdt_overlay_apply: fdt=0x808ea318 fdto=0x81f482c0
fdt_overlay_apply: fdt=0x808ea318 fdto=0x81f482c0

As for your earlier comment above, can you please try to do a larger number of runs with this setup (which seems to be always working for you), and check if the fdt/fdto addresses in fdt_overlay_apply: fdt=0x808ea318 fdto=0x81f482c0 remain constant and the same at all times, from run to run? The idea is to see if there is any variation boot to boot. If there is, this could further help prove the 'alignment theory'.

Regards, Andreas

If I remember correctly I tried it at least twice and it was constant.

I will run it automatically 100 times when I will have time for it.

Zdenek Bouska said:

If I remember correctly I tried it at least twice and it was constant.

I will run it automatically 100 times when I will have time for it.

Yes we'd need to run again multiple times. Thanks for bearing with us with those many experiments that are suggested. Will be ready to continue looking at this once I get your updated results.

Regards, Andreas

Offsets remain constant for multiple boots:

$ grep 'fdt_overlay_apply: fdt' minicom.mod.uboot.print.log.txt | sort -u
fdt_overlay_apply: fdt=0x0000000088000000 fdto=0x0000000089000000
fdt_overlay_apply: fdt=0x808eb028 fdto=0x81f482c0

I got FDT_ERR_NOTFOUND 13 times: 

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Oct 11 2023 - 10:22:07 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
SPL initial stack usage: 1416 bytes
Trying to boot from MMC2
spl_load_fit_image: Skip load 'dm': image size is 0!
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.8(release):v2.8-226-g2fcd408bb3-dirty
NOTICE:  BL31: Built : 15:00:50, Mar 25 2023
I/TC: 
I/TC: OP-TEE version: 3.20.0 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Sat Mar 25 15:17:32 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Fixing SA2UL firewall owner for GP device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Oct 11 2023 - 10:22:31 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
Detected: Silicon SR2.0
i2c_omap i2c@42120000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
Detected: AM6-IDKAPPEVM rev A
Detected: SER-PCIE2LEVM rev A
Trying to boot from MMC2
fdt_overlay_apply: fdt=0x808eb028 fdto=0x81f482c0
fdt_overlay_apply: fdt=0x808eb028 fdto=0x81f482c0
failed on fdt_overlay_apply(): FDT_ERR_NOTFOUND
failed to apply DT overlay fdt-5
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###

And 376 times it booted to Linux:

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Oct 11 2023 - 10:22:07 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
SPL initial stack usage: 1416 bytes
Trying to boot from MMC2
spl_load_fit_image: Skip load 'dm': image size is 0!
Starting ATF on ARM64 core...

NOTICE:  BL31: v2.8(release):v2.8-226-g2fcd408bb3-dirty
NOTICE:  BL31: Built : 15:00:50, Mar 25 2023
I/TC: 
I/TC: OP-TEE version: 3.20.0 (gcc version 9.2.1 20191025 (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (arm-9.10))) #1 Sat Mar 25 15:17:32 UTC 2023 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
I/TC: HUK Initialized
I/TC: Activated SA2UL device
I/TC: Fixing SA2UL firewall owner for GP device
I/TC: Enabled firewalls for SA2UL TRNG device
I/TC: SA2UL TRNG initialized
I/TC: SA2UL Drivers initialized
I/TC: Primary CPU switching to normal world boot

U-Boot SPL 2021.01-00001-g18e219dc1f-dirty (Oct 11 2023 - 10:22:31 +0200)
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.5--v08.06.05 (Chill Capybar')
Detected: Silicon SR2.0
i2c_omap i2c@42120000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
Detected: AM6-IDKAPPEVM rev A
Detected: SER-PCIE2LEVM rev A
Trying to boot from MMC2
fdt_overlay_apply: fdt=0x808eb028 fdto=0x81f482c0
fdt_overlay_apply: fdt=0x808eb028 fdto=0x81f482c0


U-Boot 2021.01-00001-g18e219dc1f-dirty (Oct 11 2023 - 10:22:31 +0200)

SoC:   AM65X SR2.0 GP
Model: Texas Instruments AM654 Base Board
Board: AM6-COMPROCEVM rev B
DRAM:  4 GiB
MMC:   sdhci@4f80000: 0, sdhci@4fa0000: 1
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA00102 cpsw_ver: 0x6BA80102 ale_ver: 0x00293904 Ports:1 mdio_freq:1000000
Detected: Silicon SR2.0
Detected: AM6-IDKAPPEVM rev A
Detected: SER-PCIE2LEVM rev A
Net:   eth1: icssg2-eth, eth0: ethernet@46000000port@1
Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
Failed to load 'boot.scr'
1490 bytes read in 2 ms (727.5 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
1 bytes read in 1 ms (1000 Bytes/s)
Already setup.
18862592 bytes read in 747 ms (24.1 MiB/s)
64327 bytes read in 6 ms (10.2 MiB/s)
11268 bytes read in 3 ms (3.6 MiB/s)
fdt_overlay_apply: fdt=0x0000000088000000 fdto=0x0000000089000000
2333 bytes read in 3 ms (758.8 KiB/s)
fdt_overlay_apply: fdt=0x0000000088000000 fdto=0x0000000089000000
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 000000008feed000, end 000000008fffffff ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]

Only change to U-Boot is that additional print.

Whole log:

minicom.mod.uboot.print.log.txt

Hi Zdenek,

thanks for the logs (and the additional experiments, I know it's A LOT of work); good to know that this particular offset remains constant and there doesn't seem to be any correlation. Let me dive into the sources again and see what else could possibly be causing it and what possible next steps could be.

Regards, Andreas

Hi Zdenek,

I did file an issue report internally (SITSW-3198) for the R&D team to investigate further. From all our discussions it seems like this is something we should be able to re-create here at TI using a standard IDK EVM, and then debug/investigate in more detail through JTAG and other means.

Also since the issue is overlay related, I wonder if removing the SER-PCIE2LEVM daughtercard (which you may or may not need for whatever you are using the EVM for) from the EVM has a direct impact here. But since you've already done a substantial amount of testing I'm hesitant asking you to do more. I did flag this as one thing for the team to look into however.

Regards, Andreas

Hi Zdenek

The dev team was able to reproduce the issue on SDK8.0 , but as mentioned are not able to reproduce the issue on latest SDKs. 

How important is this for you to root cause and and understand vs running with the latest SDK.

For us to do all the tracing on what changed will take some effort and I need to make sure it is worth everyone's time and effort. I know you have spend a lot of time on this issue, so I want to be respectful of that , but at the same point also wanted to make sure I understand the priority and the need to help root cause this, given it is likely that you will go with the latest SDK.

By any chance, have you also tried the latest SDK and have you seen any issues?

Regards

Mukul