AM6442: Booting combined R5 Application and Linux Images(ATF.OPTEE.SPL) fails when SD is used as boot device(Made some modification to the bootloader)

Hi Upendar,

Thank you for the detailed explanation.

There is no SBL_SD_LINUX example in MCU+ SDK. May I know how did you enable the Linux SBL boot flow from the SD card?

Regards,

Prashant

Hi Prashant,

Yes. you are right. I have modified the SBL_SD similar to SBL_EMMC_LINUX. Modifed the Bootloader_parseMultiCoreAppImage to parse Load only rprc files. And I have a created single app image with R5 application and Linux binraries using LinuxAppImage creator script(modified the makefile to add R5 rprc file).

Regards,

Upendar

Hi Upendar,

In that case, I think I know a possible root cause of the issue.

Any SBL as part of self reset via Bootloader_socCpuResetReleaseSelf performs the Security Handover as shown below. If successful, the security handover results in the System Firmware relinquishing the responsibility of firewall configurations & gives it to the core requesting the Security Handover.

Since you are using the SBL_SD which uses different Board Configurations than used by the Linux SBLs bootloader, the security handover is performed successfully which results in System Firmware NACKing any request for firewall configurations which is what is seen in your shared OPTEE logs.

Assuming this to be the root cause, could you please apply the following patch for SBL_SD makefile to use Linux Board Configurations & see if this indeed resolves the issue

diff --git a/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/ti-arm-clang/makefile b/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/ti-arm-clang/makefile
index f0d7186f5..250f172d1 100644
--- a/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/ti-arm-clang/makefile
+++ b/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/ti-arm-clang/makefile
@@ -243,7 +243,7 @@ BOOTIMAGE_CERT_GEN_CMD=$(PYTHON) $(MCU_PLUS_SDK_PATH)/tools/boot/signing/rom_ima
 SYSFW_PATH=$(MCU_PLUS_SDK_PATH)/source/drivers/sciclient/soc/am64x_am243x
 SYSFW_LOAD_ADDR=0x44000
 BOARDCFG_LOAD_ADDR=0x7B000
-BOARDCFG_BLOB=$(MCU_PLUS_SDK_PATH)/source/drivers/sciclient/sciclient_default_boardcfg/am64x/boardcfg_blob.bin
+BOARDCFG_BLOB=$(MCU_PLUS_SDK_PATH)/source/drivers/sciclient/sciclient_default_boardcfg/am64x/boardcfg_blob_linux.bin
 
 
 SBL_RUN_ADDRESS=0x70000000

Regards,

Prashant

Hi Prashant,

You are correct. I have built SBL_SD with your patch and the kernel is booting fine. But there are other errors. The only things I have copied to SD card are app(only Linux Binaries(ATF, OPTEE, A53 APL)) and SBL_SD. Kernel, Device Tree and U-Boot are from TI pre-built.(Boards boots fine with Complete TI Prebuilt wic image flashing).

[ 1.485502] ALSA device list:
[ 1.488561] No soundcards found.
[ 1.501076] mmc0: SDHCI controller on fa10000.mmc [fa10000.mmc] using ADMA 64-bit
[ 1.513197] mmc1: SDHCI controller on fa00000.mmc [fa00000.mmc] using ADMA 64-bit
[ 1.521282] Waiting for root device PARTUUID=0cc0f25a-02...
[ 1.574785] mmc1: new ultra high speed SDR104 SDHC card at address aaaa
[ 1.582436] mmcblk1: mmc1:aaaa SC16G 14.8 GiB
[ 1.592518] mmcblk1: p1 p2
[ 1.611065] mmc0: Command Queue Engine enabled
[ 1.615711] mmc0: new HS200 MMC card at address 0001
[ 1.621799] mmcblk0: mmc0:0001 S0J56X 14.8 GiB
[ 1.628230] mmcblk0: p1
[ 1.631599] mmcblk0boot0: mmc0:0001 S0J56X 31.5 MiB
[ 1.637442] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Quota mode: none.
[ 1.638335] mmcblk0boot1: mmc0:0001 S0J56X 31.5 MiB
[ 1.646146] VFS: Mounted root (ext4 filesystem) on device 179:2.
[ 1.652312] mmcblk0rpmb: mmc0:0001 S0J56X 4.00 MiB, chardev (240:0)
[ 1.660855] devtmpfs: mounted
[ 1.667813] Freeing unused kernel memory: 2112K
[ 1.672568] Run /sbin/init as init process
[ 1.687420] mmc1: ADMA error: 0x02000002
[ 1.691366] mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========
[ 1.697793] mmc1: sdhci: Sys addr: 0x000000c0 | Version: 0x00001004
[ 1.704219] mmc1: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[ 1.710646] mmc1: sdhci: Argument: 0x00042a20 | Trn mode: 0x0000003b
[ 1.717072] mmc1: sdhci: Present: 0x01f70000 | Host ctl: 0x0000001f
[ 1.723500] mmc1: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
[ 1.729927] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 1.736353] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000
[ 1.742778] mmc1: sdhci: Int enab: 0x03ff008b | Sig enab: 0x03ff008b
[ 1.749206] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[ 1.755633] mmc1: sdhci: Caps: 0x3de8c801 | Caps_1: 0x18002407
[ 1.762060] mmc1: sdhci: Cmd: 0x0000123a | Max curr: 0x00000000
[ 1.768487] mmc1: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x0076b27f
[ 1.774914] mmc1: sdhci: Resp[2]: 0x325b5900 | Resp[3]: 0x00000900
[ 1.781342] mmc1: sdhci: Host ctl2: 0x0000040b
[ 1.785774] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x000000008166f230
[ 1.792896] mmc1: sdhci: ============================================
[ 1.799321] mmc1: sdhci: 8166f200: DMA 0x00000000817f0000, LEN 0x2000, Attr=0x21
[ 1.806704] mmc1: sdhci: 8166f20c: DMA 0x0000000081629000, LEN 0x1000, Attr=0x21
[ 1.814088] mmc1: sdhci: 8166f218: DMA 0x000000008187c000, LEN 0x4000, Attr=0x21
[ 1.821472] mmc1: sdhci: 8166f224: DMA 0x00000000819c8000, LEN 0x0000, Attr=0x21
[ 1.828856] mmc1: sdhci: 8166f230: DMA 0x00000000819d8000, LEN 0x1000, Attr=0x21
[ 1.836239] mmc1: sdhci: 8166f23c: DMA 0x0000000000000000, LEN 0x0000, Attr=0x03
[ 1.849382] mmc1: ADMA error: 0x02000002
[ 1.853328] mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========
[ 1.859755] mmc1: sdhci: Sys addr: 0x000000c0 | Version: 0x00001004
[ 1.866181] mmc1: sdhci: Blk size: 0x00007200 | Blk cnt: 0x00000000
[ 1.872608] mmc1: sdhci: Argument: 0x00042a20 | Trn mode: 0x0000003b
[ 1.879034] mmc1: sdhci: Present: 0x01f70000 | Host ctl: 0x0000001f
[ 1.885461] mmc1: sdhci: Power: 0x0000000f | Blk gap: 0x00000080
[ 1.891887] mmc1: sdhci: Wake-up: 0x00000000 | Clock: 0x00000007
[ 1.898315] mmc1: sdhci: Timeout: 0x00000000 | Int stat: 0x00000000
[ 1.904741] mmc1: sdhci: Int enab: 0x03ff008b | Sig enab: 0x03ff008b
[ 1.911168] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[ 1.917596] mmc1: sdhci: Caps: 0x3de8c801 | Caps_1: 0x18002407
[ 1.924022] mmc1: sdhci: Cmd: 0x0000123a | Max curr: 0x00000000
[ 1.930448] mmc1: sdhci: Resp[0]: 0x00000900 | Resp[1]: 0x0076b27f
[ 1.936875] mmc1: sdhci: Resp[2]: 0x325b5900 | Resp[3]: 0x00000900
[ 1.943303] mmc1: sdhci: Host ctl2: 0x0000040b
[ 1.947735] mmc1: sdhci: ADMA Err: 0x00000000 | ADMA Ptr: 0x000000008166f230
[ 1.954856] mmc1: sdhci: ============================================
[ 1.961281] mmc1: sdhci: 8166f200: DMA 0x00000000817f0000, LEN 0x2000, Attr=0x21
[ 1.968664] mmc1: sdhci: 8166f20c: DMA 0x0000000081629000, LEN 0x1000, Attr=0x21
[ 1.976048] mmc1: sdhci: 8166f218: DMA 0x000000008187c000, LEN 0x4000, Attr=0x21
[ 1.983431] mmc1: sdhci: 8166f224: DMA 0x00000000819c8000, LEN 0x0000, Attr=0x21
[ 1.990815] mmc1: sdhci: 8166f230: DMA 0x00000000819d8000, LEN 0x1000, Attr=0x21
[ 1.998199] mmc1: sdhci: 8166f23c: DMA 0x0000000000000000, LEN 0x0000, Attr=0x03
[ 2.273202] mmc1: ADMA error: 0x02000002
[ 2.277153] mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========

I have found a thread with similar problem. https://e2e.ti.com/support/processors-group/processors/f/processors-forum/965498/am6548-mmc1-got-data-interrupt-0x00000002-error-message . I am trying to build the kernel with these changes. But I am not sure it will work for me. I think problem is elsewhere.

Anyway please provide your thoughts on this.

Thank you.

Best Regards,

Upendar

Hi Upendar,

Could you please apply the following patch in the main.c file of the SBL_SD & see if that resolves the issue

diff --git a/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c b/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c
index 35533822a..2a0d492b5 100644
--- a/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c
+++ b/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c
@@ -285,6 +285,10 @@ int main(void)
                 {
                     status = Bootloader_rprcImageLoad(bootHandle, &bootImageInfo.cpuInfo[CSL_CORE_ID_R5FSS0_0]);
                 }
+
+                extern void PowerClock_deinit();
+                PowerClock_deinit();
+
                 /* Reset self cluster, both Core0 and Core 1. Init RAMs and run the app  */
                 status = Bootloader_runSelfCpu(bootHandle, &bootImageInfo);
             }

Regards,

Prashant

Hi Prashant,

That didn't work. Genereated a different error.

Image loading done, switching to application ...
ERNOTICER: BL31: vO2.7(rRel:ease):v 2.B7.0-359-g1309c6c80
NOTICE: BL31: Built : 08:04:54, Dec 19 2023

ERROR: Unhandled External Abort received on 0x80000000 from S-EL1
ERROR: exception reason=1 syndrome=0x82000010
Unhandled Exception from EL1
x0 = 0x0000000000000000
x1 = 0x0000000000000000
x2 = 0x0000000000000000
x3 = 0x0000000000000000
x4 = 0x0000000000000000
x5 = 0x0000000000000000
x6 = 0x0000000000000000
x7 = 0x0000000000000000
x8 = 0x0000000000000000
x9 = 0x0000000000000000
x10 = 0x0000000000000000
x11 = 0x0000000000000000
x12 = 0x0000000000000000
x13 = 0x0000000000000000
x14 = 0x0000000000000000
x15 = 0x0000000000000000
x16 = 0x0000000000000000
x17 = 0x0000000000000000
x18 = 0x0000000000000000
x19 = 0x0000000000000000
x20 = 0x0000000000000000
x21 = 0x0000000000000000
x22 = 0x0000000000000000
x23 = 0x0000000000000000
x24 = 0x0000000000000000
x25 = 0x0000000000000000
x26 = 0x0000000000000000
x27 = 0x0000000000000000
x28 = 0x0000000000000000
x29 = 0x0000000000000000
x30 = 0x0000000000000000
scr_el3 = 0x0000000000000e38
sctlr_el3 = 0x0000000030cd183f
cptr_el3 = 0x0000000000000000
tcr_el3 = 0x0000000080803520
daif = 0x00000000000002c0
mair_el3 = 0x00000000004404ff
spsr_el3 = 0x00000000000003c5
elr_el3 = 0x000000009e800000
ttbr0_el3 = 0x00000000701ae800
esr_el3 = 0x0000000082000010
far_el3 = 0x000000009e800000
spsr_el1 = 0x0000000000000000
elr_el1 = 0x0000000000000000
spsr_abt = 0x0000000000000000
spsr_und = 0x0000000000000000
spsr_irq = 0x0000000000000000
spsr_fiq = 0x0000000000000000
sctlr_el1 = 0x0000000000c00801
actlr_el1 = 0x0000000000000000
cpacr_el1 = 0x0000000000000000
csselr_el1 = 0x0000000000000000
sp_el1 = 0x0000000000000000
esr_el1 = 0x0000000000000000
ttbr0_el1 = 0x0000000000000000
ttbr1_el1 = 0x0000000000000000
mair_el1 = 0x0000000000000000
amair_el1 = 0x0000000000000000
tcr_el1 = 0x0000000000800080
tpidr_el1 = 0x0000000000000000
tpidr_el0 = 0x0000000000000000
tpidrro_el0 = 0x0000000000000000
par_el1 = 0x0000000000000000
mpidr_el1 = 0x0000000080000000
afsr0_el1 = 0x0000000000000000
afsr1_el1 = 0x0000000000000000
contextidr_el1 = 0x0000000000000000
vbar_el1 = 0x0000000000000000
cntp_ctl_el0 = 0x0000000000000000
cntp_cval_el0 = 0x0000000000000000
cntv_ctl_el0 = 0x0000000000000000
cntv_cval_el0 = 0x0000000000000000
cntkctl_el1 = 0x0000000000000000
sp_el0 = 0x00000000701ab360
isr_el1 = 0x0000000000000000
dacr32_el2 = 0x0000000000000000
ifsr32_el2 = 0x0000000000000000
cpuectlr_el1 = 0x0000000000000040
cpumerrsr_el1 = 0x00000000100c0060
l2merrsr_el1 = 0x0000000011000860
cpuactlr_el1 = 0x00001000090ca000

Regards,

Upendar

Hi Upendar,

Let me look at another possible reasons for this issue. In the meantime, could you please test with the PSDK v8.6 WIC image instead of v9 image & see if the issue still comes. This will help isolate if the issue is specifically with the v9.

Regards,

Prashant

Hi Prashant.

Ok. I can try. But just to be clear, when I flashed the SD card tisdk-default-image-am64xx-evm.wic.xz from latest release, Linux booting works fine. The problem is only with SBL and Combined APP image I built.

Thank you.

Regards,

Upendar

Hi Prashant,

It works with PSDK 8.6 Image. Thank you. But could you please let me know why so that I can make the changes in my Yocto build also?

I have one more observation. The "app" that works has only linux binraies on it. I have added R5 application with it. R5 application ran and on the console output I see the data only for bl31 execution. I think it has to do with UART0 being used in both R5 and A53.  

I have made changes to k3-am642-evm.dts to use UART3 instead of UART0. I haven't tested it yet. 

My question is, if the Linux boot is successful, I should see LED23 blinking. But I don't see that.  If you have some info on this, please let me know.

I will update this thread as and when I have an update on the test with new kernel image.

Thank you. You have been very helpful.

Regards,

Upendar

Hi Upendar,

Upendar Sama said:

But could you please let me know why so that I can make the changes in my Yocto build also?

I believe when using v9 WIC image the issue comes because of the incompatibility between the MMCSD driver of MCU+ SDK v9 & the PSDK v9. Please note, the v9 was a major revision from v8 & so that brought a lot of changes including in the Linux Kernel. This might have resulted in some sort of incompatibility between the two drivers in that the MMCSD driver in PSDK v9 is not able to work with the SD card state left by the SBL_SD bootloader.

So, I suggested the previous change of including the PowerClock_deinit call which ideally should power off the module thus brining it to a clean state for it to be later consumed by the Linux Kernel. However, as you said this somehow didn't work.

Let me try to create the similar setup as yours & see if I am able to reproduce the issue.

Upendar Sama said:

I have made changes to k3-am642-evm.dts to use UART3 instead of UART0. I haven't tested it yet. 

I would recommend to change the UART port used the R5 application instead of changing in Linux. Taking hello world as an example, the UART port can be changed to UART1 as shown below

Regards,

Prashant

Hi Prashant..

Thank you. For now I will use V8.6. Since we are in evaluation phase, should not be a issue. But a solution would be great.

And yeah, I did the UART modification for SBL and R5 App. And both Linux and R5 applications are running. But for some reason I cannot see the console output of SBL on any of the COM Ports detected for J26 port. I can see Linux console output on UART0.

Thank you.

Regards,

Upendar

Hi Upendar,

For the SBL logs, have you made sure the Debug logs are allowed to come on UART & the logging UART instance is set to maybe UART0 like so

Regards,

Prashant

Hi Prashant,

UART log was already enabled. But no console output from SBL.

Regards,

Upendar

Hi Upendar,

I do see the SBL logs in the very first post of this thread. So, I would like to know the changes you have done in the SBL_SD bootloader since then that resulted in the SBL logs not coming anymore. If you could share your syscfg file of SBL_SD bootloader, I can then review the settings & see if everything is in place.

Regards,

Prashant

Hi Prashant,

The issue is solved. I have configured UART3 to different Pins and then it worked. (Changed UART3_RXD from U20 to D16, UART3_TXD from U18 to E16).

Regards,

Upendar

Hi Upendar,

This is good to know. I also realise now the PowerClock_deinit call actually power off all the modules enabled in Sysconfig. However, we are only interested in powering off the MMCSD module. So, could you please try the following patch & see if this resolves the issue with PSDK v9

diff --git a/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c b/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c
index 35533822a..5dc358596 100644
--- a/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c
+++ b/examples/drivers/boot/sbl_sd/am64x-evm/r5fss0-0_nortos/main.c
@@ -285,6 +285,10 @@ int main(void)
                 {
                     status = Bootloader_rprcImageLoad(bootHandle, &bootImageInfo.cpuInfo[CSL_CORE_ID_R5FSS0_0]);
                 }
+
+                /* Power off the SD card */
+                SOC_moduleClockEnable(TISCI_DEV_MMCSD1, 0);
+
                 /* Reset self cluster, both Core0 and Core 1. Init RAMs and run the app  */
                 status = Bootloader_runSelfCpu(bootHandle, &bootImageInfo);
             }

Regards,

Prashant

Hi Prashant,

Last patch fixed the issue. Linux booting fine. R5 Application is also running. Thank you.!.

But there is an error message. 

Image loading done, switching to application ...

ERROR: Bootloader_socCpuResetReleaseSel

GPIO LED Blink Test Started ...

R5 core where SBL running, did not reset properly.

Thank you.

Best Regards,
Upendar

Hi Upendar,

This error is caused by the following snippet

int32_t Bootloader_socCpuResetReleaseSelf(void)
{
        ...
        
        if(status==SystemP_SUCCESS)
        {
            status = Bootloader_socSecHandover();
        }
        if(status==SystemP_SUCCESS)
        {
            /* disable interrupts if enabled */
            HwiP_disable();

            /* flush all caches */
            CacheP_wbInvAll(CacheP_TYPE_ALL);

            /* execute wfi, now SYSFW will execute the above commands and reset core0 and core 1 */
            __asm__ __volatile__ ("wfi" "\n\t": : : "memory");
        }
        if(status != SystemP_SUCCESS)
        {
            DebugP_logError("CPU reset sequence failed for %s\r\n", Bootloader_socGetCoreName(CSL_CORE_ID_R5FSS0_0));
        }
    }
    return status;
}

As I said, the Bootloader_socSecHandover call will fail when using Linux SBLs which results in the non-execution of the following code in the if condition. I suggested previously to simply change the usual Board Configurations to Linux Board Configurations so that you don't have to change the driver code.

However, if you want you can simply comment that Bootloader_socSecHandover function call so that it does not request for Security Handover & so the following code in the if condition executes & a proper reset happens.

Regards,

Prashant