J721EXSOMXEVM: PDK Low Power Manager Demo Linux Boot Errors

Hi,

Couple of questions before we dig deeper:

• Are you using general purpose samples or high secure samples on your board? 
• Also can you confirm if you can boot to Linux using the default boot flow using SPL U-Boot?

Best regards,

Keerthy 

1. I am using a GP device. I am using unsigned images and they boot fine.

2. I am able to boot Linux using the default boot flow using SPL U-Boot. I was able to boot Linux originally with the default Yocto build off of the SD card with U-Boot. Then, I made the devicetree changes for the LPM demo and tested that I can still successfully boot the new image from the SD card with U-Boot. This process and check is mentioned in step 1 of my original post. Please let me know if there is something you are asking me to clarify further.

I suspect that my issue could be that my SBL is not configured correctly. As mentioned in 2B of my post, the SBL image I have is just the default that gets created when I build it as is. I have not made any changes or set any options because the LPM documentation does not instruct me to do so. However, after looking at this page on the J721E SBL and the Boot App, I realized that there are some options that might need to be changed from their default state to make the demo work. Can you confirm if this is the case? If so, could you provide instructions on which options to set and how to build?

Thanks for your help.

Hi Louie,

We are checking internally if the low power modes are having issues with the default SDK. We will keep you posted.

- Keerthy

Hi Keerthy,

thanks for checking this. Could you provide an update? We would need a working flow, which in this case needs only MCU1_0 firmware (MCU R5F) and A72 Linux, which here does not depend on OVX components or ethfw.

Thanks!

--Gunter

Hi Gunter/Louie,

I am able to reproduce the issue. For some reason the uniflash was flaky. I used the U-boot to flash all the above binaries to OSPI.

Logs on A72 On the EVM:

NOTICE:  BL31: v2.10.0(release):v2.10.0-367-g00f1ec6b87-dirty
NOTICE:  BL31: Built : 16:09:05, Feb  9 2024
ERROR:   Timeout waiting for thread SP_RESPONSE to fill
ERROR:   Thread SP_RESPONSE verification failed (-60)
ERROR:   Message receive failed (-60)
ERROR:   Failed to get response (-60)
ERROR:   Transfer send failed (-60)
ERROR:   Timeout waiting for thread SP_RESPONSE to fill
ERROR:   Thread SP_RESPONSE verification failed (-60)
ERROR:   Message receive failed (-60)
ERROR:   Failed to get response (-60)
ERROR:   Transfer send failed (-60)
ERROR:   Unable to query firmware capabilities (-60)
I/TC: 
I/TC: OP-TEE version: 4.2.0-dev (gcc version 13.3.0 (GCC)) #1 Fri Apr 12 09:51:21 UTC 2024 aarch64
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: GIC redistributor base address not provided
I/TC: Assuming default GIC group status and modifier
E/TC:0 0 k3_sec_proxy_verify_thread:108 Queue is busy
E/TC:0 0 k3_sec_proxy_recv:196 Thread SEC_PROXY_RESPONSE_THREAD verification failed. ret = -65523
E/TC:0 0 ti_sci_get_response:101 Message receive failed (-65523)
E/TC:0 0 ti_sci_do_xfer:150 Failed to get response (-65523)
E/TC:0 0 ti_sci_init:486 Unable to communicate with control firmware (-65523)
E/TC:0 0 call_initcalls:43 Initcall __text_start + 0x00070b10 failed
E/TC:0 0 k3_sec_proxy_verify_thread:108 Queue is busy
E/TC:0 0 k3_sec_proxy_recv:196 Thread SEC_PROXY_RESPONSE_THREAD verification failed. ret = -65523
E/TC:0 0 ti_sci_get_response:101 Message receive failed (-65523)
E/TC:0 0 ti_sci_do_xfer:150 Failed to get response (-65523)
E/TC:0 0 k3_sec_proxy_verify_thread:108 Queue is busy
E/TC:0 0 k3_sec_proxy_recv:196 Thread SEC_PROXY_RESPONSE_THREAD verification failed. ret = -65523
E/TC:0 0 ti_sci_get_response:101 Message receive failed (-65523)
E/TC:0 0 ti_sci_do_xfer:150 Failed to get response (-65523)
E/TC:0 0 k3_sec_proxy_verify_thread:108 Queue is busy
E/TC:0 0 k3_sec_proxy_recv:196 Thread SEC_PROXY_RESPONSE_THREAD verification failed. ret = -65523
E/TC:0 0 ti_sci_get_response:101 Message receive failed (-65523)
E/TC:0 0 ti_sci_do_xfer:150 Failed to get response (-65523)
E/TC:0 0 tee_otp_get_hw_unique_key:97 Could not get HUK
E/TC:0 0 call_initcalls:43 Initcall __text_start + 0x00070b38 failed
E/TC:0 0 k3_sec_proxy_verify_thread:108 Queue is busy
E/TC:0 0 k3_sec_proxy_recv:196 Thread SEC_PROXY_RESPONSE_THREAD verification failed. ret = -65523
E/TC:0 0 ti_sci_get_response:101 Message receive failed (-65523)
E/TC:0 0 ti_sci_do_xfer:150 Failed to get response (-65523)
E/TC:0 0 sa2ul_init:61 Failed to get SA2UL device
E/TC:0 0 call_initcalls:43 Initcall __text_start + 0x00070b40 failed
E/TC:0 0 
E/TC:0 0 Core data-abort at address 0x14 (translation fault)
E/TC:0 0  esr 0x96000005  ttbr0 0x9e8a2000   ttbr1 0x00000000   cidr 0x0
E/TC:0 0  cpu #0          cpsr 0x600003c4
E/TC:0 0  x0  000000009e875000 x1  0000000000000000
E/TC:0 0  x2  0000000000000000 x3  0000000000000000
E/TC:0 0  x4  0000000000000050 x5  000000009e892d70
E/TC:0 0  x6  ffffffffffffffb0 x7  0000000000010cb0
E/TC:0 0  x8  0000000000010cb0 x9  000000009e892f80
E/TC:0 0  x10 000000009e882070 x11 0000000000000008
E/TC:0 0  x12 0000000000000000 x13 000000009e8a3e60
E/TC:0 0  x14 0000000000000000 x15 0000000000000000
E/TC:0 0  x16 000000009e81cc5c x17 0000000000000000
E/TC:0 0  x18 0000000000000000 x19 000000009e8a41e0
E/TC:0 0  x20 000000009e8a41e8 x21 000000009e875000
E/TC:0 0  x22 000000009e875000 x23 000000009e875f00
E/TC:0 0  x24 000000009e874dc0 x25 0000000000000000
E/TC:0 0  x26 0000000000000000 x27 0000000000000000
E/TC:0 0  x28 0000000000000000 x29 000000009e8a4170
E/TC:0 0  x30 000000009e81741c elr 000000009e81742c
E/TC:0 0  sp_el0 000000009e8a4170
E/TC:0 0 TEE load address @ 0x9e800000
E/TC:0 0 Call stack:
E/TC:0 0  0x9e81742c
E/TC:0 0  0x9e807ca0
E/TC:0 0  0x9e8225fc
E/TC:0 0  0x9e807e9c
E/TC:0 0 Panic 'unhandled pageable abort' at /usr/src/debug/optee-os/4.2.0+git/core/arch/arm/kernel/abort.c:582 <abort_handler>
E/TC:0 0 TEE load address @ 0x9e800000
E/TC:0 0 Call stack:
E/TC:0 0  0x9e80817c
E/TC:0 0  0x9e81f0c0
E/TC:0 0  0x9e807884
E/TC:0 0  0x9e804a68

Logs from R5F:

Starting Sciserver..... PASSED

MCU R5F App started at 33002 usecs

PMIC initialization done.
IPC_echo_test (core : mcu1_0) .....
Waiting for Linux VDev ready...
Calling Sciclient_procBootRequestProcessor, ProcId 0x6... 
Calling Sciclient_procBootRequestProcessor, ProcId 0x7... 
Loading BootImage
BootImage completed, status = 0
Sciclient_procBootReleaseProcessor, ProcId 0x6...
Sciclient_procBootReleaseProcessor, ProcId 0x7...
Sleeping for 10 seconds after each stage
MCU2_0 booted
MCU2_1 booted
Calling Sciclient_procBootRequestProcessor, ProcId 0x8... 
Calling Sciclient_procBootRequestProcessor, ProcId 0x9... 
Calling Sciclient_procBootRequestProcessor, ProcId 0x3... 
Calling Sciclient_procBootRequestProcessor, ProcId 0x4... 
Calling Sciclient_procBootRequestProcessor, ProcId 0x30... 
Loading BootImage
BootImage completed, status = 0
Sciclient_procBootReleaseProcessor, ProcId 0x8...
Sciclient_procBootReleaseProcessor, ProcId 0x9...
Sciclient_procBootReleaseProcessor, ProcId 0x3...
Sciclient_procBootReleaseProcessor, ProcId 0x4...
Sciclient_procBootReleaseProcessor, ProcId 0x30...
C7X_0 booted
Sleeping for 10 seconds after each stage
MCU3_0 booted
MCU3_1 booted
Calling Sciclient_procBootRequestProcessor, ProcId 0x20... 
Loading BootImage
BootImage completed, status = 0
Sciclient_procBootReleaseProcessor, ProcId 0x20...
Sleeping for 20 seconds after the last stage
Boot App: Started at 2335177 usec
Boot App: Total Num booted cores = 8
Boot App: Booted Core ID #10 at 3646001 usecs
Boot App: Booted Core ID #11 at 4646000 usecs
Boot App: Booted Core ID #12 at 16345001 usecs
Boot App: Booted Core ID #13 at 17345000 usecs
Boot App: Booted Core ID #16 at 18345001 usecs
Boot App: Booted Core ID #17 at 19345000 usecs
Boot App: Booted Core ID #18 at 20345000 usecs
Boot App: Booted Core ID #0 at 35894001 usecs
Exiting rpmsg_vdevMonitorFxn!
MCU only app: Inside MCU ONLY task!
MCU only app: Disabling MAXT_OUTRG_EN for TMPSENS1:4 in MAIN domain!
MCU only app: STATE INFO :: CURRENTLY IN ACTIVE MODE!
MCU only app: LED LD5 should be ON
MCU only app: Please measure TP133 and TP134!
MCU only app: Expected values in ACTIVE mode:
TP133: HIGH
TP134: HIGH
MCU only app: Kindly unload Remoteproc modules from Linux. Type these commands in Linux CMD:
   modprobe -r ti_k3_r5_remoteproc
   modprobe -r ti_k3_dsp_remoteproc
MCU only app: After modules have been removed, press 1 and enter

Invalid input: MCU only app: Kindly unload Remoteproc modules from Linux. Type these commands in Linux CMD:
   modprobe -r ti_k3_r5_remoteproc
   modprobe -r ti_k3_dsp_remoteproc
MCU only app: After modules have been removed, press 1 and enter

From my top level observations:

1. The errors happen because the communication between A72(ATF/OPTEE) & MCU R5F(DM/LPM) is broken.
   Traces: ERROR: Timeout waiting for thread SP_RESPONSE to fill
   ERROR: Thread SP_RESPONSE verification failed (-60)
   ERROR: Message receive failed (-60)
   ERROR: Failed to get response (-60)
2. If i see the OSPI flashing map:
   

   lpm_example_freertos_mcu1_0_release.appimage> -d 3 -o 100000
   
   

   multicore_MCU2_0_MCU2_1_stage1.appimage> -d 3 -o 1FC0000
   
   But then we flash ATF/OPTEE later in between:
   

   atf_optee.appimage> -d 3 -o 1C0000

3. The above could be an issue. I have already raised a high priority internal ticket so that development team is already looking into this.

   Best Regards,
   Keerthy 

Hi Keerthy,

the above mapping with

lpm_example_freertos_mcu1_0_release.appimage> -d 3 -o 100000

atf_optee.appimage> -d 3 -o 1C0000

has been in the PDK UG since e.g. SDK8.6

Looking at the size of files

319176 Oct 18 07:57 lpm_example_freertos_mcu1_0_release.appimage (approx 312KB)

0xC0000 equates to 768KB space, so it seems there is enough space in the flash for the lpm binary, before the atf_optee.appimage starts at 0x1C0000. So there should not be an overlap.

Regards,

--Gunter

Gunter,

Yes as already discussed on that call. I agree on that. Our internal expert is looking into this. We will keep you posted.

- Keerthy

Hi Gunter/Louie,

Adding the fix here.

Please find the attachment for ……

• Lpm_patch_fix.patch  à PDK patch fix for mcu-only mode. Need to be applied on top of PDK 10.00 and lpm_example needs to be rebuild
  
  lpm_patch_fix.patch
• lpm_example_freertos_mcu1_0_release.appimage à  MCU-only mode working binary built on top of SDK10.00
  lpm_example_freertos_mcu1_0_release.appimage
• k3-j721e-common-proc-board.patch à Linux dtb patch
  k3-j721e-common-proc-board.patch

Please find the answers to the questions discussed in the previous meeting:

1 . Can boot-app directly load u-boot?

                Yes, the boot app example can load Uboot directly. Currently, it is not supported in PDK
               
2. What is the use of dummy binaries in stage 1 and stage 2 images?

                These images are minimal example binary files built for each core. This binary will boot the core and write to common memory between the remote core and mcu1_0.

                DM/mcu1_0 will poll on this memory address. Once the remote core writes to common memory, DM will print “MCU2_0 Booted”.

Where to find all the binaries needed for LPM demo:

Pre-built binaries:

$SDK_Path/packages/ti/board/utils/uniflash/target/bin/j721e_evm/uart_j721e_evm_flash_programmer_release.tiimage
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/boot/sbl/binary/j721e_evm/cust/bin/sbl_cust_img_mcu1_0_release.tiimage
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/drv/sciclient/soc/V1/tifs.bin
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/boot/sbl/example/boot_app/multicore_images/j721e_evm/multicore_MCU2_0_MCU2_1_stage1.appimage
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/boot/sbl/example/boot_app/multicore_images/j721e_evm/multicore_DSPs_MCU3_0_MCU3_1_stage2.appimage

$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/boot/sbl/example/boot_app/multicore_images/j721e_evm/multicore_MPU1_0_stage3.appimage
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/boot/sbl/tools/BootApp_binaries/linux/j721e_evm/atf_optee.appimage
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/boot/sbl/tools/BootApp_binaries/linux/j721e_evm/tikernelimage_linux.appimage 
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/boot/sbl/tools/BootApp_binaries/linux/j721e_evm/tidtb_linux.appimage
$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/board/src/flash/nor/ospi/nor_spi_patterns.bin

Only the lpm binary needs to be built:

$SDK_Path/pdk_jacinto_10_00_00_27/packages/ti/binary/lpm_example_freertos/bin/j721e_evm/lpm_example_freertos_mcu1_0_release.appimage

Thanks Chandru Dhavamani for the debug!

Regards,
Kerthy

Thank you for finding the root cause and sending the fix.

However, I still need more instructions to test this demo. Specifically, you state that "only the lpm binary needs to be built." However, this is not the case.

It is my understand that the devicetree changes listed in k3-j721e-common-proc-board.patch need to be applied to k3-j721e-common-proc-board.dts in the Linux SDK. From there, I still need instructions on how to build k3-j721e-common-proc-board.dts into a tidtb_linux.appimage file. Gunter and I are unable to locate this information in the documentation.

I found steps in this E2E post that seem to describe the process (but for the vision apps). However, I get errors when trying to run the constructappimageshlos.sh script. Please provide the complete instructions for going from the modified k3-j721e-common-proc-board.dts in the Linux SDK to getting an output tidtb_linux.appimage file in RTOS SDK.

Louie,

My bad. Attaching the working binary:

tidtb_linux.appimage

I believe you will need to recompile it yourself. This is to unblock validating the MCU_Only mode.

- Keerthy

Thanks for your support and providing the binary. I will use this for my initial test.

As requested in my previous message, could you please provide the full instructions for how to recompile the tidtb_linux.appimage? Specifically, I am looking for instructions on how to go from a an updated k3-j721e-common-proc-board.dts file to a .dtb file and finally to the tidtb_linux.appimage file. Gunter and I are unable to locate this information in the documentation.