TDA4VM: The secure boot stress test may encounter a freeze issue.

Hi Ni 

Have you tested this issue on EVM or your custom board 

If it is custom board have you tried same experiment with other boot mode to make sure it is not OSPI issue ?

Regards
Diwakar

Hi Diwakar,

1. We are conducting tests on the customer's board.

2. We will perform tests later using the SD card boot mode, but eventually, we must use the OSPI FLASH mode. Therefore, we need assistance from TI to address the issue in OSPI mode. Currently, we have observed several instances of freezing occurring during the invocation of the Sciclient_procBootAuthAndStart interface.

3. Please provide the stress test report for secure boot on your EVM board.

This issue is relatively urgent, please assist as soon as possible.

thanks.

Hi Ni 

Ni Hua said:

Switch the boot mode to ospi flash boot; conduct multiple power cycles. It is observed that sporadically, the SBL freezes with no LOG output, and after a period of freeze, it automatically resets.

You were saying you are not able to see any logs i am assuming even SBL is not getting loaded is it the case ?

If not provide more details on which stage it is getting stuck and help share the logs as well .

Regards
Diwakar

Hi Diwakar,

    The SBL freeze occurs when retrieving the TIFS version, specifically during the invocation of the SBL_SciClientInit->Sciclient_service interface. The lack of LOG printing by the SBL is due to certain LOGs not being enabled at that time.

thanks.

HI Ni 

Ni Hua said:

    The SBL freeze occurs when retrieving the TIFS version, specifically during the invocation of the SBL_SciClientInit->Sciclient_service interface. The lack of LOG printing by the SBL is due to certain LOGs not being enabled at that time.

You mean 

it is getting stuck here ? Am i right ?

Can you share the R5 and TIFS l;ogs as well ?

follow this FAQ to enable the sysfw traces.

e2e.ti.com/.../faq-system-firmware-calls-have-failed-or-crashed-what-do-i-do

Also how much is the failure rate ?

Regards
Diwakar

Hi Diwakar,

1. Yes, the freezing occurs at two locations, Sciclient_procBootAuthAndStart and the Sciclient_service position shown in your image. However, the final invocation should be the Sciclient_service interface.

2. The logs from R5 indicate a freeze at the signature verification interface Sciclient_procBootAuthAndStart.

3、The freezing occurs with a frequency of 1 in 200 attempts.

4、Obtaining logs from TIFS is currently challenging. We will provide them later. Can you replicate the issue in your testing environment?

The issue is easy to reproduce. Kindly expedite your efforts to reproduce the problem and provide a performance testing report as soon as possible.

This issue is urgent. Can you help remotely to address it?

thanks.

Hi Ni 

Ni Hua said:

4、Obtaining logs from TIFS is currently challenging. We will provide them later. Can you replicate the issue in your testing environment?

I am doing the setup at my end to see if  we can replicate this issue .Allow some time for the same .

One quick question on how many boards you are seeing this issue ?

Regards
Diwakar

Hi Diwakar,

    All tested boards exhibit the same issue, approximately three boards in total.

    Could you provide the patch for tracing TIFS? I couldn't fully understand how to proceed based on the documentation.

thanks.

Hi Diwakar,

    Have you been able to reproduce the issue of the secure boot signature verification getting stuck on your end? This issue is very urgent, and we would appreciate it if you could promptly inform us of the reproduction status.

thanks.

Hi Diwakar,

    Has the issue been reproduced? Additionally, I would like to inform you that during our testing, we observed the issue occurring within the time intervals of 1 second and 6 seconds after powering off and then powering on. The problem was replicated in both the 1-second and 6-second scenarios.

thanks.

Hi Ni 

Can you share me the images which you are using ?

Regards
Diwakar

Hi Diwakar,

    I would like to understand first, have you reproduced the issue on the demo board? Could you please provide me with the performance testing report?

thanks.

HI Ni 

The latest mcusw userguide document doesnt have the boot loader functionality with HLOS.

software-dl.ti.com/.../demo_boot_app_mcu_rtos_top.html

So what step you followed to  build  signed images that  i would like to know  form 8.5 it was planned to remove functionality of booting HLOS  and it was in the transition phase at that time.

So in order to reproduce that issue we need to have a images as you already tried.

Regards
Diwakar

Hi Diwakar,

 1、Our method of compiling the image is as follows:

    cd ti-processor-sdk-rtos-j721e-evm-08_06_00_12/vision_apps

    make sbl_mcusw_bootimage_ospi_hs

2、The path to obtain the signed image is as follows:

3、The paths to obtain other images have been explained in the initial description.

thanks.

HI Ni 

Ni Hua said:

    make sbl_mcusw_bootimage_ospi_hs

Earlier you said you are using SBL cust image

Ni Hua said:

• Target compilation is sbl_cust_img_hs.

Please share the build images which you are using for your test.

Regards
Diwakar

Hi Diwakar,

make sbl_mcusw_bootimage_ospi_hs is used to compile the can boot app, and sbl_cust_img_hs is used to compile the SBL. Do you not have any images available for testing on your end? How did you conduct your performance testing?

thanks.

Hi Diwakar,

Let me supplement my steps:

1. First, compile the SBL.
2. Compile the can boot app.
3. Compile lateapp1/lateapp2.
4. Obtain atf_optee.appimage, tikernelimage_linux.appimage, and tidtb_linux.appimage.
5. Obtain tifs.bin.
6. Obtain nor_spi_patterns.bin.
7. Burn the images to the flash, following the sequence SBL->tifs.bin->can boot app->lateapp1->lateapp2->atf_optee.appimage->tidtb_linux.appimage->tikernelimage_linux.appimage.

You should have these images. Can you test with your own images first?

thanks.

HI Ni 

Ni Hua said:

You should have these images. Can you test with your own images first?

I am not able to arrange SR 1.1 board so making the test setup on SR2.0 but we have some known issue with the SDK8.6 on SR2.0

so now i will make a setup on 9.1 SDK and will do the stress test.

Regards
Diwakar

Hi Diwakar,

   Thank you very much for your cooperation in resolving this issue.

    Could you perform stress testing on SDK 8.6? We are using SDK 08_06_00_12.

thanks.

Hi Ni 

Sorry my bad  by mistake i kept 8.5 it was 8.6.

On SR2.0 i can test with 9.1 SDK .

Or i need to look for SR1.1 board which will take time .

Regards
Diwakar

Hi Diwakar,

    Alright, you can first replicate the issue on SR2.0 based on SDK 9.1. Once you have the SR1.1 board, you can then replicate the issue based on SDK 8.6.

thanks.

Hi Diwakar,

I have conducted further tests and observed the following phenomena:

1. The issue does not reoccur when trace_dst_enables is enabled.

   

2. Our customer's board has WKUP_UART0_TXD connected to the PMIC's INT pin (low level). I noticed that when trace_dst_enables is disabled, the probability of the issue decreases after disconnecting from the PMIC. However, it is still present. I'm not sure if there is any correlation between these factors.

   

Regarding the two aspects mentioned above, could you please help analyze the root cause of the issue?

thanks.

Hi Ni 

Ni Hua said:

The issue does not reoccur when trace_dst_enables is enabled.

It will just enable the TIFS traces maybe some timing issue ? I am still struggling with setup will update you once done 

Regards
Diwakar

Hi Ni 

Can you try with this fix

sir.ext.ti.com/.../EXT_EP-11567

Regards
Diwakar

Hi Diwakar,

    Tests have been conducted based on your FIX suggestions, but the issue of the system freezing during secure startup still persists. The freezing occurs at the same location as before, with a reproduction frequency of one percent. Kindly assist in reproducing and resolving this issue as soon as possible.

thanks.

Hi Ni 

Is it necessary to boot upto Linux if we keep lateapp1 and lateapp2 will the issue comes ?

Regards
Diwakar

Hi Diwakar,

Our current reproduction method is as follows:

1、After each boot to the Linux kernel, when the kernel prints the string 'Booting Linux on,' we power off. After a 1-second interval, we power on again and repeat the test.

2、Lateapp1 and lateapp2 also need to be verified and booted on their respective cores.

thanks.

HI NI 

I understand the boot flow you are using question is it it failing during the authentication/verification of the lateapp1 or lateapp3 itself or it fails only if you add linux ?

Can you test it will removing the linux image ?

I am facing some issue with the HLOS that is the reason i am asking this.

Regards
Diwakar

Hi Diwakar,

1、During our testing, we run the entire process, including lateapp1, lateapp2, and Linux. The freeze does not necessarily occur during the signature verification of lateapp1 and lateapp2; we have also observed freezes during the verification of the Linux kernel.

2、Without removing the Linux kernel for testing, are you suggesting to disable the Linux kernel's startup and then proceed with the testing?

thanks.

Hi Ni 

Ni Hua said:

2、Without removing the Linux kernel for testing, are you suggesting to disable the Linux kernel's startup and then proceed with the testing?

Yes without booting the linux can we try reproducing the issue so that i am also will be on the same page.

Regards
Diwakar

Hi Diwakar,

    How to avoid starting the Linux kernel? Please provide your method to ensure consistency with our approach.

thanks.

Hi Ni 

Can you corrupt your linux image so that authentication fail for that and further boot will not happen or use unsigned image of kernel.

Regards
Diwakar

Hi Diwakar,

    Alright, I will conduct the test according to the method you provided. Meanwhile, I would like to know if you have been able to replicate the issue of my system getting stuck during secure boot on your end?

thanks.

Hi Diwakar,

Following the method you provided, I conducted verification without signature verification and starting the Linux kernel, and have successfully replicated the system hang issue. The log is as follows:

Please help analyze and resolve this issue. I have the following questions that require your response:

1. Have you encountered and replicated this issue? If so, please provide a solution. If not, please provide the stress testing report.
2. Why does the problem not occur when the trace configuration shown in the attached image is added? Please assist in analyzing the reason.

Please help resolve this issue as soon as possible.

thanks.

Hi NI 

Ni Hua said:

• Have you encountered and replicated this issue? If so, please provide a solution. If not, please provide the stress testing report.

I tested with PDK boot app in 9.1 as we dont have HLOS/linux boot support in can boot app .

So the test Environment is:

• Silicon SR2.0
• PDK 9.1
• SBL cust image 
• PDK boot app 
• late app1 
• late app 2 

I did a stress test for 231 times and no issue is observed.

Regards
Diwakar

Hi Diwakar,

Thank you for your testing. We have the following questions that require your response:

1、Why does adding trace configuration on the customer board prevent the system from getting stuck? Please assist in analyzing possible reasons for this.

2、Can you perform the tests on SR1.1 using SDK8.6 following the procedure I mentioned?

3、Please provide your images or detailed information on your image generation method to ensure consistency with our process.

4、May I ask why you are not using SDK8.6 for testing? Previously, it was mentioned that it doesn't support the CAN boot app. However, upon reviewing your replication with SDK9.1, I observed the use of the PDK boot app instead of the CAN boot app. In this case, why not use images such as the PDK boot app from SDK8.6 for testing?

Kindly assist in addressing each of the above questions.

thanks.

Hi Diwakar,

1、 We resolved the intermittent signature verification failure issue by enabling .trace_dst_enables and .trace_src_enables in DMSC. However, we encountered the following error reported during Linux kernel startup, with a reproducibility rate of 50%.

2、We have identified the 'ti_j721e_cpsw_virt_mac' service, and when DMSC log is enabled, there is a 50% chance of encountering a null pointer issue. Based on this observation, do you have any suggestions?

thanks.

HI Ni 

The title of the thread is deviating from the current issue can you raise a new thread for the above it help us to streamline the thread.

Also please attache full logs in the new thread.

Regards
Diwakar

Hi Diwakar,

    Okay, I will create a separate thread to track this issue.

thanks.

Hi Diwakar,

    Could you provide your test image or instructions on how to obtain the image, so that we can perform the same tests?

    Also, could you answer the following questions one by one?

thanks.

Hi Diwakar,

   Could you provide the method to obtain the tifs.bin file signed with the TI dummy key in SDK8.6?

thanks.

Hi Ni 

Ni Hua said:

   Could you provide the method to obtain the tifs.bin file signed with the TI dummy key in SDK8.6?

TIFS is signed with the TI MPK key not with the TI dummy key.

Signed TIFS comes as a prebuild ,as a part of SDK for the HS-SE devices.

Regards
Diwakar

Hi Diwakar,

    How can we use the TI Dummy key or customer key to sign an existing TIFS for our use?

thanks.

HI Ni 

The steps which i followed to create the images are mentioned in this doc 

https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/09_00_00_02/exports/docs/pdk_jacinto_09_00_00_45/docs/userguide/jacinto/boot/boot_app.html#boot-app

Rebuild images 

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/sbl_5F00_cust_5F00_img_5F00_mcu1_5F00_0_5F00_release.tiimagehttps://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/sbl_5F00_boot_5F00_app_5F00_ospi_5F00_linux_5F00_j721e_5F00_evm_5F00_mcu1_5F00_0_5F00_freertos_5F00_TestApp_5F00_release.appimage.signed

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/multicore_5F00_MCU2_5F00_0_5F00_MCU2_5F00_1_5F00_stage1.appimage.signed     https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/multicore_5F00_DSPs_5F00_MCU3_5F00_0_5F00_MCU3_5F00_1_5F00_stage2.appimage.signed

Regards
Diwakar

Hi Ni 

TIFS can not be signed at the customer end it comes from the TI.

TI  provide signed images for the system firmware.

What is the intention here for signing the TIFS ?

Regards
Diwakar

Hi Diwakar,

    Our chip model is TDA4VM88T5BALFQ1, and we are using the TI Dummy Key for programming. Which TIFS should we use?

thanks.

Hi Diwakar,

Thank you for your response. There are still the following questions that we need your assistance with:

1. For the TDA4VM88T5BALFQ1 chip model, under the usage of SDK Version: 08_06_00_12, which version of tifs.bin should be used?
2. Currently, we are using the keywriter tool to program the customer's smpk.pem into the OTP efuse area corresponding to TDA4VM88T5BALFQ1. In this scenario, how should we choose and obtain the tifs.bin? Please provide the acquisition path.
3. What is the specific model of the chip you used in testing the security boot hang issue? We need to choose a chip model consistent with yours for testing.

Hi Diwakar,

    Could you please respond to the following questions as soon as possible? Thank you.

Hi Ni Hua,

Due to holiday on 15th Jan, please expect delay in reply.

Regards,

Brijesh

Hi Hua,

pls see below for TIFS.

Generate certificate if you are using SR1.1 silicon

a0224068local@a0224068local-HP-EliteBook-830-G7-Notebook-PC:/opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/tools$ ./firmwareHeaderGen.sh j721e_sr1_1-hs

/opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/tools

Building the bin2c generation c tool

/opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/tools

Generating outer certificate for  /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/soc/sysfw/binaries/ti-fs-firmware-j721e_sr1_1-hs-cert.bin

/opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/tools /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/tools

Payload should be DMSC FW Inner certificate

DMSC_O Certificate being generated :

                        X509_CFG = ./x509-temp.cfg

                        KEY = /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/build/makerules/k3_dev_mpk.pem

                        BIN = /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/soc/sysfw/binaries/ti-fs-firmware-j721e_sr1_1-hs-cert.bin

                        CERT TYPE = DMSC_O, 3

                        CORE ID = 0

                        LOADADDR = 0x00040000

                        IMAGE_SIZE = 1684

                        BOOT_OPTIONS = 0

SUCCESS: Image /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/soc/V1/tifs_cert_sr1.1.bin generated. Good to boot

Generating the Header file for  /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/soc/sysfw/binaries/ti-fs-firmware-j721e_sr1_1-hs-enc.bin

Generating the Header file for the soc in the folder

 Converting binary file [/opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/soc/V1/tifs_sr1.1-hs-enc.bin] to C array

.. Done. (265571 bytes)

Done.

                        Note:

                        For SR1.0 silicon, use /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/soc/V1/tifs-hs-enc.bin.

                        For SR1.1 silicon, use /opt/1Twork/repository/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/pdk_jacinto_08_02_00_21/packages/ti/drv/sciclient/soc/V1/tifs_sr1.1-hs-enc.bin.

BR,

Biao