TDA4VM: Sysfw.itb Authentication Issue After SDK Migration

HI Kishore,

Are you able to authenticate latest generated sysfw.itb on 8.1 infra ?

Can you compare the certificate of both the sysfw 9.2 vs 8.1?

Can you share the logs where you are able to authenticate other images  in 9.2 infra?

Regards
Diwakar

Hi Diwakar,

Thanks for your response.

Regarding the points:

1. Can you authenticate the latest sysfw.itb on 8.1 infra?

• Yes, we have tested this, and the latest image authentication is successful on 8.1 infra.

2. Can you compare the certificates of sysfw 9.2 vs. 8.1?

• We used the pdk_jacinto_09_02_00_30/packages/ti/build/makerules/x509CertificateGen.sh script with the following OIDs:

[v3_ca]
basicConstraints = CA:true
1.3.6.1.4.1.294.1.3=ASN1:SEQUENCE:swrv
1.3.6.1.4.1.294.1.34=ASN1:SEQUENCE:sysfw_image_integrity
1.3.6.1.4.1.294.1.35=ASN1:SEQUENCE:sysfw_image_load
1.3.6.1.4.1.294.1.4=ASN1:SEQUENCE:encryption
1.3.6.1.4.1.294.1.1=ASN1:SEQUENCE:boot_seq
1.3.6.1.4.1.294.1.2=ASN1:SEQUENCE:image_integrity

Note: The same certificate (x509CertificateGen.sh) from the 8.1 SDK was used for generating the sysfw certificate for 9.2.

3. Can you share logs from 9.2 infra where other images are authenticated?

Here are the logs for image authentication:

• tiboot3.bin

  Image to be validated: /datafs/upgrade/flash_images/tiboot3.bin
  Memory Allocation for Image Validation succeeded
  VIRTUAL MMAP ADDRESS: 0x0xffff8bdb0000
  Size of image file: 196361
  MMAP of IMAGE FILE SUCCEEDED
  Base Addr: 0x8bd80000
  FIT Image format check succeeded
  Image Name: tiboot3 Addr: 0x8bd80000 Size: 196361 offset: 0
  tiboot3 image authentication status SUCCESS
  

• tispl.bin

  Image to be validated: /datafs/upgrade/flash_images/tispl.bin
  Memory Allocation for Image Validation succeeded
  VIRTUAL MMAP ADDRESS: 0x0xffffb35a0000
  Size of image file: 998119
  MMAP of IMAGE FILE SUCCEEDED
  Base Addr: 0xb34ac000
  FIT Image format check succeeded
  Image Name: atf Addr: 0xb34ac158 Size: 54588 offset: 344
  atf image authentication status SUCCESS
  Image Name: tee Addr: 0xb34b9728 Size: 433947 offset: 55080
  tee image authentication status SUCCESS
  Image Name: dm Addr: 0xb35236e4 Size: 254044 offset: 489188
  dm image authentication status SUCCESS
  Image Name: spl Addr: 0xb35617e8 Size: 240748 offset: 743400
  spl image authentication status SUCCESS
  Image Name: k3-j721e-common-proc-board.dtb Addr: 0xb359c4e8 Size: 12465 offset: 984296
  k3-j721e-common-proc-board.dtb image authentication status SUCCESS
  

Please let me know if you need any further details.

Regards,

Kishore

One more thing I would like to mention, the above successful logs are coming from ti_sci_cmd_proc_auth_boot_image()::drivers/firmware/ti_sci.c, where ti_sci_cmd_proc_auth_boot_image() returns success for tiboot3 and tispl images.

Hi Team,

Gentle reminder.

Hi Kishore,

Diwakar is on vacation, and won't be in office until mid-next week.

Kishore Mali said:

The ti_sci_cmd_proc_auth_boot_image() is returning the error "Mbox send fail -110"

Which component is performing this check? Linux Kernel?

The Mbox send fail is typically an issue if the MCU1_0 firmware is not responding. Are you able to connect to the MCU1_0 core, and check where the core is stuck at?

regards

Suman

Hi Diwakar,

Regarding, 
Which component is performing this check? Linux Kernel?
>> Yes, it is Linux Kernel.

Apart from sysfw.itb generated for sdk9.2 as mentioned above, all images are getting authenticated on sdk9.2.

Also, the same latest sdk9.2 sysfw.itb image is able to authenticate on sdk8 infra.

Regards,

Kishore

Hi Kishore,

Kishore Mali said:

Apart from sysfw.itb generated for sdk9.2 as mentioned above, all images are getting authenticated on sdk9.2.

The authentication process in general is agnostic of the data being authenticated.

Is the mbox failure triggered only during the authentication of sysfw.itb or is proceeding even before? Can you try reauthenticating repeatedly some of the previously authenticated images?

Kishore Mali said:

Also, the same latest sdk9.2 sysfw.itb image is able to authenticate on sdk8 infra.

Well, atleast this suggests there are no issues with the sysfw.itb file.

Nevertheless, I would suggest you to compare the contents of this 9.2 sysfw.itb against the previous 8.1 SDK. You can check on the image size as well as the contents using the U-Boot's dumpimage utility.

regards

Suman

Hi Team,
Thanks for your reply.

We are trying to understand if the issue points to any changes in the SDK 9.2 sysfw.itb, particularly if it demands additional OIDs or requires a specific order of OIDs to successfully validate.

Here are the details we observed:

• When flashing sysfw 8.1 in the SDK 9.2 software, all images were validated successfully.
• However, when flashing sysfw 9.2 in the SDK 9.2 software, all images passed initially, but when sysfw 9.2 itself was validated, we encountered an MBOX error. This led to subsequent failures for other images that had previously passed validation.
• Additionally, we tried removing the encryption OID in the SDK 9.2 software as sdk8.1 also not having it, but this caused the board to fail to boot.

 The difference between Point 2 and 3 is the order/sequence in which the firmware images are validated. 
 As explicitly visible from the table once the sysfw is validated, the system enters into mbox error mode after which it does not recover.
 In this sequence, when we go for soft reboot then during booting also couple of mbox errors are observed.

 It is clearly visible that when board is loaded with sdk9.2 sysfw and followed by validation of sysfw (sysfw_8.1 and sysfw_9.2), the system enters into mbox error mode. 

With these observations in mind, we have a few questions:

1. Does the SDK 9.2 sysfw.itb require additional OIDs or a specific order of OIDs to be validated correctly?
2. Is there a mandatory list of OIDs required for the proper functioning and validation in SDK 9.2?

Regarding,
Nevertheless, I would suggest you to compare the contents of this 9.2 sysfw.itb against the previous 8.1 SDK. You can check on the image size as well as the contents using the U-Boot's dumpimage utility.
>>

sdk9.2 :

1.3.6.1.4.1.294.1.3=ASN1:SEQUENCE:swrv
1.3.6.1.4.1.294.1.34=ASN1:SEQUENCE:sysfw_image_integrity
1.3.6.1.4.1.294.1.35=ASN1:SEQUENCE:sysfw_image_load
1.3.6.1.4.1.294.1.4=ASN1:SEQUENCE:encryption
1.3.6.1.4.1.294.1.1=ASN1:SEQUENCE:boot_seq
1.3.6.1.4.1.294.1.2=ASN1:SEQUENCE:image_integrity

sdk8.1:
1.3.6.1.4.1.294.1.3=ASN1:SEQUENCE:swrv
1.3.6.1.4.1.294.1.34=ASN1:SEQUENCE:sysfw_image_integrity
1.3.6.1.4.1.294.1.35=ASN1:SEQUENCE:sysfw_image_load
1.3.6.1.4.1.294.1.4=ASN1:SEQUENCE:encryption
1.3.6.1.4.1.294.1.1=ASN1:SEQUENCE:boot_seq
1.3.6.1.4.1.294.1.2=ASN1:SEQUENCE:image_integrity

Content wise:



Thanks in advance.

Regards,
Kishore

Hi Kishore,

Kishore Mali said:

We are trying to understand if the issue points to any changes in the SDK 9.2 sysfw.itb, particularly if it demands additional OIDs or requires a specific order of OIDs to successfully validate.

The required OIDs are all fixed for a given binary, so this is not an issue, and no additional OIDs are required for 9.2 sysfw.itb. 

Kishore Mali said:

Here are the details we observed:

Thanks for the detailed notes.

Kishore Mali said:

However, when flashing sysfw 9.2 in the SDK 9.2 software, all images passed initially, but when sysfw 9.2 itself was validated, we encountered an MBOX error. This led to subsequent failures for other images that had previously passed validation.

The mbox error is a result of not getting a response properly. This is usually a result of the MCU R5F core entering an exception. Once this error is encountered, the TI-SCI protocol is non-functional, and all subsequent TI-SCI messages will be non-functional (not limiting to just the authentication of further images you are testing).

Given that you are performing an authentication step through Linux, the TI-SCI request is routed through the MCU R5F firmware. What firmware do you use on MCU R5F, it appears that there is an issue with your SDK 9.2 MCU R5F firmware.

Kishore Mali said:

Additionally, we tried removing the encryption OID in the SDK 9.2 software as sdk8.1 also not having it, but this caused the board to fail to boot.

TIFS firmware is the primary Security Root-of-Trust firmware, and is an encrypted firmware. So, you cannot remove this OID, that is needed to decrypt the firmware.

Kishore Mali said:

 The difference between Point 2 and 3 is the order/sequence in which the firmware images are validated.

Your table does show the 8.1 sysfw.itb file also failing in Point 2. 

Kishore Mali said:

Does the SDK 9.2 sysfw.itb require additional OIDs or a specific order of OIDs to be validated correctly?

No

Kishore Mali said:

Is there a mandatory list of OIDs required for the proper functioning and validation in SDK 9.2?

This is not an issue with OIDs.

Kishore Mali said:

Content wise:

Thanks for this dump. Overall, nothing stands out. The 9.2 sysfw.itb files/contents are slightly larger, and this comes due to the increased size differences in certificates coming from OpenSSL 3.x on the newer Ubuntu 22.04 machines that SDK 9.x is released with.

regards

Suman

Hi Team,

We have enabled the enet lwIP example on SDK 9.2 for the MCU10 R5F firmware, and it works as expected. Every 10 seconds, we print the CPU load, as shown below.

Still the issue persists, where the sysfw.bin validation from SDK 9.2 is failing with an Mbox error when sdk9.2 based sysfw is flashed.

MCU10 Logs:

Observations:

• All other images are successfully validated as mentioned in above table also.

• The issue only occurs when flashing and validating the sysfw.bin based on SDK 9.2.

Given our observations, could you please guide us on how to resolve this issue?

Thank you for your assistance.

Best regards,
Kishore

Hi Kishore,

Kishore Mali said:

Given our observations, could you please guide us on how to resolve this issue?

Are you able to connect and debug your MCU R5F core? Can you check if your MCU R5F core is stuck in an exception?

Also, I would also recommend to enable the traces and capture the TIFS debug log from Wkup UART (if your board has support for this)?

regards

Suman

Hi Suman,

Regarding,
Are you able to connect and debug your MCU R5F core? 

>> Board has no support for this.

Also, we are able to see the regular CPU load print on MCU10 UART. If exception is there it should stuck which is not the case.

Regarding,

Also, I would also recommend to enable the traces and capture the TIFS debug log from Wkup UART (if your board has support for this)?

>> Board has no support for this.

Regards,

Kishore

Hi Kishore,

Kishore Mali said:

Are you able to connect and debug your MCU R5F core?

I am very surprised that you have no JTAG debug support, this makes it extremely hard for debug during development.

Kishore Mali said:

Also, we are able to see the regular CPU load print on MCU10 UART. If exception is there it should stuck which is not the case.

The mbox failure is only caused when the MCU R5F is not responding to the requests. Are you sure the interrupts are not disabled somehow on the MCU R5F?

Kishore Mali said:

Also, I would also recommend to enable the traces and capture the TIFS debug log from Wkup UART (if your board has support for this)?

>> Board has no support for this.

This is understandable, not every customer pins out Wkup UART on their boards. You can rely on some memory-based debug, but having no access to JTAG doesn't help your debug.

regards

Suman