AM2432: After programming OTP, TISCI_MSG_FLAG_ACK flag is not received

Hello,

Please share the Keywriter boot logs to analyze the cause of the issue. In the case of failure, there would be an error code in the logs which is useful for analyzing the failure.

Regards,

Prashant

Hello Prashant, 

We don't have the access to UART1 log. Is there any other way for us to debug this issue?  Is it possible to direct the output from UART1 to UART0? 

Hong

Prashant Shivhare said:

In the case of failure, there would be an error code in the logs which is useful for analyzing the failure.

These logs are from the SBL which comes on UART0 only.

The logs would look like the following

Starting Keywriting
Enabled VPP
keys Certificate found: 0x70042b00
Keywriter Debug Response:0x0
Success Programming Keys

Hello Prashant,

There is no error code reported from UART0 for this case. 

The DebugP_assertNoLog((status == SystemP_SUCCESS) && ((respParam.flags & TISCI_MSG_FLAG_ACK) == TISCI_MSG_FLAG_ACK)); failed is after the 

Success Programming Keys 

is printed out. 

Hong 

Hello,

Please apply the following patch to print the SYSFW logs on the UART0 console

diff --git a/source/security/sbl_keywriter/am243x-evm/r5fss0-0_nortos/keywriter_utils.c b/source/security/sbl_keywriter/am243x-evm/r5fss0-0_nortos/keywriter_utils.c
index c48be90f..cc875f39 100644
--- a/source/security/sbl_keywriter/am243x-evm/r5fss0-0_nortos/keywriter_utils.c
+++ b/source/security/sbl_keywriter/am243x-evm/r5fss0-0_nortos/keywriter_utils.c
@@ -226,6 +226,21 @@ void keywriter_setBoardCfg(void)
 	}
 }
 
+void dump_sysfw_logs()
+{
+    uint8_t* ptr = (uint8_t*)0x44043000;
+
+    DebugP_log("\r\n<<SYSFW_LOGS\r\n");
+
+    for(int32_t i = 0; i < 0x0FE0; i++)
+    {
+        DebugP_log("%c", *ptr);
+        ptr++;
+    }
+
+    DebugP_log("\r\nSYSFW_LOGS\r\n");
+}
+
 void keywriter_processKeyConfig(void)
 {
     int32_t status = SystemP_SUCCESS;
@@ -254,6 +269,8 @@ void keywriter_processKeyConfig(void)
 
 	DebugP_log("Keywriter Debug Response:0x%x \r\n", (uint32_t) response.debug_response);
 
+    dump_sysfw_logs();
+
 	if (( status == SystemP_SUCCESS ) && ( response.debug_response == 0x0U ))
 	{
 		DebugP_log("Success Programming Keys \r\n");

Regards,

Prashant

Thanks a lot, Prashant.

This will definitely help us in debugging. 

Hong 

Hello Prashant, 

I am still trying to reproduce on this issue. 

At the same time, could you please help to understand what might be issues here when we have the Success Programming Keys in the logs, 

But failed in this step next. 
DebugP_assertNoLog((status == SystemP_SUCCESS) && ((respParam.flags & TISCI_MSG_FLAG_ACK) == TISCI_MSG_FLAG_ACK));  


When the TISCI_MSG_FLAG_ACK is not received in flags, what might be the effect in terms of the keywriter? Does it mean the previous programming failed? This is kind of conflict with the "Success Programming Keys".   

Thanks, 
Hong 

Hello,

Ideally, the flags should have TISCI_MSG_FLAG_ACK if the keywriter reports "Success Programming Keys" in the logs.

Hong Zhang said:

At the same time, could you please help to understand what might be issues here when we have the Success Programming Keys in the logs

It is not feasible to assess the conditions under which this situation (if possible at all) could occur. If you could reproduce the issue & provide the SYSFW logs then we could analyse the failure.

Apart from this, may I know the followings?

• SDK version & Keywriter version you are using?
• The reason for checking out the execution state of the keywriter? Normally, if I see the keywriter reporting "Success Programming Keys", I would just assume everything went correctly.

Regards,

Prashant

Hello Prashant, 

Thanks a lot for the reply. 

I was trying to add SYSFW logs to debug this issue and seeing some other issues.

I saw this failure when programming the ext otp data with SYSFW log prints added, and could not dump the ext OTP data as well, please check your email regarding details about the ext OTP data dump issue. 

Failed programming keys, status = 0, response.debug_response = 0x20000000

For the other two questions, 

• SDK version & Keywriter version you are using?
  
  Keywriter is based on SDK 9.1 release.  keywriter binary is compiled with SDK 9.1. 
  
  
• The reason for checking out the execution state of the keywriter? Normally, if I see the keywriter reporting "Success Programming Keys", I would just assume everything went correctly.
  
  We found this issue indirectly, because we added a print after the assert of TISCI_MSG_FLAG_ACK line to print some other debug information, and I found that this line errored out and the debug information was not printed out. And we also added disabling VPP after this line, which was not executed either. 
  

Thanks a lot, 
Hong 

Hi Hong,

Hong Zhang said:

Failed programming keys, status = 0, response.debug_response = 0x20000000

Please share the SYSFW logs.

Hong Zhang said:

and could not dump the ext OTP data as well

The EXT OTP MMRs cannot be read directly. The SYSFW provides the APIs for the EXT OTP:

https://software-dl.ti.com/tisci/esd/latest/2_tisci_msgs/security/extended_otp.html

For keywriter, you do not need to readback anything. The SYSFW logs would have anything necessary to verify the OTP programming.

Regards,

Prashant

Hello Prashant, 

Thanks a lot for the reply.  

I was able to add SYSFW log print on the keywriter project with a smaller EXT OTP size of one byte. 

It might have some other issues for the response.debug_response = 0x20000000 for a larger size of EXT OTP size. Given this is not our use case, just a test case, I would like to re-focus on the issue for the TISCI_MSG_FLAG_ACK is not being received as I pointed out earlier when running the keywriter to programming OTP keys. 

We will try to reproduce on our side with the SYSFW log printed and will send this issue via email as well. 

Thanks, 
Hong 

Thanks for the update, Hong.

I will wait for your feedback on the reproducibility of the original issue.

Hi Prashant,

Thanks! I sent you an email w/ the latest status/ask.  The bottom line is that the Ack is not received and we need to understand why.  Then we need a fix/update/change so the ack is always & reliably received.

Thanks, Merril

Hello,

As I mentioned, it is not feasible to assess the conditions under which the issue could occur. It might just have been wrong observations. This is the first time such an issue is reported.

Did the customer verify that the print statement did not execute because the execution was stuck in the DebugP_assertNoLog statement or is it just a speculation?

If the customer got the "Success Programming Keys", have they verified that the OTP was really programmed correctly according to their certificate?

Regards,

Prashant

Hello Prashant, 

>If the customer got the "Success Programming Keys", have they verified that the OTP was really programmed correctly according to their certificate?
Verified today that the OTP was programmed successfully, and the HASH are all correct for both SMPK and BPMK keys. 

>Did the customer verify that the print statement did not execute because the execution was stuck in the DebugP_assertNoLog statement or is it just a speculation?
There was a DebugP_log() that was supposed to be executed after the DebugP_assertNoLog line, but it didn't get printed out. And there was a GPIO toggling after the DebugP_log(), and didn't get executed as well (This is the observation from scope). There is only this assert between the "Success Programming Keys" and the DebugP_log() we are supposed to receive. 
DebugP_assertNoLog((status == SystemP_SUCCESS) && ((respParam.flags & TISCI_MSG_FLAG_ACK) == TISCI_MSG_FLAG_ACK)); 

We have tested on another two boards with SYSLOG enabled, but we are not able to reproduce the same issue.  We are trying to reproduce on the same board, and so far have got no luck yet. 

Thanks, 
Hong 

Hello Prashant, 

I was just able to reproduce the same issue on the same board using the same binary of the keywriter. But after adding the SYSLOG print, the issue didn't occur. 

Thanks, 
Hong 

Hi Hong,

Can you please connect to the R5FSS0-0 core using the debugger & check where the core is stuck by loading the keywriter symbols?

Otherwise, maybe you can print the flags like so:

DebugP_log("flags: %d\r\n", response.flags);

Please share the keywriter logs (UART0) as well in any case.

Regards,

Prashant

Hello Prashant, 

I only got one luck to reproduce the issue, and was not able to reproduce afterwards. I don't have the JTAG debugger on that setup. 


UART0 log for successful case:

keys Certificate found: 0x7005xxxx

Keywriter Debug Response:0x0

Success Programming Keys

Set GPIO0_xx low to disable VPP

For the failed case:

keys Certificate found: 0x7005xxxx

Keywriter Debug Response:0x0

Success Programming Keys

=> Last line is missing here.  When we measured the VPP last time, and we could see VPP was not disabled.

Thanks, 
Hong

Hong Zhang said:

I only got one luck to reproduce the issue, and was not able to reproduce afterwards. I don't have the JTAG debugger on that setup.

If there is no additional debug information then there is not much we can do here to root cause the issue.

I just went through the Keywriter v09_01_00 & see that the UART0 is configured in the Interrupt mode which may be causing the issue due to the UART errata. Please configure the UART0 in polling mode & see if the issue is reproduced.

If it still does, please add debug prints like for flags value & provide boot logs if the issue is reproduced:

Otherwise, there is Keywriter v10.00.08 available so you may give that a try as well.

Regards, Prashant

Hello Prashant, 

I am not aware there is a UART errata that requires UART to be operate at polling mode, could you please help to point that out?

I am only aware of this UART bug, but this is limited to ROM code. 


I will give it a try to change to polling mode, but the issue is that with the low reproduce rate, even if the test passes, I don't know whether this is really the fix. 

Thanks, 
Hong 

Hello Prashant, 

For the UART1 interface that is used by SYSFW to output log, given we don't have access to UART1, can we not configure UART1 in the keywriter?   Will remove UART1 from the example.cfg from the keywriter cause the SYSFW to fail?


Thanks a lot, 
Hong 

Hello,

Hong Zhang said:

I am only aware of this UART bug, but this is limited to ROM code.

The Errata i2371 is specific to ROM only. But this errata originates from the Errata i2310 which affects anything using UART in interrupt mode.

https://www.ti.com/document-viewer/lit/html/SPRZ457I#GUID-C3F24986-5AED-422A-ABE2-9952EFCE1E58/TITLE-SPRZ455I2038

Hong Zhang said:

Will remove UART1 from the example.cfg from the keywriter cause the SYSFW to fail?

You may remove the UART1 from Sysconfig.

Regards,

Prashant

Hello Prashant, 

Thanks for the information. 

We removed the UART0 ISR enable and removed UART1 from the example.syscfg for the keywriter, and working to run some loops of testing with this update. The sanity check looks good so far. 

Thanks, 
Hong 

Hi Hong,

Hong Zhang said:

We removed the UART0 ISR enable and removed UART1 from the example.syscfg for the keywriter, and working to run some loops of testing with this update. The sanity check looks good so far.

This sounds good.

Please note the Keywriter v10.00.08 by default configures the UART in polling mode. It would be good to migrate to latest version for more probability of stability.

Regards,

Prashant

Hello Prashant, 

From latest test, I noticed with USB boot, even after disabling ISR for UART0, keywriter could hang at different locations. We created another thread just for this issue. I think the issue might be related. 

BTW, for keywriter v10.00.08, I see both combined mode and legacy mode are supported. Can we use legacy mode in our use case? 

Thanks, 
Hong 

Hong Zhang said:

BTW, for keywriter v10.00.08, I see both combined mode and legacy mode are supported. Can we use legacy mode in our use case?

The legacy mode in the keywriter v10.00.08 is supported only for the customers who were using JTAG to run the keywriter in earlier releases.

It is highly recommended to use the combined boot flow.

Hi Hong,

Please let us know if the issue is resolved and no further support is needed.

Thanks!