TDA4VM-Q1: TDA4VM HSFS keywriter hangs

Please be aware, that TI resource is out of office for the rest of this week, a delay in response may occur.

Regards,

kb

Hi Anders,

Anders Kagerin said:

It does not seem to be related to any function in the keywriter main.c, since it hangs after a seemingly constant amount of time even if I just do a loop with printf().

How much time are you seeing the hang after?

Is your loop code before loading the KeyWriter TIFS binary?

regards

Suman

The hang occurs roughly within a millisecond after the keywriter starts.

It only has enough time to print around  10-20 characters using UART_printf in a loop. The loop was run before/without running the TISCI init.

Enabling/Disabling the firmware load does not change the behavior.

If the firmware loading is done the other UART for traces show some hex codes, but eventually stops when the freeze occurs.

I should probably also add that I've observed this behavior on multiple boards, and using an older keywriter for the J721E EVM, and modifying the hardware to always enable EFUSE power allowed me to write the keys, and run the full SDK suite of Linux and RTOS.

So my conclusion is that it is not a hardware issue, but likely something misconfigured or missing in my attempt to create an updated Keywriter for this new board 

Hi Anders,

Anders Kagerin said:

It only has enough time to print around  10-20 characters using UART_printf in a loop. The loop was run before/without running the TISCI init.

Hmm, this seems that this is not even related to KeyWriter. 

The TIFS firmware also has a watchdog timeout and needs to be loaded within 180 seconds of the KeyWriter application running. But your issues seems to be related UART and printing traces.

The KeyWriter on TDA4VM SR1.1 is fully functional, and TI is not aware of any issues with it in general on 8.x SDKs. 

Do you have any logs from your MCU UART and WkUp UART at all?

Anders Kagerin said:

If the firmware loading is done the other UART for traces show some hex codes, but eventually stops when the freeze occurs.

Are you able to connect to the MCU1_0 core through a debugger and see where the PC is stuck at?

Anders Kagerin said:

So my conclusion is that it is not a hardware issue, but likely something misconfigured or missing in my attempt to create an updated Keywriter for this new board 

So, this was functional on some older boards, and not functional on newer boards? Has the board library and pinmux configurations all been updated for this new board?

regards

Suman

I'm using the UART_printf as a way to test the behavior of the fault. It is not causing the issue.

I printed the log for the MCU, in my first post. It's just one line before it hangs.

Same board HW, but first and second batch of builds.
The important differences between J721e EVM and our HW are mostly power related. We're using the power supply design named PDN-3F.

Since the PMIC is different, and not used for the EFUSE power supply, I have removed those parts from the J721e keywriter_utils.c.
Since the Keywriter hangs before it reaches these functions, I don't think I have removed anything required for other reasons.

The PMIC watchdog does not trigger, and I've cleared all interrupts with no improvement.

I do observe the AVS is not set in Keywriter, but I assume it uses MCU power, and not AVS_CPU or CORE power supplies?

Hi Anders,

Anders Kagerin said:

I printed the log for the MCU, in my first post. It's just one line before it hangs.

The immediate line after the UART printf is the OTP_SciClientInit() which is all about the KeyWriter TIFS initialization.

Anders Kagerin said:

We're using the power supply design named PDN-3F.

Is there a PDN difference between your first and second batch of builds?

Anders Kagerin said:

Since the PMIC is different, and not used for the EFUSE power supply, I have removed those parts from the J721e keywriter_utils.c.
Since the Keywriter hangs before it reaches these functions, I don't think I have removed anything required for other reasons.

Are you saying this hangs before the OTP_VppEn() function itself? Please ensure that the TIFS binary used for the KeyWriter image is the KeyWriter binary. The SDK has a blank ti-fs-keywriter.bin that matches the size to allow for build/linker verification.

I am not sure what you have commented or removed, but the KeyWriter functionality does require the VPP_EFUSE to be enabled, and that logic is all in the OTP_VppEn() function, which needs to be adapted for your PDN. 

Anders Kagerin said:

I do observe the AVS is not set in Keywriter, but I assume it uses MCU power, and not AVS_CPU or CORE power supplies?

Yeah, this should not matter that much or is the cause for your hang.

regards

Suman

If I comment out the first UART_printf with main.c version info i get:

OTP Keywriter ver: 21.5.2--v2021.05b (Terrific Lla
Key programm

Same PDN for both builds.

It hangs regardless of calling OTP_VppEn or not. It never reaches the key programming call.
I have updated the ti-fs-keywriter.bin according to the add-on instructions (replaced the 0-byte version).
Since I can see the version info from the ti-fs firmware, I believe it is loading properly.

Hi Anders,

Anders Kagerin said:

Same PDN for both builds.

Thanks for confirming. And there are no changes around the Crystal or other Clock related changes? Are there changes around the UART pinmuxing?

Anders Kagerin said:

It hangs regardless of calling OTP_VppEn or not. It never reaches the key programming call.

This is really strange. If you comment out the OTP_SciClientInit(), OTP_VppEn() and Sciclient_otpProcessKeyCfg() calls, do you see all the traces (the firmware just becomes an empty UART trace example?

The OTP_SciClientInit() does perform some Clock initialization and UART reconfiguration if we the hang is somehow caused within the UART driver, but this should not change between your two boards if you are using the same SDK.

regards

Suman

Compared to J721E board we use a higher 25MHz clock, but the software from J721E runs without change because the BOOTMODE pins handle all the clock settings.

UART ports are unchanged.

If I don't run any function, the hang still happens, but it just progresses a little further.
Therefor I don't think any of those calls have an effect on my issue, it's something else that causes a timeout or crash.

It does not seem to be an exception, DWWD (watchdog), PMIC watchdog, PMIC reset, and the PMIC does not report power errors.
There are PMIC interrupts from BIST done and STARTUP, but clearing these has no effect.

Is there any driver init or configuration done somewhere else than main.c?

Hi Anders,

Anders Kagerin said:

Compared to J721E board we use a higher 25MHz clock, but the software from J721E runs without change because the BOOTMODE pins handle all the clock settings.

What boot mode are you using? Have you been able to confirm the 25 MHz setting and the BOOTMODE settings from the CTRLMMR_WKUP_DEVSTAT register? 

Anders Kagerin said:

If I don't run any function, the hang still happens, but it just progresses a little further.

Has the new board gone through all board validation and checked out? I would suggest that you simply put a while loop and see if you can connect to the core, or use a non-UART for loop with minimal periodic prints after certain loop counts.

Ofcourse, the board will restart after the Watchdog fires for TIFS, but your execution is stopping way before that.

Are you able to reuse the same binaries on the previous board and verify that board does not exhibit any such issues? 

Anders Kagerin said:

Is there any driver init or configuration done somewhere else than main.c?

No, there is no other driver init/configuration other than the UART driver code and the associated pinmux.

regards

Suman

On the first boards we have verified OSPI boot mode using 25MHz oscillator. As I mentioned before, we wrote the keys using an older binary copy of the Keywriter in combination with hardware modification to enable EFUSE power 

On these boards we have a fully working Linux & RTOS with applications.

On those validated boards I did attempt to run the same keywriter I'm using now, and they did present the same timeout/hanging.

Since I had a work-around I didn't attempt to debug this issue further on those boards. Once the keys are written I also cannot run the Keywriter, so I must use a fresh board to debug.

Hi Anders,

Anders Kagerin said:

we wrote the keys using an older binary copy of the Keywriter in combination with hardware modification to enable EFUSE power 

What is the difference between the older copy and the newer copy?

Anders Kagerin said:

On those validated boards I did attempt to run the same keywriter I'm using now, and they did present the same timeout/hanging.

Hmm, this is pointing to some issue with the newer binary then.

Anders Kagerin said:

Once the keys are written I also cannot run the Keywriter, so I must use a fresh board to debug.

You have commented out all the KeyWriter related functions during your tests, and that just renders it to be like a "Hello World" example. This is not looking as a KeyWriter issue as such.

Were you able to do the simple loop tests?

If you use a newer SDK, which supports booting a HS-FS device, you might even be able to run a non KeyWriter firmware on MCU1_0 like a Sciserver_testapp_freertos firmware image.

regards

Suman

It indeed looks like a build issue.

The old keywriter works, but needs HW modification to complete key writing sucessfully
I have only the binary built by other people about a 1.5 ago, for an older board similar to J721e.

If I try to rebuild this older Keywriter using the add-on and keys, the new binary is significantly larger, and it seems the TIFS firmware fails to load, as well as hanging before reaching the end.

If I build my new Keywriter it seems to be larger than the working one, TIFS succeeds, but hangs.

I'm trying to figure out why the newer builds of Keywriter are both larger in size, and do not work.

This is the build process I am following. (I've previously update the TI FEK key and TIFS binary from the add-on)

cd <PDK>/packages/ti/boot/keywriter/scripts
./gen_keywr_cert.sh -g
cd keys
rm bmek.key bmpk.pem smek.key smpk.pem
cp ../../../../build/makerules/k3_dev_mpk.pem smpk.pem
xxd -p -r ../../../../build/makerules/k3_dev_mek.txt smek.key
cp ../ti_fek_public.pem tifekpub.pem
cd ..
./gen_keywr_cert.sh -s keys/smpk.pem --smek keys/smek.key -t keys/tifekpub.pem -a keys/aes256.key

Then from the PDK build folder (if needed I'll also build the dependecies first)

make pmic sciclient_direct board uart osal_nonos csl csl_init i2c gpio rm_pm_hal
make keywriter_img

Hi Anders,

Anders Kagerin said:

The old keywriter works, but needs HW modification to complete key writing sucessfully

I am assuming this is for the PDN to enable the VPP_EFuse. This is only in the KeyWriter application, and not the KeyWriter binary itself.

Anders Kagerin said:

If I try to rebuild this older Keywriter using the add-on and keys, the new binary is significantly larger

The Add-On should not change your rendering of the TIFS binary. You can always inspect the contents of the embedded KeyWriter binary from within your generated KeyWriter application firmware file using readelf command , and cross-compare the contents from the hexdump of the KeyWriter binary. The KeyWriter binary is linked into a specific section in your ELF file (.data.kw_firmware section).

I would expect the KeyWriter binary contents from your old image and the new image should remain the same, even if you rebuilt it anew.  

Anders Kagerin said:

the new binary is significantly larger, and it seems the TIFS firmware fails to load,

What is the size of the old image and new image? The KeyWriter binary gets linked into your KeyWriter image through the soc/j721e/ti-fs-keywriter.h file.

regards

Suman

Hi Anders,

Anders Kagerin said:

make pmic sciclient_direct board uart osal_nonos csl csl_init i2c gpio rm_pm_hal
make keywriter_img

These build commands look fine. I would also recommend an explicit clean of these same targets before you do a re-build. The PDK build doesn't rebuild a library unless a file is modified, and if there are previous existing libraries for the same targets. 

regards

Suman

Yes, I do also run the clean commands

make keywriter_img_clean

or

make clean

Hi Anders,

Anders Kagerin said:

make clean

Please note that this would clean everything, and will require you to rebuild everything as well, as long as you are aware.

regards

Suman

I'm trying to take a step back and try to build a Keywriter from a known good state.
So I unpacked ti-processor-sdk-rtos-j721e-evm-08_01_00_13.tar.gz and applied the addon and our keys.

However, I get the following error when building using "make keywriter_img"

make[1]: Entering directory '/home/anderskagerin/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/pdk_jacinto_08_01_00_36/packages/ti/boot/keywriter/build'
gap-fill=0xff -O binary /home/anderskagerin/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/pdk_jacinto_08_01_00_36/packages/ti/boot/keywriter/binary/j721e/keywriter_img_j721e_release.xer5f /home/anderskagerin/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/pdk_jacinto_08_01_00_36/packages/ti/boot/keywriter/binary/j721e/keywriter_img_j721e_release.bin
/bin/sh: 1: gap-fill=0xff: not found
make[1]: [/home/anderskagerin/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/pdk_jacinto_08_01_00_36/packages/ti/build/makerules/common.mk:630: sbl_img_bin] Error 127 (ignored)
gap-fill=0xff -O binary /home/anderskagerin/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/pdk_jacinto_08_01_00_36/packages/ti/boot/keywriter/binary/j721e/keywriter_img_j721e_release.xer5f /home/anderskagerin/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/pdk_jacinto_08_01_00_36/packages/ti/boot/keywriter/binary/j721e/keywriter_img_j721e_release.bin
/bin/sh: 1: gap-fill=0xff: not found
make[1]: [/home/anderskagerin/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/pdk_jacinto_08_01_00_36/packages/ti/build/makerules/common.mk:633: keywr_imagegen] Error 127 (ignored)

Hi Anders,

Can you please list the output of the ls command of your <ti-processor-sdk-rtos-j721e-evm-08_01_00_13> folder and your <ti-processor-sdk-rtos-j721e-evm-08_01_00_13>/psdk_rtos/scripts folders?

regards

Suman

This is from the SDK RTOS as downloaded from TI, not our software.

~/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13$ ls -lh
total 96K
drwxr-xr-x 7 anderskagerin domain users 4.0K Feb 3 2022 cg_xml_2.61.00
drwxr-xr-x 8 anderskagerin domain users 4.0K Feb 3 2022 dsplib_c66x_3_4_0_0
drwxr-xr-x 11 anderskagerin domain users 4.0K Feb 3 2022 ethfw
drwxr-xr-x 12 anderskagerin domain users 4.0K Feb 3 2022 imaging
-rw-r--r-- 1 anderskagerin domain users 391 Feb 3 2022 index.html
drwxr-xr-x 4 anderskagerin domain users 4.0K Feb 3 2022 ivision
drwxr-xr-x 7 anderskagerin domain users 4.0K Feb 3 2022 mathlib_c66x_3_1_2_1
drwxr-xr-x 6 anderskagerin domain users 4.0K Feb 3 2022 mcusw
drwxr-xr-x 6 anderskagerin domain users 4.0K Feb 3 2022 mmalib_02_02_00_03
drwxr-xr-x 5 anderskagerin domain users 4.0K Feb 3 2022 pdk_jacinto_08_01_00_36
drwxr-xr-x 5 anderskagerin domain users 4.0K Feb 3 2022 perception
drwxr-xr-x 4 anderskagerin domain users 4.0K Feb 3 2022 psdk_rtos
drwxr-xr-x 9 anderskagerin domain users 4.0K Feb 3 2022 remote_device
drwxr-xr-x 11 anderskagerin domain users 4.0K Feb 3 2022 sdl
drwxr-xr-x 18 anderskagerin domain users 4.0K Feb 3 2022 tiadalg
drwxr-xr-x 5 anderskagerin domain users 4.0K Feb 3 2022 ti-cgt-armllvm_1.3.0.LTS
drwxr-xr-x 6 anderskagerin domain users 4.0K Feb 3 2022 ti-cgt-c6000_8.3.7
drwxr-xr-x 7 anderskagerin domain users 4.0K Feb 3 2022 ti-cgt-c7000_2.0.1.STS
drwxr-xr-x 6 anderskagerin domain users 4.0K Feb 3 2022 tidl_j7_08_01_00_05
drwxr-xr-x 14 anderskagerin domain users 4.0K Feb 3 2022 tiovx
drwxr-xr-x 7 anderskagerin domain users 4.0K Feb 3 2022 uia_2_30_01_02
drwxr-xr-x 12 anderskagerin domain users 4.0K Feb 3 2022 vision_apps
drwxr-xr-x 6 anderskagerin domain users 4.0K Feb 3 2022 vxlib
drwxr-xr-x 9 anderskagerin domain users 4.0K Feb 3 2022 xdais_7_24_00_04

~/git/ti-processor-sdk-rtos-j721e-evm-08_01_00_13/psdk_rtos/scripts$ ls -lh
total 32K
-rwxr-xr-x 1 anderskagerin domain users 252 Feb 3 2022 board_env.sh
-rwxr-xr-x 1 anderskagerin domain users 1.4K Feb 3 2022 copy_from_sd_card.sh
-rwxr-xr-x 1 anderskagerin domain users 1.1K Feb 3 2022 install_data_set_to_sd_card.sh
-rwxr-xr-x 1 anderskagerin domain users 1.7K Feb 3 2022 install_to_sd_card.sh
-rwxr-xr-x 1 anderskagerin domain users 2.0K Feb 3 2022 mk-linux-card.sh
-rwxr-xr-x 1 anderskagerin domain users 2.6K Feb 3 2022 mk-qnx-card.sh
-rwxr-xr-x 1 anderskagerin domain users 7.6K Feb 3 2022 setup_psdk_rtos.sh

Thanks, that was the issue with building from ti-processor-sdk-rtos-j721e-evm-08_01_00_13 folder.
To clarify, this script is run automatically in my software repository, so should not be the original issue.

I also encountered an issue where the Keywriter binary is too big, even when just building the original, so I applied this patch (increase linker size)
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1106641/tda4vm-otp-keywriter-build-failure

However, I get the same problem. The Keywriter starts, then hangs.

MCU UART 0:

OTP Keywriter Version: 02.00.00.00 (Nov 5 2024 - 20:28:50)

WKUP UART 0:

0x400002
0x800004
0x4003005
0x4401552
0x40000B
0x800004
0x4003005
0x4401552

Hi Anders,

Anders Kagerin said:

To clarify, this script is run automatically in my software repository, so should not be the original issue.

Not sure what to make of this. You shouldn't have seen build issues if the script was run automatically and everything was installed in the right place.

Anders Kagerin said:

I also encountered an issue where the Keywriter binary is too big, even when just building the original, so I applied this patch (increase linker size)
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1106641/tda4vm-otp-keywriter-build-failure

Yes, this is a patch that was merged in our later SDK releases, and was made as part a Compiler update when it increased the program segments. Given that you have other VPP enabled code, your KeyWriter firmware size would have increased, and this is a good patch to pick up.

Anders Kagerin said:

However, I get the same problem. The Keywriter starts, then hangs.

Are you sure you have a proper certificate included in your KeyWriter binary? 

Anders Kagerin said:

MCU UART 0:

OTP Keywriter Version: 02.00.00.00 (Nov 5 2024 - 20:28:50)

Then again, I am expected to see additional traces here atleast to the point of printing the OTP Certificate trace etc., but I don't see any. So, it mostly hasn't reached that point.

I would recommend that you start out with a basic KeyWriter certificate of programming the MSV (not used in Secure Boot). 

You can use the following command to generate your certificate,

./gen_keywr_cert.sh --msv 0xC0FFE -t ti_fek_public.pem

The logs on Wkup UART will be as follows with this certificate:

0x400002
0x800004
0x4003005
0x4401552
0x40000B
0x800004
0x4003005
0x4401552
0x40000D
0x800004
0x20800000
0x20800001
0x400002
0x800004
0x4003005
0x4401552
0x409031
0x800004
#
# Decrypting extensions..
#
MPK Options:  0x0
MEK Options:  0x0
MPK Opt P1:  0x0
MPK Opt P2:  0x0
MEK Opt   :  0x0
SMPKH extension disabled
SMEK extension disabled
EXT OTP extension disabled
* BCH code & MSV: fe0fac8b

KEY CNT extension disabled

KEY REV extension disabled

SWREV extension disabled

FW CFG REV extension disabled

* KEYWR VERSION:  0x20000

#
# Programming Keys..
#

* MSV:
[u32] bch + msv:  0x0
Programmed 2/2 rows successfully
[u32] bch + msv:  0x8BAC0FFE

* SWREV:
[u32] SWREV-SYSFW:  0x1
[u32] SWREV-SBL  :  0x1
SWREV extension disabled
[u32] SWREV-SYSFW:  0x1
[u32] SWREV-SBL  :  0x1

* FW CFG REV:
[u32] SWREV-FW-CFG-REV:  0x1
SWREV SEC BCFG extension disabled
[u32] SWREV-FW-CFG-REV:  0x1

* EXT OTP:
EXT OTP extension disabled

* BMPKH, BMEK:
BMPKH extension disabled
BMEK extension disabled

* SMPKH, SMEK:
SMPKH extension disabled
SMEK extension disabled

* KEYCNT:
[u32] keycnt:  0x0
KEY CNT extension disabled
[u32] keycnt:  0x0

* KEYREV:
[u32] keyrev:  0x0
KEY REV extension disabled
[u32] keyrev:  0x0

You need to have the JTAG setup ready to debug this further.  

regards

Suman

Suman Anna said:

Are you sure you have a proper certificate included in your KeyWriter binary? 

Since it boots, I assume the ti_fek_public.pem is correct.
I assume any other key error would only lead to the wrong keys being written to the fuses?

Suman Anna said:

I would recommend that you start out with a basic KeyWriter certificate of programming the MSV (not used in Secure Boot). 

Since I don't reach the function call that actually writes the keys to fuses, does this make sense to test?

Hi Anders,

Anders Kagerin said:

Since it boots, I assume the ti_fek_public.pem is correct.

The ti_fek_public.pem is only used for the KeyWriter certificate, it is not used with booting the KeyWriter binary itself. So, this has no bearing.

Anders Kagerin said:

Since I don't reach the function call that actually writes the keys to fuses, does this make sense to test?

I agree that you are not yet reaching that point, but I would also recommend that you use this as your starting point. MSV is the most simplest KeyWriter certificate without any adverse affect to your sample.

I expect to see a few more TIFS traces before it starts seeing the certificate. 

regards

Suman

Suman Anna said:

Are you sure you have a proper certificate included in your KeyWriter binary? 

Oh, I misread your question. I'm not sure on how to verify the certificate, but I've followed the build process that I detailed earlier in this thread, and I've made sure that I'm using the SMPK and SMEK keys that my colleagues have saved as our internal development keys.

I don't have the original AES key, which I believe is used to encrypt the keys in the certificate, so I assume it will not be identical to the on in the working keywriter binary that I have. Even if correct.

If the certificate is faulty, I assume the keywriter shouldn't freeze, but instead report an error when the Sciclient_otpProcessKeyCfg is called.

Hi Anders,

Anders Kagerin said:

I don't have the original AES key, which I believe is used to encrypt the keys in the certificate, so I assume it will not be identical to the on in the working keywriter binary that I have. Even if correct.

The AES key doesn't matter. It can change from one KeyWriter run to another as well, it is only used for encryption of the keys within the certificate. The AES key is not fused into the device eFuses anywhere.

Anders Kagerin said:

If the certificate is faulty, I assume the keywriter shouldn't freeze, but instead report an error when the Sciclient_otpProcessKeyCfg is called.

Yes, correct. You don't even need to program the SMPK and SMEK to begin with to flow-flush the KeyWriter flow. The MSV I suggested would be the first baseline starting test.

Your current issue is not even reaching the state of programming the keys.

regards

Suman

Partially good news. My colleague suggested that the older (working) Keywriter was built using an older SDK.

We set up the Keywriter in SDK 8.0 (instead of SDK 8.1), and replaced it with the patched version here
https://dr-download.ti.com/software-development/software-development-kit-sdk/MD-bA0wfI4X2g/08.00.00.12/keywriter_patch.tar.gz

After building and running this Keywriter, we can confirm it does not freeze/hang! A good sign!

I'd like to figure out what the difference is, and why SDK8.1 Keywriter freezes,
but if I don't have time I might just set up a separate repository using the SDK 8.0 files, for building the Keywriter.

Hi Anders,

Anders Kagerin said:

We set up the Keywriter in SDK 8.0 (instead of SDK 8.1),

We don't have any major changed within the KeyWriter code itself in 8.1 SDK when compared to SDK 8.0, and I am not aware of anyone else reporting issues with 8.1 SDK.

Anders Kagerin said:

replaced it with the patched version here

Ok, this is from our ti.com SDK download page, I will see what changed here.

Anders Kagerin said:

but if I don't have time I might just set up a separate repository using the SDK 8.0 files, for building the Keywriter.

I still suspect some integration issues within your build system. Anyway, you might skip the 8.1 altogether and directly check with this on the newer 9.2 SDK.

regards

Suman