TDA4VM: SDK 8.06 ti-fw-firmware-j721e_sr1_1-hs-fs

Hi Spencer,

The 8.6 SDK version of the TIFS HS-FS firmwares for SR1.1 were not released and part of the ti-linux-firmware project.

I will have to check internally if these are available even. I don't see the equivalent 9.0 TIFS HS-FS versions available either.

regards

Suman  

Hi Spencer,

I do not see these internally either. Is it not possible for you to move to the latest 9.1 SDK release?

regards

Suman

Hello!

We believe we have successfully signed the sysfw.itb file - but we're still stuck in the booting process.

We've implemented an infinite loop that will toggle the voltage at test point in tiboot3.bin (r5 uboot) that should run immediately after the line:

`k3_sysfw_loader(is_rom_loaded_sysfw(&bootdata), k3_mmc_stop_clock, k3_mmc_restart_clock);` (inside of j721e_init.c)

We never see the voltage toggle on that line - so we know we aren't making it past that line. If we move it earlier in the boot process (Before this line), it works just fine. Are we failing to unlock the device? Have we misconfigured something?

MCU Uart 0 when booted via UART
(after loading firmware)

Loaded 276498 bytes
��{��íޞ��8�����៽()̟)�)kJ!�)!�!�S)o
!���kH{


ğ)�)kJ!Ȉ��Cν��k�)���{�)�������)1)����w
�����!
�k�����)�)kJ!�)o�!�S��C�gC�-�#

ğ)�)�J!Ȉ��C^���k�)���{�)�������)1)���w
�����!
Vk�����φ>)o�!�S�)o
!���kH{

Wakeup uart 0 - booted via SD card (same files):

sburtner:~/workspace/tda4/ti-processor-sdk-rtos-j721e-evm-08_06_00_12$ python3 ./pdk_jacinto_08_06_00_31/packages/ti/drv/sciclient/soc/sysfw/binaries/scripts/sysfw_trace_parser.py -l sysfw_trace.log -O
Configuring trace data version to: 0x03007
0x00710002: BasePort: TISCI_MSG_RECEIVED(TISCI Message interrupt handled): Queue ID: 49 Message ID: 2
0x00B10004: BasePort: TISCI_MSG_SENDER_HOST_ID(Message from secure host received): Queue ID: 49 Host ID: 4
0x04003007: BasePort: TRACE_DATA_VERSION(OSAL/Baseport trace data version): Trace version major: 0x03 Trace version minor: 0x007
0x04400863: BasePort: SYSFW_VERSION(System Firmware version): version: 8 subversion: 6 patch: 3
0x0071000B: BasePort: TISCI_MSG_RECEIVED(TISCI Message interrupt handled): Queue ID: 49 Message ID: b
0x00B10004: BasePort: TISCI_MSG_SENDER_HOST_ID(Message from secure host received): Queue ID: 49 Host ID: 4
0x00710005: BasePort: TISCI_MSG_RECEIVED(TISCI Message interrupt handled): Queue ID: 49 Message ID: 5
0x00B10004: BasePort: TISCI_MSG_SENDER_HOST_ID(Message from secure host received): Queue ID: 49 Host ID

Thank you,

We've stepped through with the debugger and have determined it's failing with the panic:

`Failed to set board configuration (-19)`

inside the function: `static void k3_sysfw_configure_using_fit(void *fit, struct ti_sci_handle *ti_sci)`

	/* Apply board configuration to SYSFW */
	ret = board_ops->board_config(ti_sci,
				      (u64)(u32)cfg_fragment_addr,
				      (u32)cfg_fragment_size);
	if (ret)
		panic("Failed to set board configuration (%d)\n", ret);

Any thoughts on what's causing this?

Suman is the correct person to be assigned to this, but his response over the next couple days might be limited as he is ooo at the moment.  However, just going back to the original clarification from him -- and this helps with debug -- are you doing this with the latest SDK?  

John

Hi John,  we are using PSDK 8.06.  We have received some binaries to try for `ti-fw-firmware-j721e_sr1_1-hs-fs-*.bin`, but are still having issues.  Thanks!

Can the 9.1 SDK be attempted?   

We went ahead and tried 9.1 and it works.  We have told the customer we're using 8.06.

Hi Stuart,

Thanks for the update.

Is there support still needed with 8.6 SDK?

regards

Suman

I checked with the customer we need to use 8.06 - could you support us?

Hi Stuart,

The SDK 8.6 is not supporting the HS-FS devices cleanly. This will require further debug on our side to provide you with a patch.

Stuart Burtner said:

Any thoughts on what's causing this?

The only reason this could happen here is that the board config file is not bypassing the authentication or passing in the right address to TIFS firmware.

You can try rebuilding the R5 U-Boot by adding a #define DEBUG at the top of the arch/arm/mach-k3/sysfw-loader.c and arch/arm/mach-k3/security.c files and provide the output log.

Please also check the behavior by following a similar suggestion for sysfw binary modification in the its source file to use unsigned board config binaries instead of the signed board config binaries, and regenerate the sysfw.itb. 

regards

Suman

Suman,

I can try compiling with debug.  I assume that will come out of MCU_UART0 or WKUP_UART0?  I don't have access to `MAIN_UART0` on this board, only `MAIN_UART5`.  Or can I see that with code composer studios over JTAG?

For modifying the sysfw binary - are you talking about not using both `ti-fs-firmware-j721e_sr1_1-hs-enc.bin` and `ti-fs-firmware-j721e_sr1_1-hs-cert.bin`, but instead using `ti-fs-firmware-j721e-gp.bin`?

Or are you suggesting that I compile in #debug into some of the board files?

Hi Stuart,

Stuart Burtner said:

I can try compiling with debug.  I assume that will come out of MCU_UART0 or WKUP_UART0?  I don't have access to `MAIN_UART0` on this board, only `MAIN_UART5`.  Or can I see that with code composer studios over JTAG?

The TI SPL/U-Boot, ATF, OPTEE and Linux all use the same MAIN UART for all their traces, so this would be the equivalent UART for your board, whatever that is.

Stuart Burtner said:

For modifying the sysfw binary - are you talking about not using both `ti-fs-firmware-j721e_sr1_1-hs-enc.bin` and `ti-fs-firmware-j721e_sr1_1-hs-cert.bin`, but instead using `ti-fs-firmware-j721e-gp.bin`?

No, I am not suggesting that you change the TIFS binaries. You will continue to use the HS-FS TIFS binaries. I am suggesting that you change to use the unsigned board config binaries. The its FIT image source file configuration uses signed board configuration binaries (all the remaining entries other than the sysfw.bin).

Stuart Burtner said:

Or are you suggesting that I compile in #debug into some of the board files?

This should provide a bit more verbose traces on the U-Boot itself.

regards

Suman

Suman,

Thank you for your support so far - and apologies for my delayed response.

Suman Anna said:

The TI SPL/U-Boot, ATF, OPTEE and Linux all use the same MAIN UART for all their traces, so this would be the equivalent UART for your board, whatever that is.

This makes sense - thank you.  How could I expect debug information out if its' the transfer of the device tree information that is failing? (assuming I understand the system properly of course).  I'm failing at `k3_sysfw_loader()` - which is before we get to `preloader_console_init()`. 

(An aside, I'm having issues with getting messages from u-boot out onto the console port too, but that's a different issue, based on 9.01 right now.  There's a separate post for that).

Suman Anna said:

No, I am not suggesting that you change the TIFS binaries. You will continue to use the HS-FS TIFS binaries. I am suggesting that you change to use the unsigned board config binaries. The its FIT image source file configuration uses signed board configuration binaries (all the remaining entries other than the sysfw.bin).

I think I understand.  I made the following patch to the Makefile in k3-imagegen to prevent it from signing the board config binaries:

 $(soc_objroot)/%.o: %.c | _objtree_build
-       $(CROSS_COMPILE)gcc $(CFLAGS) -c -o $@-pre-validated $<
-       python3 ./scripts/sysfw_boardcfg_validator.py -b $@-pre-validated -i -o $@ -s $(BASE_SOC) -l $@.log
+       $(CROSS_COMPILE)gcc $(CFLAGS) -c -o $@ $<
+       #python3 ./scripts/sysfw_boardcfg_validator.py -b $@-pre-validated -i -o $@ -s $(BASE_SOC) -l $@.log

Unfortunately, the system still is not making it past `k3_sysfw_loader()`.  Is it possible something unrelated is wrong (like the device tree itself?).

Thank you again,
Stuart

Hi Stuart,

Stuart Burtner said:

I'm failing at `k3_sysfw_loader()` - which is before we get to `preloader_console_init()`

Yeah ok, then traces won't help much.

Stuart Burtner said:

Unfortunately, the system still is not making it past `k3_sysfw_loader()`

Is this still failing during the processing of the board config binary? You should be definitely going past the TIFS load itself (that was your observation last time) before my suggestion to use unsigned board config binaries.

That is the last possible suggestion I have.

This is a development effort to debug this and get it functional on 8.6 SDK where the HS_FS boot on SR1.1 is not officially supported. This will take time.

regards

Suman

Suman,

Suman Anna said:

Is this still failing during the processing of the board config binary? You should be definitely going past the TIFS load itself (that was your observation last time) before my suggestion to use unsigned board config binaries.

Apologies if I have been unclear.  I believe the system never gets past `k3_sysfw_loader()`

Stuart Burtner said:

We've stepped through with the debugger and have determined it's failing with the panic:

`Failed to set board configuration (-19)`

inside the function: `static void k3_sysfw_configure_using_fit(void *fit, struct ti_sci_handle *ti_sci)`

It's been a few weeks now, but if memory serves this function is inside of k3_sysfw_loader().  So I've never made it past the sysfw_loader before.

I'm unclear as to where the TIFS load is happening, versus the board config binaries - at least as it relates to `board_init_f()` in `j721e_init.c` - could you clarify that for me? Thanks!

Hi Stuart,

Stuart Burtner said:

Apologies if I have been unclear.  I believe the system never gets past `k3_sysfw_loader()`

k3_sysfw_loader() is the overall function that deals with the processing of the sysfw.itb file, which is a combination of the TIFS binary itself as well as the 4 different board configuration binaries.

This function does invoke two other functions internally for J721E - k3_sysfw_load_using_fit() and k3_sysfw_configure_using_fit(). The first function deals with the boot of the TIFS binary, while the second one deals with the 4 board configuration binaries, all part of the same sysfw.itb file.

Stuart Burtner said:

It's been a few weeks now, but if memory serves this function is inside of k3_sysfw_loader().  So I've never made it past the sysfw_loader before.

Correct, and that's my last suggestion to not use signed binaries for the board configuration binaries. This also aligns with the binman usage in the ti-u-boot-2023.04 branch used by the 9.x SDKs. The HS-SE board config nodes do have a ti-secure node, while the HS-FS and GP board config nodes do not have a corresponding ti-secure node.

Stuart Burtner said:

I'm unclear as to where the TIFS load is happening, versus the board config binaries - at least as it relates to `board_init_f()` in `j721e_init.c` - could you clarify that for me?

It all happens inside the k3_sysfw_loader(), but inside the two separate functions as noted above.

regards

Suman 

Great - thank you - that clears things up considerably.

Yes, you are right, I've confirmed that I am making it up to - but not past - `k3_sysfw_configure_using_fit()`

I went ahead and retested 8.06 with my changes earlier to leave the board-cfg binaries unsigned (the ones from earlier), and am checking to see if I'm making it past that function.

Unfortunately, we're still stuck at that function.  I'm assuming that I've successfully compiled these binaries without security via the same patch I sent earlier:

diff --git a/Makefile b/Makefile
index 66aefd0..3687079 100644
--- a/Makefile
+++ b/Makefile
@@ -248,8 +248,8 @@ tiboot3.bin: $(TIBOOT3)
        @ln -sf $< $(BIN_DIR)/$@
 
 $(soc_objroot)/%.o: %.c | _objtree_build
-       $(CROSS_COMPILE)gcc $(CFLAGS) -c -o $@-pre-validated $<
-       python3 ./scripts/sysfw_boardcfg_validator.py -b $@-pre-validated -i -o $@ -s $(BASE_SOC) -l $@.log
+       $(CROSS_COMPILE)gcc $(CFLAGS) -c -o $@ $<
+       #python3 ./scripts/sysfw_boardcfg_validator.py -b $@-pre-validated -i -o $@ -s $(BASE_SOC) -l $@.log
 
 %.bin-signed: %.bin
        $(TI_SECURE_DEV_PKG)/scripts/secure-binary-image.sh $< $@

I went ahead and checked - and what I was doing was not dealing with the signed nature of the board config package.

I reverted my changes and made the following patch:

diff --git a/Makefile b/Makefile
index 66aefd0..2723f4d 100644
--- a/Makefile
+++ b/Makefile
@@ -137,7 +137,8 @@ SOC_LOGS=$(SOURCES:%.c=$(soc_objroot)/%.o.log)
 SOC_PVAL=$(SOURCES:%.c=$(soc_objroot)/%.o-pre-validated)
 SOC_BINS=$(soc_objroot)/sysfw.bin-$(SOC_TYPE)
 ifdef SIGN_BRDCFG
-SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin-signed)
+#SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin-signed)
+SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin)
 else
 SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin)
 endif

For a cumulative set of changes:

diff --git a/Makefile b/Makefile
index abba544..2723f4d 100644
--- a/Makefile
+++ b/Makefile
@@ -137,7 +137,8 @@ SOC_LOGS=$(SOURCES:%.c=$(soc_objroot)/%.o.log)
 SOC_PVAL=$(SOURCES:%.c=$(soc_objroot)/%.o-pre-validated)
 SOC_BINS=$(soc_objroot)/sysfw.bin-$(SOC_TYPE)
 ifdef SIGN_BRDCFG
-SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin-signed)
+#SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin-signed)
+SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin)
 else
 SOC_BINS += $(SOURCES:%.c=$(soc_objroot)/%.bin)
 endif
@@ -185,21 +186,26 @@ _bindir_build:
 
 $(SYSFW_PATH):
 	@echo "Downloading SYSFW release image..."
-	wget $(SYSFW_DL_URL)
+	#wget $(SYSFW_DL_URL)
 	@echo "Download SUCCESS!"
 
 $(SYSFW_HS_INNER_CERT_PATH):
 	@echo "Downloading HS SYSFW release certificate..."
-	wget $(SYSFW_HS_INNER_CERT_DL_URL)
+	#wget $(SYSFW_HS_INNER_CERT_DL_URL)
 	@echo "Download SUCCESS!"
 
 ifneq ($(SOC_TYPE),gp)
+
+ifneq ($(SOC_TYPE),hs-fs)
 $(SYSFW_HS_CERTS_PATH): $(SYSFW_HS_INNER_CERT_PATH)
 	@echo "Signing the SYSFW inner certificate with $(KEY) key...";
 	./gen_x509_cert.sh -d -c m3 -b $< -o $@ -l $(LOADADDR) -k $(KEY) -r $(SW_REV);
 
 $(soc_objroot)/sysfw.bin-$(SOC_TYPE): $(SYSFW_HS_CERTS_PATH) $(SYSFW_PATH) | _objtree_build
 	cat $^ > $@
+endif
+$(soc_objroot)/sysfw.bin-$(SOC_TYPE): $(SYSFW_HS_INNER_CERT_PATH) $(SYSFW_PATH) | _objtree_build
+	cat $^ > $@
 else
 $(soc_objroot)/sysfw.bin-$(SOC_TYPE): $(SYSFW_PATH) | _objtree_build
 	@echo "Signing the SYSFW release image with $(KEY) key...";
@@ -264,11 +270,11 @@ clean:
 	-rm -f $(ITS)
 	-rm -f $(TIBOOT3) $(BIN_DIR)/tiboot3.bin
 	-rm -f $(SYSFW_HS_CERTS_PATH)
-	-rm -df $(BIN_DIR)
-	-rm -df $(O)/soc/$(BASE_SOC)/$(CONFIG) && \
-	 rm -df $(O)/soc/$(BASE_SOC) && \
-	 rm -df $(O)/soc && \
-	 rm -df $(O)
+	#-rm -f $(BIN_DIR)
+	-rm -rf $(O)/soc/$(BASE_SOC)/$(CONFIG) && \
+	 rm -rf $(O)/soc/$(BASE_SOC) && \
+	 rm -rf $(O)/soc && \
+	 rm -drf $(O)
 
 .PHONY: mrproper
 mrproper: clean

Which worked so far! I haven't attempted a full boot but I'm getting past the preloader_console_init().

I've set up the console to print via MCU_UART0 (Similar to what I did on 9.01) and have found that it's stuck at ATF.  I've had to recompile ATF to deal with the UART, but I seem to be stuck before I see any prints from it.

U-Boot SPL 2021.01-00006-g8a0436f5db-dirty (Mar 13 2024 - 13:22:34 -0400)
Model: Texas Instruments K3 J721E SoC
EEPROM not available at 0x50, trying to read at 0x51
Reading on-board EEPROM at 0x51 failed 1
Board: J721EX-PM1-SOM rev E2
SYSFW ABI: 3.1 (firmware rev 0x0008 '8.6.3-v08.06.03 (Chill Capybara')
EEPROM not available at 0x50, trying to read at 0x51
Reading on-board EEPROM at 0x51 failed 1
ti_power_domain_probe(dev=41c85db8)
Trying to boot from MMC2
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Authentication passed
Starting ATF on ARM64 core...

I've gotten this to work on 9.01.00, so I'm assuming this is another HS-FS issue?

Hi Stuart,

Suman Anna said:

Correct, and that's my last suggestion to not use signed binaries for the board configuration binaries.

I am confident that this is the desired configuration for HS-FS.

Stuart Burtner said:

I went ahead and checked - and what I was doing was not dealing with the signed nature of the board config package.

The surest way of confirming this is to check the sizes of the board config binarires through the dumpimage command on the resulting sysfw.itb. The signing process typically adds about ~1500 to 1700 bytes, so the unsigned board configs should be very small (like pm-cfg.bin should be < 10 bytes).

Stuart Burtner said:

I've set up the console to print via MCU_UART0 (Similar to what I did on 9.01) and have found that it's stuck at ATF.  I've had to recompile ATF to deal with the UART, but I seem to be stuck before I see any prints from it.

Ok, this implies that you have got past the R5 SPL-side and have successfully loaded the TIFS. This might mostly be due to some missing code logic in 8.6 w.r.t dealing with signed images. You should be able to try my earlier debug suggestions now.

See if you can drop in the tispl.bin from 9.1 and go past ATF.

regards

Suman

Hi Suman,

I tried replacing the TI-SPL as you indicated - that got me through the ATF issue.  As I got further into u-boot I had more issues.

1: I couldn't rebuild fitImage properly. For now I commented it out just to see what happened

2: I notice that it can't load the firmware for the other cores unless its' secured.  That scares me.

U-Boot 2021.01-00006-g8a0436f5db-dirty (Mar 20 2024 - 15:36:39 -0400)

SoC:   J721E SR1.1 HS-FS
Model: Texas Instruments K3 J721E SoC
EEPROM not available at 0x50, trying to read at 0x51
Reading on-board EEPROM at 0x51 failed 1
Board: J721EX-PM1-SOM rev E2
DRAM:  4 GiB
Flash: 0 Bytes
MMC:   sdhci@4f80000: 0, sdhci@4fb0000: 1
Loading Environment from FAT... OK
In:    serial@40a00000
Out:   serial@40a00000
Err:   serial@40a00000
am65_cpsw_nuss ethernet@46000000: K3 CPSW: nuss_ver: 0x6BA00101 cpsw_ver: 0x6BA80100 ale_ver: 0x00293904 Ports:1 mdio_freq:1000000
EEPROM not available at 0x50, trying to read at 0x51
Reading on-board EEPROM at 0x51 failed 1
cdns,sierra serdes@5000000: PHY not found 0x7364 vs 0x0
Sierra init failed:-22
Net:   Could not get PHY for ethernet@46000000port@1: addr 0
am65_cpsw_nuss_port ethernet@46000000port@1: phy_connect() failed
No ethernet found.

Hit any key to stop autoboot:  0 
switch to partitions #0, OK
mmc1 is current device
SD/MMC found on device 1
Failed to load 'boot.scr'
483 bytes read in 7 ms (67.4 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
1 bytes read in 8 ms (0 Bytes/s)
Already setup.
i2c_write: error waiting for data ACK (status=0x116)
pca953x gpio@20: Error reading output register
GPIO: 'gpio@22_17' not found
Command 'gpio' failed: Error -121
i2c_write: error waiting for data ACK (status=0x116)
pca953x gpio@20: Error reading output register
GPIO: 'gpio@22_16' not found
Command 'gpio' failed: Error -121
k3_r5f_rproc r5f@41000000: Core 1 is already in use. No rproc commands work
k3_r5f_rproc r5f@41400000: Core 2 is already in use. No rproc commands work
Failed to load '/lib/firmware/j7-main-r5f0_0-fw-sec'
Failed to load '/lib/firmware/j7-main-r5f0_1-fw-sec'
Failed to load '/lib/firmware/j7-main-r5f1_0-fw-sec'
Failed to load '/lib/firmware/j7-main-r5f1_1-fw-sec'
Failed to load '/lib/firmware/j7-c66_0-fw-sec'
Failed to load '/lib/firmware/j7-c66_1-fw-sec'
Failed to load '/lib/firmware/j7-c71_0-fw-sec'
Failed to load '/boot/fitImage'
Wrong Image Format for bootm command
ERROR: can't get kernel image!

Hi Stuart,

Stuart Burtner said:

1: I couldn't rebuild fitImage properly. For now I commented it out just to see what happened

Which fitImage are you talking about here?

Stuart Burtner said:

2: I notice that it can't load the firmware for the other cores unless its' secured.  That scares me.

The HS-FS boot philosophy is based on having a similar flow as on a HS-SE device, so it uses signed images for any rproc firmware binaries, but the device does not actually have Customer Keys fused, so the code will recognize the image has a signature but skips authenticating it. The code really doesn't care whether the images are signed with TI Dummy Keys or with actual Customer production keys.

regards

Suman