AM69A: Intermittent Boot Failures - Frequency Handshake Timeout and SPL Hang

Hello Parth,

I am looping in our DDR expert..we will get back to you in a couple of days. 

Best Regards,

Keerthy 

Hi,

Timeout during frequency handshake
### ERROR ### Please RESET the board ###

When the issue above occurs, can you run the attached binary from the R5 core through CCS and provide the result? 

3414.tda4vh_lp4_debug_CCSoutput.zip

Thanks,
Kevin

Hello Kevin S,

Thanks for the response.

Kevin S said:

Timeout during frequency handshake
### ERROR ### Please RESET the board ###

Since this issue is random and takes a long time to reproduce, I was wondering if there’s any alternative way to help debug it - perhaps by enabling or dumping DDRSS registers in U-Boot/SPL?
In parallel, I’ll trigger the tests and plan to run the shared LP4 debug tool when the issue occurs.

P.S. : I successfully ran the tool on my setup during a normal boot (i.e., without encountering issue #1) and have attached the logs here for reference, i.e. primarily to sanity-check the LP4 debug tool on my setup.

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_LP4_5F00_Debug_5F00_Sample_5F00_run_5F00_logs_5F00_in_5F00_working_5F00_good_5F00_case.txt

Regards,
Parth P

Just to add -  in the last test cycle, I enabled additional DDRSS logs and observed the following logs on one of the three DUTs.

U-Boot SPL 2024.04-00774-g375342660d72-dirty (Jun 05 2025 - 15:49:26 +0200)
SYSFW ABI: 4.0 (firmware rev 0x000a '10.0.8--v10.00.08 (Fiery Fox)')
HW CFG: ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
0x00
k3_ddrss_probe(dev=41c675c8)
k3_ddrss_ofdata_to_priv(dev=41c675c8)
k3_ddrss_power_on(ddrss=41c6de48)
k3_ddrss memorycontroller@2990000: vtt-supply not found.
k3_lpddr4_probe: PASS
k3_lpddr4_init: PASS
--->>> LPDDR4 Initialization is in progress ... <<<---
k3_lpddr4_freq_update: received freq change req: req type = 1, req no. = 0, instance = 0
wait_for_bit_le32: Timeout (reg=114080 mask=80 wait_set=1)
Timeout during frequency handshake
### ERROR ### Please RESET the board ###

This last log provided shows the issue. Were you able to capture the debug output as well? 

Thanks,
Kevin

U-Boot SPL 2024.04-00774-g375342660d72-dirty (Jun 05 2025 - 15:49:26 +0200)
SYSFW ABI: 4.0 (firmware rev 0x000a '10.0.8--v10.00.08 (Fiery Fox)')
HW CFG: ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
ti_dm6441_gpio gpio@42110000: pinctrl_select_state_full: uclass_get_device_by_phandle_id: err=-19
0x00
k3_ddrss_probe(dev=41c675c8)
k3_ddrss_ofdata_to_priv(dev=41c675c8)
k3_ddrss_power_on(ddrss=41c6de48)
k3_ddrss memorycontroller@2990000: vtt-supply not found.
k3_lpddr4_probe: PASS
k3_lpddr4_init: PASS
--->>> LPDDR4 Initialization is in progress ... <<<---
k3_lpddr4_freq_update: received freq change req: req type = 1, req no. = 0, instance = 0
wait_for_bit_le32: Timeout (reg=114080 mask=80 wait_set=1)
Timeout during frequency handshake
### ERROR ### Please RESET the board ###

Hi Kevin S,

We were able to reproduce the reported DDRSS Issue#1 and successfully captured the LP4 debug logs via JTAG.

Please find below the LP4 debug logs for both scenarios (Boot Stuck and Boot OK) on the same DUT:

DDRSS Issue#1 (Boot Stuck): /cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_LP4_5F00_Debug_5F00_Logs_5F00_When_5F00_DDR_5F00_Issue_5F00_Boot_5F00_Stuck_5F00_10062025.txt

Working (Boot OK):/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_LP4_5F00_Debug_5F00_Logs_5F00_Good_5F00_case_5F00_Boot_5F00_OK_5F00_10062025.txt

Looking forward to the analysis followed by some proposals of possible causes and solutions.
Thank you.

Regards,
Parth P

Hi Parth,

Thanks, can you also upload your register settings. If you used the spreadsheet version of the tool, please provide the filled-in spreadsheet used to generate your register settings. If you used SYSCONFIG, please provide the *.sysconfig files in addition to the .dtsi or .h files.

Based on the log file provided, it appears that command bus training failed for DDRSS2. 

A few initial suggestions:

1. Try adjusting the CA/CS IO settings. If not already, can you try a combination where CS drive strength = CA/CS ODT = 2x CA drive strength
   1. Example: CS drive = 80 ohms, CA/CS ODT = 80 ohms, CA drive = 40 ohms
2. Try reducing the CA VREF training range
   1. Change DDR register configuration file with following modifications (where n could be 0, 1, 2, or 3):
      #define DDRSSn_PI_199_DATA 0x24182418
      #define DDRSSn_PI_200_DATA 0x01012418

Regards,
Kevin

Thanks a lot, Kevin S, for the quick response.

Sysconfig file: /cfs-file/__key/communityserver-discussions-components-files/791/Toradex_5F00_Aquila_5F00_AM69_5F00_DDRSS_5F00_Register_5F00_Configuration_5F00_32GB.zip

LPDDR4 configuration DTSI :  https://git.toradex.com/cgit/u-boot-toradex.git/tree/arch/arm/dts/k3-am69-aquila-lpddr4-4266.dtsi?h=toradex_ti-u-boot-2024.04

Regarding your initial suggestions,

Kevin S said:

Try adjusting the CA/CS IO settings. If not already, can you try a combination where CS drive strength = CA/CS ODT = 2x CA drive strength

1. Example: CS drive = 80 ohms, CA/CS ODT = 80 ohms, CA drive = 40 ohms

I believe this setting is already in place, as these configurations were carried over unchanged from the AM69-SK setup. Please double confirm from the shared files.

Kindly review the attached Sysconfig file and let us know if anything else needs adjustment.

Kevin S said:

Try reducing the CA VREF training range

1. Change DDR register configuration file with following modifications (where n could be 0, 1, 2, or 3):
   #define DDRSSn_PI_199_DATA 0x24182418
   #define DDRSSn_PI_200_DATA 0x01012418

In the meantime, I’ll try reducing the CA VRAF training range and will share the results afterward.

Thank you.

Regards,
Parth P

Hi Parth,

I have a very similar problem as you. I temporarily solved it by reducing the DDR frequency from 2133 MHz to 2000 MHz or using only the first 2 DDR chips at full speed.

I have a problem to load the program 3414.tda4vh_lp4_debug_CCSoutput.zip into the R5 core. Please, can you post here the procedure to load it in both cases DDRAM OK/NOK? Thank you.

Hello Juraj Golej,

I followed the similar steps outlined in the link below to load the LP4 debug binary via CCS:

https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-j784s4/09_01_00_06/exports/docs/psdk_rtos/docs/user_guide/ccs_setup_j784s4.html

To collect the logs:

• Connect the XDS110 debug probe to the hardware
• Launch the target configuration in CCS
• Reset the R5 CPU
• Load the GEL files
• Finally, load and run the tda4vh_lp4_debug-temp.out binary

For more detailed guidance, TI experts should be able to assist further.

Regards,
Parth P

Hi Kevin S,

Just a quick update:

Parth Pancholi said:

Kevin S said:

Try reducing the CA VREF training range

1. Change DDR register configuration file with following modifications (where n could be 0, 1, 2, or 3):
   #define DDRSSn_PI_199_DATA 0x24182418
   #define DDRSSn_PI_200_DATA 0x01012418

In the meantime, I’ll try reducing the CA VRAF training range and will share the results afterward.

This change appears to improve the behavior for Issue #1.

Could you please clarify which parameter in SysConfig corresponds to these register values? In other words, how can I generate these values automatically via SysConfig?

Thank you.

Regards,
Parth P

Hello Keerthy J

I’m awaiting Kevin S's  response on the query below.

Parth Pancholi said:

Could you please clarify which parameter in SysConfig corresponds to these register values? In other words, how can I generate these values automatically via SysConfig?

While I am waiting for the answer to that, I’d appreciate your help with Issue #2, which I reported in the original post which in my opinion not related to DDRSS/RAM memory configs.

As mentioned, the issue#2 occurs during the transition from SPL to U-Boot proper, where the boot process gets stuck.
I managed to capture R5 trace logs during the failure, and they appear concerning. Based on these traces, it seems there’s a definite issue that may require attention from the TI side.

R5 Traces in RAW when issue#2 happens: /cfs-file/__key/communityserver-discussions-components-files/791/R5_5F00_Traces_5F00_200625.txt
R5 Traces Parsed when issue#2 happens: /cfs-file/__key/communityserver-discussions-components-files/791/R5_5F00_Traces_5F00_Parsed_5F00_200625.txt

Looking forward to your guidance.

Thank you.

Regards,
Parth P

Hello Keerthy J

I was able to reproduce Issue #2 two more times and have collected the corresponding R5 traces for each occurrence.
While these logs do not contain clear exception signatures like the one I previously shared, I'm including them here in case they help identify potential root causes for Issue #2.

R5 Traces in RAW when issue#2 happens:
Hit1:/cfs-file/__key/communityserver-discussions-components-files/791/R5_5F00_Traces_5F00_23062025_5F00_Failed_5F00_Try1.txt
Hit2:/cfs-file/__key/communityserver-discussions-components-files/791/R5_5F00_Traces_5F00_23062025_5F00_Failed_5F00_Try2.txt

R5 Traces Parsed when issue#2 happens:
Hit1:/cfs-file/__key/communityserver-discussions-components-files/791/R5_5F00_Traces_5F00_Parced_5F00_23062025_5F00_Failed_5F00_Try1.txt
Hit2:/cfs-file/__key/communityserver-discussions-components-files/791/R5_5F00_Traces_5F00_Parced_5F00_23062025_5F00_Failed_5F00_Try2.txt

Thank you.

Awaiting the response with an analysis followed by some proposals of possible causes and solutions.

Regards,
Parth P

Hi Keerthy J

Regarding the issue #2, where the system gets stuck during the SPL to U-Boot proper transition. enabling 'CONFIG_K3_QOS' in the U-Boot R5 configuration appears to improve the behavior. TI has made a similar change in the following commit:

https://git.ti.com/cgit/ti-u-boot/ti-u-boot/commit/?h=ti-u-boot-2024.04&id=3d51439612d1ed516e3ceb7091293e90dc71b952 

However, it's unclear and a bit surprising that the absence of this configuration could lead to boot failures and system instability.

Could you please clarify how 'CONFIG_K3_QOS' is related to this issue?
In other words, do you believe that not enabling 'CONFIG_K3_QOS' could result in such reboot stability problems ?
If so, shouldn't 'CONFIG_K3_QOS' be enabled by default, possibly via imply for 'ARCH_K3', similar to how 'TI_SECURE_DEVICE' is handled ?

Looking forward to your response.

Thanks.

Regards,

Parth P

Parth,

I will dig into this. I will get back on a couple of days. 

Best Regards,

Keerthy 

Hi Keerthy J,

Just a quick update. Even with 'CONFIG_K3_QOS' enabled, I still encountered issue #2, where the system hangs during the transition from SPL to U-Boot. This suggests that enabling 'CONFIG_K3_QOS' does not fully resolve the issue.

I was able to capture R5 trace logs during a failure, and they again show FWL exceptions, similar to previous occurrences.

Additionally, I observed that both MCU_RESETSTATZ and RESETSTATZ SoC pins remain low when this issue occurs indicating that SoC is not out of reset, while all PMIC rails appear to be at their expected voltages.

R5 Traces (with FWL exceptions):

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_R5_5F00_Trace_5F00_QOS_5F00_250625.txt

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_R5_5F00_Trace_5F00_QOS_5F00_250625_5F00_Parsed.txt

U-Boot console logs:

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_QOS_5F00_250625_5F00_Uboot.txt

As we are approaching the final stages of product development, resolving this system instability has become a high priority. I kindly request you to prioritize this issue. Please feel free to reach out if further details or testing from my side would help.

I would appreciate any insights you can provide regarding the R5 FWL exceptions.

Thank you for your understanding.

Regards,
Parth P

Parth,

Thanks for the additional details. I am looping in our security expert w.r.t firewall exception.

- Keerthy

Hi Parth,

We're having difficulty analyzing the TIFS traces you provided because there's an inconsistency between the firewall ID reporting the exception and the address range that firewall should be protecting.

For example:

According to the TISCI documentation (https://software-dl.ti.com/tisci/esd/latest/6_topic_user_guides/firewall_faq.html#how-do-i-debug-firewall-issues),,) 

when examining your traces:

Exception addr 0x45B00000
FWL Exception 0x1027F00   -> Firewall id 639
0x30000
0x1FFFEB0    -->  transaction address.
0x0
0x2201
0x10

The logs indicate that the A72 processor is attempting to write to address 0x1FFFEB0. However, the firewall raising the exception has ID 639, which according to documentation (https://software-dl.ti.com/tisci/esd/latest/5_soc_doc/j784s4/firewalls.html#list-of-region-based-firewalls)) should be protecting a completely different address range, not 0x1FFFEB0.

This mismatch makes it challenging for us to properly diagnose the issue.

How you are running test  from uboot logs sometimes it is booting till linux and in some case it is just resetting after uboot.

Regards
Diwakar

Diwakar Dhyani

What can we do to make debugging this easier and ensure we get faster to a solution? Is there any big time lost on back and forth that we can cut with some meeting? Can we send you our hardware?  Have you tried reproducing the issue on your HW (as you can read from this discussion it's not a systematic issue, it requires quite some time to reproduce)? Any other suggestion?

Hello Diwakar Dhyani,

Diwakar Dhyani said:

The logs indicate that the A72 processor is attempting to write to address 0x1FFFEB0. However, the firewall raising the exception has ID 639, which according to documentation (https://software-dl.ti.com/tisci/esd/latest/5_soc_doc/j784s4/firewalls.html#list-of-region-based-firewalls)) should be protecting a completely different address range, not 0x1FFFEB0.

Thanks for reviewing the logs and sharing your initial analysis.
It’s clear that there’s either a misconfiguration or a bug in how TISCI is functioning in the system. Given the unexpected behavior, I would not question the test setup itself. Rather, I would like to believe that low-level firmware like TISCI should operate independently of specific system use cases. If that’s not the case, it points to something that needs to be addressed.

Diwakar Dhyani said:

How you are running test  from uboot logs sometimes it is booting till linux and in some case it is just resetting after uboot.

The test setup is straightforward. A test GPIO is toggled in U-Boot, which is externally monitored by test hardware. This test hardware then triggers a power cycle after a random delay of 5 to 10 seconds. In the shared U-Boot logs, after Line #4136, there are no further attempts to boot Linux. The issue is seen at Line #91640, indicating a significant number of iterations to analyze and back trace.

Though I have limited knowledge of TISCI, I would assume that its firewall mappings should remain consistent regardless of the HLOS in use (U-Boot or Linux).

We would greatly appreciate guidance on the next steps to help resolve this issue, as outlined below.

fd said:

Diwakar Dhyani

What can we do to make debugging this easier and ensure we get faster to a solution? Is there any big time lost on back and forth that we can cut with some meeting? Can we send you our hardware?  Have you tried reproducing the issue on your HW (as you can read from this discussion it's not a systematic issue, it requires quite some time to reproduce)? Any other suggestion?

Looking forward to your response.

Thank you.

Regards,
Parth P

Hello Diwakar Dhyani,

I have another set of R5 trace logs with FWL exceptions that might provide further insight.
In this test, both CONFIG_K3_QOS and lockstep mode for R5F were enabled, aligning our platform more closely with TI’s J784S4 EVM configuration.
With these changes, the R5 traces revealed more specific and concerning exceptions, as shown below.

FWL Bit  0x1D
Exception addr  0x45B0B800
FWL Exception  0x1141400
0x00030000:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x030000
0x706F9C10: Power Management/FAIL(Action failed):   DEVICE_OFF(Device has been Turned OFF): Device ID: 3120144
0x00000000:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x000000
0x00C82201:   BasePort: Unknown Action: 0x03 MSG:0x082201
0x00000010:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x000010
...
...
FWL Bit  0x1E
Exception addr  0x45B0B000
FWL Exception  0x1112000
0x00070000:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x070000
0x30C07E10:   Security/FAIL(Action failed): SEC_BOOT(Points of failures during secure boot api call): 0x01 => Certificate length > ASN1P_IMAX, 0x02 => Issue fetching certificate, 0x3 => Issue with Hash operation, 0x4 => Hash comparison fails: 0
0x00000000:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x000000
0x00002201:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x002201
0x00000004:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x000004

FWL Bit  0x1E
Exception addr  0x45B0B000
FWL Exception  0x1112000
0x00070000:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x070000
0x30B1FE10:   Security/FAIL(Action failed): SEC_INIT(Security post boardconfig initialization): Start(0) or End(1): 0
0x00000000:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x000000
0x00002201:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x002201
0x00000004:   BasePort: INIT_COMPLETE(OSAL/Baseport init complete): MSG:0x000004
...
...

R5 Traces (with FWL exceptions):

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_R5_5F00_Traces_5F00_Parsed_5F00_26062025.txt

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_R5_5F00_Traces_5F00_26062025.txt

U-Boot console logs:

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_U_2D00_boot_5F00_26062025.txt

Here are the corresponding U-Boot source details for our hardware

• Repo: https://git.toradex.com/cgit/u-boot-toradex.git/?h=toradex_ti-u-boot-2024.04
• Board cfg files: https://git.toradex.com/cgit/u-boot-toradex.git/tree/board/toradex/aquila-am69?h=toradex_ti-u-boot-2024.04
• U-boot DTs:
  • https://git.toradex.com/cgit/u-boot-toradex.git/tree/arch/arm/dts/k3-am69-r5-aquila-dev.dts?h=toradex_ti-u-boot-2024.04
  • https://git.toradex.com/cgit/u-boot-toradex.git/tree/arch/arm/dts/k3-am69-aquila-dev-binman.dtsi?h=toradex_ti-u-boot-2024.04
  • https://git.toradex.com/cgit/u-boot-toradex.git/tree/arch/arm/dts/k3-am69-aquila-dev-u-boot.dtsi?h=toradex_ti-u-boot-2024.04 

Looking forward to your response.

Thank you.

Regards,
Parth P

Hi Parth,

Allow me some time to check.

One more question are you seeing similar behavior with TI EVM as well?

Regards
Diwakar

Hello Diwakar Dhyani,

Diwakar Dhyani said:

are you seeing similar behavior with TI EVM as well?

I don’t have access to the J784S4 EVM, so I can’t comment on its behavior.

On the AM69-SK, I did observe system instability in the past (quite some time ago) related to DDRSS issue #1. However, given the nature of the current issue, I believe it has little to no dependency on the hardware itself.
Our software stack is very closely aligned with TI’s, and I’ve personally reviewed configuration differences between the TI EVM and our platform without finding anything concerning.
That said, I would really appreciate a closer review by one of the TI experts to be doubly sure. (The source links are shared in the previous response)

Feel free to reaching me out in case of any further information.

Thank you.

Regards,
Parth P

HI Parth,

i analysed TIFS traces , there are multiple firewall exception and after that we are seeing that authentication failure, which is very unlikely as the authentication is already passed.So i assume there is some issue with traces.

I think we need to figure out why A72 is doing these random transaction in case of  failing case, due to these random transaction we are seeing firewall exception.

Also still there a mismatch between the firewall exception address and the firewall id in the logs which you shared.

From uboot logs it seems that authentication for uboot and dtb is passing, but when uboot start executing on A72 the issue seems to appear.

Regards
Diwakar

Hi Diwakar Dhyani,

Diwakar Dhyani said:

So i assume there is some issue with traces.

Could you please elaborate on what might be wrong with the current traces ?
In other words, how can we ensure that we’re capturing the proper R5F traces? Below are the changes I’ve made to enable R5F trace logging.

diff --git a/board/toradex/aquila-am69/board-cfg.yaml b/board/toradex/aquila-am69/board-cfg.yaml
index 17b517118d3e..9a37360d612e 100644
--- a/board/toradex/aquila-am69/board-cfg.yaml
+++ b/board/toradex/aquila-am69/board-cfg.yaml
@@ -33,5 +33,5 @@ board-cfg:
         subhdr:
             magic: 0x020C
             size: 8
-        trace_dst_enables: 0x00
-        trace_src_enables: 0x00
+        trace_dst_enables: 0x0d
+        trace_src_enables: 0x3f

Diwakar Dhyani said:

Also still there a mismatch between the firewall exception address and the firewall id in the logs which you shared.

Could the issue be related to TISCI itself, or possibly due to how the traces are being collected?
How can we ensure we're capturing accurate traces? I’ve attempted attaching JTAG to both R5 and A72 at the time of failure, but without success.

Diwakar Dhyani said:

when uboot start executing on A72 the issue seems to appear.

This is perfectly clear and well understood.

Thank you.

Regards,
Parth P

Hi Parth,

can you add a while loop in board_init_f function in  common/board_f.c file and re-compile uboot. Using JTAG load uboot symbol file and see if it even reaching that or not.
Something like this.

uboot symbol file located  under :board-support/<ti-u-boot>/build/a72/u-boot

As we are seeing firewall exception before even getting the uboot version print.Just wanted to check what exactly it is executing.

Regards
Diwakar

Hello Diwakar Dhyani,

I followed your suggestion, but it appears that execution doesn't even reach board_init_f().

Diwakar Dhyani said:

As we are seeing firewall exception before even getting the uboot version print.Just wanted to check what exactly it is executing.

From the snippet above, it seems the system is getting stuck in the vsnprintf_internal() function within lib/vsprintf.c, which aligns with the observed blank console when U-Boot proper is expected to print its version information.

Thank you.

Regards,
Parth P

Hi Parth 

Version print will come after console init in  board_init_f(). function that strange that is stuck in print before that.

Is this getting stuck at the same location every time, i.e at vsnprintf_internal() ?

Regards
Diwakar

Hello Diwakar Dhyani,

Diwakar Dhyani said:

Is this getting stuck at the same location every time, i.e at vsnprintf_internal() ?

It's difficult to say for sure. I was fortunate to catch a system stuck at the issue and ready for debugging and share observations to you.
However, the system doesn't always hang at the same point. For instance, in one case, the SoC hadn't come out of reset when the issue occurred, which made it impossible to attach the JTAG. (Please check below response.)

Parth Pancholi said:

Additionally, I observed that both MCU_RESETSTATZ and RESETSTATZ SoC pins remain low when this issue occurs indicating that SoC is not out of reset, while all PMIC rails appear to be at their expected voltages.

R5 Traces (with FWL exceptions):

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_R5_5F00_Trace_5F00_QOS_5F00_250625.txt

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_R5_5F00_Trace_5F00_QOS_5F00_250625_5F00_Parsed.txt

U-Boot console logs:

/cfs-file/__key/communityserver-discussions-components-files/791/Aquila_5F00_AM69_5F00_QOS_5F00_250625_5F00_Uboot.txt

Thank you.

Regards,
Parth P

Hi Parth 

Parth Pancholi said:

It's difficult to say for sure. I was fortunate to catch a system stuck at the issue and ready for debugging and share observations to you.
However, the system doesn't always hang at the same point.

How much is the rate at which you are seeing this firewall issue vs any other issue ? 

Also how much time it take to reproduce this issue ?

Regards
Diwakar

Hello Diwakar Dhyani,

Diwakar Dhyani said:

How much is the rate at which you are seeing this firewall issue vs any other issue ? 

In all observed cases during long-duration power cycle tests (100% of time), the failure consistently results in an FWL exception - there have been no other types of issues seen now.
I just wanted to re-emphasize this point in case it got lost across our previous exchanges, especially since the system behavior varies noticeably when the FWL exception occurs.

Diwakar Dhyani said:

Also how much time it take to reproduce this issue ?

It typically takes just one overnight power cycle test (around 12 to 14 hours) to encounter the issue, and it can occur at any point during that time-frame.

Thank you.

Regards,
Parth P

Parth,

Have you tried slowing DDR frequency down to see if that has any impact? As an example, maybe try 3733 or 3200 MT/s data rate.

A new DDR DTSI file using the register configuration tool would be required to reduce the frequency. 

This suggestion is because DDR issues were previously identified, and while the initialization failures seem to be resolved, it is possible the current issue may be related to marginal DDR READ / WRITE issues. If the issue is resolved by reducing the DDR frequency, then we can focus the attention on the DDR interface. If this experiment has no impact on the failure, then it is possible the DDR interface is not related.

Regards,
Kevin