Dear TI experts,
we have a problem with SBL and SciClient using TI RTOS SDK7.3.
We know that this SDK is deprecated but we cannot go with SDK8.x because we need an RTOS at A53 (no Linux).
The issue appeared when we migrated from SDK6.3 to SDK7.3, using our custom board.
It is a sporadic issue: in average, about every ~500 boot cycles some boards do not boot.
Instead, boot phase is "frozen" and after about 180 seconds, the ROM bootloader starts rebooting with secondary boot medium, which is in our case UART. First boot medium is QSPI from NOR flash, using the OSPI periphieral.
In some cases the ROM bootloader does not trigger, but the SBL somehow gets stuck endlessly.
We already spend a lot of time (!) on this issue and tried to drill it down via UART_printf() inside SBL.
This issue usually appears in one of 2 functions:
1. either in Sciclient_procBootGetProcessorState() called by SBL_ConfigMcuLockStep(false)
2. or in SBL_ReleaseCore() called by SBL_ReleaseAllCores()
The most seen effect is that the SciClient function just doesn't return until the ROM bootloader watchdog triggers.
If the issue appears in the 1. or the 2. function listed above seem to depend on how many UART_printf() we added.
So it sounds also like a timing issue, since each UART_printf() adds some delay.
When the issue is in the first mentioned function I was able to deeper analyze:
Sciclient_procBootGetProcessorState() calls Sciclient_serviceSecureProxy(), which has certain while() loops.
The second while() loop is waiting for the answer from DMSC:
int32_t Sciclient_serviceSecureProxy(const Sciclient_ReqPrm_t *pReqPrm,
Sciclient_RespPrm_t *pRespPrm)
{
...
/* Check if some message is received*/
while (((HW_RD_REG32(Sciclient_threadStatusReg(rxThread)) &
CSL_SEC_PROXY_RT_THREAD_STATUS_CUR_CNT_MASK) - initialCount) <= 0U)
{
if (timeToWait > 0U)
{
timeToWait--;
if (g_u32DebugVerboseFlag) UART_printf("SciW2x loop\n");
}
else
{
status = CSL_ETIMEOUT;
break;
}
}
...
}
In the normal case this loop is not entered at all because the while() condition is initially false.
But in the error case this loop is looping 'endlessly' until the ROM bootloader watchdog triggers.
I proved this with a UART_printf() inside the loop and in the error case I get for 180 seconds the UART loop output and then the reset occures followed by UART chip ident string and 'CCCC...'.
The tricky thing is that it appears quote sproadic, so that always an endurance test of multiple hours is needed which switches a device off/on continuously.
We also have complete systems with 20 or more devices, but there it is not possible to connect UARTs. But with such complete systems it is much more likely to reproduce the issue. Typically is appears on some device much more often than on others. On some devices it seem not to appear at all.
To ensure that we have no hardware issue we run the same systems with an older firmware with SDK6.3 and then everything is absolutely stable: No issue during 5 days continuously switching off/on the whole system (~ 100.000 total boots)
We already analyzed some potential general root causes:
- Do we have an issue when reading from NOR flash? Answer is no. Checked by copying image to DDR, then doing checksums, then booting image from DDR instead via OSPI FFS.
- Do we have a problem with DDR RAM access? Normally DDR RAM is not used by SBL. We also did RAM tests.
- Do we have a problem by starting A53 firmware? No, checked via several methods, e.g. a very early A53 boot delay function.
- Do we have an SBL internal memory range issue? No, checked linker file and map files.
Now some questions to TI:
- Is there a known issue in DMSC firmware in PDK 7.3 for AM65x?
- Is there a known issue in SciClient library in PDK 7.3 for AM65x?
- How could we further drill down this issue?
- Would it make sense to use the TISCI "Trace Layer" feature? If yes, could you support us here?
Regards, Ruediger
