• State TI Thinks Resolved
  • Replies 3 replies
  • Answers 1 answer
  • Subscribers 100 subscribers
  • Views 153 views
  • Users 0 members are here
Support feedback
Options
Options
Related

J722SXH01EVM: J722S + TI RTOS SDK 11.02.00.11 DSP crash under non-cached read-modify-write stress

Delio Brignoli
Intellectual 685 points

Part Number: J722SXH01EVM

On our production hardware (J722S) sustained r/w access to a non-cached memory area can lock up the DSP.

We have been developing with this hardware for more than year and have been running a release candidate DSP firmware for a while with no issue until recently when we changed increased the speed at which we exchange data between Linux and the DSP. The lock up in production can present itself in various ways: DSP exception dump in the debug trace, binary garbage in DSP debug trace, silent DSP crash (Linux still running), total lockup of Linux and DSP. Total lockup of Linux and DSP disappeared after removing access to the entire DDR from the DSP.
1st:
[c75ss0] 75.614277s : Exception at 0x0000000000000000
[c75ss0] 75.614289s : TSR at time of exception: 0x0000000000000000
[c75ss0] 75.614301s : Internal exception:
[c75ss0] 75.614310s : IERR=0x0000000000002000
[c75ss0] 75.614319s : IEAR=0x00000000a3536464
[c75ss0] 75.614339s : Address exception
[c75ss0] 75.614350s : Out-of-range address on DMC memory address
2nd:
[c75ss0] 77.987807s : Exception at 0x0000000000000000
[c75ss0] 77.987857s : IEAR=0x00000000a35360a4
[c75ss0] 77.987889s : Out-of-range address on DMC memory address

Decoded on one real image map: IEAR=0xa3536464 ≈ vTaskSwitchContext+0x164 (faulting load of ListItem_t.pvOwner via a bad/NULL pxIndex?) but this may not be relevant because the issue results in overwriting large chunks of memory so the task list corruption may just be one possible consequence.

I have distilled the issue down to a minimal firmware that reproduces the issue (attached). The firmware sets up an unused RPMsg endpoint and then hammers a non-cached memory area with rmw access. The RPMsg endpoint seems required to make the issue visible. My suspicion is that the issue may exist independently of RPMsg, but once triggered with RPMsg endpoint configured it results in widespread memory corruption which makes the issue impossible to ignore.

Is this a known issue? Can someone help with resolving it? Thank you.
  • 0
    TI__Genius 17600 points

    Hi,

    Decoding the exception register.

    1.IERR: Indicates it is a address out of range excpetion.

    2.Out of range error on DMC, indicating "• Out-of-range address on L1D memory address".

    Here the issue is , you are performing read m write on a non cacheable address. so this seems weird.

    1.Is the issue consistent on all the boards??.

    2.Could you post your MMU mapping.

    3.Is the issue reproducible on Our j722s EVM or only in your custom board.

    [Edited]:

    4.Is issue reproduced if only write is being done??.

    5.Are there any other master accessing the same region??.

    Regards,

    Manojna

  • 0
    TI__Intellectual 2850 points

    Hi Delio,

    Delio Brignoli said:
    Is this a known issue? Can someone help with resolving it?

    We have reproduced this on a stock J722S EVM and confirmed the fix, using your reproduction firmware unmodified apart from one line.

    Root cause is the C75x interrupt-context overrun. The hardware writes the interrupt context 8 KiB upward from TCSP, and the FreeRTOS port derives each task's TCSP from that task's own stack buffer. When the buffer is not 8 KiB aligned the port takes a branch intended to maximise usable stack and places TCSP at the top of the aligned region, leaving less than 8 KiB of real buffer above it. The context save then writes past the end of the array. Nothing checks it.

    On your wedge build the hammer stack starts at A3414180 and ends at A3424180. We read TCSP directly from the CPU and it is A3424000, only 180 bytes of buffer above it, against a context dump of about 4C0. We watched the task's own task object at A3424180 go from all zeros before the run to saved register values after it, so the TCB is overwritten by the context save. Garbage TCB gives garbage ready-list pointers, which is why your exception decodes into the scheduler. The core then dies at soft_reset.

    Rebuilding with only the stack alignment attribute changed to 8 KiB, and the same failing BSS pin order, TCSP is still A3424000 but the buffer now ends at A3426000, so the full 8 KiB is inside it. The task object stays valid and the firmware logs rounds= continuously. That single attribute is the whole difference between a dead DSP and one running normally.

    Recommended action is to align every task stack buffer to 8 KiB, not just the two in the reproduction. We are following up internally on the SDK side and on the API guidance, which currently recommends 32-byte alignment and is not sufficient on C75x.

    Regards,
    Shabary S Sundar