The issue is in the ipc_rpmsg_echo_linux project running on the DM R5F core. Specifically, in the "IPC Echo" task, which runs on the DM R5F alongside the DM task. Here is the problematic code:

    /* This API MUST be called by applications when its ready to talk to Linux */
    status = RPMessage_waitForLinuxReady(SystemP_WAIT_FOREVER);
    
    DebugP_assert(status==SystemP_SUCCESS);

A bug in RPMessage_waitForLinuxReady allows the function to exit after 49 days if Linux never initializes the RPMsg infrastructure (where the “correct” behavior is to loop forever inside RPMessage_waitForLinuxReady). When DebugP_assert catches the error, it crashes the DM R5F.

Details

Linux must initialize the RPMsg infrastructure before a non-Linux core can do RPMsg inter-processor communication (IPC) with Linux. The "IPC Echo" task uses function RPMessage_waitForLinuxReady to wait for the signal that the RPMsg IPC infrastructure has been initialized. If Linux never initializes the RPMsg infrastructure with the DM R5F, then the "IPC Echo" task should stay in the RPMessage_waitForLinuxReady function forever.

However, the RPMessage_waitForLinuxReady function compares the RTOS 1msec tick counter value against the argument that is passed in. If the tick counter ever equals the argument, the function times out and exits.

SystemP_WAIT_FOREVER = 0xFFFF_FFFF. A 1msec counter reaches 0xFFFF_FFFF after 49 days. At that point, RPMessage_waitForLinuxReady returns with status = SystemP_TIMEOUT.

DebugP_assert catches the error (status = SystemP_TIMEOUT != SystemP_SUCCESS). Then DebugP_assert freezes the entire DM R5F core (both the "IPC Echo" task and the DM task) by disabling interrupts and entering an infinite loop.

+1 Nick Saulnier 10 months ago

TI__Guru** 108410 points

Is this a silicon bug?

No. This crash is caused by firmware code, not by hardware.

+1 Nick Saulnier 10 months ago

TI__Guru** 108410 points

When was the issue fixed?

AM62x, AM62Ax, AM62Dx, AM62Px

The issue is fixed starting in SDK 11.0 and at https://git.ti.com/cgit/processor-firmware/ti-linux-firmware/tree/ti-dm?h=ti-linux-firmware after tag 11.00.00.

In this version of the firmware, function RPMessage_waitForLinuxReady tests the argument that is passed in. If argument = SystemP_WAIT_FOREVER, the function will not exit unless the Linux RPMsg infrastructure has been initialized.

AM67, AM67A

The issue is fixed starting in SDK 11.1 and at https://git.ti.com/cgit/processor-firmware/ti-linux-firmware/tree/ti-dm?h=ti-linux-firmware starting with this commit.

Note that AM67 & AM67A use the firmware binary at ti-linux-firmware/ti-dm/j722s

+1 Nick Saulnier 10 months ago

TI__Guru** 108410 points

Our design meets all the conditions for a crash, but we do not want to update the DM R5F firmware. Will our system will actually crash after running for 49 days?

In our tests, the DM R5F core in an affected system crashed 100% of the time after the 1msec counter value reached SystemP_TIMEOUT = 0xFFFF_FFFF.

+1 Nick Saulnier 10 months ago

TI__Guru** 108410 points

The A53 cores in my design are not running Linux. Could my design still be affected?

Yes.

Let's assume that your design meets all of the other conditions listed in [FAQ] [Alert] DM R5F can crash in certain conditions: AM62x, AM62Ax, AM62Dx, AM62Px, AM67, AM67A . So we will just be looking at the condition "RPMsg IPC communication is not initialized between the DM R5F and Linux".

The DM R5F IPC Echo firmware is waiting for a message in the resource table indicating that someone has initialized the RPMsg infrastructure that would typically be used to communicate with Linux. If the DM R5F does not receive that message after 49 days, then the DM R5F would crash, regardless of whether the A53 cores were running Linux, MCU+ SDK, or something else.

+1 Nick Saulnier 10 months ago in reply to Nick Saulnier

TI__Guru** 108410 points

What are typical scenarios where RPMsg IPC communication is not initialized between the DM R5F and Linux?

DM R5F uses a different communication protocol to send and receive device management and power management messages with the other cores. So the RPMsg protocol is not actually used by the DM task.

There are multiple reasons that a design might not initialize RPMsg communication between Linux and the DM R5F, such as:

Remoteproc & RPMsg kernel drivers are disabled in the Linux kernel configs
The DM R5F node has been removed or set "status = disabled" from the Linux devicetree file (this does not impact DM task operation as long as the DM R5F memory allocations in the Linux devicetree file are preserved)
The 1MB memory allocation for the RPMsg VIRTIO buffers between Linux and DM R5F have been removed from the Linux devicetree

+1 Nick Saulnier 7 months ago

TI__Guru** 108410 points

We are writing custom DM R5F firmware. Could our design be affected?

If you are writing custom firmware, you will not be affected by this specific bug unless you add RPMsg IPC code to the DM R5F firmware, and then do not initialize the RPMsg infrastructure.

HOWEVER, it is still possible for your code to crash the DM task. Please find some best practices for writing custom DM R5F firmware at [FAQ] How to avoid crashing the DM R5F when writing custom DM R5F firmware .