• State TI Thinks Resolved
  • Locked Locked
  • Replies 6 replies
  • Answers 1 answer
  • Subscribers 100 subscribers
  • Views 343 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM6441: Industrial Comms SDK, EthernetIP and Tunneling random crash

Philip Kees
Prodigy 145 points

Part Number: AM6441

Hi,

we have a AM64 device that uses EthernetIP from the Industrial Comms SDK on an R5 core and Linux on the A53 core. 
Our application is different from the TI example since our fieldbus application runs on Linux, and is connected via RpMsg to the EthernetIP application on the R5 core. 
This worked without issues. 

After adding the raw ethernet tunneling between the R5 and A53 cores based on the TI tunneling example from the SDK, we observed random crashes on the R5 during the start of the application, producing the following logs on Linux side:
image.png

This happens quite unreliable (Test system with ~10 devices constantly rebooting lead to at least one case during a day of testing)

Since we already used RpMsg on the R5 core for the fieldbus IPC, there was already a call to waitForLinuxReady there. When integrating the TI tunneling example, a second call to waitForLinuxReady was added in the tunnel task.
It seems that waitForLinuxReady does not only wait, but also reset the vrings between R5 and A53, so we removed the call from the tunnel task and are simply waiting on a semaphore until the other call to waitForLinux proceeds. 
With this change we did not see any further crashes over the course of several days. 

For me it is not clear how two calls to waitForLinuxReady could cause this, since it was guaranteed by our application that no endpoints are announced to Linux before both waitForLinux calls were done and therefore no RpMsg communication happended that could affect the vring states(?).

Since we couldn't reliably reproduce the crash even before this change, we can not be sure if it is reliably fixed. It would be great if someone with deeper knowledge of the RpMsg drivers could point us to the mechanism that might cause the behavior seen in the screenshot, when 2 calls to waitForLinuxReady reset the vrings twice.

Thank you and kind regards
Philip Kees

  • 0
    TI__Expert 3640 points

    Hi Philip

    We have re-assigned the thread to the relevant expert. Please ping the thread again if there's no response in a couple of days.

    Regards
    Archit

  • 0
    TI__Guru** 110030 points

    Hello Philip,

    Since I am primarily a Linux guy rather than an MCU+ guy, I will give some initial thoughts, and then pass your thread over to another team member to discuss.

    Is there a specific reason that that you need to have two separate waitForLinuxReady calls?

    Even if your R5F code has multiple tasks, I would expect it to have a single resource table file for the entire R5F instance. Our ipc_rpmsg_echo_linux example puts waitForLinuxReady in the beginning of the main function, and then spawns off the individual tasks after Linux confirms that it is ready.
    https://github.com/TexasInstruments/mcupsdk-core/blob/next/examples/drivers/ipc/ipc_rpmsg_echo_linux/ipc_rpmsg_echo.c 

    I would try the code with a single waitForLinuxReady call, either before the different tasks are started, or by doing something like your system where one task waits for waitForLinuxReady to return, and then the other task does not try to initialize RPMsg endpoints until after the first task signals to it that Linux is ready.

    But why is my system crashing though?

    I would expect that inconsistent behavior might be related to a race condition somewhere. But I am not sure where to look based on your description.

    Some thoughts that may have nothing to do with your usecase:

    (rscTable->vdev.status == 0x7U) is used by waitForLinuxReady. If Linux sets the value, one MCU+ task reads this value, and then modifies the value before the other MCU+ task can read the value, that could force the task that runs second to get stuck in an infinite loop within the waitForLinuxReady function.

    If there are other dependencies between your tasks, having the tasks start running at slightly different times could cause issues if you don't have ways to enforce order of execution.

    Anything else? 

    I have not spent any time with this rpmsg_eth code, but please check your code for sending data. "incoming message too big" makes me concerned you are losing data. Remember that each RPMsg message between Linux + MCU+ core is limited to 496 bytes (512 bytes - 16 Byte header)

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    TI__Intellectual 1430 points

    Hello,
    Thank you for your query. The concerned expert is Out of Office due to **TI India** Holiday.
    Please expect a delay in response. We appreciate your patience and understanding.

    Best regards,
    TI E2E Support Team
    ---
    *This is an automated notification.*

  • 0
    TI__Guru 63078 points
    Philip Kees said:
    it was guaranteed by our application that no endpoints are announced to Linux before both waitForLinux calls were done and therefore no RpMsg communication happended that could affect the vring states(?).

    How are you making sure that the RPMsg only happens after both the waitForReadyLinux call finish execution?

    Is it possible in your application, that task1 comes out of the waitForReadyLinux() call and start doing IPC, meanwhile the task2 is still waiting inside waitForLinuxReady() call to finish?

    As the WaitForReadyLinux() API reset the VRING buffer on Linux side. Suppose the task1 started IPC and changed the index of vring buffer, but task2 after waitForReadyLinux call resets the vring buffer index. The crash could be happening because of this inconsistency of the Vring buffers index.

    Regards,

    Tushar

  • 0 in reply to Tushar Thakur
    Prodigy 145 points

    Hi Tushar,

    Tushar Thakur said:
    How are you making sure that the RPMsg only happens after both the waitForReadyLinux call finish execution?

    Let’s consider the following scenario:

    • Task A (priority 3) starts Task B (priority 6).
    • Task B gets scheduled immediately and enters waitForLinuxReady.
    • After waitForLinuxReady, Task B waits on a semaphore that Task A must signal.
    • Task A then reaches waitForLinuxReady.
    • Once Linux is ready, Task A announces its endpoints and starts communicating.
    • Task A signals the semaphore for Task B.
    • Task B resumes and announces its endpoint.

    This setup should work in both situations:

    1. Linux is not ready yet → both tasks block in waitForLinuxReady.
    2. Linux is already ready → Task B passes waitForLinuxReady immediately and waits on the semaphore.

    In theory, because Task B has higher priority, it should always reach the semaphore before Task A finishes waitForLinuxReady. That ensures no endpoints get announced before any potential vring reset caused by waitForLinuxReady.

    After thinking it through again, the only case where this ordering might break is the following timing edge case:

    • Task B blocks in waitForLinuxReady for exactly one FreeRTOS tick because Linux is not ready.
    • Linux becomes ready during that tick, before Task A reaches waitForLinuxReady.
    • This gives Task A a window of up to one FreeRTOS tick (1 ms) to proceed, announce endpoints, and start communication before Task B gets scheduled and triggers the vring reset.

    This window is very slim, which might explain why the issue happens only rarely across many (hundreds to thousands) successful reboots.

    Since you mentioned that resetting the vrings after IPC has started could produce the behavior we’re seeing, this timing scenario seems plausible to me.

    If this matches your understanding as well, I think we can consider the question resolved.

    Kind regards
    Philip

  • 0 in reply to Philip Kees
    TI__Guru 63078 points

    Yes, The above does explains the behavior which you are observing. As it is very rare to meet the timing sequence, the reproducibility of the issue is random or rare.

    Instead of calling the WaitForLinuxReady API twice from two different task, do the following.

    1. Call the WaitForLinuxReady from TaskB which is highest priority.
    2. Signal a semaphore from TaskB after WaitForLinuxReady API call.
    3. Inside TaskA wait for semaphore to be signaled by TaskB.

    The above sequence will make sure that WaitForLinuxReady API is called only once and proper synchronization is also achieved between the tasks.

    Hope the above helps. 

    Regards,

    Tushar