SK-AM62B-P1: How to enable graceful shutdown of remote cores with multithreading in FreeRTOS threads

Hello Shreyan,

I will be commenting from the Linux side.

From the Linux side, it would look exactly the same:

Linux sends a shutdown message to the M4F core, and then Linux waits to get an ACK signal from the M4F core before shutting it down. If Linux does not receive an ACK signal within a couple of seconds, it times out and does NOT power the core off (that's the "graceful" part of the shutdown - we only shut off a core when the core is in a known good state).

So from a structural standpoint, I assume the M4F code would want every task to get to a known good state, and then send the ACK signal after every task is in a known good state.

Take a look at the ipc_rpmsg_echo_linux code for an example  

Here's the file I'm looking at:

mcu_plus_sdk_am62x_09_02_01_06$ cd examples/drivers/ipc/ipc_rpmsg_echo_linux
mcu_plus_sdk_am62x_09_02_01_06/examples/drivers/ipc/ipc_rpmsg_echo_linux$ vi ipc_rpmsg_echo.c

We create multiple tasks (though the multiple tasks use the same task function):

void ipc_rpmsg_create_recv_tasks()
{
...
    /* Create the tasks which will handle the ping service */
    TaskP_Params_init(&taskParams);
    taskParams.name = "RPMESSAGE_PING";
    taskParams.stackSize = IPC_RPMESSAGE_TASK_STACK_SIZE;
    taskParams.stack = gIpcTaskStack[0];
    taskParams.priority = IPC_RPMESSAGE_TASK_PRI;
    /* we use the same task function for echo but pass the appropiate rpmsg handle to it, to echo messages */
    taskParams.args = &gIpcRecvMsgObject[0];
    taskParams.taskMain = ipc_recv_task_main;

    status = TaskP_construct(&gIpcTask[0], &taskParams);
    DebugP_assert(status == SystemP_SUCCESS);

    TaskP_Params_init(&taskParams);
    taskParams.name = "RPMESSAGE_CHAR_PING";
    taskParams.stackSize = IPC_RPMESSAGE_TASK_STACK_SIZE;
    taskParams.stack = gIpcTaskStack[1];
    taskParams.priority = IPC_RPMESSAGE_TASK_PRI;
    /* we use the same task function for echo but pass the appropiate rpmsg handle to it, to echo messages */
    taskParams.args = &gIpcRecvMsgObject[1];
    taskParams.taskMain = ipc_recv_task_main;

    status = TaskP_construct(&gIpcTask[1], &taskParams);
    DebugP_assert(status == SystemP_SUCCESS);

and we can see that there is code in the ipc_recv_task_main task function to ONLY send the ack and close all drivers after each receive task has exited:

void ipc_recv_task_main(void *args)
{
...
    /* wait for messages forever in a loop */
    while(1)
    {
        /* set 'recvMsgSize' to size of recv buffer,
        * after return `recvMsgSize` contains actual size of valid data in recv buffer
        */
        recvMsgSize = IPC_RPMESSAGE_MAX_MSG_SIZE;
        status = RPMessage_recv(pRpmsgObj,
            recvMsg, &recvMsgSize,
            &remoteCoreId, &remoteCoreEndPt,
            SystemP_WAIT_FOREVER);

        if (gbShutdown == 1u)
        {
            break;
        }
...
    }

    gRecvTaskExitCounter++;
    // this counter means that only the last task to exit will close the drivers
    // and send the shutdown ACK
    if (gRecvTaskExitCounter >= IPC_RPMESSAGE_NUM_RECV_TASKS)
    {
        /* Follow the sequence for gracefull shutdown for the last recv task */
        DebugP_log("[IPC RPMSG ECHO] Closing all drivers and going to WFI ... !!!\r\n");

        /* Close the drivers */
        Drivers_close();

        /* deinit system */
        System_deinit();

        /* ACK the suspend message */
        IpcNotify_sendMsg(gbShutdownRemotecoreID, IPC_NOTIFY_CLIENT_ID_RP_MBOX, IPC_NOTIFY_RP_MBOX_SHUTDOWN_ACK, 1u);
#if (__ARM_ARCH_PROFILE == 'R') ||  (__ARM_ARCH_PROFILE == 'M')
        /* For ARM R and M cores*/
        __asm__ __volatile__ ("wfi"   "\n\t": : : "memory");
#endif
#if defined(BUILD_C7X)
        asm("    IDLE");
#endif
    }
    vTaskDelete(NULL);
}

What I do not understand 

I do not understand the logic of how we actually get to that shutdown code though.

1) There is a separate mailbox callback - when does this get called, and when does code in ipc_recv_task_main get called?

void ipc_rp_mbox_callback(uint16_t remoteCoreId, uint16_t clientId, uint32_t msgValue, void *args)
{
    if (clientId == IPC_NOTIFY_CLIENT_ID_RP_MBOX)
    {
        if (msgValue == IPC_NOTIFY_RP_MBOX_SHUTDOWN) /* Shutdown request from the remotecore */
        {
            gbShutdown = 1u;

2) I think the code in ipc_recv_task_main only tests if (gbShutdown == 1u) after RPMessage_recv() returns from receiving a message. But if that is true... how does that work with shutdown messages?

My assumption is that we should only be exiting RPMessage_recv() when we receive an RPMsg for that specific endpoint:
I think
task "RPMESSAGE_PING" should only exit RPMessage_recv when it receives an RPMsg for endpoint 13.
task "RPMESSAGE_CHAR_PING" should only exit RPMessage_recv when it receives an RPMsg for endpoint 14.

However, it looks like the Linux driver code only sends a mailbox, NOT an RPMsg, when it sends a shutdown message:
linux kernel, drivers/remoteproc$ vi ti_k3_m4_remoteproc.c

/*
 * Stop the M4 remote processor.
 *
 * This function puts the M4 processor into reset, and finishes processing
 * of any pending messages. This callback is invoked only in remoteproc mode.
 */
static int k3_m4_rproc_stop(struct rproc *rproc)
{
        unsigned long to = msecs_to_jiffies(3000);
        struct k3_m4_rproc *kproc = rproc->priv;
        u32 msg = (u32)(uintptr_t) RP_MBOX_SHUTDOWN;
        struct device *dev = kproc->dev;
        int ret;
        u32 stat = 0;

        reinit_completion(&kproc->shut_comp);
        ret = mbox_send_message(kproc->mbox, (void *) (uintptr_t)msg);
        if (ret < 0) {
                dev_err(dev, "PM mbox_send_message failed: %d\n", ret);
                return ret;
        }

        ret = wait_for_completion_timeout(&kproc->shut_comp, to);
        if (ret == 0) {
                dev_err(dev, "%s: timedout waiting for rproc completion event\n", __func__);
        };

        mbox_free_channel(kproc->mbox);

        ret = readx_poll_timeout(is_core_in_wfi, kproc, stat, stat, 200, 2000);
        if (ret)
                return ret;

        k3_m4_rproc_reset(kproc);

        return 0;
}

I am sending your thread to another team member who knows more about MCU+ SDK to comment more on my questions 1) and 2)

Regards,

Nick

Hi Shreyan,

Nick Saulnier said:

1) There is a separate mailbox callback - when does this get called, and when does code in ipc_recv_task_main get called?

When Linux tries to power off remote cores, it first sends the IPC message to remote core.

To receive and handle this IPC message we register a callback(i.e. ipc_rp_mbox_callback) in remote core. This callback will be called when IPC message is sent from the Linux to remote core.

On the callback we check for the IPC message and if it for Shutdown, unblock the RPMessage for all the RPMsg objects used in the code.

On the main thread where the IPC is happening, break all the loops when gbShutdown == 1. Close all the drivers and send an acknowledgment to the Linux core that core is ready for shutdown. 

Deinit the system and go to WFI/IDLE mode.

Please refer the ipc_rpmsg_echo_linux example that implements the graceful shutdown method.

Hope the above information helps.

Regards,

Tushar

Tushar & Nick:

 There is valuable information in this exchange -- even for AM64x. I will also mention to Bin to see how this can be rolled into AM64x Academy.

later

Jim

Hello Jim,
Thanks for the suggestion - I have already tagged this information to get added to the multicore module.

Hello Tushar,

Thanks for the explanation. So what I was missing was these lines of code, which forces the RPMessage_recv() to return for each of the tasks that has been opened:

void ipc_rp_mbox_callback(uint16_t remoteCoreId, uint16_t clientId, uint32_t msgValue, void *args)
{
    if (clientId == IPC_NOTIFY_CLIENT_ID_RP_MBOX)
    {
        if (msgValue == IPC_NOTIFY_RP_MBOX_SHUTDOWN) /* Shutdown request from the remotecore */
        {
            gbShutdown = 1u;
            gbShutdownRemotecoreID = remoteCoreId;
            // unblock the RPMessage_Object for task RPMESSAGE_PING
            RPMessage_unblock(&gIpcRecvMsgObject[0]);
            // unblock the RPMessage_Object for task RPMESSAGE_CHAR_PING
            RPMessage_unblock(&gIpcRecvMsgObject[1]);

            // if main task freertos_main is still running, and that task
            // is still in function ipc_rpmsg_send_messages, then
            // gIpcAckReplyMsgObjectPending == 1u
            // in that case, unblock the RPMessage_Object for freertos_main
            if (gIpcAckReplyMsgObjectPending == 1u)
                RPMessage_unblock(&gIpcAckReplyMsgObject);
        }

But that leads to another couple of questions.

1) The project calls Drivers_close() in 2 of the 3 tasks (freertos_main, and whichever RPMESSAGE task gets de-initialized last). Is that expected? If so, why? If not, should we file a bug so that Drivers_close() gets called for all 3 tasks?

2) It seems like there is a potential race condition introduced by ipc_rp_mbox_callback. I assume we want System_deinit() to be called by the very last task to be running, but it looks like that call could potentially be reached before freertos_main is able to delete itself. Should we file a bug to modify the mbox_callback like this?

void ipc_rp_mbox_callback(uint16_t remoteCoreId, uint16_t clientId, uint32_t msgValue, void *args)
{
    if (clientId == IPC_NOTIFY_CLIENT_ID_RP_MBOX)
    {
        if (msgValue == IPC_NOTIFY_RP_MBOX_SHUTDOWN) /* Shutdown request from the remotecore */
        {
            gbShutdown = 1u;
            gbShutdownRemotecoreID = remoteCoreId;

            // first, check if freertos_main is still running
            // is there a better way to check for this? If the only reason we have variable
            // gIpcAckReplyMsgObjectPending is to allow other tasks to know that the free_rtos
            // main is still running, then this variable should really be set at the beginning
            // of ipc_rpmsg_send_messages, and not cleared until the end of ipc_rpmsg_send_messages,
            // right?
            if (gIpcAckReplyMsgObjectPending == 1u)
            {
                RPMessage_unblock(&gIpcAckReplyMsgObject);
                // there are still a bunch of print statements in ipc_rpmsg_send_messages
                // after the RPMessage returns. This will take some time to execute.
                // how to wait until we know that ipc_rpmsg_send_messages has returned?
            }
            // once ipc_rpmsg_send_messages has returned, task freertos_main should
            // be deleted soon after. So now it is safe to shut down the other tasks
            RPMessage_unblock(&gIpcRecvMsgObject[0]);
            RPMessage_unblock(&gIpcRecvMsgObject[1]);

Please note that I am NOT an RTOS developer, so that sample code could be wrong, and I could be misunderstanding what is going on here.

Thanks,

Nick

I will add one additional note: I am not sure whether the freertos_main task is supposed to complete immediately after initializing the system. If that is the case, perhaps the "correct" change is to move the "ipc_rpmsg_send_messages" code to a new task, allow freertos_main to complete immediately after system initialization, and add code to check for whether that new task is still running before sending the ACK signal.

Dear Nick,

Thank you for the detailed response,

Regards,

Shreyan