PROCESSOR-SDK-AM64X: remoteproc not able to power up the remote processor (R50-0) after shutdown

Tool/software:

Dear TIers,

We have the following setup:

0) TMDS64EVM as hardware

1) Processor SDK 10.00.07.04 +  the corresponding Yocto for Linux, Kernel version 6.6.32-ti-01287

2) R5-0-0 and R5-0-1 running in split mode

3) Custom firmware, running on R5-0-0 from SRAM+TMC, with FreeRTOS

4) Implemented RPMSG, respectively IPC between Linux and R5

5) Implemented (at least up to 90%) graceful shutdown, i.e. "shutdown" signal from Linux is captured, system uninitialized and confirmation is being sent back to Linux

Code samples below:

* IPC callback

    void m_cb(uint32_t remoteCoreId, uint16_t clientId, uint32_t msgValue, [[maybe_unused]]int32_t crcStatus, [[maybe_unused]] void *args)
    {
        IpcManager* mgr = static_cast<IpcManager*>(args);
        if (clientId == (uint16_t)IPC::CID::IPC_NOTIFY_CLIENT_ID_RP_MBOX)
        {
            if (msgValue == (uint32_t)IPC::MsgType::IPC_NOTIFY_RP_MBOX_SHUTDOWN)
            {
                mgr->Shutdown(remoteCoreId);
            }
        }
    }

* Response

 void doShutdown()
    {
        //ACK the suspend message
        IpcNotify_sendMsg(m_shutdownCoreId, (uint16_t)CID::IPC_NOTIFY_CLIENT_ID_RP_MBOX, (uint32_t)MsgType::IPC_NOTIFY_RP_MBOX_SHUTDOWN_ACK, 1u);

        Board_driversClose();
        Drivers_close();

        //Disable interrupts
        HwiP_disable();

        /* For ARM R and M cores*/
        __asm__ __volatile__ ("wfi"   "\n\t": : : "memory");
    }

The issue we are facing:

Apparently, the graceful shutdown works as expected, because we see in dmesg output, that the remoteproc shutdown does not complain about timeout:

Curiously, we have to stop at first R5-0-1 (78200000.r5f) and then R5-0-0 (78000000.r5f), otherwise the driver complains. Anyway, that is not (directly) the problem. The problem is, that after stopping the R5-0-0, we are not able to start it again. Not just that, but the complete board hangs. The problem apparently comes in a very early stage, because the R5-0-0 does not come out of reset an there are only 2 lines in dmesg:

...and then the complete system hangs.

Questions:

• Is that normal, that R5-0-1 must be stopped at first?
• Is that the correct stop/start procedure?
• Is there something else that we must do to enable the repeated rproc fw load and boot? 
• And of course, why is the remoteproc driver not able to load the same fw twice, not to mention to be able to boot it?

Thank you and I will be very happy to have a deeper discussion on this topic in the following days.

Regards,

Angel