AM625: MCU_PLUS_SDK_AM62x_08_06_00_18: Loading M4F examples from remoteproc fail

Hi Nick

Thanks for quickly replay

I would like to use "Approach #2: Manually add an empty resource table to the MCU+ Project ". I tested the patches 0001-Enable-Linux-load-no-IPC.-debug-logs-enabled.patch and 0001-Enable-Linux-load-no-IPC-no-debug-logs.patch. Booting the kernel the example works fine, i see the "Hello World!" message printed on M4F console but when i tried to stop the remote processor from linux user space i see the following error:

cat /sys/class/remoteproc/remoteproc0/state
running

echo stop > /sys/class/remoteproc/remoteproc0/state
[ 40.147127] k3-m4-rproc 5000000.m4fss: k3_m4_rproc_stop : timed out waiting for rproc completion event
[ 40.156543] remoteproc remoteproc0: can't stop rproc: -16

I observed the same behavior using "Approach #1: Add Linux IPC to the MCU+ Project "

Regards,

Francesco

Hello Francesco,

Summary 

Graceful shutdown of remote cores is not supported until SDK 9.0. For now, please reboot the processor if you want to load new firmware into the remote core.

More details

Putting this Linux academy link here in case it is useful to you going forward: https://dev.ti.com/tirex/explore/node?node=A__AISILbMWZ4d84U2oSmXcdA__linux_academy_am62x__XaWts8R__LATEST)

Technically, graceful shutdown of remote cores is not supported in SDK 8.6 and earlier SDK releases (as per the note in the above Linux Academy page, and in the Linux SDK IPC docs). We are planning on officially supporting graceful shutdown of remote cores in AM64x and AM62x starting in SDK 9.0.

However, the behavior you are observing is new to SDK 8.6. If I recall my discussion with the developers correctly, they might have finished adding support for gracefully shutting down M4F ONLY on AM62x ONLY before SDK 8.6 - that is why you see Linux waiting to hear back from the M4F before shutting it down (instead of just blindly powering off the core, like you would see in SDK 8.5 and earlier).

In order for graceful shutdown to work, from the Linux side the driver needs to notify the remote core that it will be shut down soon (to give the remote core time to release all its resources and go into a known good state). But then from the remote core side, the remote core must be able to RECEIVE that message and know what to do with it. My suspicion is that the remote core side of graceful shutdown is not being added to the project with the information currently in my FAQ.

The developers haven't provided any documentation for how this works since they're still in the process of implementing it. But I'll ask them if there is something like an easy SysConfig setting you can flip to add the remote core side of graceful shutdown support to your project.

Regards,

Nick

Hello Francesco,

I ran a quick test with the RPMsg Echo binary that is packaged into the AM62x SDK 8.6 default filesystem image, and it seems to shut down gracefully:

root@am62xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc0/state
[  309.550932] remoteproc remoteproc0: stopped remote processor 5000000.m4fss

Can I get you to check to see if the ipc_rpmsg_echo_linux project in the MCU+ SDK will also shut down gracefully? If so, we might be able to do a diff of the project files to figure out how to enable graceful shutdown (e.g., example.syscfg). When I do a quick scan of the SDK 8.6 ipc_rpmsg_echo_linux project's example.syscfg, I don't see any obvious changes from SDK 8.5, but I have not done a deep dive at this point.

Regards,

Nick

Hi Nick

The ipc_rpmsg_echo_linux project in the MCU+ SDK shut down gracefully even for me

Can you test an M4F example other than ipc_rpmsg_echo_linux? for example hello_world

However i found a workaround that fixes the M4F examples where the resource table is not declared and even the shut down issue:

https://github.com/FrancescoFerraro/ti-linux-kernel/commit/c471142bc66a8277f6be1d84af7c1228acd77e98

https://github.com/FrancescoFerraro/ti-linux-kernel/commit/077cb3b86367fa8ecf670eb818ecd260f7a0cb0a

With these patches no changes to the M4F examples are needed. What do you think about the above patches?

Thanks

Francesco

Hello Francesco,

It looks like the code difference is in the project's .c file, examples/drivers/ipc/ipc_rpmsg_echo_linux/ipc_rpmsg_echo.c. You can see the difference by doing a diff between the SDK 8.5 code and the SDK 8.6 code, if you want to try to get it working.

Please keep in mind that graceful shutdown is NOT officially supported by TI until SDK 9.0. It may not be completely tested yet, and if you have trouble getting it working, I will not be able to offer support.

Regards,

Nick

In terms of the patches you linked:

These are not changes that I will be able to support. The suggestion would be to wait until SDK 9.0 for graceful shutdown to be added, or to check out my response above: e2e.ti.com/.../4593999

1) enabling Remoteproc to load M4F and R5F code without requiring a resource table:

PRU cores can be loaded without a resource table. A while back, I asked the developers whether it was feasible to add the same functionality for M4F and R5F cores. Their response was that it would require significant modifications to support both with and without:

"The PRUs are considered to be microcontrollers compared to the R5Fs and DSPs which are considered General Purpose...and these will require the generic IPC. The non-IPC firmwares for R5Fs and DSPs would be a very niche usecase. Do you have a strong reason to support the non resource-table firmwares for R5F and M4F? You would have to unplumb a lot in the driver to ensure the support with and without... PRUs follow the client-driver boot methodology where you have an PRU Application driver establish its own communication with its corresponding firmware.....that is not expected for R5Fs/DSPs or IPUs/DSPs on Vayu (AM57x). Also, be mindful that PRUs do not have an late-attach/IPC-only scheme."

2. removing the checks for remote core ready for shutdown:

Even if Linux didn't wait for an ACK before shutting down the remote cores, graceful shutdown would still not be supported. Why?

a) resource allocation: when the AM62x remote cores come up, if they are using any peripherals or system resources, they request those resources from the device manager code running on the R5F. Those resources are then locked, and nobody else is able to use them. If the remote cores are not brought down gracefully, then those resources are not released. That means the next time the remote core boots and requests the resources, its request will be denied because the device manager thinks the peripherals are still in use.

b) "known good" state: When the remote core is shut down without warning, it is not placed in a known good state. We have seen kernel crashes and other instability issues when bringing a remote core down and then back up again without rebooting the entire processor.

Regards,

Nick

Hi Nick

Thanks for you support

Basically i'm trying to port the SDK 8.6 to my AM62x Board to support my customers but now i'm a bit confused.

If I understand correctly the M4F/R5F examples should all be based on the ipc_rpmsg_echo_linux example to work fine. Also the nortos examples to implement the ipc_rp_mbox_callback function to support shutdown from linux. I experimented with this trying to fix the hello_world example.

When do you plan to release the SDK 9.0?

Regards

Francesco