AM625: CCS simulates IPC exception

Cherry Zhou

Part Number: AM625

Hi team,

Here's an issue from the customer may need your help:

1) Simulates IPC functionality with TI development boards and uses IPC_rpmsg_echo_Linux program from MCU_plus_SDK_AM62x_08_04_00_16. Replace the newly compiled am62-MCU-m4f0_0-FW in the /lib/firmware/ directory, and the M-core runs the IPC program properly after going back online. It can be seen the following print information:

IPC RPMSG ECHO] Dec 15 2022 09:13:26
[IPC RPMSG ECHO] Remote Core waiting for messages at end point 13 ... !!!
[IPC RPMSG ECHO] Remote Core waiting for messages at end point 14 ... !!!

2) Simulates the project using CCS and finds that the program is stuck on the statement " status = RPMessage_waitForLinuxReady(SystemP_WAIT_FOREVER);"

It is understood that the M-Core waits for the A-Core to complete initialization of IPC-related shared memory, etc., and cannot complete subsequent simulations.

The official default hardware and SDK packages are used, and nothing else is modified and simulating other programs with CCS is good.

Could you help check this case? Thanks.

Best Regards,

Cherry

over 1 year ago

0 Nick Saulnier over 1 year ago

TI__Guru 71415 points

Hello Cherry,

RPMessage_waitForLinuxReady is waiting for Linux to add information to the M4's resource table. Linux usually adds that information when the RemoteProc driver is probed during Linux boot.

If CCS is loading new firmware to the remote cores AFTER Linux remoteproc already set up the remote cores, I suspect that the information that Linux wrote to the resource table is getting overwritten. So either do not overwrite the resource table (i.e., use CCS to attach to the firmware Linux started), or force Linux to re-add the resource table information after the resource table gets overwritten (i.e., use CCS to load new firmware to M4, then get Linux to do something else).

See this thread for more information:

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1180017/am623-am623-mcu-sdk

Regards,

Nick

0 Annie Liu over 1 year ago in reply to Nick Saulnier

TI__Genius 10295 points

Hi Nick,

Nick Saulnier said:
do not overwrite the resource table

Customer wouldn't know how to do it? Put am62-mcu-m4f0_0-fw that does not contain other routines of ipc in /lib/firmware, In this way, the M-core program starts by default does not contain IPC. Then use ccs to simulate the routine with ipc code, the test found that it is still stuck. This issue is not resolved.

Nick Saulnier said:
force Linux to re-add the resource table information after the resource table gets overwritten

After ccs loads the ipc routine, the program freezes. At this time, uninstall the IPC-related modules on the linux shell interface. Then reloads the corresponding module. Simulate the operation of adding resource table information again. It is OK now. But it’s too inconvenient to do this. Customer has to do it every time you simulate.

Apart from the second method of uninstalling and reloading ipc-related modules, is there any other easier way to achieve it?

Thanks,

Annie

0 Nick Saulnier over 1 year ago in reply to Annie Liu

TI__Guru 71415 points

Hello Annie,

First off:

Good, I am glad the customer has a way to start working on development.

I have verified with developers that if CCS writes new firmware to a remote core, then CCS DOES overwrite the resource table information that Linux provided.

second, let me re-explain what is going on by default (WITHOUT CCS)

Please point the customers to this getting started documentation, where we run IPC firmware just fine:
https://dev.ti.com/tirex/explore/node?node=A__ACKtJpWz8b53RwT-PJ4qmA__linux_academy_am62x__XaWts8R__LATEST

What is happening?

1) Power up the board. The processor goes through all the uboot steps mentioned here: https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/08_04_01_03/exports/docs/linux/Foundational_Components/U-Boot/UG-General-Info.html#boot-flow

2) Once uboot finishes, it starts booting the Linux kernel. The Linux kernel goes through the devicetree binary (dtb) file you generated with your dts files located at arch/arm64/boot/dts/ti/k3-am62...

3) When Linux gets to the device tree node for the M4F core, Linux starts (or "probes") the RemoteProc driver to initialize the remote core.

Remote core initialization includes:
a) enable all clocks to the remote core
b) initialize all remote core settings
c) load the firmware for the remote core
d) write all the memory sections used by the remote core (specificied in the linker.cmd file for the remote core project)
e) start the remote core
f) go to the resource table that was written as a part of step d). Update the resource table with the VIRTIO locations

Notice step f). After the RemoteProc driver finishes, the M4 is able to run RPMsg examples as described in the Linux Academy link above.

OK so what does that mean if I use CCS to reload new firmware into the remote core?

Regardless of what you do with CCS, Linux already enabled the clocks:

a) enable all clocks to the remote core

If you use CCS to stop a core and reload firmware, this is what happens:

b) initialize all remote core settings
c) load the firmware for the remote core
d) write all the memory sections used by the remote core (specificied in the linker.cmd file for the remote core project)
e) start the remote core

Notice step f) is missing. RPMsg will no longer work, because CCS deleted the resource table Linux wrote in, and added a new resource table without the needed information.

Regards,

Nick

0 Nick Saulnier over 1 year ago in reply to Nick Saulnier

TI__Guru 71415 points

Hello Annie,

Please note that both options are ONLY needed if you want to watch what the code is doing in CCS during development. Neither is needed for the final product.

option 2 described in the post I linked is the option the customer says takes too much time:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1180017/am623-am623-mcu-sdk/4446143#4446143

option 1 is faster. Instead of CCS loading new firmware into the remote core, it just watches the currently running firmware. I do not know how to explain option 1 more clearly than I do in these responses:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1180017/am623-am623-mcu-sdk/4447744#4447744
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1180017/am623-am623-mcu-sdk/4446143#4446143

Regards,

Nick

0 Cherry Zhou over 1 year ago in reply to Nick Saulnier

TI__Mastermind 22235 points

Hi Nick,

Regarding the first case you provided:

During CCS simulation, do not perform this step when simulating because the CPU reset step causes Linux to remove the information registered in the resource table.

The customer tests this way: place am62-MCU-m4f0_0-FW with the routine containing IPC in /lib/firmware and simulate, but not reset the CPU and load the IPC out file directly. However, the problem is still stuck on the Linux-pending statement to register the IPC resource table.

Thanks and regards,

Cherry