AM6442: Creating multiple rpmsg endpoints (resp. channels) to exchange data between A53 Linux and R5Fss0_0 freertos

Hello Daniel,

1) Which version of the SDK are you using? (note that the rpmsg userspace library for Linux kernel 6.1 is still under development, so I can only comment on the Linux kernels in our released SDKs).

2) can you please clarify what changes to the MCU+ code you are doing, and what output you are observing with your statement "I can announce the second Endpoint alone,when no other endpoint is announced before, but not as the second one."

In general, yes, you should be able to define multiple channels that can use the same set of RPMsg VIRTIO buffers to communicate between different tasks running on different cores.

Regards,

NIck

Hi Nick,

thank you for your answer and your questions!

To point 1):

I am using MCU+ SDK Version 08_02_00_31 and Ti Kernel 5.10.100-rt62 from here (TI_TAG 08.02.00.003-rt): 08.02.00.003-rt Release

To point 2):

I did not made any changes to the MCU+ SDK Code. It is only referenced and used in my Code Composter Studio project.

Sorry my explanation was not clear. It is possible to create and announce Endpoint with numbers 14 and 15 individually. But when I create Endpoint with number 15 and announce it to Linux after Endpoint 14 was successfully created and announced (and its device /dev/rpmsg_ctrl0 was added in Linux), nothing happens. No message about endpoint number 15 is shown on the Linux console as before in the individual case. Also no device /dev/rpmsg_ctrl1 is added.

See attached figure for further clarification. 

Do you have any more suggestions for me? Am I doing something wrong?

Many thanks or your help in advance!

Daniel

Hey there Daniel,

Got it. I ran out of time today to run tests on my side, I am planning to run tests tomorrow. Please ping the thread if I have not replied by Thursday or Friday.

Regards,

Nick

Hi Nick,

thanks for your time and your support! Here's the Ping ;)

Daniel

Hey Daniel,

I apologize for the continued delays here. I made the mistake of updating my Ubuntu system and things are finally (mostly) setup. Trying for another day in the lab tomorrow.

Regards,

Nick

Hey Nick,

Thank you for the reply! Do you habe some new insights for me?
Were you able to spend some time on this problem?

Thanks a lot!

Daniel

Hello Daniel,

Thank you for your persistence in following up here.

Test Setup

I am currently working on SDK 8.6 for both Linux, and MCU+ SDKs (using the MCU+ SDK 8.6.0.45, which was released June 13). I would expect the results to be similar for you on SDK 8.2.

I wrote a patch to the MCU+ project ipc_rpmsg_echo_linux, but I did not have time to run tests today. Could you try it out and see if it allows you to interact with multiple endpoints? Based on reading through the RPMsg code, this is the way I would expect us to create multiple endpoints on a single remote core.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_Linux_5F00_RPMsg_5F00_Echo_2D00_add_2D00_additional_2D00_endpoints.patch

In case you are unfamiliar with applying git files, I follow this process on the Linux terminal:

$ cd location/of/mcu+/sdk
$ git status
// we see no git repo has been created yet
$ git init
$ git add .
// create the initial commit by adding everything in the SDK
// this works best with a fresh SDK installation
$ git branch
// the master branch is the only branch
$ git branch featureBranchName
$ git checkout featureBranchName
// copy over the patch file
$ git apply patchFileName

Regards,

Nick

Hi Nick, 

thank you for your reply and for your further suggestions and instructions!

I will try out the modified example by tomorrow or at least next wednesday. 

I believe I still have the corresponding linux program (rpmsg_char_simple) already deployed, as I also started with this example.

I'll let you know about the result!

Regards,

Daniel

Hey Nick,

I was able to apply the patch and compile the example program. I can test it by tomorrow and let you know if it works as it should.

Daniel

Sounds good, thanks for the update Daniel!

Please keep us updated on whether it works. Once we have something working, I'll turn this into an FAQ to make it easier for future engineers to find.

Regards,

Nick

So I have some results:

When the binary for the R5f is loaded by linux during the boot process it shows that all the 3 desired channels are created:

[   15.881572] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd
[   15.881837] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe
[   15.881979] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev2 addr 0xf
[   15.882093] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev3 addr 0x10
[   15.950668] virtio_rpmsg_bus virtio0: rpmsg host is online

The other ping-pong channel is also exchanging its messages:

 ............................................................................ sg 92 (src: 0xd)
[   17.774577] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 93 (src: 0xd)
[   17.774815] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 94 (src: 0xd)
[   17.775023] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 95 (src: 0xd)
[   17.775231] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 96 (src: 0xd)
[   17.775440] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 97 (src: 0xd)
[   17.775676] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 98 (src: 0xd)
[   17.775884] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 99 (src: 0xd)
[   17.776094] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: incoming msg 100 (src: 0xd)
[   17.776104] rpmsg_client_sample virtio0.ti.ipc4.ping-pong.-1.13: goodbye!

Testing the program by using the corresponding rpmsg_char_simple app shows that the first endpoint works well, but the second and third do not:

First (rpmsg_chrdev, Endpt number 14):

root@-target-:/home/-user-/ti-example-multiple-channels# ./rpmsg_char_simple -r 2 -n 10 -d rpmsg_chrdev -p 14

Created endpt device rpmsg-char-2-849, fd = 3 port = 1025
Exchanging 10 messages with rpmsg device ti.ipc4.ping-pong on rproc id 2 ...

Sending message from Linux #0: hello there 0!
Receiving message echoed by R5F #0: hello there 0!
Sending message from Linux #1: hello there 1!
Receiving message echoed by R5F #1: hello there 1!
Sending message from Linux #2: hello there 2!
Receiving message echoed by R5F #2: hello there 2!
Sending message from Linux #3: hello there 3!
Receiving message echoed by R5F #3: hello there 3!
Sending message from Linux #4: hello there 4!
Receiving message echoed by R5F #4: hello there 4!
Sending message from Linux #5: hello there 5!
Receiving message echoed by R5F #5: hello there 5!
Sending message from Linux #6: hello there 6!
Receiving message echoed by R5F #6: hello there 6!
Sending message from Linux #7: hello there 7!
Receiving message echoed by R5F #7: hello there 7!
Sending message from Linux #8: hello there 8!
Receiving message echoed by R5F #8: hello there 8!
Sending message from Linux #9: hello there 9!
Receiving message echoed by R5F #9: hello there 9!

Communicated 10 messages successfully on rpmsg-char-2-849

TEST STATUS: PASSED

Second (rpmsg_chrdev2, Endpt number 15):

root@-target-:/home/-user-/ti-example-multiple-channels# ./rpmsg_char_simple -r 2 -n 10 -d rpmsg_chrdev2 -p 15

_rpmsg_char_find_ctrldev: rpmsg directory doesn't exist under virtio0.rpmsg_chrdev2.-1.15
Can't create an endpoint device: No such file or directory
TEST STATUS: FAILED

Third (rpmsg_chrdev3, Endpt number 16):

root@-target-:/home/-user-/ti-example-multiple-channels# ./rpmsg_char_simple -r 2 -n 10 -d rpmsg_chrdev3 -p 16
_rpmsg_char_find_ctrldev: rpmsg directory doesn't exist under virtio0.rpmsg_chrdev3.-1.16
Can't create an endpoint device: No such file or directory
TEST STATUS: FAILED

I searched in the rootfs for further infomation. In the remoteproc section for the core are all endpoints listed:

root@-target-:/home/-user-/ti-example-multiple-channels# ls -l /sys/class/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/
total 0
-r--r--r-- 1 root root 4096 Aug  7 16:14 device
lrwxrwxrwx 1 root root    0 Aug  7 16:14 driver -> ../../../../../../../../../bus/virtio/drivers/virtio_rpmsg_bus
-r--r--r-- 1 root root 4096 Aug  7 16:14 features
-r--r--r-- 1 root root 4096 Aug  7 16:14 modalias
drwxr-xr-x 2 root root    0 Aug  7 16:14 power
-r--r--r-- 1 root root 4096 Aug  7 16:14 status
lrwxrwxrwx 1 root root    0 Aug  7 13:25 subsystem -> ../../../../../../../../../bus/virtio
-rw-r--r-- 1 root root 4096 Aug  7 13:25 uevent
-r--r--r-- 1 root root 4096 Aug  7 16:14 vendor
drwxr-xr-x 4 root root    0 Aug  7 13:25 virtio0.rpmsg_chrdev.-1.14
drwxr-xr-x 3 root root    0 Aug  7 13:25 virtio0.rpmsg_chrdev2.-1.15
drwxr-xr-x 3 root root    0 Aug  7 13:25 virtio0.rpmsg_chrdev3.-1.16
drwxr-xr-x 3 root root    0 Aug  7 13:25 virtio0.ti.ipc4.ping-pong.-1.13

The links to this endpoints are also all created under /sys/bus/rpmsg/devices:

root@-target-:/home/-user-/ti-example-multiple-channels# ls -l /sys/bus/rpmsg/devices/                                                                                                                                                                  

total 0
lrwxrwxrwx 1 root root 0 Aug  7 14:33 virtio0.rpmsg_chrdev.-1.14 -> ../../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.rpmsg_chrdev.-1.14
lrwxrwxrwx 1 root root 0 Aug  7 14:33 virtio0.rpmsg_chrdev2.-1.15 -> ../../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.rpmsg_chrdev2.-1.15
lrwxrwxrwx 1 root root 0 Aug  7 14:33 virtio0.rpmsg_chrdev3.-1.16 -> ../../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.rpmsg_chrdev3.-1.16
lrwxrwxrwx 1 root root 0 Aug  7 13:25 virtio0.ti.ipc4.ping-pong.-1.13 -> ../../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.ti.ipc4.ping-pong.-1.13

Nevertheless there is only one rpmsg device created (rpmsg_ctrl0) which points to the first created endpoint with number 14:

root@-target-:/home/-user-/ti-example-multiple-channels# ls -l /dev/rpmsg*
crw------- 1 root root 239, 0 Aug  7 13:25 /dev/rpmsg_ctrl0

root@-target-:/home/-user-/ti-example-multiple-channels# ls -l /sys/class/rpmsg/
total 0
lrwxrwxrwx 1 root root 0 Aug  7 13:39 rpmsg_ctrl0 -> ../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.rpmsg_chrdev.-1.14/rpmsg/rpmsg_ctrl0

So I believe these deveices must be created to work correctly, do they?

Compared to my own project it can be seen that you are using diffenrent service names for the endpoints (rpmsg_chrdev<2|3>) while I was using the same name  rpmsg_chrdev for both (Have to change this of course).

Also it uses different freeRTOS Task while I am using one single task, which calls the subprogramms dealing with the endpoints cyclically.

Have you some more ideas regarding the sample program? Again thank you for your effort!

Regards 

Daniel

Hello Daniel,

Thank you for the detailed testing and observations! That is helpful for me when discussing with the Linux & MCU+ developers.

Good question on whether more than one endpoint can be in a single task. I am double-checking with the MCU+ developers. It sounds like the TYPICAL design is to have one endpoint per task, but I do not know if "typical" means "this is the only way to do it", or "this is one way to do it": https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/08_06_00_45/exports/docs/api_guide_am64x/IPC_GUIDE.html#autotoc_md408

I am also reaching out to see if another team member more experienced with MCU+ development can check my code, as well as pinging the Linux developer. I'll try to run tests on my side on Friday.

Regards,

Nick

Hello Nick,

you're welcome. I am waiting for more news.

Maybe one more annotation from my side: Is it possible that the service name rpmsg_chrdev might be a keyword? I remember that I tried different names for it (like you did in the example with rpmsg_chrdev2 and rpmsg_chrdev3) and I commented in my code that this is a keyword, because it was the only working naming for me, even with only one endpoint.

Perhaps this might help.

Regards 

Daniel

Hello Daniel,

I expect to hear back from the developers on Monday.

That is a good question. I'll add it to the list of questions to the Linux developer.

Regards,

Nick

Actually, I got a weekend update from the Linux developer! Can I get you to retry, but giving all the endpoints the same service name of rpmsg_chrdev?

From the developer:

"

The service name needs to be 'rpmsg_chrdev' and it can have multiple endpoints. I believe, AM62A and J7 vision apps use multiple endpoints. The virtio sysfs listing in your thread above looks fine. but since there is no 'rpmsg-chrdev2' rpmsg device driver, you do not see /dev/rpmsg' char device nodes.

"

Regards,

Nick

Hello Nick, 

thanks for the reply! I will try it by tomorrow .

I will keep you updated here.

Regards

Daniel

Sounds good, looking forward to the update!

Regards,

Nick

Hello Nick,

I have positive feedback for you and your team!

We can now talk to all the Endpoints with service name rpmsg_chrdev on R5fss0_0!

root@-target-:/home/-user-/ti-example-multiple-channels# ./rpmsg_char_simple -r 2 -n 4 -d rpmsg_chrdev -p 14
Created endpt device rpmsg-char-2-799, fd = 3 port = 1025
Exchanging 4 messages with rpmsg device ti.ipc4.ping-pong on rproc id 2 ...

Sending message from Linux #0: hello there 0!
Receiving message echoed by R5F #0: hello there 0!
Sending message from Linux #1: hello there 1!
Receiving message echoed by R5F #1: hello there 1!
Sending message from Linux #2: hello there 2!
Receiving message echoed by R5F #2: hello there 2!
Sending message from Linux #3: hello there 3!
Receiving message echoed by R5F #3: hello there 3!

Communicated 4 messages successfully on rpmsg-char-2-799

TEST STATUS: PASSED


root@-target-:/home/-user-/ti-example-multiple-channels# ./rpmsg_char_simple -r 2 -n 4 -d rpmsg_chrdev -p 15
Created endpt device rpmsg-char-2-801, fd = 3 port = 1025
Exchanging 4 messages with rpmsg device ti.ipc4.ping-pong on rproc id 2 ...

Sending message from Linux #0: hello there 0!
Receiving message echoed by R5F #0: hello there 0!
Sending message from Linux #1: hello there 1!
Receiving message echoed by R5F #1: hello there 1!
Sending message from Linux #2: hello there 2!
Receiving message echoed by R5F #2: hello there 2!
Sending message from Linux #3: hello there 3!
Receiving message echoed by R5F #3: hello there 3!

Communicated 4 messages successfully on rpmsg-char-2-801

TEST STATUS: PASSED


root@-target-:/home/-user-/ti-example-multiple-channels# ./rpmsg_char_simple -r 2 -n 4 -d rpmsg_chrdev -p 16
Created endpt device rpmsg-char-2-802, fd = 3 port = 1025
Exchanging 4 messages with rpmsg device ti.ipc4.ping-pong on rproc id 2 ...

Sending message from Linux #0: hello there 0!
Receiving message echoed by R5F #0: hello there 0!
Sending message from Linux #1: hello there 1!
Receiving message echoed by R5F #1: hello there 1!
Sending message from Linux #2: hello there 2!
Receiving message echoed by R5F #2: hello there 2!
Sending message from Linux #3: hello there 3!
Receiving message echoed by R5F #3: hello there 3!

Communicated 4 messages successfully on rpmsg-char-2-802

TEST STATUS: PASSED

Compared to the previous example all three character devices are created ...

root@-target-:/home/-user-/ti-example-multiple-channels# ls -l /dev/rpmsg_ctrl*
crw------- 1 root root 239, 0 Aug  7 13:25 /dev/rpmsg_ctrl0
crw------- 1 root root 239, 1 Aug  7 13:25 /dev/rpmsg_ctrl1
crw------- 1 root root 239, 2 Aug  7 13:25 /dev/rpmsg_ctrl2

... and linked to the corresponding buffers:

root@-target-:/home/-user-/ti-example-multiple-channels# ls -l /sys/class/rpmsg/
total 0
lrwxrwxrwx 1 root root 0 Aug  7 16:03 rpmsg_ctrl0 -> ../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.rpmsg_chrdev.-1.14/rpmsg/rpmsg_ctrl0
lrwxrwxrwx 1 root root 0 Aug  7 16:03 rpmsg_ctrl1 -> ../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.rpmsg_chrdev.-1.15/rpmsg/rpmsg_ctrl1
lrwxrwxrwx 1 root root 0 Aug  7 16:03 rpmsg_ctrl2 -> ../../devices/platform/bus@f4000/bus@f4000:r5fss@78000000/78000000.r5f/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/virtio0.rpmsg_chrdev.-1.16/rpmsg/rpmsg_ctrl2

root@-target-:/home/-user-/ti-example-multiple-channels# ls -l /sys/class/remoteproc/remoteproc1/remoteproc1#vdev0buffer/virtio0/
total 0
-r--r--r-- 1 root root 4096 Aug  7 16:01 device
lrwxrwxrwx 1 root root    0 Aug  7 16:01 driver -> ../../../../../../../../../bus/virtio/drivers/virtio_rpmsg_bus
-r--r--r-- 1 root root 4096 Aug  7 16:01 features
-r--r--r-- 1 root root 4096 Aug  7 16:01 modalias
drwxr-xr-x 2 root root    0 Aug  7 16:01 power
-r--r--r-- 1 root root 4096 Aug  7 16:01 status
lrwxrwxrwx 1 root root    0 Aug  7 13:25 subsystem -> ../../../../../../../../../bus/virtio
-rw-r--r-- 1 root root 4096 Aug  7 13:25 uevent
-r--r--r-- 1 root root 4096 Aug  7 16:01 vendor
drwxr-xr-x 4 root root    0 Aug  7 13:25 virtio0.rpmsg_chrdev.-1.14
drwxr-xr-x 4 root root    0 Aug  7 13:25 virtio0.rpmsg_chrdev.-1.15
drwxr-xr-x 4 root root    0 Aug  7 13:25 virtio0.rpmsg_chrdev.-1.16
drwxr-xr-x 3 root root    0 Aug  7 13:25 virtio0.ti.ipc4.ping-pong.-1.13

From now on I would say it is my part to port this working example into my project.

I will document the differenties found and will let you know here about the results (As written above I already tried the same rpmsg_chrdev name for all endpoints, which did not work. Perhaps it has something to do with the different/individual freeRTOS tasks..).

Many thanks for your help so far! When I was successful I will mark these thread as solved of course.

Have a nice week and best regards,

Daniel

Hello Daniel,

Thanks for the update! Glad to hear that things are moving in the right direction. I will double-check with the MCU+ developers on the "are different tasks required" question.

Regards,

Nick

Hello Daniel,

The MCU+ developer said that you do need a separate task for each endpoint. From our conversation:

"

We need a seperate RPMessage_Object for each local end point. Each object will have different semaphore. Receive has to wait on a semaphore to be posted from ISR. So we need a separate task for waiting on each semaphore.

"

That is an unexpected observation! I would have assumed that order of operations does not matter. I am checking with the developer to see if that is expected.

Regards,

Nick

Hello Nick,

thanks for the information about the different tasks. Even if I don't have different tasks, I have two seperate RPMessage_Object for each local endpoint.

I will share further feedback here for you .

Regards,

Daniel

Hey Daniel, I have created an FAQ based on our discussion. Let me know if I got anything wrong or if you have anything to add:

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1245326/faq-rpmsg-how-to-add-multiple-endpoints-to-a-remote-core-running-rtos

Thanks for the questions!

Regards,

Nick

Hi Nick,

the FAQ looks fine! This will be helpful for others doing the same as I do.
I will keep you updated about the "Different RTOS Tasks required?" question, as far as I was able to test it properly.

Have a nice weekend!

Daniel