PROCESSOR-SDK-AM335X: RPMsg Echo Example fails on Beaglebone

Hello Filip,

Hmm, interesting. I will need a day or so to hook up my AM335x SDK 9.1 setup and double-check the RPMsg communication.

The PRU is unique among our remote cores that are initialized by the remoteproc driver in that the PRU can be loaded with or without a resource table, as discussed here: https://software-dl.ti.com/processor-sdk-linux/esd/AM335X/09_01_00_001/exports/docs/common/PRU-ICSS/Resource_Tables.html

So it is entirely possible that something is going on around the resource table and the initialization of the VRINGs that are used to do RPMsg communication. If so, I would not be surprised that non-RPMsg examples work just fine.

Regards,

Nick

Hello Filip,

I am able to replicate your observations on AM335x Linux SDK 9.1:

root@am335x-evm:~# uname -a
Linux am335x-evm 6.1.46-g1d4b5da681 #1 PREEMPT Thu Oct 19 10:19:08 UTC 2023 armv7l armv7l armv7l GNU/Linux

root@am335x-evm:~# head /sys/class/remoteproc/remoteproc
remoteproc0/ remoteproc1/
root@am335x-evm:~# cat /sys/class/remoteproc/remoteproc0/state
offline
root@am335x-evm:~# cat /sys/class/remoteproc/remoteproc1/state
offline
root@am335x-evm:~# cat /sys/class/remoteproc/remoteproc0/firmware
am335x-pru0-fw
root@am335x-evm:~# cat /sys/class/remoteproc/remoteproc1/firmware
am335x-pru1-fw
root@am335x-evm:~# ls -al /lib/firmware/
total 14488
drwxr-xr-x  5 root root    4096 Mar  9  2018 .
drwxr-xr-x 10 root root    4096 Mar  9  2018 ..
-rw-r--r--  1 root root    2040 Mar  9  2018 LICENCE.ibt_firmware
-rw-r--r--  1 root root    2046 Mar  9  2018 LICENCE.iwlwifi_firmware
-rw-r--r--  1 root root      73 Mar  9  2018 am335x-bone-scale-data.bin
-rw-r--r--  1 root root      17 Mar  9  2018 am335x-evm-scale-data.bin
-rw-r--r--  1 root root  230332 Mar  9  2018 am335x-pm-firmware.elf
lrwxrwxrwx  1 root root      47 Mar  9  2018 am335x-pru0-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interrupt0.out
lrwxrwxrwx  1 root root      47 Mar  9  2018 am335x-pru1-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interrupt1.out
-rw-r--r--  1 root root      41 Mar  9  2018 am43x-evm-scale-data.bin
drwxr-xr-x  2 root root    4096 Mar  9  2018 intel
-rw-r--r--  1 root root  918268 Mar  9  2018 iwlwifi-3160-17.ucode
-rw-r--r--  1 root root 2448976 Mar  9  2018 iwlwifi-8000C-34.ucode
-rw-r--r--  1 root root 2428004 Mar  9  2018 iwlwifi-8000C-36.ucode
-rw-r--r--  1 root root 2440780 Mar  9  2018 iwlwifi-8265-34.ucode
-rw-r--r--  1 root root 2436632 Mar  9  2018 iwlwifi-8265-36.ucode
-rw-r--r--  1 root root 2678092 Mar  9  2018 iwlwifi-9260-th-b0-jf-b0-34.ucode
-rw-r--r--  1 root root 2521412 Mar  9  2018 iwlwifi-9260-th-b0-jf-b0-38.ucode
-rw-r--r--  1 root root 1489652 Mar  9  2018 iwlwifi-9260-th-b0-jf-b0-46.ucode
drwxr-xr-x  2 root root    4096 Mar  9  2018 pru
-rw-r--r--  1 root root    4728 Mar  9  2018 regulatory.db
-rw-r--r--  1 root root    1182 Mar  9  2018 regulatory.db.p7s
drwxr-xr-x  2 root root    4096 Mar  9  2018 ti-connectivity
root@am335x-evm:~# echo start > /sys/class/remoteproc/remoteproc0/state
[  154.951240] remoteproc remoteproc0: powering up 4a334000.pru
[  155.043825] remoteproc remoteproc0: Booting fw image am335x-pru0-fw, size 72852
[  155.066542] pru-rproc 4a334000.pru: error -ENXIO: IRQ vring not found
[  155.083705] remoteproc remoteproc0: unable to get vring interrupt, status = -6
[  155.101768] remoteproc remoteproc0: can't start rproc 4a334000.pru: -6
[  155.120626] remoteproc remoteproc0: Boot failed: -6
-sh: echo: write error: No such device or address

I also see the automated test failing when it was run back in October 2023:

https://software-dl.ti.com/cicd-report/linux/index.html?section=test&platform=am335x&snapshot=cicd.kirkstone.202310191725&board=am335x-evm&variant=Linux&test=12894

I am following up with our developers. Please ping the thread if I have not provided an update within a week.

Regards,

Nick

I am also able to replicate your observation that if a firmware does not involve the resource table or the VRINGs, then it can be loaded fine.

root@am335x-evm:~# ls /lib/firmware/pru/PRU_
PRU_ADS8688_Controller.out     PRU_RPMsg_Echo_Interrupt0.out
PRU_ADS8688_Interface.out      PRU_RPMsg_Echo_Interrupt1.out
PRU_Halt.out
root@am335x-evm:~# cd /lib/firmware/
root@am335x-evm:/lib/firmware# ln -sf /lib/firmware/pru/PRU_Halt.out am335x-pru0-fw
root@am335x-evm:/lib/firmware# ln -sf /lib/firmware/pru/PRU_Halt.out am335x-pru1-fw
root@am335x-evm:/lib/firmware# ls -al
total 14488
drwxr-xr-x  5 root root    4096 Jan  1 00:08 .
drwxr-xr-x 10 root root    4096 Mar  9  2018 ..
-rw-r--r--  1 root root    2040 Mar  9  2018 LICENCE.ibt_firmware
-rw-r--r--  1 root root    2046 Mar  9  2018 LICENCE.iwlwifi_firmware
-rw-r--r--  1 root root      73 Mar  9  2018 am335x-bone-scale-data.bin
-rw-r--r--  1 root root      17 Mar  9  2018 am335x-evm-scale-data.bin
-rw-r--r--  1 root root  230332 Mar  9  2018 am335x-pm-firmware.elf
lrwxrwxrwx  1 root root      30 Jan  1 00:07 am335x-pru0-fw -> /lib/firmware/pru/PRU_Halt.out
lrwxrwxrwx  1 root root      30 Jan  1 00:08 am335x-pru1-fw -> /lib/firmware/pru/PRU_Halt.out
-rw-r--r--  1 root root      41 Mar  9  2018 am43x-evm-scale-data.bin
drwxr-xr-x  2 root root    4096 Mar  9  2018 intel
-rw-r--r--  1 root root  918268 Mar  9  2018 iwlwifi-3160-17.ucode
-rw-r--r--  1 root root 2448976 Mar  9  2018 iwlwifi-8000C-34.ucode
-rw-r--r--  1 root root 2428004 Mar  9  2018 iwlwifi-8000C-36.ucode
-rw-r--r--  1 root root 2440780 Mar  9  2018 iwlwifi-8265-34.ucode
-rw-r--r--  1 root root 2436632 Mar  9  2018 iwlwifi-8265-36.ucode
-rw-r--r--  1 root root 2678092 Mar  9  2018 iwlwifi-9260-th-b0-jf-b0-34.ucode
-rw-r--r--  1 root root 2521412 Mar  9  2018 iwlwifi-9260-th-b0-jf-b0-38.ucode
-rw-r--r--  1 root root 1489652 Mar  9  2018 iwlwifi-9260-th-b0-jf-b0-46.ucode
drwxr-xr-x  2 root root    4096 Mar  9  2018 pru
-rw-r--r--  1 root root    4728 Mar  9  2018 regulatory.db
-rw-r--r--  1 root root    1182 Mar  9  2018 regulatory.db.p7s
drwxr-xr-x  2 root root    4096 Mar  9  2018 ti-connectivity

root@am335x-evm:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc0/state
[  510.801345] remoteproc remoteproc0: powering up 4a334000.pru
[  510.925629] remoteproc remoteproc0: Booting fw image am335x-pru0-fw, size 22624
[  510.934005] remoteproc remoteproc0: remote processor 4a334000.pru is now up
root@am335x-evm:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc1/state
[  515.981246] remoteproc remoteproc1: powering up 4a338000.pru
[  516.029440] remoteproc remoteproc1: Booting fw image am335x-pru1-fw, size 22624
[  516.037657] remoteproc remoteproc1: remote processor 4a338000.pru is now up

Regards,

Nick

Great! Thank you for your help!

Hello Filip,

Thanks again for reporting this behavior. Here's a quick update before I take off for a 3 day weekend:

Further debug

I was able to confirm that examples with just INTC configuration also load and run fine with PRU_Direct_Connectx examples:

If the example works properly, values from the PRU0 buffer defined here: https://git.ti.com/cgit/pru-software-support-package/pru-software-support-package/tree/examples/am335x/PRU_Direct_Connect0/PRU0_Direct_Connect.c
will show up in the .bcc memory of PRU1. For an explanation about how I calculated the memory offset, refer to the PRU Getting Started Labs:
https://software-dl.ti.com/processor-sdk-linux/esd/AM335X/09_01_00_001/exports/docs/common/PRU-ICSS/PRU-Getting-Started-Labs_Lab5.html#debugging-the-pru-from-linux-core

root@am335x-evm:/lib/firmware# uname -a
Linux am335x-evm 6.1.46-g1d4b5da681 #1 PREEMPT Thu Oct 19 10:19:08 UTC 2023 armv7l armv7l armv7l GNU/Linux
root@am335x-evm:/lib/firmware# ls -al
...
lrwxrwxrwx  1 root root      41 Jan  2 19:54 am335x-pru0-fw -> /lib/firmware/pru/PRU_Direct_Connect0.out
lrwxrwxrwx  1 root root      41 Jan  2 18:56 am335x-pru1-fw -> /lib/firmware/pru/PRU_Direct_Connect1.out
...
root@am335x-evm:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc0/state
[ 2011.631308] remoteproc remoteproc0: powering up 4a334000.pru
[ 2011.756959] remoteproc remoteproc0: Booting fw image am335x-pru0-fw, size 43452
[ 2011.766839] remoteproc remoteproc0: remote processor 4a334000.pru is now up
root@am335x-evm:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc1/state
[ 2016.971233] remoteproc remoteproc1: powering up 4a338000.pru
[ 2017.017425] remoteproc remoteproc1: Booting fw image am335x-pru1-fw, size 25760
[ 2017.025247] remoteproc remoteproc1: remote processor 4a338000.pru is now up
root@am335x-evm:/lib/firmware# devmem2 0x4a302100 // PRU1 stores .bcc data in DMEM1 at offset 0x100
/dev/mem opened.
Memory mapped at address 0xb6f95000.
Read at address  0x4A302100 (0xb6f95100): 0xABABABAB
root@am335x-evm:/lib/firmware# devmem2 0x4a302104
/dev/mem opened.
Memory mapped at address 0xb6fab000.
Read at address  0x4A302104 (0xb6fab104): 0xAAAAAAAA
root@am335x-evm:/lib/firmware# devmem2 0x4a302108
/dev/mem opened.
Memory mapped at address 0xb6fcb000.
Read at address  0x4A302108 (0xb6fcb108): 0x12345678

What happened? 

As far as I can tell, there was a mix up with the folks who were porting AM335x SDK 8.2 to SDK 9.1. We descoped PRU Ethernet for SDK 9.1 ONLY (it is enabled for SDK 8.2 and earlier, and we will re-enable it in AM335x SDK 10.x, which is tentatively scheduled for around Oct 2024). But it looks like someone saw the word "PRU" and thought we were descoping PRU programming entirely, instead of only a very specific usecase.

What next?

Unfortunately, it might be a while before I can tell you exactly what is going on or provide a fix. We're wrapping up development on Linux kernel 6.1 as the team cleans up the upcoming SDK 9.2 for some of our other AM6x devices, and then the team will pivot to migrating to the next Linux kernel version.

I obviously don't want to ask you to wait until October, and a new kernel version, to get going with your PRU development. We'll try to spend some more time looking into this sometime in the next few weeks. Hopefully it's an easy fix, and I'm able to get an update pushed to https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/?h=ti-linux-6.1.y-cicd to enable you and other people working on AM335x Linux SDK 9.1. If it's not an easy fix, we might have to adjust the plan.

Soooo... can you do anything in the meantime?

YES. 

If your application doesn't need IPC (inter-processor communication) with Linux, you can just avoid adding a resource table to your PRU project and it should run on SDK 9.1 just fine. For more information about how to do that, refer to the PRU Getting Started Labs:
https://software-dl.ti.com/processor-sdk-linux/esd/AM335X/09_01_00_001/exports/docs/common/PRU-ICSS/PRU-Getting-Started-Labs.html 

Even if you do need IPC with Linux, sometimes RPMsg is not the right IPC for your specific usecase - but if you aren't an experienced PRU developer, it's a lot easier to start with RPMsg than trying to write a custom IPC.

The last time we made a major change to the Linux pru remoteproc driver was when going from Linux kernel 5.4 -> kernel 5.10 (AM335x SDK 8.3). So if your application does need IPC with Linux, you can just start development with SDK 8.3. From a PRU firmware side, the exact same PRU code should work on SDK 8.3 and SDK 9.1 (once it's fixed) as long as you are using a version of the PRU Software Support Package that is v6.0.0 or later
https://git.ti.com/cgit/pru-software-support-package/pru-software-support-package/

Regards,

Nick

Thank you for the update, Nick! The reason I am reporting this is that we have a project using the PRU, currently on Yocto Dunfell (kernel 4.9) which we are porting to Kirkstone (kernel 6.1). During porting we noticed that the RPMsg communication broke. We are looking into an alternative IPC, but it would obviously be nice if RPMsg got fixed. 

Hello Filip,

Understood. I'm gunna plan to give you another update next week. Feel free to ping the thread if you haven't heard a reply in the next 7-10 days, that will help keep me honest and make sure your thread stays towards the top of my TODO list.

Regards,

Nick

Hello Filip,

Update: no progress so far between vacations & needing to prioritize other projects. We will definitely fix this issue for the AM335x SDK 10.1 release. We will not be able to push out a CI/CD release on AM335x Linux kernel 6.1 with the fix, but my hope is that once the team fixes the issue on the next kernel baseline, we can backport to kernel 6.1 and then I can post the patches to an e2e FAQ post.

I'll be on vacation for the rest of March, but feel free to check in during April to see if there have been any updates.

Regards,

Nick

Hello Filip,

Monthly update - we haven't started on the PRU stuff yet, but the developer has said they'll try to take an initial look next week. If we make any fun progress I'll let you know, otherwise I'll shoot to give another update a month from now.

Regards,

Nick

Thanks for the update, Nick! In the meantime we have implemented our own IPC, so we have a solution at the moment. But it's good to hear that this is getting fixed. 

Hello Filip,

Glad to hear that you were able to get another implementation working! That helps me drive the right priority with the development team - this needs to get fixed eventually, but (at the moment) it is not blocking any customers.

I'll continue providing updates on this thread as a way to keep us accountable, even though yall have implemented an alternative solution.

Regards,

Nick