PROCESSOR-SDK-AM437X: The remoteproc is starting very late during boot

Hello Eduardo,

Can you tell us a bit more about what the PRU is doing in your usecase? e.g., PRU Ethernet? Are you running custom firmware? Is your custom firmware interacting with pinmuxing, etc?

TI has only validated initializing the PRUs with remoteproc during Linux boot.

There are a couple potential options that come to mind:

1) reduce overall boot time (this is not my expertise, but I can reassign your thread as needed)

2) probe the PRU remoteproc driver earlier in the kernel boot flow (I haven't tried doing this, not sure how much control we have over sequencing)

3) Initialize PRUs during uboot. This is NOT supported by TI, but I do know of some AM335x customers who were able to hack uboot to initialize their PRU cores then. I'll be limited in the help I can offer, but I can dig around and try to give you some pointers if you want to try this.

Regards,

Nick

Hello Nick,

Thank you for your suggestions.

We are using our custom PRU firmware which is interacting with pins that were configured via the AM4372_IOPAD in our DTS file (pinctrl-single,pins) (I am not expert on DTS files). Our main application is currently managing the PRU (firmware loading and startup) via "/sys/class/remoteproc/remoteproc1/" in user space. The current implementation works but only after the remoteproc is up and running, which is happening very late. Our board uses the AM4376ZDN.

I've been studying this documentation from TI:

• RemoteProc and RPMsg

Based on that, I was wondering we could initialize the PRU, with our custom firmware, much earlier.

1) Agree, that's what I have been working on, and I already got some systemd improvements, unfortunately the remoteproc seems to be one of the last modules initialized. This is something was going to look next and see what systemd service is starting the remoteproc.

2) Good idea, I haven't done that myslef but I will give a try; I wonder this is being affected by systemd.

3) Interesting idea, I will keep that in mind.

Regards,

Eduardo

Hello Eduardo,

Here are some additional threads that may be useful:

Enable auto-boot of PRU cores, so firmware is loaded as soon as Linux remoteproc driver is probed instead of waiting for userspace application 

This might be the quickest thing to try, modify the Linux driver & rebuild kernel / kernel modules:

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/943886/re-am4376-linux-5-4-how-to-auto-load-pru-firmware/3490293#3490293 

It looks like kernel 5.10 uses the same basic driver structure as 5.4, so you could make the same change to
board-support/linux-5.10.100+gitAUTOINC+7a7a3af903-g7a7a3af903/drivers/remoteproc/pru_rproc.c

        /*
         * rproc_add will auto-boot the processor normally, but this is not
         * desired with PRU client driven boot-flow methodology. A PRU
         * application/client driver will boot the corresponding PRU
         * remote-processor as part of its state machine either through the
         * remoteproc sysfs interface or through the equivalent kernel API.
         */
        /*rproc->auto_boot = false; */
        rproc->auto_boot = true;

Other potentially useful links

Booting PRU during uboot 

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1254407/am3358-prevent-pru-from-stopping-during-kernel-boot

Options for auto-booting PRU cores - custom Linux driver 

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/634904/linux-am3358-running-pru-remoteproc-at-boot-time

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/909445/tmdsidk574-pru-not-started-when-linux-boots

Hello Nick,

Thank you, your suggestion looks very promising!

I found the code you mentioned at board-support/linux-5.10.100+gitAUTOINC+7a7a3af903-g7a7a3af903/drivers/remoteproc and added the same changes plus a debug message:

	 * remoteproc sysfs interface or through the equivalent kernel API.
	 */
	/* rproc->auto_boot = false; */
	rproc->auto_boot = true;
	dev_info(dev, "auto_boot = true - pdev->name %s fw_name %s\n", pdev->name, fw_name);

I ran a full clean Yocto build to make sure those changes in the Kernel were included in the final image, but I am not seeing that message with dmesg, and the remoteproc is still not starting sooner. I am certainly doing something wrong; tomorrow I will resume investigating more. I am still hopeful this is the right solution.

Thank you for sending all those helpful links.

Regards,

Eduardo

Hello Nick,

I was finally able to test that change. The PRU auto boot works, but unfortunately the remoteproc initialization is still happening late, for example:

[   26.650399] pru-rproc 54434000.pru: auto_boot = true - pdev->name 54434000.pru fw_name am437x-pru1_0-fw
[   26.650670] remoteproc remoteproc1: 54434000.pru is available
[   26.650957] remoteproc remoteproc1: Direct firmware load for am437x-pru1_0-fw failed with error -2
[   26.650982] remoteproc remoteproc1: powering up 54434000.pru
[   26.651032] remoteproc remoteproc1: Direct firmware load for am437x-pru1_0-fw failed with error -2
[   26.651045] remoteproc remoteproc1: request_firmware failed: -2

But that's okay, at least now I know where the remoteproc source code is in the Kernel. I think the issue is with systemd, I will keep investigating more.

Hello Eduardo,

Glad to hear you are making progress. For the auto-boot, Linux error 2 means the remoteproc driver did not find a firmware to download in the place it was looking.

Remoteproc looks for the firmware in the filesystem under /lib/firmware, based on the firmware name in the Linux devicetree:
arch/arm/boot/dts/am4372.dtsi

                                pru1_0: pru@34000 {
                                        compatible = "ti,am4376-pru";
                                        reg = <0x34000 0x3000>,
                                              <0x22000 0x400>,
                                              <0x22400 0x100>;
                                        reg-names = "iram", "control", "debug";
                                        firmware-name = "am437x-pru1_0-fw";
...
                                };

                                pru1_1: pru@38000 {
                                        compatible = "ti,am4376-pru";
...
                                        firmware-name = "am437x-pru1_1-fw";
etc

I usually like to leave the firmware name alone, and I just add a symbolic link named am437x-pru1_0-fw, etc to /lib/firmware pointing to my actual firmware. e.g.,

$ cd /lib/firmware
$ // let's say you put firmware in /lib/firmware/pru
$ ln -sf /lib/firmware/pru/my_pru_firmware_1_0.out am437x-pru1_0-fw

Good luck with SystemD.

Regards,

Nick

Hello Nick,

Thank you for your explanation and suggestions.

Thank you for pointing out those details about the am4372.dtsi. The symbolic link is a great idea.

Quick update:

For now, I disabled the PRU auto_boot and implemented a systemd service that starts the remoteproc Kernel modules much earlier, but I am still experimenting with:

[Service]
Type=oneshot
StandardOutput=tty
ExecStart=insmod /lib/modules/5.10.100-g7a7a3af903/kernel/drivers/soc/ti/pruss.ko
ExecStart=insmod /lib/modules/5.10.100-g7a7a3af903/kernel/drivers/remoteproc/pru_rproc.ko

It seems to be initializing correctly:

[    4.912645] systemd[1]: Mounted Kernel Configuration File System.
[    5.378262] pru-rproc 54434000.pru: pru_rproc_probe: auto_boot = 0 - pdev->name 54434000.pru fw_name am437x-pru1_0-fw
[    5.378534] remoteproc remoteproc0: 54434000.pru is available
[    5.381422] pru-rproc 54438000.pru: pru_rproc_probe: auto_boot = 0 - pdev->name 54438000.pru fw_name am437x-pru1_1-fw
[    5.381641] remoteproc remoteproc1: 54438000.pru is available
[    5.386933] pru-rproc 54474000.pru: pru_rproc_probe: auto_boot = 0 - pdev->name 54474000.pru fw_name am437x-pru0_0-fw
[    5.387176] remoteproc remoteproc2: 54474000.pru is available
[    5.391009] pru-rproc 54478000.pru: pru_rproc_probe: auto_boot = 0 - pdev->name 54478000.pru fw_name am437x-pru0_1-fw
[    5.393881] remoteproc remoteproc3: 54478000.pru is available
[    5.403963] systemd[1]: start-pru.service: Succeeded.
[    5.427885] systemd[1]: Started PRU modules startup service.

This is still a working in progress, but I just wanted to give you a quick update.

Regards,

Eduardo

Hello Eduardo,

Glad to hear things are moving forward for you! I don't have a ton of experience with SystemD, so I won't be able to help there, but I appreciate the updates.

Let me know if other questions pop up about PRU or how the hardware is behaving.

Also, just FYI: the team and I are working on a PRU academy throughout this year. The goal is to take the PRU Getting Started Labs and expand them into a much more comprehensive discussion of the PRU subsystem and how to write PRU code (e.g., reads/writes, GPI/GPO, INTC, broadside accelerators, Linux + PRU, etc):
https://software-dl.ti.com/processor-sdk-linux/esd/AM437X/09_03_05_02/exports/docs/common/PRU-ICSS/PRU-Getting-Started-Labs.html

If you've got any feedback for us on the existing PRU experience I'd love to hear it. I'll have to focus the academy on more recent devices like AM243x/AM64x, AM263x, & AM62x, but most concepts on AM62x will also apply on AM335x/AM437x/AM57x.

Regards,

Nick

Hello Nick,

Thank you!

One problem I am facing now is this error when trying to load of the firmware via sysfs in user space:

[   11.825878] pru-rproc 54438000.pru: can't change firmware while running

That same code was working fine before I made that change on systemd where I started both the pruss.ko and pru_rproc.ko modules much earlier during boot. It seems as if another PRU firmware was already running. I will keeping investigating more.

Great, I will start to look at that link since it is related to what I am working on now.

Very nice, the PRU academy will certainly be very helpful to all developers. Thank you for you and the TI team working on the PRU academy, very much appreciated!

Regards,

Eduardo

Hello Eduardo,

Hmm, yes that would seem to indicate that firmware is already running on the PRU cores. I don't currently have an AM437x running, so I'll show some terminal output from AM62x.

First off, I assume you still have prints enabled during boot? I would expect you to see remoteproc driver output if the PRU remoteproc driver is actually loading firmware into the PRU cores.

Here are some Linux commands that might be helpful while debugging:

root@am62xx-evm:~# dmesg | grep remoteproc
...
[    8.259308] remoteproc remoteproc2: 30074000.pru is available
[    8.270489] remoteproc remoteproc3: 30078000.pru is available

root@am62xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
...
==> /sys/class/remoteproc/remoteproc2/name <==
30074000.pru

==> /sys/class/remoteproc/remoteproc3/name <==
30078000.pru

root@am62xx-evm:~# cat /sys/class/remoteproc/remoteproc2/state
offline
root@am62xx-evm:~# cat /sys/class/remoteproc/remoteproc2/firmware
am62x-pru0-fw
root@am62xx-evm:~# ls -al /lib/firmware/
total 25496
...
lrwxrwxrwx  1 root root      51 Mar  9  2018 am62x-pru0-fw -> /usr/lib/firmware/pru/PRU_RPMsg_Echo_Interrupt0.out
lrwxrwxrwx  1 root root      51 Mar  9  2018 am62x-pru1-fw -> /usr/lib/firmware/pru/PRU_RPMsg_Echo_Interrupt1.out

Regards,

Nick

Hello Eduardo,

Glad to hear that things are working for you!

I'll mark this thread as resolved, since it sounds like you are successfully initializing PRU earlier in the boot process. If we need to have any more discussion about that subject, feel free to continue the conversation here.

If you have any feedback or questions based on the PRU Getting Started Labs, please create a new e2e thread and ask for me there.

Regards,

Nick

Hello Nick,

Thank you very much for your help, Nick! I might be still putting more updates about my PRU investigation here.

Best regards,

Eduardo

Hello Nick,

Since this issue is related to starting the remoteproc earlier, I thought to post here. I can create a new conversation if you prefer.

I've been doing A/B testing with and without starting the remoteproc earlier.

Earlier:

[   66.377976] virtio_rpmsg_bus virtio0: rpmsg host is online
[   66.378154]  remoteproc1#vdev0buffer: registered virtio0 (type 7)
[   66.378170] remoteproc remoteproc1: remote processor 54438000.pru is now up

Late:

[  135.989416]  remoteproc1#vdev0buffer: registered virtio0 (type 7)
[  135.989444] remoteproc remoteproc1: remote processor 54434000.pru is now up
[  136.007425] virtio_rpmsg_bus virtio0: creating channel rpmsg-pru addr 0x20
[  136.008337] virtio_rpmsg_bus virtio0: rpmsg host is online
[  136.071744] rpmsg_pru virtio0.rpmsg-pru.-1.32: new rpmsg_pru device: /dev/rpmsg_pru32

I am trying to figure it out why with an earlier remoteproc start I am not seeing the new rpmsg_pru device: /dev/rpmsg_pru32 after the PRU had been successfully started.

April/4/25 update: it seems obvious now, but the issue was with the PRU core assignment to a sys class remoteproc handler. For some reason, with the earlier start, the remoteproc1 was assigned to 54438000.pru, but with the late start the remoteproc1 was assigned to 54434000.pru. Our user space code was relying on this path /sys/class/remoteproc/remoteproc1 to load and start the PRU, but the actual PRU core was different in both cases. I haven't yet had the chance to verify why we can only use 54434000.pru core for getting things working with the rpmsg_pru device but once I changed the path to /sys/class/remoteproc/remoteproc0 the rpmsg_pru virtio0.rpmsg-pru started to work with the earlier remoteproc start.

The last improvement to be made, is to start the rpmsg_pru virtio0.rpmsg-pru sooner, as of now, the user space code makes few sys attempts before the rpmsg_pru virtio0.rpmsg-pru.-1.32 is created:

[   48.735769] virtio_rpmsg_bus virtio0: creating channel rpmsg-pru addr 0x20
[   48.736029] virtio_rpmsg_bus virtio0: rpmsg host is online
[   48.736149]  remoteproc0#vdev0buffer: registered virtio0 (type 7)
[   48.736167] remoteproc remoteproc0: remote processor 54434000.pru is now up
[   48.810940] rpmsg_pru virtio0.rpmsg-pru.-1.32: new rpmsg_pru device: /dev/rpmsg_pru32

Regards,

Eduardo