• State Resolved
  • Locked Locked
  • Replies 7 replies
  • Subscribers 99 subscribers
  • Views 573 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Original question:

AM3358: PRU Started with U-Boot Stops When Kernel Starts

AM3358: Prevent PRU from stopping during kernel boot

António Oliveira
Prodigy 165 points
Part Number: AM3358

We are running a custom u-boot which is able to boot the PRU. We can see in the board that the PRU is running while u-boot is running its shell.

The image is built using poky kirkstone with meta-ti-bsp and kernel linux-ti-staging_5.10

From u-boot we load the text and data segments from the firmware file, and also load a blob into the shared ram of the PRU. Then we boot the PRU and can confirm that it does run by watching the board's behaviour.

However, the moment that the kernel starts booting, we see that the PRU stops working in the initial settings. 

If we configure the devicetree for the correct firmware and patch pru-rproc.c to set `rproc->auto_boot = true` the PRU does start during kernel boot, but there is a big gap between the time is stops (~3s mark) and the time it starts again (~17s).

Is there anything we can try to understand:

a) why/where the PRU is being stopped as part of the kernel boot

b) what can be done to prevent it from happening and preserve the PRU state during boot.

Notes:

It is OK if we can restart the PRU from the kernel as long as the gap between STOP/START is less than, say, 500ms.

It is also OK-ish to have the rproc driver coming up with state not maching the actual PRU operation as long as we can restart it from userspace to finally converge. While not elegant it is a tradeoff that we can accept.

At this point we have validated that the pinmux settings are OK and that the PRU runs both from u-boot and from the kernel. What is missing is the ability to run continuously (or with a minor interruption <500ms) when booting the kernel from u-boot.

Any other hints about where to look to are greatly appreciated.

Thanks!

  • 0
    TI__Guru** 100540 points

    Hello Antonio,

    Just to set expectations: As we stated in the "PRU Started with U-Boot Stops When Kernel Starts" thread you linked above, what you are doing has NOT been validated by TI. I will make some suggestions to try to help, but otherwise you are on your own since TI has neither written code for your usecase, nor validated that your usecase will work the way you are describing.

    For future readers, TI has not implemented uboot rproc support for AM335x loading PRU cores before the Linux boot starts. However, we DO have that functionality implemented for other cores in other processors.

    Let's use AM62x as an example. AM62x SDK 8.6 added support for uboot rproc driver booting M4F core as mentioned here: https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/08_06_00_42/exports/docs/devices/AM62X/linux/Release_Specific_Release_Notes.html with documentation for it officially added here: https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/09_00_00_03/exports/docs/linux/Foundational_Components/U-Boot/UG-RemoteProc.html

    The Linux driver for M4F checks to see if the M4F core is already running during Linux boot time. If M4F is already running, then Linux remoteproc does NOT reset the M4F processor. Perhaps you could compare the PRU remoteproc driver to the M4F remoteproc driver to get some ideas on how you want to edit the PRU remoteproc driver to check whether the PRU cores are running before putting them into a reset state during initialization.

    In AM62x SDK 8.6:
    linux-5.10.168+gitAUTOINC+2c23e6c538-g2c23e6c538/drivers/remoteproc/ti_k3_m4_remoteproc.c
    linux-5.10.168+gitAUTOINC+2c23e6c538-g2c23e6c538/drivers/remoteproc/pru_rproc.c

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 165 points

    Hi Nick,

    Many thanks for your reply. We do understand what we are on our own and this question is in fact related to the other in the link you cited.

    Our issue at the moment is that something stops the PRU early in kernel boot. We tried printing out the value of the PRU_CTRL_CTRL register, which reads as `0x00000001` at the end of `pru_rproc_probe()` which is an indication that the PRU is inactive. Thing is, this is already >10 seconds after the board behaviour clearly shows that the PRU is stopped.

    So: we know that the PRU is stopped when the pru_rproc.c driver is probing and we tweaked it to auto-start the PRU at that point, which is OK-ish

    We still have the issue that something else causes the PRU to stop, a few seconds before the pru_rproc driver actually kicks in. Our goal with this request is to have some hint about where to look at so that we can make tweaks as needed in there too.

    Thanks in advance for any ideas or pointers in that regard,

    António

  • 0 in reply to António Oliveira
    TI__Guru** 100540 points

    Hello António,

    Interesting! Off the top of my head I am not sure what would be stopping the PRU cores earlier in the kernel boot process, before the remoteproc driver is getting probed. I am reassigning your thread to another team member to see if they have any ideas about what might be going on. Feel free to ping the thread if you do not get a response by the end of the week.

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 165 points

    Hi Nick, thanks.

    I've been away from this task for a couple of weeks and will return to it soon. Like you pointed out, I'm pinging this thread to see if there's some suggestion of things to look into.

    One thing that occurred to me is whether early in boot the initialization of clocks is stopping the clock that drives the PRU. OR if by some power management feature the peripheral is disabled. I'm not experienced at any of those topics, but that's where gut tells me to start looking in the absence of any other kind of advice.

    If there's anything else that I poke and explore, please just let me know.

    Thanks!

    António

  • 0 in reply to António Oliveira
    TI__Guru** 100540 points

    Edited Sept 6 2023 based on António's later response

    Hello António,

    Thanks for the ping. There was some confusion in the reassignment, so I'll grab your thread back for now.

    Try disabling / deleting PRU nodes from the Linux kernel devicetree

    The first suggestion I got from my teammates is to disable / remove all the PRU related Linux kernel devicetree nodes, and see if the PRU cores are still getting shut down during the boot process. If Linux is not aware of a peripheral or subsystem, they do NOT expect that Linux would cut the clocks to that subsystem. You can check the dtb output to make sure that the associated nodes are actually disabled or deleted. I am NOT sure if you would get different behavior from disabling, as opposed to just deleting all references to the PRU subsystem. The fix was adding property ti,no-reset-on-init. See António's response below.

    I am able to decompile the dtb file on my Linux PC with a command like 
    dtc -I dtb filename.dtb | less

    If you are using RPMsg to communicate between Linux and PRU, I would expect that this would prevent you from using RPMsg. But you can at least see if the cores stop getting shut down.

    Isolating what is causing the behavior during boot time 

    If you want to take a deep dive to figure out exactly what is going on, you would want to start from a minimal kernel image (e.g., disable / delete EVERY peripheral node other than the critical ones to boot to a Linux), and see if the PRU cores are still getting shut down. Potentially useful resources:

    https://dev.ti.com/tirex/explore/node?node=A__AAH6hGIXbUwhj18pp-affA__linux_academy_am62x__XaWts8R__LATEST

    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1259528/faq-sk-am62-how-to-build-initramfs-into-kernel-image

    Regards,

    Nick

  • 0 in reply to Nick Saulnier
    Prodigy 165 points

    Hi Nick,

    Thanks for your pointers, with suggestion to altogether disable/remove the pru devicetree node, I found the solution rather quickly.

    I'm posting the solution here, just in case it may be of help to anyone reading this and looking for the same sort of solution.

    From what I see, this also may apply to other modules where the same kind of behaviour is needed (don't reset when the kernel is loading).

    When I was about to delete the node for the PRU - pruss_tm - I was having a look at it and opened ti-sysc.txt DT bindings doc to learn about some of its properties and that's when I came across this:

    - ti,no-reset-on-init	interconnect target module should not be reset at init

    So I gave it a try and and it worked like a charm! I can now boot the board and in the transition from u-boot to the kernel and finally userspace I can see the display constantly being driven by the PRU without any kind of glitches or blanks.

    So the solution is to basically add the following to the device tree:

    // assuming this:
#include "am33xx.dtsi"


// add this:

&pruss_tm {
    ti,no-reset-on-init;
};

    Caveat

    While the above snippet ensures that the PRU isn't reset on boot, it will show up as offline on sysfs after boot is complete.

    To get it into a consistent state either start the PRU from userspace:

    # assuming the firmware name is already set and is valid...
echo start > /sys/class/remoteproc/remoteproc1/state

    Or patch pru_rproc.c as follows:

    diff --git a/drivers/remoteproc/pru_rproc.c b/drivers/remoteproc/pru_rproc.c
index 1cf71a932249..f1dd1f742108 100644
--- a/drivers/remoteproc/pru_rproc.c
+++ b/drivers/remoteproc/pru_rproc.c
@@ -1141,7 +1141,7 @@ static int pru_rproc_probe(struct platform_device *pdev)
 	 * remote-processor as part of its state machine either through the
 	 * remoteproc sysfs interface or through the equivalent kernel API.
 	 */
-	rproc->auto_boot = false;
+	rproc->auto_boot = true;
 
 	pru = rproc->priv;
 	pru->dev = dev;

    YMMV depending on the application running on the PRU, whether it can be restarted without ill effects, etc. Either way, those tricks may be useful to others.

    Thanks for the support,

    António

  • 0 in reply to António Oliveira
    TI__Guru** 100540 points

    Hey António,

    Thanks for responding back with your solution, you taught me a cool trick today! I'll edit my earlier reply based on your feedback.

    Regards,

    Nick