This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM6422: Linux will got hang if restart the remote R5 core based on the SDK9.02

Part Number: AM6422

Tool/software:

Hi Ti Team,

 

The linux will got hang if restart the remote R5 core.

Here are the test steps:

  1. Using the ipc_rpmsg_echo_linux as R5 firmware (mcu sdk 09_02_01_05)
  2. Start remote core R5 with below command, start successfully.

echo start>/sys/class/remoteproc/remoteproc0/state

  1. Stop remote core R5 with below command, stop successfully.

echo stop>/sys/class/remoteproc/remoteproc0/state

  1. Restart the remote core R5 with step 2, will get hang immediately.

 

By tracing the kernel, which should be blocked by the below code in the ti_sci_do_xfer.

k3_r5_lockstep_release-->ti_sci_cmd_get_device-->ti_sci_do_xfer

 

Have you encounter this issue?

Is there any existing patch to fix it?

 

 

Thanks,

Tiger

  • Hi Nick

    Any update about this issue?

  • Hello Tiger,

    Can you please attach your terminal output so I can see exactly what you are seeing?

    Is this a custom board or a TI EVM? If it is a TI EVM, can you confirm which one?

    What version of software is running on the Linux A53 cores?

    Which R5F core are you restarting? What is the state of the other R5F core?
    I believe that R5F core0 in a subsystem needs to be booted up before booting R5F core1, and also R5F core0 cannot be shut off if R5F core1 is still running.

    Regards,

    Nick

  • Hi Nick,

    Here is the terminal log for your ref:

    root@localhost:~# ls /lib/firmware/ -l
    total 700
    lrwxrwxrwx 1 root root 25 Sep 18 01:22 am64-main-r5f0_0-fw -> /root/am64-main-r5f0_0-fw
    lrwxrwxrwx 1 root root 25 Sep 14 08:57 am64-main-r5f1_0-fw -> /root/am64-main-r5f1_0-fw

    root@localhost:~# dmesg|tail
    [ 543.620479] remoteproc remoteproc0: stopped remote processor 78000000.r5f
    [ 557.041060] remoteproc remoteproc0: powering up 78000000.r5f
    [ 557.041470] remoteproc remoteproc0: Booting fw image am64-main-r5f0_0-fw, size 148616
    [ 557.046292] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [ 557.050033] virtio_rpmsg_bus virtio0: rpmsg host is online
    [ 557.050092] rproc-virtio rproc-virtio.1.auto: registered virtio0 (type 7)
    [ 557.050102] remoteproc remoteproc0: remote processor 78000000.r5f is now up
    [ 557.058047] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd
    [ 557.058306] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe
    [ 562.388428] remoteproc remoteproc0: stopped remote processor 78000000.r5f

    root@localhost:~# echo start>/sys/class/remoteproc/remoteproc0/state
    root@localhost:~# echo stop>/sys/class/remoteproc/remoteproc0/state
    root@localhost:~#
    root@localhost:~# echo start>/sys/class/remoteproc/remoteproc0/state

    [Tiger]: Linux hang here, no any output in the console after several R5 core restarts

    Note:

    1. Not using the TI EVM board, our software version should be using the 09.02.01.

    2. The above testing is only using core0 (the remoteproc0 is link to the core 0).

        This hang will happen too when test with multiple cores, but we can ignore multiple cores case here to narrow down issue.

    3. The first restart might be not hang sometimes, with this scenario, we can try several times, the issue will reproduce soon later.

        In my side, the linux hang will appear in almost 100% with second or third restarts. 

    Are you able to reproduce this issue on your side?

    Thanks,

    Tiger

  • Hello Tiger,

    Hmm, I am seeing your reported behavior on SDK 10.0 with the default filesystem image and the binaries that are bundled into the filesystem by default.

    The output does not look correct to me. I am getting a "timeout waiting for rproc completion event" message, which SHOULD mean that remoteproc refuses to restart the remote core (because of graceful shutdown, discussed here: https://dev.ti.com/tirex/explore/node?a=7qm9DIS__LATEST&node=A__AVt2qZLTY3BgKr3D4YIv8w__AM64-ACADEMY__WI1KRXP__LATEST ). However, it looks like the remote core is still being forced off.

    root@am64xx-evm:~#
    head /sys/class/remoteproc/remoteproc*/name
    ==> /sys/class/remoteproc/remoteproc0/name <==
    5000000.m4fss
    
    ==> /sys/class/remoteproc/remoteproc1/name <==
    78000000.r5f
    ...
    ==> /sys/class/remoteproc/remoteproc2/name <==
    78200000.r5f
    ...
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    
    [602661.859629] platform 78200000.r5f: k3_r5_rproc_stop: timeout waiting for rproc completion event
    [602661.862122] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~#
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [602683.363624] platform 78000000.r5f: k3_r5_rproc_stop: timeout waiting for rproc completion event
    [602683.363857] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [602696.928024] remoteproc remoteproc1: powering up 78000000.r5f
    [602696.929099] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 434436
    // output freezes here

    I am running out of time today, otherwise I would dig through the R5F's remoteproc driver to see what has changed in recent releases. That is good information that the same behavior is seen on SDK 9.2.1.

    Please ping the thread if I do not respond within a couple of business days - there is a lot going on, and I do not want to lose this thread.

    Regards,

    Nick

  • Thanks Nick to help reproduce similar issue on the SDK10.0.

    I will keep touch with you by this thread.

  • Hello Tiger,

    I have verified with the developer that there is an issue with the R5F remoteproc driver - looks like a patch that was added for some other devices with R5F broke graceful shutdown for AM64x on SDK 9.2.1 and SDK 10. We will fix this for SDK 10.1.

    For now, during your development, please load new firmware into the R5F by pointing the remoteproc link to the updated firmware, and rebooting the board, as discussed here:
    https://dev.ti.com/tirex/explore/node?a=7qm9DIS__LATEST&node=A__AdAyuKWUWVV5j4wBc7C6XA__AM64-ACADEMY__WI1KRXP__LATEST

    Regards,

    Nick

  • Hi Nick,

    Could you please help share the patch when it ready? May I know which patch to cause this issue on SDK9.2.1?

    How do I reach to you?

    Thanks,

    Tiger

  • Hello Tiger,

    This thread is a good place to reach me.

    1) Help me understand your usecase here. Are you doing your production release on kernel 6.1 / Linux SDK 9.2.1, and you need the ability to reboot the cores during runtime? Or are you just rebooting cores for your development and debug?

    If you are just using this for development and debug, it is probably easier to simply reboot the board.

    I took a quick look through the commit log for both SDK 9.2.1 (kernel 6.1) & SDK 10.0 (kernel 6.6). I see some patches messing around with graceful shutdown that might have introduced bugs in SDK 10.0, but I don't see anything obvious for SDK 9.2.1.
    https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/log/drivers/remoteproc/ti_k3_r5_remoteproc.c?h=ti-linux-6.1.y-cicd
    https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/log/drivers/remoteproc/ti_k3_r5_remoteproc.c?h=ti-linux-6.6.y-cicd 

    2) Just to double check, can you do this on the Linux terminal to verify exactly which version of the Linux kernel is running on the board?
    $ uname -a

    Regards,

    Nick

  • Hi Nick,

    Need the ability to reboot the cores during runtime. We don't want to reboot the whole board when update R5 firmware or start/stop remote core during runtime.

    And for development, yes it's better we can rebooting cores runtime, which will speed up the development too.

    Please refer to the info as below for uname -a command for SDK9.

    Linux localhost 6.1.82-rt27-00005-g53f003dcf768-dirty #37 SMP PREEMPT_RT Wed Sep 18 10:00:18 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

    Thanks,

    Tiger

  • Hello Tiger,

    Can you please confirm for me exactly where you are getting your Linux driver code?

    e.g.,
    From the SDK download page? https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM64X/09.02.01.10
    From the ti-linux-kernel git repo? If so, which branch? https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/?h=ti-rt-linux-6.1.y-cicd
    Somewhere else?

    I cannot promise a specific timeframe for providing a fix. However, it sounds like the developer has spent a bit of time looking into this and has some commits they are investigating. I am asking them for more information.

    Regards,

    Nick

  • Hi Nick,

    Share SDK package name for your ref as below:

    ti-processor-sdk-linux-rt-am64xx-evm-09.02.01.09-Linux-x86-Install.bin

    Thanks,

    Tiger

  • Hello Tiger,

    Thanks for the confirmation. It might take a couple of days to get back to you. Please ping the thread if I have not replied by Wednesday of next week.

    Regards,

    Nick

  • Any update to avoid close this issue automatically?

  • Hello Tiger,

    E2E threads lock automatically after 30 days of no activity, so no need to ping every day to keep the thread "alive". If any of your threads lock and you need to continue the conversation, feel free to create a new thread and point to the original thread. Our team will either unlock your original thread to continue the conversation there, or continue the conversation on the new thread.

    I'll reply back in a day or so, as discussed last week.

    Regards,

    Nick

  • Hello Tiger,

    Question

    I see you testing core0 of the R5F subsystem. Can you please confirm the state of core1 of the R5F subsystem? Was it already off? Or were you trying to turn core0 on and off while core1 was running?

    SDK 9.2.1.10 - unable to replicate your observations 

    Test with firmware that does not support graceful shutdown: driver times out, does not stop the R5F firmware.

    root@am64xx-evm:~# uname -a
    Linux am64xx-evm 6.1.83-ti-gc1c2f1971fbf #1 SMP PREEMPT Mon May 13 20:19:51 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
    root@am64xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
    ==> /sys/class/remoteproc/remoteproc0/name <==
    78000000.r5f
    
    ==> /sys/class/remoteproc/remoteproc1/name <==
    5000000.m4fss
    ...
    ==> /sys/class/remoteproc/remoteproc2/name <==
    78200000.r5f
    ...
    root@am64xx-evm:~# ls -al /lib/firmware/
    ...
    lrwxrwxrwx  1 root root      55 Mar  9  2018 am64-main-r5f0_0-fw -> /lib/firmware/mcusdk-benchmark_demo/am64-main-r5f0_0-fw
    ...
    lrwxrwxrwx  1 root root      55 Mar  9  2018 am64-main-r5f0_1-fw -> /lib/firmware/mcusdk-benchmark_demo/am64-main-r5f0_1-fw
    
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc2/state
    running
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc0/state
    running
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [ 1079.408548] platform 78200000.r5f: k3_r5_rproc_stop: timeout waiting for rproc completion event
    [ 1079.417291] remoteproc remoteproc2: can't stop rproc: -16
    -sh: echo: write error: Device or resource busy
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc2/state
    running
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc0/state
    [ 1196.400559] platform 78000000.r5f: k3_r5_rproc_stop: timeout waiting for rproc completion event
    [ 1196.409302] remoteproc remoteproc0: can't stop rproc: -16
    -sh: echo: write error: Device or resource busy
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc0/state
    running
    

    test with firmware that does support graceful shutdown.

    The prebuilt binaries for AM64x SDK 9.2.1.10 and earlier are accidentally from
    an earlier MCU+ SDK release that does NOT support graceful shutdown.
    Starting in SDK 10.0, each Linux SDK should have prebuilt binaries from
    the same version of MCU+ SDK. I manually built the ipc_rpmsg_echo_linux
    project in MCU+ SDK 9.2.1.5 and copied it to the filesystem:
    
    root@am64xx-evm:~# ls
    ipc_rpmsg_echo_linux.r5f0_0.out  ipc_rpmsg_echo_linux.r5f0_1.out
    root@am64xx-evm:~# cd /lib/firmware/
    root@am64xx-evm:/lib/firmware# ln -sf ~/ipc_rpmsg_echo_linux.r5f0_0.out am64-main-r5f0_0-fw
    root@am64xx-evm:/lib/firmware# ln -sf ~/ipc_rpmsg_echo_linux.r5f0_1.out am64-main-r5f0_1-fw
    root@am64xx-evm:/lib/firmware# reboot -f
    ...
    root@am64xx-evm:~# uname -a
    Linux am64xx-evm 6.1.83-ti-gc1c2f1971fbf #1 SMP PREEMPT Mon May 13 20:19:51 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
    root@am64xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
    ==> /sys/class/remoteproc/remoteproc0/name <==
    78000000.r5f
    ...
    ==> /sys/class/remoteproc/remoteproc2/name <==
    78200000.r5f
    ...
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc0/state
    running
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc2/state
    running
    
    // start and stop core1 while core0 stays up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  178.434497] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  189.573023] remoteproc remoteproc2: powering up 78200000.r5f
    [  189.579606] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 422812
    [  189.600707] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  189.611200] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  189.616873] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  189.625232] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  189.634776] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  189.636961] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  195.710699] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  198.449033] remoteproc remoteproc2: powering up 78200000.r5f
    [  198.455419] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 422812
    [  198.467254] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  198.477943] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  198.483658] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  198.491605] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  198.498795] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    [  198.506013] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  202.178446] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  204.401029] remoteproc remoteproc2: powering up 78200000.r5f
    [  204.407432] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 422812
    [  204.419160] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  204.434656] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  204.440365] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  204.447036] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  204.452634] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    [  204.459461] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  208.234417] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    
    // start and stop core0 while core1 stays off
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc0/state
    [  213.466431] remoteproc remoteproc0: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc0/state
    [  239.353027] remoteproc remoteproc0: powering up 78000000.r5f
    [  239.359520] remoteproc remoteproc0: Booting fw image am64-main-r5f0_0-fw, size 422816
    [  239.371526] rproc-virtio rproc-virtio.0.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  239.384756] virtio_rpmsg_bus virtio0: rpmsg host is online
    [  239.390441] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd
    [  239.400970] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe
    [  239.408383] rproc-virtio rproc-virtio.0.auto: registered virtio0 (type 7)
    [  239.416091] remoteproc remoteproc0: remote processor 78000000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc0/state
    [  243.966465] remoteproc remoteproc0: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc0/state
    [  247.037058] remoteproc remoteproc0: powering up 78000000.r5f
    [  247.043469] remoteproc remoteproc0: Booting fw image am64-main-r5f0_0-fw, size 422816
    [  247.055166] rproc-virtio rproc-virtio.0.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  247.065628] virtio_rpmsg_bus virtio0: rpmsg host is online
    [  247.071298] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd
    [  247.079047] rproc-virtio rproc-virtio.0.auto: registered virtio0 (type 7)
    [  247.086239] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe
    [  247.093442] remoteproc remoteproc0: remote processor 78000000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc0/state
    [  249.494449] remoteproc remoteproc0: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc0/state
    [  251.633035] remoteproc remoteproc0: powering up 78000000.r5f
    [  251.639447] remoteproc remoteproc0: Booting fw image am64-main-r5f0_0-fw, size 422816
    [  251.651054] rproc-virtio rproc-virtio.0.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  251.661661] virtio_rpmsg_bus virtio0: rpmsg host is online
    [  251.662781] virtio_rpmsg_bus virtio0: creating channel ti.ipc4.ping-pong addr 0xd
    [  251.667345] rproc-virtio rproc-virtio.0.auto: registered virtio0 (type 7)
    [  251.681550] remoteproc remoteproc0: remote processor 78000000.r5f is now up
    [  251.684428] virtio_rpmsg_bus virtio0: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc0/state
    [  254.809344] remoteproc remoteproc0: stopped remote processor 78000000.r5f
    

    SDK 9.2.1.9 - did not have time to test today 

    Will plan to test tomorrow.

    Regards,

    Nick

  • Hello Tiger,

    I am unable to replicate your observations on any SDK 9.2.1, regular Linux or RT Linux (where the two SDK 9.2.1 versions can be downloaded here: https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM64X/09.02.01.09.

    I did notice that your Linux version does not match any of our SDK versions (you reported Linux localhost 6.1.82-rt27-00005-g53f003dcf768-dirty #37 SMP PREEMPT_RT Wed Sep 18 10:00:18 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux). Are you sure this Linux kernel version is not from something else, like your Linux PC? And that you are not actually running Linux SDK 10.0 on the AM64x EVM?

    SDK 9.2.1.9 (NOT RT Linux) - unable to replicate results 

    root@am64xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
    ...
    ==> /sys/class/remoteproc/remoteproc1/name <==
    78000000.r5f
    ...
    ==> /sys/class/remoteproc/remoteproc2/name <==
    78200000.r5f
    ...
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc1/state
    running
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc2/state
    running
    
    // start and stop core1 while core0 is running
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [   85.754423] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [   94.464627] remoteproc remoteproc2: powering up 78200000.r5f
    [   94.471231] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [   94.483379] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [   94.493976] virtio_rpmsg_bus virtio2: rpmsg host is online
    [   94.495031] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [   94.503396] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [   94.515291] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [   94.516581] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [   98.294326] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  100.652622] remoteproc remoteproc2: powering up 78200000.r5f
    [  100.659012] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  100.670339] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  100.682830] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  100.682892] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  100.698793] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  100.705833] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  100.715538] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  104.126001] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  111.380618] remoteproc remoteproc2: powering up 78200000.r5f
    [  111.387076] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  111.398250] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  111.410740] virtio_rpmsg_bus virtio2: rpmsg host is online                                                                                                       [90/7190]
    [  111.411844] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  111.416455] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  111.432597] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    [  111.435541] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  114.722002] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    
    // start and stop core0 while core1 is off
    [  118.730136] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  128.256606] remoteproc remoteproc1: powering up 78000000.r5f
    [  128.263161] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  128.274587] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  128.288391] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  128.294077] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  128.301976] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  128.312495] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    [  128.321805] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [  131.338108] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  134.180615] remoteproc remoteproc1: powering up 78000000.r5f
    [  134.186993] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  134.198904] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  134.209687] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  134.215382] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  134.223196] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  134.230383] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    [  134.237577] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [  137.430053] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  162.536619] remoteproc remoteproc1: powering up 78000000.r5f
    [  162.542990] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  162.552533] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  162.565474] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  162.566240] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  162.571140] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  162.583179] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    [  162.587720] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    
    // try different combinations of core0/core1 being off or on
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  166.852604] remoteproc remoteproc2: powering up 78200000.r5f
    [  166.859008] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  166.870106] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  166.882562] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  166.882649] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  166.888225] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  166.904469] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    [  166.907276] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [  173.646002] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  181.380645] remoteproc remoteproc1: powering up 78000000.r5f
    [  181.387021] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  181.398916] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  181.409936] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  181.415672] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  181.427948] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  181.436905] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    [  181.445856] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [  187.602030] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  192.288616] remoteproc remoteproc1: powering up 78000000.r5f
    [  192.295011] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  192.307818] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  192.322023] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  192.327715] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  192.335723] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    [  192.348467] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  192.356333] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    

    RT Linux SDK 9.2.1.9 - unable to replicate results

    //SDK 9.2.1.9 RT Linux
    root@am64xx-evm:~# uname -a
    Linux am64xx-evm 6.1.80-rt26-ti-rt-g3c08dbfd7bfd #1 SMP PREEMPT_RT Wed Mar 20 14:44:35 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
    root@am64xx-evm:~# ls
    r5f0_0_ipc_rpmsg_echo_linux_count.out  r5f0_1_ipc_rpmsg_echo_linux_count.out
    root@am64xx-evm:~# cd /lib/firmware/
    root@am64xx-evm:/lib/firmware# ls
    LICENCE.ibt_firmware      am64-main-r5f1_0-fw-sec  am64x-pru1_0-fw    am64x-txpru0_1-fw      iwlwifi-8000C-34.ucode             mcusdk-benchmark_demo  tee-raw.bin
    LICENCE.iwlwifi_firmware  am64-main-r5f1_1-fw      am64x-pru1_1-fw    am64x-txpru1_0-fw      iwlwifi-8000C-36.ucode             pru                    tee.bin
    am64-main-r5f0_0-fw       am64-main-r5f1_1-fw-sec  am64x-rtu0_0-fw    am64x-txpru1_1-fw      iwlwifi-8265-34.ucode              regulatory.db          tee.elf
    am64-main-r5f0_0-fw-sec   am64-mcu-m4f0_0-fw       am64x-rtu0_1-fw    bl32.bin               iwlwifi-8265-36.ucode              regulatory.db.p7s      ti-connectivity
    am64-main-r5f0_1-fw       am64-mcu-m4f0_0-fw-sec   am64x-rtu1_0-fw    bl32.elf               iwlwifi-9260-th-b0-jf-b0-34.ucode  tee-header_v2.bin      ti-ipc
    am64-main-r5f0_1-fw-sec   am64x-pru0_0-fw          am64x-rtu1_1-fw    intel                  iwlwifi-9260-th-b0-jf-b0-38.ucode  tee-pageable_v2.bin    ti-pruss
    am64-main-r5f1_0-fw       am64x-pru0_1-fw          am64x-txpru0_0-fw  iwlwifi-3160-17.ucode  iwlwifi-9260-th-b0-jf-b0-46.ucode  tee-pager_v2.bin
    root@am64xx-evm:/lib/firmware# ln -sf ~/r5f0_0_ipc_rpmsg_echo_linux_count.out am64-main-r5f0_0-fw
    root@am64xx-evm:/lib/firmware# ln -sf ~/r5f0_1_ipc_rpmsg_echo_linux_count.out am64-main-r5f0_1-fw
    root@am64xx-evm:/lib/firmware# reboot -f
    ...
    root@am64xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
    ==> /sys/class/remoteproc/remoteproc0/name <==
    5000000.m4fss
    
    ==> /sys/class/remoteproc/remoteproc1/name <==
    78000000.r5f
    ...
    ==> /sys/class/remoteproc/remoteproc2/name <==
    78200000.r5f
    ...
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc1/state
    running
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc2/state
    running
    
    // test stopping core1 while core0 is on
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [   88.843984] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [   94.391643] remoteproc remoteproc2: powering up 78200000.r5f
    [   94.394469] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    root@am64xx-evm:~# [   94.397404] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [   94.406425] virtio_rpmsg_bus virtio2: rpmsg host is online
    [   94.406574] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [   94.406584] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [   94.407059] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [   94.409531] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    root@am64xx-evm:~# [   97.470261] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  100.233667] remoteproc remoteproc2: powering up 78200000.r5f
    [  100.235733] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  100.238545] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  100.241147] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  100.241260] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  100.241269] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  100.242165] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  100.242402] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    root@am64xx-evm:~# [  104.204524] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  106.555682] remoteproc remoteproc2: powering up 78200000.r5f
    [  106.556479] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  106.561703] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  106.571623] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  106.571676] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  106.571686] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  106.572328] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  106.579270] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    root@am64xx-evm:~# [  108.664234] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    
    // test stopping core0 while core1 is off
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [  115.983151] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  121.155647] remoteproc remoteproc1: powering up 78000000.r5f
    [  121.157911] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    root@am64xx-evm:~# [  121.160815] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  121.170513] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  121.170577] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  121.170586] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    [  121.171513] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  121.172505] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [  124.870138] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  126.729666] remoteproc remoteproc1: powering up 78000000.r5f
    [  126.731691] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  126.736077] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  126.738571] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  126.738622] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    root@am64xx-evm:~# [  126.738631] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    [  126.738773] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  126.738889] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    root@am64xx-evm:~# [  128.873173] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  138.926652] remoteproc remoteproc1: powering up 78000000.r5f
    [  138.927500] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  138.941975] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    root@am64xx-evm:~# [  138.944963] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  138.945345] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  138.945361] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    [  138.945663] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  138.947862] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    
    // test a couple configurations of core0 and core1 being off and on
    echo start > /sys/class/remoteproc/remoteproc2/state
    [  142.370679] remoteproc remoteproc2: powering up 78200000.r5f
    [  142.371460] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  142.376998] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  142.383863] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  142.383948] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  142.384066] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  142.384075] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  142.384278] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    root@am64xx-evm:~# [  145.780428] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  154.632095] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  157.161650] remoteproc remoteproc2: powering up 78200000.r5f
    [  157.163746] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  157.176536] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  157.179916] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  157.179966] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  157.179974] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  157.180153] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  157.180378] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  159.846565] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  161.681660] remoteproc remoteproc2: powering up 78200000.r5f
    [  161.683698] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  161.696159] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  161.700353] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  161.700402] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  161.700411] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  161.700735] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  161.700919] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  163.222164] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    

    RT Linux SDK 9.2.1.10 - unable to replicate results

    root@am64xx-evm:~# uname -a
    Linux am64xx-evm 6.1.83-rt28-ti-rt-g96b0ebd82722 #1 SMP PREEMPT_RT Mon May 13 23:06:24 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
    
    root@am64xx-evm:~# ls
    r5f0_0_ipc_rpmsg_echo_linux_count.out  r5f0_1_ipc_rpmsg_echo_linux_count.out
    root@am64xx-evm:~# cd /lib/firmware/
    root@am64xx-evm:/lib/firmware# ls
    LICENCE.ibt_firmware      am64-main-r5f1_0-fw-sec  am64x-pru1_0-fw    am64x-txpru0_1-fw      iwlwifi-8000C-34.ucode             mcusdk-benchmark_demo  tee-raw.bin
    LICENCE.iwlwifi_firmware  am64-main-r5f1_1-fw      am64x-pru1_1-fw    am64x-txpru1_0-fw      iwlwifi-8000C-36.ucode             pru                    tee.bin
    am64-main-r5f0_0-fw       am64-main-r5f1_1-fw-sec  am64x-rtu0_0-fw    am64x-txpru1_1-fw      iwlwifi-8265-34.ucode              regulatory.db          tee.elf
    am64-main-r5f0_0-fw-sec   am64-mcu-m4f0_0-fw       am64x-rtu0_1-fw    bl32.bin               iwlwifi-8265-36.ucode              regulatory.db.p7s      ti-connectivity
    am64-main-r5f0_1-fw       am64-mcu-m4f0_0-fw-sec   am64x-rtu1_0-fw    bl32.elf               iwlwifi-9260-th-b0-jf-b0-34.ucode  tee-header_v2.bin      ti-ipc
    am64-main-r5f0_1-fw-sec   am64x-pru0_0-fw          am64x-rtu1_1-fw    intel                  iwlwifi-9260-th-b0-jf-b0-38.ucode  tee-pageable_v2.bin    ti-pruss
    am64-main-r5f1_0-fw       am64x-pru0_1-fw          am64x-txpru0_0-fw  iwlwifi-3160-17.ucode  iwlwifi-9260-th-b0-jf-b0-46.ucode  tee-pager_v2.bin
    root@am64xx-evm:/lib/firmware# ln -sf ~/r5f0_0_ipc_rpmsg_echo_linux_count.out am64-main-r5f0_0-fw
    root@am64xx-evm:/lib/firmware# ln -sf ~/r5f0_1_ipc_rpmsg_echo_linux_count.out am64-main-r5f0_1-fw
    root@am64xx-evm:/lib/firmware# reboot -f
    
    root@am64xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
    ==> /sys/class/remoteproc/remoteproc0/name <==
    5000000.m4fss
    
    ==> /sys/class/remoteproc/remoteproc1/name <==
    78000000.r5f
    ...
    ==> /sys/class/remoteproc/remoteproc2/name <==
    78200000.r5f
    ...
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc1/state
    running
    root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc2/state
    running
    
    // try turning core1 off and on while core0 is up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [   76.350981] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [   83.973722] remoteproc remoteproc2: powering up 78200000.r5f
    [   83.975955] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [   83.978779] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [   83.987456] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [   83.987656] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# [   83.990021] virtio_rpmsg_bus virtio2: rpmsg host is online
    [   83.990160] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [   83.990172] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [   87.804235] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [   89.717760] remoteproc remoteproc2: powering up 78200000.r5f
    [   89.718565] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [   89.727705] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [   89.733579] virtio_rpmsg_bus virtio2: rpmsg host is online
    [   89.733634] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [   89.733643] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [   89.734345] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    root@am64xx-evm:~# [   89.734561] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    root@am64xx-evm:~# [   91.839334] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    
    // try turning core0 off and on while core1 is off
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [   99.114181] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  104.685733] remoteproc remoteproc1: powering up 78000000.r5f
    [  104.687859] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  104.700601] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  104.704182] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  104.704236] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  104.704245] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    [  104.704272] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    root@am64xx-evm:~# [  104.704454] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    [  107.737271] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  109.559745] remoteproc remoteproc1: powering up 78000000.r5f
    [  109.561777] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  109.574473] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  109.577868] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  109.577924] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  109.577933] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    [  109.578118] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    root@am64xx-evm:~# [  109.578253] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    root@am64xx-evm:~# [  111.874209] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  114.342724] remoteproc remoteproc1: powering up 78000000.r5f
    [  114.343519] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  114.356729] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  114.361459] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  114.361666] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# [  114.362135] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  114.362185] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  114.362194] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    root@am64xx-evm:~# [  116.948235] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    
    // try turning core1 off and on while core0 is off
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  123.212729] remoteproc remoteproc2: powering up 78200000.r5f
    [  123.213549] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  123.227661] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  123.234369] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  123.234578] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    [  123.235061] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  123.235111] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    root@am64xx-evm:~# [  123.235120] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  129.245443] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  131.155905] remoteproc remoteproc2: powering up 78200000.r5f
    [  131.163762] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  131.174892] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  131.181626] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  131.181682] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  131.181692] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  131.182516] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  131.184945] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    [  133.132422] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  135.750739] remoteproc remoteproc2: powering up 78200000.r5f
    [  135.752853] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  135.765572] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a1000000
    [  135.768626] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  135.768683] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    root@am64xx-evm:~# [  135.768692] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  135.769219] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  135.769461] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc2/state
    root@am64xx-evm:~# [  137.676250] remoteproc remoteproc2: stopped remote processor 78200000.r5f
    
    // try turning core0 off and on while core1 is on
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc2/state
    [  420.391735] remoteproc remoteproc2: powering up 78200000.r5f
    [  420.393790] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 398148
    [  420.406509] rproc-virtio rproc-virtio.1.auto: assigned reserved memory node r5f-dma-memory@a1000000
    root@am64xx-evm:~# [  420.409184] virtio_rpmsg_bus virtio1: creating channel ti.ipc4.ping-pong addr 0xd
    [  420.410054] virtio_rpmsg_bus virtio1: rpmsg host is online
    [  420.413559] rproc-virtio rproc-virtio.1.auto: registered virtio1 (type 7)
    [  420.413579] remoteproc remoteproc2: remote processor 78200000.r5f is now up
    [  420.414599] virtio_rpmsg_bus virtio1: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    -sh: echo: write error: Invalid argument
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  432.928734] remoteproc remoteproc1: powering up 78000000.r5f
    [  432.930000] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    root@am64xx-evm:~# [  432.943431] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  432.945042] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  432.945104] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  432.945168] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  432.945444] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    [  432.945717] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    root@am64xx-evm:~# [  436.358335] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  439.217725] remoteproc remoteproc1: powering up 78000000.r5f
    [  439.219876] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  439.224332] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  439.232569] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  439.232628] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  439.232637] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    root@am64xx-evm:~# [  439.232970] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    [  439.233129] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    root@am64xx-evm:~# [  441.245271] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    root@am64xx-evm:~# echo start > /sys/class/remoteproc/remoteproc1/state
    [  443.331739] remoteproc remoteproc1: powering up 78000000.r5f
    [  443.333781] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 398152
    [  443.338039] rproc-virtio rproc-virtio.2.auto: assigned reserved memory node r5f-dma-memory@a0000000
    [  443.347810] virtio_rpmsg_bus virtio2: rpmsg host is online
    [  443.347864] rproc-virtio rproc-virtio.2.auto: registered virtio2 (type 7)
    [  443.347873] remoteproc remoteproc1: remote processor 78000000.r5f is now up
    [  443.348773] virtio_rpmsg_bus virtio2: creating channel ti.ipc4.ping-pong addr 0xd
    root@am64xx-evm:~# [  443.355647] virtio_rpmsg_bus virtio2: creating channel rpmsg_chrdev addr 0xe
    root@am64xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc1/state
    root@am64xx-evm:~# [  445.207343] remoteproc remoteproc1: stopped remote processor 78000000.r5f
    

    Regards,

    Nick