AM6252: Load M4 firmware generate from MCU RTOS SDK

Hello Tony,

Apologies for the delayed response. If the customer is just trying to build and run the IPC Echo firmware, there is no need to update the memory allocations in the devicetree file. The Linux devicetree file and the M4F project configs will be aligned.

Information around building and running the MCU+ SDK examples is in a webinar I gave last year. Please take a look at https://training.ti.com/process-inter-processor-communication , starting around 34 minutes, 30 seconds. That will include information, including what generated binary can be loaded. I included the links to the associated MCU+ SDK docs at the bottom of the slides. If you watch long enough, I show in the terminal what files are used, and how to load them.

Unfortunately I do not have that information about building the MCU+ application written down at this point. We document building the Linux userspace and kernel space applications at https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/08_03_00_19/exports/docs/linux/Foundational_Components_IPC62x.html#rpmsg-examples .

Regards,

Nick

Nick Saulnier said:

If the customer is just trying to build and run the IPC Echo firmware, there is no need to update the memory allocations in the devicetree file.

It is the customer wanted, don't use IPC, would like to use Mailbox directly in firmware.

so just put generated .out to /lib/firmware/pdk-ipc/, Linux will load it automatically?

Hello Tony,

On using just a mailbox for IPC

To clarify: We currently provide RPMsg as an IPC mechanism between remote cores and Linux userspace. That involves a mailbox used to notify the remote core, and a 512 Byte RPMsg message with the content of the information.

We do provide a mailbox-only driver for MCU+ to MCU+ cores (applicable to AM64x, not applicable to AM62x at this point in time). However, at this point we have not written a driver to expose just mailboxes up to Linux userspace. Information on those AM64x MCU+ drivers is here: https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/08_02_00_31/exports/docs/api_guide_am64x/IPC_GUIDE.html

On loading a specific firmware into a remote core

Linux looks for a specific firmware name when initializing a remote core. The new AM62x Linux Academy may be a bit easier to understand than the SDK docs. Please point the customer here:
https://dev.ti.com/tirex/explore/node?node=AO2IbV53aUarnRvmyK9fxQ__XaWts8R__LATEST
sections
Evaluating Linux > Booting Remote Cores from Linux
and
Evaluating Linux > Tour of TI Linux > IPC.

Regards,

Nick

Nick, 

#1. There is not such section on on-line AM62x academy:

While AM64x has:

#2. Build out gpio_led_blink_am62x-sk_m4fss0-0_freertos_ti-arm-clang.out  from RTOS SDK example, and copy to /lib/firmware/pdk-ipc/

I follow steps in user guide, get error as below:

root@am62xx-evm:~# ^C
root@am62xx-evm:~# ls /lib/firmware/pdk-ipc/
gpio_led_blink_am62x-sk_m4fss0-0_freertos_ti-arm-clang.out
ipc_echo_baremetal_test_mcu2_0_release_strip.xer5f
ipc_echo_testb_mcu1_0_release_strip.xer5f
root@am62xx-evm:~#

ls -l /lib/firmware/
total 46716
-rw-r--r-- 1 root root 2040 May 30 2022 LICENCE.ibt_firmware
-rw-r--r-- 1 root root 2046 May 30 2022 LICENCE.iwlwifi_firmware
lrwxrwxrwx 1 root root 63 May 30 2022 am62-main-r5f0_0-fw -> /lib/firmware/pdk-ipc/ipc_echo_te
stb_mcu1_0_release_strip.xer5f
lrwxrwxrwx 1 root root 80 May 30 2022 am62-mcu-m4f0_0-fw -> /lib/firmware/pdk-ipc/gpio_led_bli
nk_am62x-sk_m4fss0-0_freertos_ti-arm-clang.out
lrwxrwxrwx 1 root root 47 May 30 2022 am62x-pru0-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interr
upt0.out
lrwxrwxrwx 1 root root 47 May 30 2022 am62x-pru1-fw -> /lib/firmware/pru/PRU_RPMsg_Echo_Interr
upt1.out

root@am62xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
==> /sys/class/remoteproc/remoteproc0/name <==
5000000.m4fss

==> /sys/class/remoteproc/remoteproc1/name <==
30074000.pru

==> /sys/class/remoteproc/remoteproc2/name <==
30078000.pru
root@am62xx-evm:~# echo stop > /sys/class/remoteproc/remoteproc0/state
-sh: echo: write error: Invalid argument
root@am62xx-evm:~# ^C
root@am62xx-evm:~# echo start > /sys/class/remoteproc/remoteproc0/state
[ 406.622201] remoteproc remoteproc0: powering up 5000000.m4fss
[ 406.629035] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 244496
[ 406.659384] remoteproc remoteproc0: Boot failed: -22
-sh: echo: write error: Invalid argument
root@am62xx-evm:~#

Hello Tony,

Apologies for the delayed response, looks like my reply from last week stayed as a draft instead of hitting the "Reply" button!

Hmm, the IPC files were definitely merged into the AM62x Linux Academy project. There were some issues with graphics that meant we could not integrate new pull requests for a couple weeks back in June, let me check to see if that has been fixed yet.

In the meantime, please take a look at the latest AM64x Linux Academy project:
https://dev.ti.com/tirex/explore/node?node=AGqWQH5YSU6rleHcUT-o3w__7qm9DIS__LATEST

Please note that you need to reboot the AM62x for now when loading a new binary into the remote core as per the note in the documentation:
"The RemoteProc driver does not support a graceful shutdown of R5 and M4 cores in the current Linux Processor SDK. For now, it is recommended to reboot the board to load new binaries into an R5F or M4F core."

Regards,

Nick

Nick, 

even reboot and did not report error in log, but the status is offline, and can't start.

[ 6.353962] k3-m4-rproc 5000000.m4fss: assigned reserved memory node m4f-dma-
memory@9cb00000
[ 6.368794] k3-m4-rproc 5000000.m4fss: configured M4 for remoteproc mode
[ 6.385733] k3-m4-rproc 5000000.m4fss: local reset is deasserted for device
[ 6.398365] remoteproc remoteproc0: 5000000.m4fss is available
[ 6.421126] remoteproc remoteproc0: powering up 5000000.m4fss
[ 6.427148] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size
244496
[ 6.724872] PVR_K: 209: Read BVNC 33.15.11.3 from HW device registers

root@am62xx-evm:~# cat /sys/class/remoteproc/remoteproc0/state
offline

root@am62xx-evm:~# echo start > /sys/class/remoteproc/remoteproc0/state
[ 2173.202355] remoteproc remoteproc0: powering up 5000000.m4fss
[ 2173.209017] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size
244496
[ 2173.238620] remoteproc remoteproc0: Boot failed: -22
-sh: echo: write error: Invalid argument
root@am62xx-evm:~#

root@am62xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
==> /sys/class/remoteproc/remoteproc0/name <==
5000000.m4fss

==> /sys/class/remoteproc/remoteproc1/name <==
30074000.pru

==> /sys/class/remoteproc/remoteproc2/name <==
30078000.pru
root@am62xx-evm:~#

BTW: we did not enable IPC in the MCU example. if enabled IPC, all are OK. 

so how to load and run M4 firmware without IPC enabled?

Hello Tony,

Update on AM62x Linux Academy files 

The updated version of AM62x Linux Academy is finally live, but for some reason it is not getting populated as the "latest version". The link to the latest as of right now is here: https://dev.ti.com/tirex/explore/node?a=XaWts8R__8.3.0.0%20v2&node=AO2IbV53aUarnRvmyK9fxQ__XaWts8R__8.3.0.0%20v2&r=XaWts8R__LATEST

On debugging the customer issue 

To confirm, when the customer loads an unmodified TI example like gpio_led_blink or gpio_input_interrupt, do those firmwares load automatically during Linux boot? If not, which TI firmware is observed as not loading?

Please provide specific details about "did not enable IPC" and "enabled IPC".

Regards,

Nick

Nick,

Nick Saulnier said:

To confirm, when the customer loads an unmodified TI example like gpio_led_blink or gpio_input_interrupt, do those firmwares load automatically during Linux boot? If not, which TI firmware is observed as not loading?

it is software link to new .out file, so the specified file loaded from boot log. but state is offline if not add IPC in ccs project.

Nick Saulnier said:

Please provide specific details about "did not enable IPC" and "enabled IPC"

Enable IPC means add IPC in .syscfg and add .resource table in cmd file SECTIONS.

Notes: change the DDR_0 ORIGIN to be same as .k3-am625-sk.dts.

If not enable IPC in CCS project, the .out is load according to boot log, but the MCU state is Offline.

Nick, 

Did you reproduced it?

Hello Tony,

I did not have time to try running an unmodified gpio_input_interrupt or gpio_led_blink example today. I will give it a try on Friday.

Regards,

Nick

Hello Tony,

Ok, I am able to replicate your test results on my side.

root@am62xx-evm:~# cat /sys/class/remoteproc/remoteproc0/state
offline
root@am62xx-evm:~# head /sys/class/remoteproc/remoteproc*/name
==> /sys/class/remoteproc/remoteproc0/name <==
5000000.m4fss

==> /sys/class/remoteproc/remoteproc1/name <==
30074000.pru

==> /sys/class/remoteproc/remoteproc2/name <==
30078000.pru
root@am62xx-evm:~# dmesg | grep remoteproc
[    5.469592] k3-m4-rproc 5000000.m4fss: configured M4 for remoteproc mode
[    5.508487] remoteproc remoteproc0: 5000000.m4fss is available
[    5.536651] remoteproc remoteproc0: powering up 5000000.m4fss
[    5.542897] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 239668
[    7.240644] remoteproc remoteproc1: 30074000.pru is available
[    7.248180] remoteproc remoteproc2: 30078000.pru is available
root@am62xx-evm:~# echo [  129.254789] Initializing XFRM netlink socket
dmesg | grep boot
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    1.535294] printk: bootconsole [ns16550a0] disabled
[    1.612590] 0x000000000000-0x000000080000 : "ospi.tiboot3"
[    1.625723] 0x000000280000-0x000000680000 : "ospi.u-boot"
[    2.088477] mmcblk0boot0: mmc0:0001 S0J56X partition 1 31.5 MiB
[    2.094581] mmcblk0boot1: mmc0:0001 S0J56X partition 2 31.5 MiB
[    3.995718] systemd[1]: Starting Start psplash boot splash screen...
[    9.388826] wlcore: firmware booted (Rev 8.9.0.0.86)
root@am62xx-evm:~# dmesg | grep Boot
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[    0.092908] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[    0.104264] CPU2: Booted secondary processor 0x0000000002 [0x410fd034]
[    0.115516] CPU3: Booted secondary processor 0x0000000003 [0x410fd034]
[    5.542897] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 239668
root@am62xx-evm:~# echo start > /sys/class/remoteproc/remoteproc0/state
[  236.038662] remoteproc remoteproc0: powering up 5000000.m4fss
[  236.044926] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 239668
[  236.082388] remoteproc remoteproc0: Boot failed: -22
-sh: echo: write error: Invalid argument
root@am62xx-evm:~# ls -al /lib/firmware/
total 46720
drwxr-xr-x  7 root root    4096 May 19  2022 .
drwxr-xr-x 10 root root    4096 May 17  2022 ..
-rw-r--r--  1 root root    2040 May 17  2022 LICENCE.ibt_firmware
-rw-r--r--  1 root root    2046 May 17  2022 LICENCE.iwlwifi_firmware
lrwxrwxrwx  1 root root      63 May 17  2022 am62-main-r5f0_0-fw -> /lib/firmware/pdk-ipc/ipc_echo_testb_mcu1_0_release_strip.xer5f
lrwxrwxrwx  1 root root      54 May 19  2022 am62-mcu-m4f0_0-fw -> /lib/firmware/pdk-ipc/gpio_input_interrupt.release.out

I will do some more digging on Monday.

Regards,

Nick

Hello Tony,

I apologize, I was not able to make much progress on this before leaving on vacation for the rest of August. I am reassigning this to our SW apps team members who are focused on the MCU+ SDK side to drive progress for the next couple of weeks.

I suspect there is something going on with the MCU+ project settings. I don't see anything different between main.c files, maybe Linux expects the MCU+ core to have specific sysconfig settings?

I also realized I only tested an example that is bare metal (gpio_input_interrupt). I did not test the RTOS version of gpio_led_blink to see if it would load (i.e., if this is a freertos vs bare metal issue).

Please ping the thread if you do not get another update within a couple of days.

Regards,

Nick

Nick,

It is same with rtos example. 

as long as with ipc enabled in example. can start anytime:

root@am62xx-evm:/lib/firmware# sudo ln -sf gpio_led_blink_am62x-sk_m4fss0-0_free
rtos_ti-arm-clang-ipc.out am62-mcu-m4f0_0-fw
root@am62xx-evm:/lib/firmware# sync
root@am62xx-evm:/lib/firmware# echo start > /sys/class/remoteproc/remoteproc0/st
ate
[ 1085.853378] remoteproc remoteproc0: powering up 5000000.m4fss
[ 1085.906891] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size
301908
[ 1085.931867] remoteproc0#vdev0buffer: assigned reserved memory node m4f-dma-m
emory@9cb00000
[ 1085.940641] remoteproc0#vdev0buffer: registered virtio0 (type 7)
[ 1085.946877] remoteproc remoteproc0: remote processor 5000000.m4fss is now up
root@am62xx-evm:/lib/firmware# [ 1085.957398] virtio_rpmsg_bus virtio0: rpmsg ho
st is online

root@am62xx-evm:/lib/firmware#

I compared map file, the major difference is IPC example has a .resource_table section, while no ipc example has not.

Hello Tony,

I am back from vacation, but I do not have full access to my EVMs this week. When I went through the RemoteProc driver with the developer, it looks like the remoteproc driver does require the resource table to be present in the remote core binary. Can you try modifying the linker.cmd file of the gpio example to match the linker.cmd file of the ipc example?

i.e.,
examples/drivers/gpio/gpio_input_interrupt/am62x-sk/m4fss0-0_nortos/ti-arm-clang/linker.cmd
to match
examples/drivers/ipc/ipc_rpmsg_echo_linux/am62x-sk/m4fss0-0_freertos/ti-arm-clang/linker.cmd

rebuild the gpio example and see if it will successfully load.

Depending on what you see, we will need to update our documentation to make this clear.

Regards,

Nick

Nick,

Project doesn't generate .resource_table section without enabling IPC. so it doesn't take effect.

Nick,

Still need your help.

Hello Tony,

I have not been able to test, since I have not had the correct resources in my remote location. I hope to be able to go into the office to test tomorrow or the next day.

To confirm, will it load if you manually change the linker.cmd file? (without changing the sysconfig settings)

I would be curious to see if the output binary size changes when Linux IPC is enabled or disabled.

Regards,

Nick

Nick,

OK, wait for you.

Nick Saulnier said:

To confirm, will it load if you manually change the linker.cmd file? (without changing the sysconfig settings)

although change .cmd, as there is not such a section in project, it won't generate such section in binary.

Hello Tony,

It took a couple days of messing around with CCS for me to catch up to you, apologies for not posting updates during that process. Your previous e2e questions were very helpful as I was getting everything running. I'll talk about where we are at now, and potential things to try.

First off, I have validated your observations:

1) Just changing the linker.cmd file does not generate a reserved DDR section for the Linux driver to see, and the project fails to load

2) gpio_led_blink can be loaded by Linux after updating both the linker.cmd file and example.syscfg file to add the IPC module:

const ipc             = scripting.addModule("/drivers/ipc/ipc");

Next steps

The easiest solution is to just make sure IPC is included in projects that are loaded by AM62x ARM core (potentially also applies to AM64x, I have not tested at this point). When I look at the map files, it looks like adding IPC to a release build adds about 0x1000 of code to M4F IRAM and 0x2200 of code to M4F DRAM.

Another option is to modify the Linux driver. The default Linux error outputs on my build are not set to see exactly where in the remoteproc driver the firmware is failing to be loaded. Before modifying the Linux driver, I would suggest changing the kernel logging levels as documented here: https://training.ti.com/debugging-embedded-linux-systems-training-series

Finally, we could try seeing if we can get the M4F project to reserve space in DDR without having to include the IPC module. That might allow the Linux driver to load the remote core firmware. I am reaching out to the MCU+ developers to see if they have any suggestions there.

I want to continue making progress here. I'm on vacation Monday, but please ping the thread if I have not responded by the middle of next week.

Regards,

Nick

Nick, 

We need a pure M4 firmware loader. waiting for you.

Hi Tony, as per Nick's earlier message he's OOO today Monday. Please allow another day or two  for a response. Thanks.

Hello Tony,

Still waiting to hear back from the MCU+ developers on adding a resource table to the remote core firmware binary without including the IPC modules. I have pinged them again.

I have verified this behavior for both AM62x & AM64x, for both M4F and R5F.

Regards,

Nick

Nick,

Thanks for update, we verified it works also. 

How to you think change Linux rpmsg driver to support firmware without resource table?

Hello Tony,

I am checking with the Linux developer on feasibility of modifying the RemoteProc driver to not require a resource table.

To confirm: Does your design need something that is specifically NOT addressed by the workaround described in my previous response? (in other words, what is the added value we get by asking the developers to do additional work to modify the Linux driver?)

Regards,

Nick

Nick,

Thanks for your help, customer can adopt the workaround by now.

But we think firmware loader should be a stand alone function, not relate to firmware itself. and the unused resource table exist there looks stranger although occupied memory is not larger.

Hello Tony,

Understood.

As a bit of background, the workaround above is very similar to what we required to load PRU binaries for every software release before Linux kernel 5.10 (even if RPMsg was not used, each PRU project had to have an empty resource table). For more information on resource tables with PRU firmware, reference
https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/08_04_01_03/exports/docs/common/PRU-ICSS/Resource_Tables.html

I am not sure why the developer changed the PRU remoteproc driver to be able to load with or without a resource table, but did not change R5F/M4F remoteproc driver. Discussions are ongoing. Please ping the thread if I have not provided an update by next week.

Regards,

Nick

Hello Tony,

Discussions are still ongoing with the developers. The developer is on vacation for the rest of November, so I won't have any more updates about plans for the future of the driver for a while.

It sounds like it would take a good amount of effort to modify the R5F and M4F drivers so that they could support booting cores with and without resource tables. So the question we are dealing with now is, "What is the added value for supporting non resource-table firmwares, and it is worth the effort?"

For the short term, I am moving forward on documenting the "no IPC resource table" workaround described in my previous response: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1114668/am6252-load-m4-firmware-generate-from-mcu-rtos-sdk/4360349#4360349

Regards,

Nick