AM625: DM R5F via rpmsg_char_simple

Hello John,

The first step is making sure that the firmware running on the DM R5F actually is running RPMsg Echo code. If you are using the default Linux filesystem image, then your AM62x DM R5F will be running the IPC Echo example in a separate task, alongside the Device Management (DM) task.

Please start by trying out the steps in our AM62x academy, Linux module, page IPC Example:
https://dev.ti.com/tirex/explore/node?node=A__AXINfJJ0T8V7CR5pTK41ww__AM62-ACADEMY__uiYMDcq__LATEST

Since DM R5F is included in the Linux devicetree file, I don't need to bind/unbind. RemoteProc already "attached" to the DM R5F, and set up IPC communication with the DM R5F during boot:

root@am62xx-evm:~# uname -a
Linux am62xx-evm 6.1.80-ti-g2e423244f8c0 #1 SMP PREEMPT Wed Mar 20 14:43:33 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
root@am62xx-evm:~# dmesg | grep remoteproc
[    5.698173] k3-m4-rproc 5000000.m4fss: configured M4 for remoteproc mode
[    5.727342] remoteproc remoteproc0: 5000000.m4fss is available
[    5.744247] remoteproc remoteproc0: powering up 5000000.m4fss
[    5.750309] remoteproc remoteproc0: Booting fw image am62-mcu-m4f0_0-fw, size 55016
[    5.804783] remoteproc remoteproc0: remote processor 5000000.m4fss is now up
[    6.130999] remoteproc remoteproc1: 78000000.r5f is available
[    6.148848] remoteproc remoteproc1: attaching to 78000000.r5f
[    6.289572] remoteproc remoteproc1: remote processor 78000000.r5f is now attached
[    8.260595] remoteproc remoteproc2: 30074000.pru is available
[    8.294837] remoteproc remoteproc3: 30078000.pru is available
root@am62xx-evm:~# cat /sys/class/remoteproc/remoteproc1/state
attached

When I check on SDK 9.1, it runs by default:

root@am62xx-evm:~# rpmsg_char_simple -r 9 -n 1
Created endpt device rpmsg-char-9-510371, fd = 4 port = 1025
Exchanging 1 messages with rpmsg device rpmsg-char-9-510371 on rproc id 9 ...

Sending message #0: hello there 0!
Receiving message #0: hello there 0!

Communicated 1 messages successfully on rpmsg-char-9-510371

TEST STATUS: PASSED
root@am62xx-evm:~# rpmsg_char_simple -r 15 -n 1
Created endpt device rpmsg-char-15-510399, fd = 4 port = 1025
Exchanging 1 messages with rpmsg device rpmsg-char-15-510399 on rproc id 15 ...

Sending message #0: hello there 0!
Receiving message #0: hello there 0!

Communicated 1 messages successfully on rpmsg-char-15-510399

TEST STATUS: PASSED

I would first start by making sure that you can run the TI example for RPMsg IPC with the default settings. Then, once you have a "known good" software project, you can start modifying the code for your own usecase.

For followup lessons on IPC & writing remote core code, I would refer to the AM62x academy's multicore module, like page "How to develop with RPMsg IPC"
https://dev.ti.com/tirex/explore/node?node=A__AVjm7chph.4Q-bCWodAr.w__AM62-ACADEMY__uiYMDcq__LATEST 

If you need guidance on rebuilding DM R5F code, you can refer to this work-in-progress FAQ. I am still debugging the "moving memory allocations around" part of the FAQ, so please ignore that for now. But all the other parts of the FAQ should be helpful for you:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1317407/faq-am625-how-do-i-modify-the-ddr-memory-footprint-of-the-dm-r5f

Regards,

Nick

Hi Nick,

I can confirm that using the prebuild images, I can run rpmsg_char_simple -r 15. But, when I compile the DM R5F firmware it didn't work. Below are the steps that I did.

1. First, I deleted the prebuilt-images/am62xx-evm/ti-dm/am62xx/ipc_echo_testb_mcu1_0_release_strip.xer5f.signed and just leave the ipc_echo_testb_mcu1_0_release_strip.xer5f.  Recompiled the u-boot, transfer the new images to the sd card and boot the board - it works.

2. I compiled the ipc_rpmsg_echo from the source - I am using mcu_plus_sdk_am62x_09_00_00_19 and ti-processor-sdk-linux-am62xx-evm-09.01.00.08.

Then, I ran the following in the mcu_plus_sdk_am62x_09_00_00_19 directory:

$make clean

$export DEVICE_TYPE=GP
$ make -s PROFILE=release

I replaced the ipc_echo_testb_mcu1_0_release_strip.xer5f by copying the new binary.

cp ipc_rpmsg_echo.release.out ~/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/prebuilt-images/am62xx-evm/ti-dm/am62xx/ipc_echo_testb_mcu1_0_release_strip.xer5f

Then, recompile the u-boot, transfer the new images into the sd card. The eval did not show any error/s.

[ 7.460017] remoteproc remoteproc1: attaching to 78000000.r5f
[ 7.497249] remoteproc remoteproc1: remote processor 78000000.r5f is now attached
[ 8.645880] remoteproc remoteproc2: 30074000.pru is available
[ 8.653031] remoteproc remoteproc3: 30078000.pru is available

But, when I try to ping the R5F.

$ rpmsg_char_simple -r 15

could not find the matching rpmsg_ctrl device for virtio0.rpmsg_chrdev.-1.14

Then, I tried the other firmware am62-wkup-r5f0_0-fw. I copied it to ~/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/prebuilt-images/am62xx-evm/ti-dm/am62xx/ipc_echo_testb_mcu1_0_release_strip.xer5f

Rebuild the u-boot and tried it again. Unfortunately, it didn't work as well.

What step/s did I miss when compiling the R5F firmware? Do I need to run syscfg-gui?. Could you give me some guidedance?

Regards,

John

Hello John,

Thank you for providing additional details. You are building a different project, so it makes sense that the Linux IPC Echo example does not work!

ipc_rpmsg_echo is specifically for MCU+ to MCU+ RPMsg echo
ipc_rpmsg_echo_linux is for Linux to MCU+ RPMsg echo example

Please try again with the ipc_rpmsg_echo_linux example and see if that works for you.

Regards,

Nick

Hi Nick,

Thanks for the info. I tried the ipc_rpmsg_echo_linux. When I boot the device I see some crashes and it looks like it related to ti-sci

[ 0.954308] ti-sci 44043000.system-controller: ABI: 3.1 (firmware rev 0x0009 '9.1.8--v09.01.08 (Kool Koala)')
[ 1.058349] ti-sci-intr bus@f0000:interrupt-controller@a00000: Interrupt Router 3 domain created
[ 1.067633] ti-sci-inta 48000000.interrupt-controller: Interrupt Aggregator domain 28 created
[ 9.757672] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_get_device_exclusive+0x18/0x2c)
[ 9.769309] ti-sci 44043000.system-controller: Mbox send fail -110
[ 10.749608] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_put_clock+0x18/0x24)
[ 10.749670] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_get_device_exclusive+0x18/0x2c)
[ 10.759540] ti-sci 44043000.system-controller: Mbox send fail -110
[ 10.759567] ti-sci-clk 44043000.system-controller:clock-controller: unprepare failed for dev=186, clk=6, ret=-110
[ 10.786837] ti-sci 44043000.system-controller: Mbox send fail -110
[ 11.133562] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_put_device+0x18/0x24)
[ 11.143552] ti-sci 44043000.system-controller: Mbox send fail -110
[ 11.293602] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: xudma_navss_psil_pair+0x2c/0x40)
[ 11.303733] ti-sci 44043000.system-controller: RM_PSIL:Mbox send fail -110
[ 11.773620] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_determine_rate+0x68/0xf0)
[ 11.783716] ti-sci 44043000.system-controller: Mbox send fail -110
[ 11.789964] ti-sci-clk 44043000.system-controller:clock-controller: determine-rate failed for dev=135, clk=0, ret=-110
[ 11.800744] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_put_device+0x18/0x24)
[ 11.809582] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_get_device_exclusive+0x18/0x2c)

While showing the said errors in ti-sci, I can still ping the m4 but not the r5f.

The output size of the r5f binaries:

150324 Apr 16 10:02 am62-wkup-r5f0_0-fw

1367012 Apr 16 10:02 ipc_rpmsg_echo_linux.release.out

Regards,

John

Hello John,

Hmm, those outputs indicate that the DM R5F firmware is not running, or that something bad happened during boot that stopped the firmware from working properly.

Have you made any modifications to the DM R5F firmware or memory allocations? Or are you just building and integrating the unmodified project into uboot?

Please try using the unmodified project with unmodified memory allocations (including in the linux devicetree file) first. If you are unable to get it working, please provide me with your exact steps so I can try to replicate your observations on my side.

Regards,

Nick

Hi Nick,

I did not make any modification at all. I just re-download the https://www.ti.com/tool/download/MCU-PLUS-SDK-AM62X/09.00.00.19 and extract it in ~/ti directory. Then, went into the ti/mcu_plus_sdk_am62x_09_00_00_19 and run:

$ make -s PROFILE=release -C examples/drivers/ipc/ipc_rpmsg_echo_linux/am62x-sk/r5fss0-0_freertos/ti-arm-clang

copy the ipc_rpmsg_echo_linux.release.out to ti-processor-sdk-linux-am62xx-evm-09.00.00.03/board-support/prebuilt-images/am62xx-evm/ti-dm/am62xx/ipc_echo_testb_mcu1_0_release_strip.xer5f

Then, recompile the u-boot. I think you can easily verified it...

Could you try to replicate it?.

Regards,

John

Also, I have the following installed in my ti directory to build mcu_plus_sdk_am62x_09_00_00_19.

Also, I am using the GP evm board.

The ipc_rpmsg_echo_linux r5f on mcu_plus_sdk_am62x_09_02_00_38 works for me.

thanks

Hi Nick,

Just to mentioned, my custom AM625 single core has a 512MB of memory. I did change the memory reservation in u-boot and in kernel dts file based on 512M capacity and my understanding.

At this moment, I am using the AM62-SKEVM board and I modified default addresses that defined in the following:
optee-os
trusted-firmware-a
u-boot
kernel

Then, in u-boot - it shows the following info. after I changed the mapping.

U-Boot 2023.04-00007-gad23e126dc-dirty (Apr 16 2024 - 16:33:30 -0700)

SoC: AM62X SR1.0 GP
Model: Texas Instruments AM625 SK
EEPROM not available at 80, trying to read at 81
Board: AM62-SKEVM rev E3
DRAM: 512 MiB
Core: 72 devices, 32 uclasses, devicetree: separate
MMC: mmc@fa10000: 0, mmc@fa00000: 1
Loading Environment from nowhere... OK
In: serial
Out: serial
Err: serial
Net: eth0: ethernet@8000000port@1
Hit any key to stop autoboot: 0
=> bdinfo
boot_params = 0x0000000000000000
DRAM bank = 0x0000000000000000
-> start = 0x0000000080000000
-> size = 0x0000000020000000
flashstart = 0x0000000000000000
flashsize = 0x0000000000000000
flashoffset = 0x0000000000000000
baudrate = 115200 bps
relocaddr = 0x000000009b61a000
reloc off = 0x000000001ae1a000
Build = 64-bit
current eth = ethernet@8000000port@1
ethaddr = 34:08:e1:80:86:4d
IP addr = <NULL>
fdt_blob = 0x00000000995eb760
new_fdt = 0x00000000995eb760
fdt_size = 0x000000000000f620
Video = dss@30200000 inactive
multi_dtb_fit= 0x0000000000000000
lmb_dump_all:
memory.cnt = 0x1
memory[0] [0x80000000-0x9fffffff], 0x20000000 bytes flags: 0
reserved.cnt = 0x4
reserved[0] [0x8c000000-0x8c0fffff], 0x00100000 bytes flags: 0
reserved[1] [0x8c400000-0x8dffffff], 0x01c00000 bytes flags: 4
reserved[2] [0x8e100000-0x8f97ffff], 0x01880000 bytes flags: 4
reserved[3] [0x985e6000-0x9fffffff], 0x07a1a000 bytes flags: 0
devicetree = separate
arch_number = 0x0000000000000000
TLB addr = 0x000000009bff0000
irq_sp = 0x00000000995eaff0
sp start = 0x00000000995eaff0
Early malloc usage: 3c28 / 8000

I made changes in u-boot/kernel reserved-memory:

reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;

ramoops@8C000000 {
compatible = "ramoops";
reg = <0x00 0x8C000000 0x00 0x00100000>;
record-size = <0x8000>;
console-size = <0x8000>;
ftrace-size = <0x00>;
pmsg-size = <0x8000>;
};

rtos_ipc_memory_region: ipc-memories@8C100000 {
compatible = "shared-dma-pool";
reg = <0x00 0x8C100000 0x00 0x00300000>;
no-map;
};

mcu_m4fss_dma_memory_region: m4f-dma-memory@8C400000 {
compatible = "shared-dma-pool";
reg = <0x00 0x8C400000 0x00 0x100000>;
no-map;
};

mcu_m4fss_memory_region: m4f-memory@8C500000 {
compatible = "shared-dma-pool";
reg = <0x00 0x8C500000 0x00 0xe00000>;
no-map;
};

wkup_r5fss0_core0_dma_memory_region: r5f-dma-memory@8D300000 {
compatible = "shared-dma-pool";
reg = <0x00 0x8D300000 0x00 0x00100000>;
no-map;
};

wkup_r5fss0_core0_memory_region: r5f-memory@8D400000 {
compatible = "shared-dma-pool";
reg = <0x00 0x8D400000 0x00 0x00c00000>;
no-map;
};

secure_tfa_ddr: tfa@8E100000 {
reg = <0x00 0x8E100000 0x00 0x80000>;
alignment = <0x1000>;
no-map;
};

secure_ddr: optee@8E180000 {
reg = <0x00 0x8E180000 0x00 0x01800000>; /* for OP-TEE */
alignment = <0x1000>;
no-map;
};
};

After the changes, the u-boot and the kernel able to reserved the memory without any errors and the kernel able to boot all the way to the filesystem.
The only exception is that the R5F firmware isn't working because it showing the ti-sci errors

[ 12.859803] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_get_device_exclusive+0x18/0x2c)
[ 12.871051] ti-sci 44043000.system-controller: Mbox send fail -110
[ 12.891898] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: sci_clk_recalc_rate+0x48/0xb0)
[ 12.901843] ti-sci 44043000.system-controller: Mbox send fail -110
[ 12.911411] ti-sci-clk 44043000.system-controller:clock-controller: recalc-rate failed for dev=191, clk=0, ret=-110

For the R5F, I modified the linker.cmd of the ipc_rpmsg_echo_linux/am62x-sk/r5fss0-0_freertos and I made only changes in MEMORY Section (please see below)

MEMORY
{
R5F_TCMA_VEC (RWIX) : ORIGIN = 0x00000000 LENGTH = 0x00000040
R5F_TCMA (RWIX) : ORIGIN = 0x00000040 LENGTH = 0x000077C0
R5F_TCMA_TRACE_BUFF (RWIX) : ORIGIN = 0x00007800 LENGTH = 0x0000800
R5F_TCMB_VEC (RWIX) : ORIGIN = 0x41010000 LENGTH = 0x00000040
R5F_TCMB (RWIX) : ORIGIN = 0x41010040 LENGTH = 0x000077C0
R5F_TCMB_TRACE_BUFF (RWIX) : ORIGIN = 0x41017800 LENGTH = 0x0000800
HSM_RAM (RWIX) : ORIGIN = 0x43C00000 LENGTH = 0x3FF00

/* DDR for FS Stub binary [ size 32.00 KB ] */
//orig->DDR_FS_STUB (RWIX) : ORIGIN = 0x9DC00000 LENGTH = 0x00008000
/* DDR for DM R5F code/data [ size 10 MB + 992 KB ] */
//orig->DDR (RWIX) : ORIGIN = 0x9DC08000 LENGTH = 0x00AF8000
//orig->DDR_IPC_RESOURCE_TABLE_LINUX (RWIX) : ORIGIN = 0x9DB00000 LENGTH = 0x00001000

//orig->DDR_IPC_VRING_RTOS (RWIX) : ORIGIN = 0x9C800000, LENGTH = 0x00300000

/* DDR for FS Stub binary [ size 32.00 KB ] */
DDR_FS_STUB (RWIX) : ORIGIN = 0x8E000000 LENGTH = 0x00008000
/* DDR for DM R5F code/data [ size 10 MB + 992 KB ] */
DDR (RWIX) : ORIGIN = 0x8E008000 LENGTH = 0x00AF8000
DDR_IPC_RESOURCE_TABLE_LINUX (RWIX) : ORIGIN = 0x8D400000 LENGTH = 0x00001000

DDR_IPC_VRING_RTOS (RWIX) : ORIGIN = 0x8C100000, LENGTH = 0x00300000
}

Could you tell me the missing part in order to work the R5F firmware?.

Regards,

John

I tried to adjust the default memory size from 2GB to 512 and saved it. But, when I bundled the image in u-boot it didn't work... the u-boot tries to boot but it hang...

Hi Nick,

I was trying to use the ccstudio and when I tried to run it it shows this errors:
CortexA53_0: File Loader: Verification failed: Values at address 0x0000000041010040 do not match Please verify target memory and memory map.
CortexA53_0: GEL: File: /home/jtobias/workspace_v12/ipc_rpmsg_echo_linux_am62x-sk_r5fss0-0_freertos_ti-arm-clang/Debug/ipc_rpmsg_echo_linux_am62x-sk_r5fss0-0_freertos_ti-arm-clang.out: a data verification error occurred, file load failed.

Just wonder if you could show me how to fix my error. Attached is my test project.

Regards,

John

ipc_rpmsg_echo_linux_am62x-sk_r5fss0-0_freertos_ti-arm-clang.tar.gz

Hello John,

Ok. Just to make sure I understand:

When you are using the default Linux devicetree file (i.e., 2GB of DDR), and when you are building the unmodified ipc_rpmsg_echo_linux r5f project in mcu_plus_sdk_am62x_09_02_00_38, then the R5F code builds and runs just fine, right?

What about when you use the default Linux devicetree file (i.e., 2GB of DDR), and the unmodified ipc_rpmsg_echo_linux r5f project in MCU+ SDK 9.0? Does the R5F code also build and run just fine?

Are you only seeing the "Mbox send fail" error when moving the DM R5F memory allocation from the default location of 0x9db00000, or reducing the memory allocation from the default size of 0x00c00000?

The reason I am asking is because that is exactly the behavior that I was observing while working on the FAQ I linked above:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1317407/faq-am625-how-do-i-modify-the-ddr-memory-footprint-of-the-dm-r5f

So if you are also running into that behavior, that gives me an excuse to spend more time debugging the FAQ. But it also means that I do not have an immediate solution for you. We will need to work together to figure out how to move your DM R5F memory allocation around.

Regards,

Nick

Hi NIck,

Yeah, I think we are in the same page - I need more info 'TODOs for the future' in the link you provided. I really need your expertise to resolve the issue.

I thought by changing the addresses (DDR_FS_STUB, DDR, DDR_IPC_RESOURCE_TABLE_LINUX and DDR_IPC_VRING_RTOS) in linker.cmd will fix the issue for R5F.

Although, in M4FSS it worked.

Regards,

John

Hello John,

Next steps

Understood. Today (and probably tomorrow) I am busy validating a fix for the AM62x watchdog driver, so I probably will not be able to re-start experiments on modifying the DM R5F until next week.

For now, I will share my experiments from last month, and my notes from an initial debug with our device management SW guy. If you run any additional experiments today or tomorrow, please let me know what you observe. As needed, attach the patches with your exact changes to Linux dts, MCU+ project, & uboot binman.

Notes from debug session 

MCU+ linker.cmd file needs an update:
DDR_FS_STUB --> this is only needed if low power mode is needed
DDR --> this is used by the DM R5F program during runtime

Binman needs an update
dm load & entry addresses. These are just where the firmware is INITIALLY loaded when being copied out of the filesystem. The actual address does not matter
tifsstub: the address DOES matter, but they expect this would only cause issues when initiating deep sleep and trying to resume
remember to update the filename used

dts memory allocations need to match the DM R5F's allocations

We SHOULD be able to use the R5F's .map file in the generated folder to accurately see EXACTLY how much memory is actually used by the program (i.e., there should NOT be any additional "hidden" memory that is required for the code to run)

I saw issues both when reducing the memory allocation, AND when moving the memory allocation. They suggested first trying to move the data without changing the memory allocation size - once that is working, then try reducing the memory allocation.

Notes from experiments 

I think these were run with AM62x Linux & MCU+ SDK 9.1. It looks like I ran these tests without modifying the binman file. I definitely experimented with updating the binman as well later on, but I do not see any recorded notes for those tests.

Default dts values:

                wkup_r5fss0_core0_memory_region: r5f-memory@9db00000 {
                        compatible = "shared-dma-pool";
                        reg = <0x00 0x9db00000 0x00 0x00c00000>;
                        no-map;
                };

TEST 1: Unmodified ipc_rpmsg_echo_linux, DTS at 0x9db00000, allocate 0x200000
WORKS

TEST 2: modified hello_world, DTS at 0x9db00000, allocate 0x200000
hello_world linker updated like
  DDR_FS_STUB           9db00000   00008000  00008000  00000000  RWIX
  DDR                   9db08000   00048000  00043c02  000043fe  RWIX
i.e., on R5F side only allocate 0x50000
WORKS

Both examples working so far

TESTs 3 & 4: modified hello_world, DTS at 0x9db00000, allocate 0x60000 & 0x100000
DOES NOT WORK

re-validate TESTs 1 & 2, still working

TESTs 5 & 6: modified mailbox project (custom) & ipc_rpmsg_echo_linux, DTS at 0x9c800000, allocate 0x200000
modified IPC project to move the DM firmware data to 0x9c800000, moved the IPC_RTOS_shared_memory to 0x9db00000
DOES NOT WORK

Regards,

Nick

Hello John,

Apologies for the delay. The watchdog driver testing is wrapping up now (if you will be using a Linux watchdog in your design, you can see updates here: https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1345810/am625-how-to-enable-hardware-watchdog-in-am625-and-set-a-timeout-to-30-sec/5166063#5166063)

I am going to pivot towards working on your thread tomorrow.

Regards,

Nick

Hi Nick,

Thanks for for the message. I forgot to update you that I was able to change the memory address of the R5F.

Regards,

John

Hello John,

Glad to hear that you got it working!

1) Did you need to modify anything other than these 3 files?
* MCU+ linker.cmd file
* Binman file
* dts memory allocations need to match the DM R5F's allocations

2) Did you try to reduce the memory allocation, or did you just move the memory address?

3) Would you be willing to share your code diffs with me so I can try to replicate? If you're not comfortable posting here, you could send those to me over direct message.

Regards,

Nick