[FAQ] AM625: How do I modify the DDR memory footprint of the DM R5F?

Base knowledge 

How do I allocate DDR memory between Linux and a remote core? 

AM62x academy > Multicore module > How to allocate memory
AM62Ax academy > Multicore module > How to allocate memory

How do use the MCU+ SDK to build the DM R5F firmware? 

Any MCU+ example that has the option to build for the DM R5F can be used to build the firmware. The important thing is that the project needs to call sciServer_init to start the DM task.

Refer to 

AM62x MCU+ SDK docs > Developing applications on Device Manager/Wake-up R5 core
AM62Ax MCU+ SDK docs > Developing applications on Device Manager/Wake-up R5 core
AM62Px MCU+ SDK docs > Developing applications on Device Manager/Wake-up R5 core

How do I use the .map file to see how much memory my MCU+ application actually uses?

Refer to the "How to allocate memory" link above, section "Optimizing Memory Usage"

How does the DM R5F firmware work? What resources are actually needed? 

What is the DM R5F doing? 

The DM R5F runs a dedicated RTOS task that is responsible for resource management and power management. For example whenever another core on the device wants to use a specific resource (e.g., Main domain UART 0), that core makes a request to the DM software running on the DM R5F. The DM R5F then decides whether to accept, or deny the request. This task is critical! If the DM task stops running, then the processor will stop working.

Users can also program additional RTOS tasks to run on the DM R5F alongside the DM task. These additional RTOS tasks are NOT critical like the DM task is. Different MCU+ SDK examples will run different tasks on the DM R5F.

As an example: For AM62x & AM62Px, the DM R5F firmware running in the default Linux filesystem is built using the MCU+ SDK example
examples/drivers/ipc/ipc_rpmsg_echo_linux

So in our out-of-the-box example, there is a second RTOS task that is running the RPMsg Echo example. That is why you can run the Linux IPC Echo Example with the DM R5F:
AM62x academy > Linux module > Evaluating Linux > IPC Example
AM62Px academy > Linux module > Evaluating Linux > IPC Example

How does the DM R5F start running? 

The DM R5F is booted differently than other remote cores (i.e., non-Linux cores running alongside the Linux A53s). Other remote cores must be initialized by either the Linux remoteproc driver, or MCU+ SBL. However, because the DM R5F is so important, it gets loaded earlier in the boot process.

That means that DM R5F does NOT depend on the Linux remoteproc driver to start running.

Default Linux memory regions 

Linux allocates memory regions for all remote cores as if the cores are 1) initialized by Linux remoteproc, and 2) as if the cores are using RPMsg IPC to communicate with Linux. Because of that, Linux carves out 2 memory regions for each remote core: the "IPC (Virtio/Vring buffers)" memory region, and the "external code/data mem" memory region.

How does the DM R5F use memory? 

The DM R5F continues running when the processor is in a low power mode. Since DDR is not powered up during low power mode, the DM R5F firmware must run out of the local TCM memory during low power. The data that goes into the TCM memory is called a "stub" in the linker.cmd file.

Most of the rest of the DM R5F code goes into DDR.

The DM R5F does NOT use RPMsg IPC when it is communicating with the other processor cores. That means that the dedicated DM task does NOT need the "IPC (Virtio/Vring buffers)" memory region that Linux allocates by default.

How do I reduce the DDR memory footprint of the DM R5F? 

PART 1: Build the DM R5F firmware with the smallest MCU+ example

First, let's verify that the hello_world example will initialize the sciServer:
examples/hello_world/am62x-sk/r5fss0-0_freertos/main.c

void hello_world_main(void *args);

void main_thread(void *args)
{
...
    sciServer_init();

Now let's build the DM R5F firmware with the hello_world example. I'll be building from the Linux terminal:

$ make -s -C examples/hello_world/am62x-sk/r5fss0-0_freertos/ti-arm-clang

Check how much memory is actually needed 

The memory regions are defined in the linker.cmd file:
examples/hello_world/am62x-sk/r5fss0-0_freertos/ti-arm-clang/linker.cmd

Let's see how much data is actually being placed in each memory region:

vi examples/hello_world/am62x-sk/r5fss0-0_freertos/ti-arm-clang/hello_world.release.map

MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  R5F_TCMA_VEC          00000000   00000040  00000000  00000040  RWIX
  R5F_TCMA              00000040   000077c0  00002db8  00004a08  RWIX
  R5F_TCMA_TRACE_BUFF   00007800   00000800  00000800  00000000  RWIX
  R5F_TCMB_VEC          41010000   00000040  00000040  00000000  RWIX
  R5F_TCMB              41010040   000077c0  00002db8  00004a08  RWIX
  R5F_TCMB_TRACE_BUFF   41017800   00000800  00000000  00000800  RWIX
  HSM_RAM               43c00000   0003ff00  00000000  0003ff00  RWIX
  DDR_FS_STUB           9dc00000   00008000  00008000  00000000  RWIX
  DDR                   9dc08000   00af8000  00043c02  00ab43fe  RWIX

DDR_FS_STUB and DDR are the only regions that are in DDR. 0x8000 + 0x43C02 = 0x4BC02. Let's set aside 0x50000 for the R5F memory.

Change the linker.cmd file

Then we can update the linker command file like this:

vi examples/hello_world/am62x-sk/r5fss0-0_freertos/ti-arm-clang/linker.cmd

    /* DDR for FS Stub binary [ size 32.00 KB ] */
    DDR_FS_STUB    (RWIX)      : ORIGIN = 0x9DC00000 LENGTH = 0x00008000
    /* DDR for DM R5F code/data [ size 288 KB ] */
    DDR            (RWIX)      : ORIGIN = 0x9DC08000 LENGTH = 0x00048000

and double-check that the memory allocation still looks good:

$ make -s -C examples/hello_world/am62x-sk/r5fss0-0_freertos/ti-arm-clang clean
$ make -s -C examples/hello_world/am62x-sk/r5fss0-0_freertos/ti-arm-clang
$ vi examples/hello_world/am62x-sk/r5fss0-0_freertos/ti-arm-clang/hello_world.release.map

MEMORY CONFIGURATION

         name            origin    length      used     unused   attr    fill
----------------------  --------  ---------  --------  --------  ----  --------
  R5F_TCMA_VEC          00000000   00000040  00000000  00000040  RWIX
  R5F_TCMA              00000040   000077c0  00002db8  00004a08  RWIX
  R5F_TCMA_TRACE_BUFF   00007800   00000800  00000800  00000000  RWIX
  R5F_TCMB_VEC          41010000   00000040  00000040  00000000  RWIX
  R5F_TCMB              41010040   000077c0  00002db8  00004a08  RWIX
  R5F_TCMB_TRACE_BUFF   41017800   00000800  00000000  00000800  RWIX
  HSM_RAM               43c00000   0003ff00  00000000  0003ff00  RWIX
  DDR_FS_STUB           9dc00000   00008000  00008000  00000000  RWIX
  DDR                   9dc08000   00048000  00043c02  000043fe  RWIX

PART 2: Modify the Linux devicetree file 

Now let's update the memory allocations in the Linux devicetree. We are making 2 major changes:

1) Set the DM R5F as "disabled" from the Linux perspective

2) Reduce the DDR memory allocations to match the linker.cmd file

If a devicetree node is set as "disabled", that does NOT mean that Linux actively tries to remove power from the disabled core. It just means that Linux ignores that devicetree node, and assumes that it is disabled.

Set the DM R5F as disabled 

Modify the devicetree file like this:

diff --git a/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi b/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
index 6f102b430..09203519a 100644
--- a/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
+++ b/arch/arm64/boot/dts/ti/k3-am62x-sk-common.dtsi
@@ -487,27 +469,11 @@ cpsw3g_phy0: ethernet-phy@0 {
 };
&mailbox0_cluster0 {
        mbox_m4_0: mbox-m4-0 {
                ti,mbox-rx = <0 0 0>;
                ti,mbox-tx = <1 0 0>;
        };
 
-       mbox_r5_0: mbox-r5-0 {
-               ti,mbox-rx = <2 0 0>;
-               ti,mbox-tx = <3 0 0>;
-       };
 };
 
...

-&wkup_r5fss0_core0 {
-       mboxes = <&mailbox0_cluster0 &mbox_r5_0>;
-       memory-region = <&wkup_r5fss0_core0_dma_memory_region>,
-                       <&wkup_r5fss0_core0_memory_region>;
+&wkup_r5fss0 {
+       status = "disabled";
 };

Reduce the DDR memory allocations 

Since we are not using RPMsg IPC to communicate with the DM R5F, the "IPC (Virtio/Vring buffers)" memory region can be removed. The "external code/data mem" memory region can be reduced to 0x50000.

// treat this like a diff file

-                wkup_r5fss0_core0_dma_memory_region: r5f-dma-memory@9da00000 {
-                        compatible = "shared-dma-pool";
-                        reg = <0x00 0x9da00000 0x00 0x00100000>;
-                        no-map;
-                };

                wkup_r5fss0_core0_memory_region: r5f-memory@9db00000 {
                        compatible = "shared-dma-pool";
-                        reg = <0x00 0x9db00000 0x00 0x00c00000>;
+                        reg = <0x00 0x9db00000 0x00 0x00050000>;
                        no-map;
                };

PART 3: build the output files and test 

If the board is using the Linux SDK SPL boot:

Update the Uboot binman file to point to the new binary.

e.g. for AM625-sk:
arch/arm/dts/k3-am625-sk-binman.dtsi
ti-dm node, update the filename entry
it looks like the filename is relative to board-support/prebuilt-images/am62xx-evm, so you would copy the new DM firmware somewhere in there.

then Follow the steps to integrate the DM R5F firmware into uboot here:
https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/09_01_00_08/exports/docs/linux/Foundational_Components/U-Boot/UG-General-Info.html

For this example, we only need to rebuild tiboot3-am62x-hs-fs-evm.bin:

~/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/ti-u-boot-2023.04+gitAUTOINC+b0d717b732-gb0d717b732$ export UBOOT_DIR=/home/user/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/ti-u-boot-2023.04+gitAUTOINC+b0d717b732-gb0d717b732
~/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/ti-u-boot-2023.04+gitAUTOINC+b0d717b732-gb0d717b732$ export TI_LINUX_FW_DIR=/home/user/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/prebuilt-images/am62xx-evm
~/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/ti-u-boot-2023.04+gitAUTOINC+b0d717b732-gb0d717b732$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- am62x_evm_r5_defconfig O=$UBOOT_DIR/out/r5
~/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/ti-u-boot-2023.04+gitAUTOINC+b0d717b732-gb0d717b732$ make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabihf- O=$UBOOT_DIR/out/r5 BINMAN_INDIRS=$TI_LINUX_FW_DIR
~/ti-processor-sdk-linux-am62xx-evm-09.01.00.08/board-support/ti-u-boot-2023.04+gitAUTOINC+b0d717b732-gb0d717b732$ ls out/r5/
...
tiboot3-am62x-hs-fs-evm.bin

Then we'll copy this file into the boot partition of the SD card.

Build the Linux devicetree. If using a fitImage, build it into the fitImage as described here:
https://software-dl.ti.com/processor-sdk-linux/esd/AM62X/09_01_00_08/exports/docs/linux/Foundational_Components_Kernel_Users_Guide.html#compiling-the-device-tree-binaries

run and test

TODOs for the future 

This doesn't really cover moving the base address of the DM R5F. Note that the fs_stub region in the linker.cmd file should probably be updated to match the region in the binman file? Though it doesn't seem like the binman file's dm node's address 0x8900_0000 needs to match the linker.cmd file... needs more work here

ping