[FAQ] AM64X: How to combine R5F PRU_ICSSG Ethernet with Linux A53 cores

GETTING STARTED

1) Download and install both SDKS

2) Set up the EVM, create an SD card, and boot the EVM as discussed in the AM64x EVM Quick Start Guide.

3) Before making any software changes, verify that Linux can load firmware into the R5F, that the R5F firmware can run, and that the out-of-the-box Inter-Processor Communication (IPC) demo works as expected.

AM64x academy > Linux > Booting Remote Cores

AM64x academy > Linux > IPC example

What steps are required to get Linux running alongside an R5F core that controls a network interface? 

We will follow the steps in AM64x academy > Multicore > Application Development on Remote Cores:

1) Modify the MCU+ example so that it can be loaded by Linux

2) Fix all resource conflicts (peripherals, memory)

3) Update the resource configuration settings so that the networking DMA channels are assigned to the R5F core instead of Linux

MCU+ PROJECT UPDATES

Now, let's work on RTOS side.

For this FAQ, we will use the AM64x MCU+ SDK: Enet Lwip ICSSG Example.

SDK 10.1 path: mcu_plus_sdk_am64x_10_01_xxxx\source\networking\enet\core\examples\lwip\enet_lwip_icssg
SDK 9.2.1 path: mcu_plus_sdk_am64x_09_02_01_05/examples/networking/lwip/enet_lwip_icssg

Modify the remote core application to be loaded by Linux

Enable IPC to get a resource table

We need to add a resource table to the MCU+ project. From academy page "The resource table", we know that:

• A remote core application must have a resource table in order to be initialized by the Linux Remoteproc driver
• The resource table is generated by the remote core and included in the remote core binary.
• An easy way to add a resource table is to enable IPC with Linux in SysConfig

Follow the steps at "Add Linux IPC to the remote core project".

Updated sysconfig file is attached further down.

Update the linker.cmd file - Add a DDR section for resource table

The above changes will generate the resource table. Now, a DDR section must be allocated for the resource table in the linker.cmd file.

The resource table address must align with the beginning of the “external code/data mem” section in the Linux devicetree file. Refer to "Allocating memory in Linux" for more information. For additional memory layout information please check AM64x MCU+ SDK: Memory Map Layout.

    GROUP  :   {

    .resource_table : {

    } palign(4096)

    } > DDR_0

}

DDR_0     : ORIGIN = 0xA0100000 , LENGTH = 0x10000 

MSRAM     : ORIGIN = 0x70082F00 , LENGTH = 0xB9700 (Note: in linker command we start after 0x70080000 however if SBL is not used it can start at 0x70000000)

Linker command file example:

25087.linker.cmd

Allocate peripherals - Fix peripheral allocation conflicts

This project has 2 conflicts with Linux: both R5F & Linux request UART0, and I2C1. Each peripheral can only be controlled by 1 software instance, so the peripheral must be removed from the SysConfig project settings, or disabled in the Linux devicetree.

1) Resolve UART0 conflict

You can see that main_uart0 is used by Linux in arch/arm64/boot/dts/ti/k3-am642-evm.dts

Let's resolve the conflict by removing UART0 from the R5F project.

• Option 1: Remove UART from Sysconfig by unchecking "Enable UART Log"

• Option 2: or, if using UART Logs, you can also change to another UART instance that is not used by Linux

2) Resolve I2C1 conflict

main_i2c1 is also used by Linux in arch/arm64/boot/dts/ti/k3-am642-evm.dts

For this exercise, we will delete the I2C1 module from SysConfig. Another possible solution is to disable I2C1 in the Linux devicetree by changing "status" from "okay" to "disabled". (shown below in red)

• Option 1:

• Option 2: Change Linux DTS:

&main_i2c1 {
    status = "disabled";
    pinctrl-names = "default";
    pinctrl-0 = <&main_i2c1_pins_default>;

Sysconfig example file:

6545.example.syscfg

Rebuild the MCU+ project 

Build the modified R5F project following the steps here:
https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/latest/exports/docs/api_guide_am64x/GETTING_STARTED_BUILD.html 

LINUX

Allocate peripherals - Fix peripheral allocation conflicts

DTS and DTSI files are in the Linux SDK at
ti-processor-sdk-linux-am64xx-evm-xxxxxxx/board-support/ti-linux-kernel-xxxxxx/arch/arm64/boot/dts/ti/k3-am64*

Disable "icssg" in k3-am642-evm.dts. The PRU_ICSSG instances are owned by the R5F

 &icssg1_iep0 {

pinctrl-names = "disabled";
pinctrl-0 = <&icssg1_iep0_pins_default>;

};

&icssg0 {

status = "disabled";

}; 

&icssg1 {

status = "disabled";

}; 

&icssg1_eth {

status = "disabled";

};

Allocate memory - fix memory allocation conflicts

Create region "r5f0_0_sram" in DTSI file, and add a link to the SRAM region from the devicetree node main_r5fss0_core0. This ensures that Linux does not overwrite the R5F memory.

• Create MSRAM regions for R5F in k3-am64-main.dtsi

r5f0_0_sram: r5f0_0_sram@0 {

reg = <0x0 0x180000>;

};

• Add "r5f0_0_sram" region in k3-am642-evm.dts

&main_r5fss0_core0 {

mboxes = <&mailbox0_cluster2 &mbox_main_r5fss0_core0>;

memory-region = <&main_r5fss0_core0_dma_memory_region>,

<&main_r5fss0_core0_memory_region>;

sram = <&r5f0_0_sram>;                        

};

0842.mod_k3-am642-evm.dts 0842.mod_k3-am64-main.dtsi

Resource Manager: Assign networking DMA channels to R5F

1)  Resource configuration changes - Update the <ti-u-boot-folder>/board/ti/rm-cfg.yaml file to assign the PKTDMA channels for PRU Ethernet to R5F instead of the Linux A53 cores. Below useful links

• FAQ: (+) [FAQ] How to use K3 resource partitioning tool with Processor SDK v9.0 or v9.1? - Processors forum - Processors - TI E2E support forums
• E2E other tips: (+) PROCESSOR-SDK-AM64X: ENET open failure from R5 when A53 is running - Processors forum - Processors - TI E2E support forums
  • Note: The steps in the link above are for R5F0_1 core, make sure to add the resources to MAIN_0_R5_1 (means R5F subsystem 0 core 0). 

2) After modifying resource manager yaml file using steps from 1), copy this new *.yaml in ~/ti-processor-sdk-linux-am64xx-evm-xxxx/board-support/ti-u-boot-xxxx/board/ti/am64x

• $ cp mod_rm-cfg.yaml rm-cfg.yaml

0842.mod_rm-cfg.yaml

After changes in rm-cfg.yaml, k3-am642-evm.dts, and k3-am64-main.dtsi "make linux" and "make u-boot". Additional details here: Build U-Boot

• ~/ti-processor-sdk-linux-am64xx-evm-xxxx$ make linux
• ~/ti-processor-sdk-linux-am64xx-evm-xxxx$ make u-boot

 Note: dtb file will be generated in the same location the dts file is present

Copy new boot files to SD card

• ~/ti-processor-sdk-linux-am64xx-evm-xxxx/board-support/u-boot-build$ cp a53/tispl.bin /media/alice/boot/
• ~/ti-processor-sdk-linux-am64xx-evm-xxxx/board-support/u-boot-build$ cp a53/u-boot.img /media/alice/boot/
• ~/ti-processor-sdk-linux-am64xx-evm-xxxx/board-support/u-boot-build$ cp r5/tiboot3.bin /media/alice/boot/

Update R5F firmware on the EVM's filesystem

In SD card filesystem, copy *.out and create a new firmware's symbolic link. Example steps below. 

• Using WinSCP (or alike) copy enet_lwip_icssg_am64x-evm_r5fss0-0_freertos_ti-arm-clang.out in /lib/firmware/mcusdk-benchmark_demo
• "cd /lib/firmware"
• remove OOB symbolic link: "rm am64-main-r5f0_0-fw"
• create new symbolic link: "ln -sf /lib/firmware/mcusdk-benchmark_demo/enet_lwip_icssg_am64x-evm_r5fss0-0_freertos_ti-arm-clang.out am64-main-r5f0_0-fw"
• Confirm symbolic link "ls -l"
  • am64-main-r5f0_0-fw -> /lib/firmware/mcusdk-benchmark_demo/enet_lwip_icssg_am64x-evm_r5fss0-0_freertos_ti-arm-clang.out

TEST

Reboot EVM, check new R5F firmware is loaded, you should see new filesize

• root@am64xx-evm:~# dmesg | grep remoteproc

[    7.153455] k3-m4-rproc 5000000.m4fss: configured M4 for remoteproc mode

[    7.382876] remoteproc remoteproc0: 5000000.m4fss is available

[    7.462955] remoteproc remoteproc0: powering up 5000000.m4fss

[    7.468814] remoteproc remoteproc0: Booting fw image am64-mcu-m4f0_0-fw, size 86084

[    7.529858] remoteproc remoteproc0: remote processor 5000000.m4fss is now up

[    7.746279] platform 78000000.r5f: configured R5F for remoteproc mode

[    7.752373] remoteproc remoteproc1: 78000000.r5f is available

[    7.827354] remoteproc remoteproc1: powering up 78000000.r5f

[    7.827390] remoteproc remoteproc1: Booting fw image am64-main-r5f0_0-fw, size 3071612

[    7.831056] remoteproc remoteproc1: remote processor 78000000.r5f is now up

[    7.859219] platform 78200000.r5f: configured R5F for remoteproc mode

[    7.860675] remoteproc remoteproc2: 78200000.r5f is available

[    7.863756] remoteproc remoteproc2: powering up 78200000.r5f

[    7.863791] remoteproc remoteproc2: Booting fw image am64-main-r5f0_1-fw, size 141772

[    7.867293] remoteproc remoteproc2: remote processor 78200000.r5f is now up

[    7.938997] platform 78400000.r5f: configured R5F for remoteproc mode

[    8.095898] remoteproc remoteproc3: 78400000.r5f is available

[    8.110907] remoteproc remoteproc3: powering up 78400000.r5f

[    8.116886] remoteproc remoteproc3: Booting fw image am64-main-r5f1_0-fw, size 93260

[    8.151913] platform 78600000.r5f: configured R5F for remoteproc mode

[    8.158498] remoteproc remoteproc3: remote processor 78400000.r5f is now up

[    8.335912] remoteproc remoteproc4: 78600000.r5f is available

[    8.344783] remoteproc remoteproc4: powering up 78600000.r5f

[    8.350875] remoteproc remoteproc4: Booting fw image am64-main-r5f1_1-fw, size 91520

[    8.399547] remoteproc remoteproc4: remote processor 78600000.r5f is now up

• root@am64xx-evm:~# cat /sys/class/remoteproc/remoteproc1/state

Running

If there is any issues such as kernel panic or firmware not loading correctly, please check academy: Debug the remote core alongside Linux for additional tips.

Academy is great resource for multicore development. Below some recommended links four further study:

Application development on Remote Cores > Modify the remote core application to be loaded by Linux
https://dev.ti.com/tirex/explore/node?node=A__AaxbsDidZkzMnkMKgzk7bQ__AM64-ACADEMY__WI1KRXP__LATEST 

How to allocate peripherals:
https://dev.ti.com/tirex/explore/node?node=A__AaxbsDidZkzMnkMKgzk7bQ__AM64-ACADEMY__WI1KRXP__LATEST