AM6442: SRAM allocation & ICSSG ethernet driver failure

Jared Weyer

Part Number: AM6442

Hi,

I have some questions regarding SRAM requirements of the ICSSG ethernet driver. For some background, my goal is to allocate an SRAM bank for each R5 core. To accomplish the I have modified the device tree as follows:

oc_sram: sram@70000000 {
compatible = "mmio-sram";
reg = <0x00 0x70000000 0x00 0x200000>;
#address-cells = <1>;
#size-cells = <1>;
ranges = <0x0 0x00 0x70000000 0x200000>;

r5f0_0_sram: r5f0_1-sram@80000 {
reg = <0x80000 0x40000>;
};

r5f0_1_sram: r5f1_0-sram@c0000 {
reg = <0xc0000 0x40000>;
};

r5f1_0_sram: r5f1_1-sram@100000 {
reg = <0x100000 0x40000>;
};

r5f1_1_sram: r5f0_0-sram@140000 {
reg = <0x140000 0x40000>;
};

tfa-sram@1c0000 {
reg = <0x1c0000 0x20000>;
};

dmsc-sram@1e0000 {
reg = <0x1e0000 0x1c000>;
};

sproxy-sram@1fc000 {
reg = <0x1fc000 0x4000>;
};
};

My understading is that the above should be a valid allocation of the SRAM banks as the ICSSG ethernet driver should only require 384 KB of SRAM per this documentation:

SoC	Mode	SRAM Required per ICSSG Instance
AM64X	Emac Mode	192 KB

Note that I am using remoteproc to interface with the R5 cores, and therefore, it is my understanding that the first two 256 KB banks should be free to use as they are not reserved for a SBL. However, when attempting to boot with the above setup the Kernel panics:

[ 9.563647] icssg-prueth ethernet: couldn't get ti,pa-stats syscon regmap
[ 9.578398] SError Interrupt on CPU1, code 0x00000000bf000002 -- SError
[ 9.578434] CPU: 1 UID: 0 PID: 28 Comm: kworker/u8:1 Tainted: G O 6.12.8-gbba00fec-00044-g133701b6ee76 #1
[ 9.578450] Tainted: [O]=OOT_MODULE
[ 9.578454] Hardware name: PHYTEC phyBOARD-Electra-AM64x RDK (DT)
[ 9.578462] Workqueue: events_unbound deferred_probe_work_func
[ 9.578494] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 9.578503] pc : __memset_io+0x48/0x90
[ 9.578518] lr : prueth_probe+0x378/0x980 [icssg_prueth]
[ 9.578537] sp : ffff80008168bb10
[ 9.578541] x29: ffff80008168bb30 x28: ffff000001499800 x27: ffff00006fd8dbf0
[ 9.578554] x26: ffff8000797219c8 x25: ffff00006fd8ea10 x24: ffff00006fd8e620
[ 9.578567] x23: ffff000001499810 x22: ffff0000070330c0 x21: 0000000000040000
[ 9.578579] x20: ffff000007033080 x19: 0000000000000000 x18: 0000000000000000
[ 9.578590] x17: 0000000000000000 x16: 1fffe000001e1ec1 x15: 0000000000000001
[ 9.578602] x14: ffffffffffffffff x13: 666c652e77662d68 x12: 74657572702d3175
[ 9.578613] x11: 0000000000000000 x10: ffff80007a114985 x9 : 0000000000000008
[ 9.578625] x8 : 0000000000000040 x7 : ffffffffffffffff x6 : ffff800081bc0000
[ 9.578636] x5 : 0000000000000000 x4 : 0000000000000c70 x3 : ffff800081bbf398
[ 9.578647] x2 : 0000000000040000 x1 : 0000000000000000 x0 : ffff800081b80000
[ 9.578662] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 9.578670] CPU: 1 UID: 0 PID: 28 Comm: kworker/u8:1 Tainted: G O 6.12.8-gbba00fec-00044-g133701b6ee76 #1
[ 9.578680] Tainted: [O]=OOT_MODULE
[ 9.578683] Hardware name: PHYTEC phyBOARD-Electra-AM64x RDK (DT)
[ 9.578687] Workqueue: events_unbound deferred_probe_work_func
[ 9.578701] Call trace:
[ 9.578706] dump_backtrace+0x90/0xe8
[ 9.578721] show_stack+0x18/0x24
[ 9.578731] dump_stack_lvl+0x34/0x8c
[ 9.578750] dump_stack+0x18/0x24
[ 9.578761] panic+0x390/0x3a4
[ 9.578773] nmi_panic+0x40/0x8c
[ 9.578784] arm64_serror_panic+0x64/0x70
[ 9.578796] do_serror+0x3c/0x70
[ 9.578803] el1h_64_error_handler+0x30/0x48
[ 9.578814] el1h_64_error+0x64/0x68
[ 9.578822] __memset_io+0x48/0x90
[ 9.578830] platform_probe+0x68/0xc4
[ 9.578842] really_probe+0xbc/0x29c
[ 9.578852] __driver_probe_device+0x78/0x12c
[ 9.578862] driver_probe_device+0xd8/0x15c
[ 9.578872] __device_attach_driver+0xb8/0x134
[ 9.578882] bus_for_each_drv+0x88/0xe8
[ 9.578893] __device_attach+0xa0/0x190
[ 9.578904] device_initial_probe+0x14/0x20
[ 9.578914] bus_probe_device+0xac/0xb0
[ 9.578923] deferred_probe_work_func+0x88/0xc0
[ 9.578933] process_one_work+0x148/0x288
[ 9.578948] worker_thread+0x2cc/0x3d4
[ 9.578959] kthread+0x110/0x114
[ 9.578971] ret_from_fork+0x10/0x20
[ 9.578983] SMP: stopping secondary CPUs
[ 9.579018] Kernel Offset: disabled
[ 9.579021] CPU features: 0x00,00000080,00200000,4200420b
[ 9.579030] Memory Limit: none
[ 9.846574] ---[ end Kernel panic - not syncing: Asynchronous SError Interrupt ]---

The documentation above states that the driver should exit cleanly with "-ENOMEM error". I've briefly looked at the source code and it seems like `gen_pool_alloc_algo` is returning a non-null address, which is why the driver does not exit cleanly with -ENOMEM.

In my attempts to debug I adjusted the SRAM allocation to look like this (Shifted all R5 SRAM allocations down one bank) :

oc_sram: sram@70000000 {
compatible = "mmio-sram";
reg = <0x00 0x70000000 0x00 0x200000>;
#address-cells = <1>;
#size-cells = <1>;
ranges = <0x0 0x00 0x70000000 0x200000>;

r5f0_0_sram: r5f0_1-sram@c0000 {
reg = <0xc0000 0x40000>;
};

r5f0_1_sram: r5f1_0-sram@100000 {
reg = <0x100000 0x40000>;
};

r5f1_0_sram: r5f1_1-sram@140000 {
reg = <0x140000 0x40000>;
};

r5f1_1_sram: r5f0_0-sram@180000 {
reg = <0x180000 0x40000>;
};

tfa-sram@1c0000 {
reg = <0x1c0000 0x20000>;
};

dmsc-sram@1e0000 {
reg = <0x1e0000 0x1c000>;
};

sproxy-sram@1fc000 {
reg = <0x1fc000 0x4000>;
};
};

The above actually works, and I do not get the same Kernel panic on boot.

QUESTIONS:

1) Why does the ethernet driver need 768 KB of SRAM to work? Shouldn't it only need 384 KB, making two 256 KB banks more than enough?

2) Is there a way to explictly allocate SRAM for the ethernet driver instead of the driver allocating the first available unallocated region? Looking at the source code it seems like there is mulitple algorithms that can be utilized for the allocation of SRAM, or maybe I could do the following:

oc_sram: sram@70000000 {
compatible = "mmio-sram";
reg = <0x00 0x70000000 0x00 0x200000>;
#address-cells = <1>;
#size-cells = <1>;
ranges = <0x0 0x00 0x70000000 0x200000>;

prueth_sram: prueth-sram@00000 {
reg = <0x00000 0x80000>;
};

r5f0_0_sram: r5f0_1-sram@c0000 {
reg = <0xc0000 0x40000>;
};

...

ethernet {
compatible = "ti,am642-icssg-prueth";
pinctrl-names = "default";
pinctrl-0 = <&icssg0_rgmii1_pins_default>, <&icssg0_rgmii2_pins_default>;

interrupt-parent = <&icssg0_intc>;
interrupts = <24 0 2>, <25 1 3>;
interrupt-names = "tx_ts0", "tx_ts1";

sram = <&prueth_sram>;

...

Thanks,

Jared

11 months ago

0 Nick Saulnier 11 months ago

TI__Guru** 110990 points

Hello Jared,

You are going down the right path, but we may need to experiment with where you are defining the SRAM for the PRU Ethernet. Please let me know what ends up working!

Several notes:

Allocating SRAM for R5F cores

Defining the SRAM regions is sufficient to prevent Linux from overwriting the R5F SRAM with the PRU Ethernet data. However, if you want Linux to write data into those memory regions during R5F initialization, you should make sure to also reference the SRAM from the R5F Linux devicetree nodes.

My example code in the AM64x Multicore academy still needs to be updated, so please refer to this post in the meantime:
PROCESSOR-SDK-AM64X: v10.01.10.04: Problems with adding R5FSS0_0 SRAM to devicetree when following AM64x Academy

Future readers, you can refer to AM64x academy > multicore > Application Development on Remote Cores > Allocate memory (but do check the patch to make sure it corresponds with the linked e2e thread. Not sure when I'll have time to update the academy docs, so you may be reading this before I am able to update):
https://dev.ti.com/tirex/explore/node?node=A__AXdlNVv-3hxEuToJ8O8ivg__AM64-ACADEMY__WI1KRXP__LATEST

Allocating memory for PRU Ethernet

768kB sounds like too much. Per ICSSG subsystem, we should need 192kB for Ethernet endpoint, or 256kB for Ethernet switch. So I would expect a max of 512kB needed if you have 4 PRU Ethernet ports implementing 2 switches.

Yes, you should be able to define a specific SRAM region and pass it to the PRU Ethernet driver. I have not tested this myself on hardware, but you can see how I expect it to look in this e2e thread:
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1307841/re-am6548-allocating-sram-for-pru-ethernet/4967459#4967459

Regards,

Nick

0 Jared Weyer 11 months ago in reply to Nick Saulnier

Prodigy 197 points

Hi,

Just got around to checking if explicitly allocating sram for ICSSG works, and it does.

For future readers. This is the device tree patch I ended up with:

arch/arm64/boot/dts/ti/k3-am64-main.dtsi | 20 +++++++++++++++++++
.../dts/ti/k3-am642-phyboard-electra-rdk.dts | 2 +-
2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/boot/dts/ti/k3-am64-main.dtsi b/arch/arm64/boot/dts/ti/k3-am64-main.dtsi
index f585eda7fa16..3c00828b7fa5 100644
--- a/arch/arm64/boot/dts/ti/k3-am64-main.dtsi
+++ b/arch/arm64/boot/dts/ti/k3-am64-main.dtsi
@@ -24,6 +24,26 @@ oc_sram: sram@70000000 {
#size-cells = <1>;
ranges = <0x0 0x00 0x70000000 0x200000>;

+ ethernet_sram: ethernet-sram@00000 {
+ reg = <0x00000 0xc0000>;
+ };
+
+ r5f0_0_sram: r5f0_0-sram@c0000 {
+ reg = <0xc0000 0x40000>;
+ };
+
+ r5f0_1_sram: r5f0_1-sram@100000 {
+ reg = <0x100000 0x40000>;
+ };
+
+ r5f1_0_sram: r5f1_0-sram@140000 {
+ reg = <0x140000 0x40000>;
+ };
+
+ r5f1_1_sram: r5f1_1-sram@180000 {
+ reg = <0x180000 0x40000>;
+ };
+
tfa-sram@1c0000 {
reg = <0x1c0000 0x20000>;
};
diff --git a/arch/arm64/boot/dts/ti/k3-am642-phyboard-electra-rdk.dts b/arch/arm64/boot/dts/ti/k3-am642-phyboard-electra-rdk.dts
index bc8e1ce11047..2eb1ca0149b2 100644
--- a/arch/arm64/boot/dts/ti/k3-am642-phyboard-electra-rdk.dts
+++ b/arch/arm64/boot/dts/ti/k3-am642-phyboard-electra-rdk.dts
@@ -67,7 +67,7 @@ ethernet {
interrupts = <24 0 2>, <25 1 3>;
interrupt-names = "tx_ts0", "tx_ts1";

- sram = <&oc_sram>;
+ sram = <&ethernet_sram>;
firmware-name = "ti-pruss/am65x-sr2-pru0-prueth-fw.elf",
"ti-pruss/am65x-sr2-rtu0-prueth-fw.elf",
"ti-pruss/am65x-sr2-txpru0-prueth-fw.elf",
--

Thanks for your help Nick!

- Jared

0 Nick Saulnier 11 months ago in reply to Jared Weyer

TI__Guru** 110990 points

Hello Jared,

Glad to hear that things are working for you, and thank you for posting your solution!

I am still concerned about the 768kB of memory allocated for the Ethernet interface. If you have the time, can I get you to test with ethernet_sram of size 256kB (0x40000) and 192kB (0x30000)?

I am already planning to add a note to update the SDK docs to say that the buffer needs to be on 64kB alignment. But if 192kB & 256kB do not work as documented, we (TI) needs to do additional digging to make sure we don't need to fix something in our code.

Regards,

Nick

0 Jared Weyer 11 months ago in reply to Nick Saulnier

Prodigy 197 points

Hey Nick,

Yeah, I will test this later today.

Thanks,

Jared

0 Jared Weyer 11 months ago in reply to Nick Saulnier

Prodigy 197 points

Hey Nick,

Turns out the above answer is wrong. I never looked at the actual state of the PRU ethernet interface. With the above device tree setup, the driver fails with 'unable to get SRAM pool':

prueth->sram_pool = of_gen_pool_get(np, "sram", 0);
if (!prueth->sram_pool) {
dev_err(dev, "unable to get SRAM pool\n");
ret = -ENODEV;

goto put_mem;
}

This is because the pool property must be set (see below). Since it failed here, I wasn't getting to the code that was causing the kernel panic, and thus I was fat and happy because I only cared about the R5 SRAM carveouts and didn't care about the status of PRU eth. Looking into this more, I found that the below DT actually works:

&cbass_main {
	oc_sram: sram@70000000 {
		compatible = "mmio-sram";
		reg = <0x00 0x70000000 0x00 0x200000>;
		#address-cells = <1>;
		#size-cells = <1>;
		ranges = <0x0 0x00 0x70000000 0x200000>;

        ethernet_sram: ethernet-sram@00000 {
            compatible = "mmio-sram";
            reg = <0x00000 0x40000>;
            pool;
        };

        r5f0_0_sram: r5f0_0-sram@c0000 {
            reg = <0xc0000 0x40000>;
        };

        r5f0_1_sram: r5f0_1-sram@100000 {
            reg = <0x100000 0x40000>;
        };

        r5f1_0_sram: r5f1_0-sram@140000 {
            reg = <0x140000 0x40000>;
        };

        r5f1_1_sram: r5f1_1-sram@180000 {
            reg = <0x180000 0x40000>;
        };

		tfa-sram@1c0000 {
			reg = <0x1c0000 0x20000>;
		};

		dmsc-sram@1e0000 {
			reg = <0x1e0000 0x1c000>;
		};

		sproxy-sram@1fc000 {
			reg = <0x1fc000 0x4000>;
		};
	};

I didn't get around to testing with 192KB, but 256KB works great!

[ 9.263114] icssg-prueth ethernet: couldn't get ti,pa-stats syscon regmap
[ 9.277631] icssg-prueth ethernet: port 1: using random MAC addr: 2a:8c:c4:1b:88:0e
[ 9.290835] icssg-prueth ethernet: port 2: using random MAC addr: 2a:9c:84:56:38:2b
[ 9.362106] icssg-prueth ethernet: TI PRU ethernet driver initialized: dual EMAC mode

I will test 192KB tomorrow.

Cheers,

-Jared

0 Jared Weyer 11 months ago in reply to Jared Weyer

Prodigy 197 points

192KB fails with ENOMEM:

[ 199.979100] icssg-prueth ethernet: unable to allocate MSMC resource
[ 200.002070] icssg-prueth ethernet: probe with driver icssg-prueth failed with error -12

Here is the SRAM node:

ethernet_sram: ethernet-sram@00000 {
    reg = <0x00000 0x40000>;
    pool;
};

This is 64KB aligned so I'm not sure why this isn't working. A quick check of the code shows that the documentation is correct in that 192KB should be enough memory:

#define PRUETH_EMAC_BUF_POOL_SIZE SZ_8K
#define PRUETH_EMAC_POOLS_PER_SLICE 24
#define PRUETH_EMAC_BUF_POOL_START 8
#define PRUETH_NUM_BUF_POOLS 8
#define PRUETH_EMAC_RX_CTX_BUF_SIZE SZ_16K /* per slice */
#define MSMC_RAM_SIZE \
(2 * (PRUETH_EMAC_BUF_POOL_SIZE * PRUETH_NUM_BUF_POOLS + \
PRUETH_EMAC_RX_CTX_BUF_SIZE * 2))

...

	msmc_ram_size = MSMC_RAM_SIZE;
	prueth->is_switchmode_supported = prueth->pdata.switch_mode;
	if (prueth->is_switchmode_supported)
		msmc_ram_size = MSMC_RAM_SIZE_SWITCH_MODE;

	/* NOTE: FW bug needs buffer base to be 64KB aligned */
	prueth->msmcram.va =
		(void __iomem *)gen_pool_alloc_algo(prueth->sram_pool,
						    msmc_ram_size,
						    gen_pool_first_fit_align,
						    &gp_data);

So maybe gen_pool_alloc_algo isn't working as expected.

Cheers,

Jared

0 Nick Saulnier 11 months ago in reply to Jared Weyer

TI__Guru** 110990 points

Hello Jared,

Thank you for the followup here. I have not actually tried this, so I was unaware of the pool requirement!

Ok, so 256kB is working for dual EMAC mode, but 192kB is failing. I will take this back to the team to discuss.

Is there anything else you need from our side?

Regards,

Nick

0 Jared Weyer 11 months ago in reply to Nick Saulnier

Prodigy 197 points

All good on my side!

Thanks,

Jared

0 Nick Saulnier 10 months ago in reply to Jared Weyer

TI__Guru** 110990 points

Hello Jared,

Sounds good, thanks for creating this thread.

Since you are unblocked, I have filed 2 separate tickets for the team (update the Linux SDK docs with more details, and look into the 192kB vs 256kB issue) at a lower priority. It might be a bit before the team digs into what is going on, so I will not plan to provide additional updates on this thread. Future readers, if you want an update, create a new e2e thread and reference this one.

Feel free to reach back out in a new thread if something else pops up!

Regards,

Nick

0 Nick Saulnier 10 months ago in reply to Nick Saulnier

TI__Guru** 110990 points

For future readers, the Linux driver defaults to enabling switch mode, which is why 256kB is required by default. if you went into the Linux driver and modified it to disable switch mode, you would be able to use 192kB instead.

we will document this starting in SDK 11.2 of the SDK docs. In the meantime, future readers, please refer to Jared's example from Nov 12, not his example from Nov 7: RE: AM6442: SRAM allocation & ICSSG ethernet driver failure

Regards,

Nick

Processors

Processors forum

AM6442: SRAM allocation & ICSSG ethernet driver failure