PROCESSOR-SDK-AM65X: RSTP issue with several boards

Hello Milan,

Double-checking on this: Have you tested RSTP on a previous version of the SDK and seen different behavior? Or is this the first time you are testing it on AM65x?

I am not sure if the information on that page was ever actually tested on AM65x/AM64x (which is one reason we removed that page in the SDK 9.1 version of the AM65x Linux SDK docs). We do have a single line on turning STP on and off in the main SDK docs:

https://software-dl.ti.com/processor-sdk-linux/esd/AM65X/09_01_00_01/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/PRU_ICSSG_Ethernet_Switch.html

Regards,

Nick

Yes we did a try on previous version first and there is same error. Then we did a try on latest release but we followed doc from SDK 8.x... All the time we did a tests on AM65x... I will try process setup in your link and I will let you know

Hello Milan, sounds good. If needed, I can ask around to see what testing has been done from our side on STP performance and expected terminal output.

Regards,

Nick

Thread has been unlocked.

Responding to "We did a test with AM64x and we do not have this kind of issue like on AM65x... we think it is issue with PRU on AM65x"

Was the AM64x test also on Linux kernel 6.1, or on a more recent Linux kernel version?

Could I get you to confirm the board revision for the AM65x (Assuming you are using the AM65x EVM instead of a custom board)?

Regards,

Nick

Yes it is AM65X EVM but SR1 as we do not have latest EVM... On AM64x we are based on latest TI SDK 11 with kernel 6.12. In case of AM65 it is SDK 9 as this one is only available from TI... I need to  see if I can check with some our custom images based on new kernel version 6.12 for AM65x but this is not something that is coming from TI

Hello Milan,

I have been told that there are firmware fixes for RSTP that were added to the AM64x firmware at some point, but not added to AM65x firmware for any existing SDK release. These RSTP fixes have been added to the AM65x firmware for the upcoming SDK 11.2 release. The fixes are not guaranteed to resolve the behavior you are observing, but they should get the AM64x and AM65x firmware to be identical (so if the behavior is fixed on AM64x kernel 6.12, that seems like an encouraging sign for AM65x kernel 6.12).

However, this firmware is written for AM65x SR2.0, where the PRU subsystem is very different than AM65x SR1.0 (e.g., 6 PRU cores per subsystem instead of 4).

Regards,

Nick

Hi Nick,

OK we are testing this with AM65x EVM GP SR2.0 

k3conf --cpuinfo

AM65x SR2.0 

This should be fine as we have SR1.0 but on HS EVM. we cannot use our AM64x HS EVM with your SDK and this is why we did with GP only

Hello Milan,

Got it.

It looks like the am65x-sr2 firmware was last updated 9 days ago. I am not sure if this firmware would work with Linux kernel 6.1, or if it would only work with Linux kernel 6.12. I will ask the firmware team if these binaries are the ones with RSTP fixes:
https://git.ti.com/cgit/processor-firmware/ti-linux-firmware/tree/ti-pruss?h=ti-linux-firmware

Regards,

Nick

The firmware team says those are the latest binaries. We are releasing the AM65x Linux SDK 11.2 soon (I think within the next week), so the team suggests waiting for the SDK 11.2 release to test the latest PRU binaries.

Regards,

Nick

SDK 11.2 is now live on TI.com

Thanks Nick... we will test this on our side too

Hi Nick

We tried to test this new SDK release 

https://www.ti.com/tool/PROCESSOR-SDK-AM65X

We took default image tisdk-default-image-rt-am65xx-evm-11.02.05.02.rootfs.wic.xz but here PRU is not available at all... It seems PRU does not work on this image

 root@am65xx-evm:~# dmesg | grep -i pru
[ 8.271861] remoteproc remoteproc1: b034000.pru is available
[ 8.284042] remoteproc remoteproc3: b00a000.txpru is available
[ 8.284862] remoteproc remoteproc4: b038000.pru is available
[ 8.286013] remoteproc remoteproc6: b00c000.txpru is available
[ 8.288915] remoteproc remoteproc7: b134000.pru is available
[ 8.294920] remoteproc remoteproc9: b10a000.txpru is available
[ 8.300325] remoteproc remoteproc10: b138000.pru is available
[ 8.309337] remoteproc remoteproc12: b10c000.txpru is available
[ 8.316146] remoteproc remoteproc13: b234000.pru is available
[ 8.322362] remoteproc remoteproc15: b20a000.txpru is available
[ 8.337248] remoteproc remoteproc16: b238000.pru is available
[ 8.362390] remoteproc remoteproc18: b20c000.txpru is available  

root@am65xx-evm:~# ifconfig -a
eth0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 40:2e:71:3f:3e:45 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 90 bytes 7514 (7.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 90 bytes 7514 (7.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Are we missing something here? We just used official image without any modification...

Hello Milan,

Do you see anything in your boot logs about the PRU Ethernet driver?

It does not look like your board has detected a link up. The PRU firmware has not actually been loaded into the PRU cores in your dmesg output, but I would expect to see that if I would expect to see the PRU cores get loaded with PRU firmware.

I am building an SD card now that I can use to run tests tomorrow. Please keep me honest and ping the thread if I do not get back to you on Friday.

Regards,

Nick

Hi Nick, 

yes no any logs about PRU, we did not go in more investigation. We used prebuild images downloaded from TI SDK.

We did not build image, it is default TI image

Hello Milan,

I am able to replicate your observations. Thank you for reporting this so I could dig into it.

The problem

There are 2 issues:

1) ICSSG2 Ethernet nodes are moved from k3-am654-base-board.dts to k3-am654-icssg2.dtso in kernel 6.12

2) Uboot name_overlay argument does not seem to be updated from older kernel versions, when devicetree files were under /boot/dtb/ instead of under /boot/dtb/ti/

The fix 

Please update the uEnv.txt file in the SD card's boot partition like this:

name_overlays=ti/k3-am654-icssg2.dtbo ti/k3-am654-idk.dtbo ti/k3-am654-pcie-usb2.dtbo

I am still figuring out next steps for whether adding documentation for this is enough, or if I need to push the team to update the default filesystem with fixed name_overlay entries.

Regards,

Nick

(if you are just referring to the email, please make sure to reference .dtbo and not .dtso, I had a typo in the initial response)

Hi Nick

Yes this will setup properly PRU... we need to do tests too... But we notice during the boot next issue. This is not PRU issue but it is network related and we need to see if this can make some issues on our RSTP tests... Do you see this kind of logs on your side?

[ 9.660670] ------------[ cut here ]------------
[ 9.660692] WARNING: CPU: 1 PID: 12 at /net/devlink/core.c:382 0xffff8000809ab170
[ 9.660709] Modules linked in: ecc rfkill irq_pruss_intc ti_am335x_adc kfifo_buf icssg_prueth sch_taprio sch_mqprio_lib hsr icssg icss_iep pru_rproc crct10dif_ce omap_mailbox omap_hwspinlock phy_can_6
[ 9.660846] CPU: 1 UID: 0 PID: 12 Comm: kworker/u16:0 Tainted: G O 6.12.49-ti-rt-g1a86d36433ea #1
[ 9.660861] Tainted: [O]=OOT_MODULE
[ 9.660867] Hardware name: Texas Instruments AM654 Base Board (DT)
[ 9.660874] Workqueue: events_unbound 0xffff8000805a8530
[ 9.660882] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 9.660894] pc : 0xffff8000809ab170
[ 9.660897] lr : 0xffff8000809ab118
[ 9.660901] sp : ffff00080015ba90
[ 9.660905] x29: ffff00080015ba90 x28: 00000000ffffffed x27: ffff000807b2ed18
[ 9.660919] x26: ffff000807b2ed18 x25: ffff000807b2ecc0 x24: ffff00087ffa60c0
[ 9.660933] x23: ffff000808ded958 x22: ffff00087ffa6478 x21: 00000000ffffffed
[ 9.660946] x20: ffff80008102e020 x19: ffff0008053e6000 x18: 0000000000000000
[ 9.660958] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000002
[ 9.660970] x14: 0000000000000040 x13: ffff0008000dab40 x12: 0000000000000000
[ 9.660983] x11: 00000000000000c0 x10: 0000000000000002 x9 : 0000000000000001
[ 9.660995] x8 : 0000000000000238 x7 : 0000000000000000 x6 : 0000000000000046
[ 9.661006] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000001
[ 9.661018] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[ 9.661030] Call trace:
[ 9.661035] 0xffff8000809ab170
[ 9.661039] 0xffff8000809ab19c
[ 9.661043] 0xffff80007930d8d4
[ 9.661047] 0xffff80007930e18c
[ 9.661053] 0xffff8000805ab974
[ 9.661057] 0xffff8000805a9084
[ 9.661061] 0xffff8000805a92d4
[ 9.661065] 0xffff8000805a9464
[ 9.661069] 0xffff8000805a95ec
[ 9.661073] 0xffff8000805a7168
[ 9.661077] 0xffff8000805a9948
[ 9.661081] 0xffff8000805a9b94
[ 9.661084] 0xffff8000805a80a8
[ 9.661089] 0xffff8000805a85b8
[ 9.661093] 0xffff800080053b60
[ 9.661097] 0xffff8000800543f8
[ 9.661101] 0xffff80008005c648
[ 9.661105] 0xffff800080014864
[ 9.661109] ---[ end trace 0000000000000000 ]---

Hello Milan,

I did not get any outputs like this the few times that I booted the AM65x SR2 IDK EVM with regular Linux (I forgot that you would probably be using an RT Linux image).

Do you see it each boot, or is the behavior inconsistent?

Regards,

Nick

Hi Nick,

yes it is on each boot... 

we did basic test with RSTP and we did not see issue from this ticket... we will do more test in coming days to confirm

Still we have this dump but I did not see impact on RSTP test

Hello Milan,

Ok. I will create an SD card from the prebuilt RT Linux image tomorrow and see if I can see your behavior. Let me know if anything else occurs to you that might help me see your behavior (e.g., if you are using something other than the latest version of the EVM, if you have Ethernet cables plugged in a certain way, anything that might be easy for me to replicate).

Regards,

Nick

Hi Nick,

Nothing specific, we just put this in uEnv.txt...

name_overlays=ti/k3-am654-icssg2.dtbo ti/k3-am654-idk.dtbo ti/k3-am654-pcie-usb2.dtbo

Even with network/interface down, we see same issue

Correct, this resolve issue... thanks Nick Saulnier