AM6442: XDP Zero Copy for CPSW

Hello Artur, 

Apologies for the delay in response

Artur Toews said:

My current project include "meta-ti:11.02.13", but unfortunately my device crashes as soon as I try to load my bpfprog.o with libxdp. After the crash I cannot do anything, even my serial console is frozen.

Is there a log of the crash that you can share? My expectation is that if your bpf program cannot be loaded due to the kernel configurations necessary for XDP ZC not being enabled, it shouldn't typically result in an entire kernel crash. Additionally, have you validated that there are no issues with your bpfprog.o and that it is tested to be working for XDP zero copy use on perhaps a host PC first?

You are correct that the requirements are the same when using CPSW for XDP zero copy. Have you ensured that these kernel configurations are enabled? https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/PRU_ICSSG_XDP.html#how-to-run-xdp-with-pru-icssg 

-Daolin

Hello Daolin,

My kernel config:

zcat /proc/config.gz | grep -E "XDP|BPF"
CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# BPF subsystem
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT is not set
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
# CONFIG_BPF_PRELOAD is not set
# end of BPF subsystem
CONFIG_CGROUP_BPF=y
CONFIG_XDP_SOCKETS=y
CONFIG_XDP_SOCKETS_DIAG=m
CONFIG_NETFILTER_BPF_LINK=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_NET_CLS_BPF is not set
# CONFIG_NET_ACT_BPF is not set
CONFIG_BPF_STREAM_PARSER=y
# HID-BPF support
# end of HID-BPF support
# CONFIG_TEST_BPF is not set

It looks like it freezes up completely, so nothing works anymore. No console, no network. Even the timer-controlled LEDs stop flashing.

This is what I captured with "journalctl" :

Apr 30 06:07:13 am6442 systemd-logind[325]: New session c3 of user admin.
Apr 30 06:07:13 am6442 systemd-journald[149]: Forwarding to syslog missed 61 messages.
Apr 30 06:07:13 am6442 systemd[1]: Started Session c3 of User admin.
Apr 30 06:07:23 am6442 kernel: am65-cpsw-nuss 8000000.ethernet eth1: entered promiscuous mode
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: btf_vmlinux is malformed
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 0: R1=ctx() R10=fp0
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: ;  @ xdp-dispatcher.c:118
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 0: (bf) r6 = r1                       ; R1=ctx() R6_w=ctx()
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: ;  @ xdp-dispatcher.c:120
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 1: (18) r8 = 0xffff000007aeaf20       ; R8_w=map_value(map=xdp_disp.rodata,ks=4,vs=124)
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 3: (71) r7 = *(u8 *)(r8 +2)           ; R7_w=1 R8_w=map_value(map=xdp_disp.rodata,ks=4,vs=124)
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 4: (b7) r0 = 2                        ; R0_w=2
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: ;  @ xdp-dispatcher.c:123
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 5: (15) if r7 == 0x0 goto pc+136      ; R7_w=1
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: ;  @ xdp-dispatcher.c:125
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 6: (bf) r1 = r6                       ; R1_w=ctx() R6_w=ctx()
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: 7: (85) call pc+135
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: btf_vmlinux is malformed
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: R1 type=ctx expected=fp
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: Caller passes invalid args into func#1 ('prog0')
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: -- END PROG LOAD LOG --
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: prog 'xdp_dispatcher': failed to load: -22
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: failed to load object '/usr/lib/arm-linux-gnueabihf/bpf/xdp-dispatcher.o'
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libxdp: Failed to load dispatcher: Invalid argument
Apr 30 06:07:23 am6442 kernel: ------------[ cut here ]------------
Apr 30 06:07:23 am6442 kernel: Unbalanced enable for IRQ 140
Apr 30 06:07:23 am6442 kernel: WARNING: CPU: 1 PID: 30 at /kernel/irq/manage.c:790 0xffff800080090820
Apr 30 06:07:23 am6442 kernel: Modules linked in: can_raw can xt_addrtype xt_mark xt_conntrack xt_comment bridge stp llc rfkill iptable_filter xt_MASQUERADE xt_nat xt_tcpudp iptable_nat nf_nat ip_tables x_tables onboard_usb_dev rpmsg_ctrl rpmsg_char cdns3 cdns_usb_common at25 crct10dif_ce cdns3_ti rti_wdt ti_k3_r5_remoteproc ti_k3_m4_remoteproc ti_k3_common k3_j72xx_bandgap sa2ul authenc omap_hwspinlock omap_mailbox spi_omap2_mcspi cryptodev(O) fuse drm drm_panel_orientation_quirks backlight nfnetlink ipv6
Apr 30 06:07:23 am6442 kernel: CPU: 1 UID: 0 PID: 30 Comm: ksoftirqd/1 Tainted: G           O       6.12.57-ti-rt-01365-gc52c5589aab8-dirty #1
Apr 30 06:07:23 am6442 kernel: Tainted: [O]=OOT_MODULE
Apr 30 06:07:23 am6442 kernel: Hardware name: Turck Automation Platform based on AM642 (rev0) (DT)
Apr 30 06:07:23 am6442 kernel: pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Apr 30 06:07:23 am6442 kernel: pc : 0xffff800080090820
Apr 30 06:07:23 am6442 kernel: lr : 0xffff800080090820
Apr 30 06:07:23 am6442 kernel: sp : ffff0000001ebb20
Apr 30 06:07:23 am6442 kernel: x29: ffff0000001ebb20 x28: ffff000001bf5778 x27: ffff000001bf5778
Apr 30 06:07:23 am6442 kernel: x26: ffff0000001ebd10 x25: ffff800081078000 x24: ffff000001bf5748
Apr 30 06:07:23 am6442 kernel: x23: 0000000000000040 x22: ffff000001bf5880 x21: ffff000001bf4080
Apr 30 06:07:23 am6442 kernel: x20: 000000000000008c x19: ffff0000018b4c00 x18: ffffffffffffffff
Apr 30 06:07:23 am6442 kernel: x17: 0000000000000000 x16: 0000000000000000 x15: ffff0000001eb778
Apr 30 06:07:23 am6442 kernel: x14: 0000000000000000 x13: ffff800081095ac0 x12: 00000000000005c4
Apr 30 06:07:23 am6442 kernel: x11: 00000000000001ec x10: ffff8000810edac0 x9 : ffff800081095ac0
Apr 30 06:07:23 am6442 kernel: x8 : 00000000000001ed x7 : ffff8000810edac0 x6 : 80000000fffff000
Apr 30 06:07:23 am6442 kernel: x5 : ffff00003ff896d0 x4 : 0000000000000000 x3 : 0000000000000027
Apr 30 06:07:23 am6442 kernel: x2 : 0000000000000027 x1 : ffff0000001ad880 x0 : 0000000100000001
Apr 30 06:07:23 am6442 kernel: Call trace:
Apr 30 06:07:23 am6442 kernel:  0xffff800080090820
Apr 30 06:07:23 am6442 kernel:  0xffff800080090880
Apr 30 06:07:23 am6442 kernel:  0xffff80008077671c
Apr 30 06:07:23 am6442 kernel:  0xffff800080972cb8
Apr 30 06:07:23 am6442 kernel:  0xffff800080972f38
Apr 30 06:07:23 am6442 kernel:  0xffff80008003dd6c
Apr 30 06:07:23 am6442 kernel:  0xffff80008003e3a4
Apr 30 06:07:23 am6442 kernel:  0xffff800080063888
Apr 30 06:07:23 am6442 kernel:  0xffff80008005cdf0
Apr 30 06:07:23 am6442 kernel:  0xffff800080014864
Apr 30 06:07:23 am6442 kernel: ---[ end trace 0000000000000000 ]---
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libxdp: Falling back to loading single prog without dispatcher
Apr 30 06:07:23 am6442 kernel[314]: [  778.482500] ------------[ cut here ]------------
Apr 30 06:07:23 am6442 codesyscontrol-turck[940]: libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Apr 30 06:07:23 am6442 kernel[314]: [  778.482517] Unbalanced enable for IRQ 140
Apr 30 06:07:23 am6442 kernel[314]: [  778.482541] WARNING: CPU: 1 PID: 30 at /kernel/irq/manage.c:790 0xffff800080090820
Apr 30 06:07:23 am6442 kernel[314]: [  778.482552] Modules linked in: can_raw can xt_addrtype xt_mark xt_conntrack xt_comment bridge stp llc rfkill iptable_filter xt_MASQUERADE xt_nat xt_tcpudp iptable_nat nf_nat ip_tables x_tables onboard_usb_dev rpmsg_ctrl rpmsg_char cdns3 cdns_usb_common at25 crct10dif_ce cdns3_ti rti_wdt ti_k3_r5_remoteproc ti_k3_m4_remoteproc ti_k3_common k3_j72xx_bandgap sa2ul authenc omap_hwspinlock omap_mailbox spi_omap2_mcspi cryptodev(O) fuse drm drm_panel_orientation_quirks backlight nfnetlink ipv6
Apr 30 06:07:23 am6442 kernel[314]: [  778.482683] CPU: 1 UID: 0 PID: 30 Comm: ksoftirqd/1 Tainted: G           O       6.12.57-ti-rt-01365-gc52c5589aab8-dirty #1
Apr 30 06:07:23 am6442 kernel[314]: [  778.482695] Tainted: [O]=OOT_MODULE
Apr 30 06:07:23 am6442 kernel[314]: [  778.482699] Hardware name: Turck Automation Platform based on AM642 (rev0) (DT)
Apr 30 06:07:23 am6442 kernel[314]: [  778.482704] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Apr 30 06:07:23 am6442 kernel[314]: [  778.482712] pc : 0xffff800080090820
Apr 30 06:07:23 am6442 kernel[314]: [  778.482715] lr : 0xffff800080090820
Apr 30 06:07:23 am6442 kernel[314]: [  778.482718] sp : ffff0000001ebb20
Apr 30 06:07:23 am6442 kernel[314]: [  778.482721] x29: ffff0000001ebb20 x28: ffff000001bf5778 x27: ffff000001bf5778
Apr 30 06:07:23 am6442 kernel[314]: [  778.482732] x26: ffff0000001ebd10 x25: ffff800081078000 x24: ffff000001bf5748
Apr 30 06:07:23 am6442 kernel[314]: [  778.482743] x23: 0000000000000040 x22: ffff000001bf5880 x21: ffff000001bf4080
Apr 30 06:07:23 am6442 kernel[314]: [  778.482753] x20: 000000000000008c x19: ffff0000018b4c00 x18: ffffffffffffffff
Apr 30 06:07:23 am6442 kernel[314]: [  778.482764] x17: 0000000000000000 x16: 0000000000000000 x15: ffff0000001eb778
Apr 30 06:07:23 am6442 kernel[314]: [  778.482774] x14: 0000000000000000 x13: ffff800081095ac0 x12: 00000000000005c4
Apr 30 06:07:23 am6442 kernel[314]: [  778.482784] x11: 00000000000001ec x10: ffff8000810edac0 x9 : ffff800081095ac0
Apr 30 06:07:23 am6442 kernel[314]: [  778.482794] x8 : 00000000000001ed x7 : ffff8000810edac0 x6 : 80000000fffff000
Apr 30 06:07:23 am6442 kernel[314]: [  778.482804] x5 : ffff00003ff896d0 x4 : 0000000000000000 x3 : 0000000000000027
Apr 30 06:07:23 am6442 kernel[314]: [  778.482814] x2 : 0000000000000027 x1 : ffff0000001ad880 x0 : 0000000100000001
Apr 30 06:07:23 am6442 kernel[314]: [  778.482824] Call trace:
Apr 30 06:07:23 am6442 kernel[314]: [  778.482828]  0xffff800080090820
Apr 30 06:07:23 am6442 kernel[314]: [  778.482833]  0xffff800080090880
Apr 30 06:07:23 am6442 kernel[314]: [  778.482836]  0xffff80008077671c
Apr 30 06:07:23 am6442 kernel[314]: [  778.482839]  0xffff800080972cb8
Apr 30 06:07:23 am6442 kernel[314]: [  778.482842]  0xffff800080972f38
Apr 30 06:07:23 am6442 kernel[314]: [  778.482845]  0xffff80008003dd6c
Apr 30 06:07:23 am6442 kernel[314]: [  778.482848]  0xffff80008003e3a4
Apr 30 06:07:23 am6442 kernel[314]: [  778.482851]  0xffff800080063888
Apr 30 06:07:23 am6442 kernel[314]: [  778.482855]  0xffff80008005cdf0
Apr 30 06:07:23 am6442 kernel[314]: [  778.482858]  0xffff800080014864
Apr 30 06:07:23 am6442 kernel[314]: [  778.482862] ---[ end trace 0000000000000000 ]---

BR
Artur

Hi Artur, 

Thanks for sharing your kernel config and logs

Daolin Qiu said:

You are correct that the requirements are the same when using CPSW for XDP zero copy. Have you ensured that these kernel configurations are enabled? https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/latest/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/PRU_ICSSG_XDP.html#how-to-run-xdp-with-pru-icssg 

Looking at your kernel config, it appears you are missing some configurations that was specified in this link. Most notably the "CONFIG_DEBUG_INFO_BTF" is missing from your kernel config which is likely contributing to your error "btf_vmlinux is malformed" from your logs.

-Daolin

Hi Daolin,

I cannot enable DEBUG_INFO_BTF in my custom yocto-project.

I already added this fragment: 

CONFIG_DEBUG_INFO_BTF=y
CONFIG_BPF_PRELOAD=y
CONFIG_BPF_PRELOAD_UMD=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_LSM=y
CONFIG_DEBUG_INFO_REDUCED=n
CONFIG_FTRACE=y
CONFIG_XDP_SOCKETS=y

But the option is still disabled. 

BR
Artur

Hi Artur,

Artur Toews said:

I cannot enable DEBUG_INFO_BTF in my custom yocto-project.

There might be some dependencies that is causing DEBUG_INFO_BTF not to be enabled. Have you double checked if there are any kernel configs that DEBUG_INFO_BTF depend on also being enabled? Also are you using menuconfig to enable DEBUG_INFO_BTF? menuconfig will show you the configs that it depends on, just adding the config manually won't account for the dependencies.

From the past I remember that below sequence (in exactly in this order) what what I needed but you should double check on menuconfig in case this sequence no longer works.

1. Disable CONFIG_DEBUG_INFO_REDUCED 

2. Install PAHOLE on your host PC if not already installed (sudo apt install) - I might be misremembering here but I believe this is necessary because a PAHOLE config is necessary for CONFIG_DEBUG_INFO_BTF

3. Enable necessary config dependencies CONFIG_DEBUG_INFO_BTF has (check with menuconfig)

3. Enable CONFIG_DEBUG_INFO_BTF

-Daolin

Hi Daolin,

it seems that it works even without the CONFIG_DEBUG_INFO_BTF. 
My test-app (codesyscontrol) loads the BPF prog. 

That means loading the BPF prog is not the cause for the crash. From my understanding it must be the memory initialisation.

Can a wrong memory initialisation cause the crash?

-Artur

Hi Daolin,

I was able to capture a kernel panic:

[ 22.962251] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000018
[ 22.962278] Mem abort info:
[ 22.962280] ESR = 0x0000000096000006
[ 22.962284] EC = 0x25: DABT (current EL), IL = 32 bits
[ 22.962290] SET = 0, FnV = 0
[ 22.962294] EA = 0, S1PTW = 0
[ 22.962297] FSC = 0x06: level 2 translation fault
[ 22.962301] Data abort info:
[ 22.962303] ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[ 22.962307] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 22.962311] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 22.962317] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000093574000
[ 22.962322] [0000000000000018] pgd=080000009356b003, p4d=080000009356b003, pud=080000009357d003, pmd=0000000000000000
[ 22.962341] Internal error: Oops: 0000000096000006 [#1] PREEMPT_RT SMP
[ 22.962348] Modules linked in: can_raw can xt_addrtype xt_mark xt_conntrack xt_comment bridge stp llc rfkill iptable_filter xt_MASQUERADE xt_nat xt_tcpudp iptable_nat nf_nat ip_tables x_tables onboard_usb_dev rpmsg_ctrl rpmsg_char cdns3 cdns_usb_common at25 crct10dif_ce cdns3_ti ti_k3_m4_remoteproc rti_wdt ti_k3_r5_remoteproc ti_k3_common k3_j72xx_bandgap sa2ul authenc omap_hwspinlock omap_mailbox spi_omap2_mcspi cryptodev(O) drm fuse drm_panel_orientation_quirks backlight nfnetlink ipv6
[ 22.962473] CPU: 1 UID: 0 PID: 30 Comm: ksoftirqd/1 Tainted: G O 6.12.57-ti-rt-01365-gc52c5589aab8-dirty #1
[ 22.962486] Tainted: [O]=OOT_MODULE
[ 22.962489] Hardware name: Turck Automation Platform based on AM642 (rev0) (DT)
[ 22.962494] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 22.962503] pc : 0xffff800080773200
[ 22.962507] lr : 0xffff8000807731cc
[ 22.962510] sp : ffff0000001ebb90
[ 22.962514] x29: ffff0000001ebb90 x28: ffff000001cac128 x27: 00000000000000a0
[ 22.962525] x26: 0000000000000002 x25: 0000000000000000 x24: 0000000000000000
[ 22.962535] x23: 0000000000000000 x22: 0000000000000000 x21: ffff000001cac080
[ 22.962546] x20: ffff000001cac080 x19: 0000000000000000 x18: 0000000000000000
[ 22.962556] x17: 0000000000000000 x16: 0000000000000000 x15: 00000000f72ddb48
[ 22.962566] x14: 0000000000000000 x13: ffff0000001ad900 x12: 0000000000000000
[ 22.962576] x11: 00000000000000c0 x10: 0000000000000980 x9 : ffff0000001ebd80
[ 22.962586] x8 : ffff0000001ebd90 x7 : 0000000000000000 x6 : 00000000000001e1
[ 22.962596] x5 : ffff000001cac2e8 x4 : ffff000001cac2e8 x3 : 0000000000000640
[ 22.962607] x2 : 0000000000000000 x1 : 00000000fffbc5d2 x0 : ffff000001ce1d00
[ 22.962618] Call trace:
[ 22.962622] 0xffff800080773200
[ 22.962625] 0xffff8000809720b8
[ 22.962628] 0xffff800080972338
[ 22.962632] 0xffff80008003dd6c
[ 22.962635] 0xffff80008003e3a4
[ 22.962638] 0xffff800080063888
[ 22.962641] 0xffff80008005cdf0
[ 22.962644] 0xffff800080014864
[ 22.962653] Code: b9435800 9b230800 f9404be2 f9400418 (f9400f00)
[ 22.962659] ---[ end trace 0000000000000000 ]---
[ 22.962666] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[ 22.962671] SMP: stopping secondary CPUs
[ 22.962686] Kernel Offset: disabled
[ 22.962689] CPU features: 0x00,00000080,00000000,4200400b
[ 22.962695] Memory Limit: none
[ 23.250477] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

Hi Daolin,

It just occurred to me that we're using a 32-bit userspace application that initializes this XDP memory. Could that cause any issues on the 64-bit kernel?

-Artur

Hi Artur, 

Just to clarify, previously you were seeing a kernel panic with the message "Unbalanced enable for IRQ 140" and now you are seeing message "Unable to handle kernel NULL pointer dereference at virtual address"? 

In both cases, approximately how long did it take before the kernel panic appear after starting your application that uses XDP zero copy?

Artur Toews said:

it seems that it works even without the CONFIG_DEBUG_INFO_BTF. 
My test-app (codesyscontrol) loads the BPF prog. 

Artur Toews said:

My current project include "meta-ti:11.02.13", but unfortunately my device crashes as soon as I try to load my bpfprog.o with libxdp. After the crash I cannot do anything, even my serial console is frozen.

I remember seeing kernel panic issues related to "Unbalanced enable for IRQ" occur with our SDK 11.2 release from which the patches from https://lore.kernel.org/all/20260220041431.372610-1-s-vadapalli@ti.com/ is meant to fix this issue, the patches are awaiting being accepted upstream. This might already be backported to SDK 12.0. You can try applying the patches from the link to your SDK 11.2 build or try moving to latest SDK 12.0. I would recommend first trying to apply the patches.

Artur Toews said:

It just occurred to me that we're using a 32-bit userspace application that initializes this XDP memory. Could that cause any issues on the 64-bit kernel?

The application you use should technically be built for 64-bit, my understanding is that 32-bit built application shouldn't be able to start running on AM64x Linux kernel (not specifically XDP memory but simply due to architecture incompatibility).

-Daolin

Hi Daolin,

I found the root cause. It was a missing patch in kernel 6.12 which I found in 6.18.
PENDING: net: ethernet: ti: Fix port lookup to avoid OOB/NULL derefer… · TexasInstruments/ti-linux-kernel@77c46e9

Now I only see the "Unbalanced enable for IRQ" warning.
I tried to apply the path from you link:

diff --git a/drivers/net/ethernet/ti/am65-cpsw-nuss.c b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
index 5924db6be3fe..8785ab40f157 100644
--- a/drivers/net/ethernet/ti/am65-cpsw-nuss.c
+++ b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
@@ -1570,8 +1570,8 @@ static irqreturn_t am65_cpsw_nuss_rx_irq(int irq, void *dev_id)
 {
 	struct am65_cpsw_rx_flow *flow = dev_id;
 
-	flow->irq_disabled = true;
 	disable_irq_nosync(irq);
+	flow->irq_disabled = true;
 	napi_schedule(&flow->napi_rx);
 
 	return IRQ_HANDLED;
-- 
2.51.1

but I still get the "Unbalanced enable for IRQ" warning:

------------[ cut here ]------------
[   34.365063] Unbalanced enable for IRQ 64
[   34.365085] WARNING: CPU: 1 PID: 30 at /kernel/irq/manage.c:790 0xffff800080090820
[   34.365097] Modules linked in: xt_addrtype xt_mark can_raw can xt_conntrack xt_comment bridge stp llc rfkill iptable_filter xt_MASQUERADE xt_nat xt_tcpudp iptable_nat nf_nat ip_tables x_tables onboard_usb_dev rpmsg_ctrl rpmsg_char cdns3 cdns_usb_common at25 crct10dif_ce cdns3_ti ti_k3_m4_remoteproc ti_k3_common rti_wdt ti_k3_r5_remoteproc k3_j72xx_bandgap sa2ul authenc omap_hwspinlock omap_mailbox spi_omap2_mcspi cryptodev(O) drm drm_panel_orientation_quirks backlight fuse nfnetlink ipv6
[   34.365224] CPU: 1 UID: 0 PID: 30 Comm: ksoftirqd/1 Tainted: G           O       6.12.57-ti-rt-01367-g67cae3885e40-dirty #1
[   34.365237] Tainted: [O]=OOT_MODULE
[   34.365240] Hardware name: Turck Automation Platform based on AM642 (rev0) (DT)
[   34.365246] pstate: 800000c5 (Nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   34.365255] pc : 0xffff800080090820
[   34.365258] lr : 0xffff800080090820
[   34.365261] sp : ffff0000001ebb50
[   34.365264] x29: ffff0000001ebb50 x28: 0000000000000040 x27: 0000000000000003
[   34.365274] x26: ffff000001ec3600 x25: 0000000000000000 x24: ffff000001995000
[   34.365285] x23: 0000000000000008 x22: ffff00001ae70000 x21: ffff000001cf8080
[   34.365295] x20: 0000000000000040 x19: ffff000001965e00 x18: ffffffffffffffff
[   34.365306] x17: 0000000000000000 x16: 0000000000000000 x15: ffff0000001eb7a8
[   34.365316] x14: 0000000000000000 x13: ffff800081095ac0 x12: 000000000000057c
[   34.365326] x11: 00000000000001d4 x10: ffff8000810edac0 x9 : ffff800081095ac0
[   34.365336] x8 : 00000000000001d5 x7 : ffff8000810edac0 x6 : 80000000fffff000
[   34.365346] x5 : ffff00003ff896d0 x4 : 0000000000000000 x3 : 0000000000000027
[   34.365356] x2 : 0000000000000027 x1 : ffff0000001ad880 x0 : 0000000100000001
[   34.365367] Call trace:
[   34.365371]  0xffff800080090820
[   34.365375]  0xffff800080090880
[   34.365378]  0xffff8000807735e8
[   34.365381]  0xffff8000809720d0
[   34.365384]  0xffff800080972350
[   34.365387]  0xffff80008003dd6c
[   34.365390]  0xffff80008003e3a4
[   34.365393]  0xffff800080063888
[   34.365396]  0xffff80008005cdf0
[   34.365400]  0xffff800080014864
[   34.365403] ---[ end trace 0000000000000000 ]---

I also seems that I randomly miss some pakets. I'm not sure if this is related to the IRQ issue. 

-Artur

Hi Artur,

Apologies for the delayed response

Artur Toews said:

I found the root cause. It was a missing patch in kernel 6.12 which I found in 6.18.
PENDING: net: ethernet: ti: Fix port lookup to avoid OOB/NULL derefer… · TexasInstruments/ti-linux-kernel@77c46e9

Good find, I had forgotten to bring up that 6.12 implementation has an issue when XDP is used on the second port which the patch you identified is meant to solve.

"[   34.365063] Unbalanced enable for IRQ 64"

Can you share "cat /proc/interrupts" to help identify what IRQ 64 is showing up as? The patch I shared was meant to address "Unbalanced enable" issues for CPSW RX IRQ that is handling traffic on XDP zero copy port.

Can you also try the below variation of the patch I previously shared?

From 789aa2abb9589bbf9f7da4b39ac217747f12e1e8 Mon Sep 17 00:00:00 2001
From: Siddharth Vadapalli <s-vadapalli@ti.com>
Date: Tue, 10 Feb 2026 14:09:21 +0530
Subject: [PATCH] net: ethernet: ti: am65-cpsw-nuss: fix ordering in Tx and Rx
 IRQ handlers

In both am65_cpsw_nuss_rx_irq() and am65_cpsw_nuss_tx_irq(), the status
of the interrupt is set to 'disabled' before it is actually disabled by
disable_irq_nosync(). As a result, the following scenario is possible:

	CPU0				CPU1
1.	Enters Tx IRQ handler		Running TX NAPI for previous pkts
2.	Sets tx_chn->irq_disabled	Sees tx_chn->irq_disabled
3.	Invokes disable_irq_nosync	Invokes	enable_irq

	At this point, before CPU0 has actually disabled the IRQ, CPU1
	executes enable_irq leading to the following error:
	"Unbalanced enable for IRQ N"

Fix this by disabling the IRQ by executing 'disable_irq_nosync()' before
updating the value of the 'irq_disabled' variable.

Signed-off-by: Siddharth Vadapalli <s-vadapalli@ti.com>
---
 drivers/net/ethernet/ti/am65-cpsw-nuss.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/ti/am65-cpsw-nuss.c b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
index a91774d4c044..41016c137c31 100644
--- a/drivers/net/ethernet/ti/am65-cpsw-nuss.c
+++ b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
@@ -1968,8 +1968,9 @@ static irqreturn_t am65_cpsw_nuss_rx_irq(int irq, void *dev_id)
 {
 	struct am65_cpsw_rx_flow *flow = dev_id;
 
-	atomic_set(&flow->irq_disabled, true);
 	disable_irq_nosync(irq);
+	atomic_set(&flow->irq_disabled, true);
+	smp_mb__after_atomic();
 	napi_schedule(&flow->napi_rx);
 
 	return IRQ_HANDLED;
@@ -1979,8 +1980,9 @@ static irqreturn_t am65_cpsw_nuss_tx_irq(int irq, void *dev_id)
 {
 	struct am65_cpsw_tx_chn *tx_chn = dev_id;
 
-	atomic_set(&tx_chn->irq_disabled, true);
 	disable_irq_nosync(irq);
+	atomic_set(&tx_chn->irq_disabled, true);
+	smp_mb__after_atomic();
 	napi_schedule(&tx_chn->napi_tx);
 
 	return IRQ_HANDLED;
-- 
2.51.1

-Daolin

Hi Daolin,

thanks for the shared patch. Is this patch already available in the kernel repo? 
I'm missing the patches which changed "flow->irq_disabled = true" to "atomic_set(&flow->irq_disabled, true)".

-Artur

 BTW: are there other known issues with XDP? 

Hi Artur,

Artur Toews said:

thanks for the shared patch. Is this patch already available in the kernel repo? 
I'm missing the patches which changed "flow->irq_disabled = true" to "atomic_set(&flow->irq_disabled, true)".

As mentioned, this patch I shared in my last response is simply an older variation of the first patch I shared in an earlier response. The first patch I shared is the final patch that got upstreamed for 6.18 kernel. The patch I shared in my last response is not upstreamed because it was proved from internal testing that the first patch I shared solves the "Unbalanced enable IRQ" issue.

Artur Toews said:

BTW: are there other known issues with XDP? 

From my understanding two issues are fixed with XDP from 6.12 to 6.18 kernel

1. 2nd port XDP setup leads to "Unable to handle kernel NULL pointer dereference at virtual address" 

2. CPSW RX IRQ that is handling traffic on XDP zero copy port leads to race condition causing "Unbalanced enable for IRQ" issue

XDP is a relatively new feature we added to CPSW driver so as we perform more edge case testing, the feature will mature. At this point in time, I have not observed other identified issues with the CPSW XDP implementation but this may change as we perform more edge case testing.

One question for you, are you planning on having promiscuous mode enabled for the ethernet port you are configuring for XDP?

Daolin Qiu said:

Can you share "cat /proc/interrupts" to help identify what IRQ 64 is showing up as? The patch I shared was meant to address "Unbalanced enable" issues for CPSW RX IRQ that is handling traffic on XDP zero copy port.

Can you also check this?

-Daolin

Hi Daolin,

Daolin Qiu said:

As mentioned, this patch I shared in my last response is simply an older variation of the first patch I shared in an earlier response. The first patch I shared is the final patch that got upstreamed for 6.18 kernel. The patch I shared in my last response is not upstreamed because it was proved from internal testing that the first patch I shared solves the "Unbalanced enable IRQ" issue.

I'm wondering because the current code does not use the "atomic_set" function:
https://github.com/TexasInstruments/ti-linux-kernel/blob/c52c5589aab8c376f975b1feb7b8746b7b6624bd/drivers/net/ethernet/ti/am65-cpsw-nuss.c#L1972

But that means the 6.18.y kernel should not show the unbalanced issue?

Daolin Qiu said:

One question for you, are you planning on having promiscuous mode enabled for the ethernet port you are configuring for XDP?

Yes, we have promiscuous mode enabled. Why are you asking?

Daolin Qiu said:

Can you share "cat /proc/interrupts" to help identify what IRQ 64 is showing up as?

cat /proc/interrupts
           CPU0       CPU1
 11:     714880     765776     GICv3  30 Level     arch_timer
 14:       1203          0     GICv3  66 Level     4d000000.mailbox thr_012
 23:       7612          0     GICv3 193 Level     20000000.i2c
 60:          0          0  MSI-INTA 1970688 Level     485c0000.dma-controller chan1
 64:     312599          0  MSI-INTA 1970704 Level     8000000.ethernet-tx0
 72:          0        124  MSI-INTA 1970712 Level     8000000.ethernet-tx1
 80:          1          0  MSI-INTA 1970720 Level     8000000.ethernet-tx2
 88:          0         37  MSI-INTA 1970728 Level     8000000.ethernet-tx3
 96:         18          0  MSI-INTA 1970736 Level     8000000.ethernet-tx4
104:          0         10  MSI-INTA 1970744 Level     8000000.ethernet-tx5
112:          3          0  MSI-INTA 1970752 Level     8000000.ethernet-tx6
120:          0         96  MSI-INTA 1970760 Level     8000000.ethernet-tx7
128:          0          0  MSI-INTA 1970776 Level     485c0000.dma-controller chan4
136:         32          0  MSI-INTA 1971712 Level     485c0000.dma-controller chan0
140:     720466          0  MSI-INTA 1971728 Level     8000000.ethernet-rx0
156:          0          0  MSI-INTA 1971752 Level     485c0000.dma-controller chan2
157:          0          0  MSI-INTA 1971753 Level     485c0000.dma-controller chan3
164:        513          0     GICv3 212 Level     2820000.serial
165:          0          0     GICv3 213 Level     2830000.serial
166:          0          0     GICv3 134 Level     8000000.ethernet
176:          0          0     GICv3 129 Level     39000000.cpts
177:      14408          0     GICv3 165 Level     mmc0
181:          0          0     GICv3 166 Level     mmc1
276:          0          0     GICv3 204 Level     20100000.spi
277:      19758          0     GICv3 112 Level     mbox-main-r5fss0-core0, mbox-main-r5fss0-core1
278:          0          0     GICv3 116 Level     mbox-main-r5fss1-core0, mbox-main-r5fss1-core1
279:          0          0     GICv3 108 Level     mbox-m4-0
283:         40          0     GICv3 220 Level     xhci-hcd:usb1
285:          0          0     GICv3 228 Level     f400000.usb
IPI0:    299165     306233       Rescheduling interrupts
IPI1:      5349       8556       Function call interrupts
IPI2:         0          0       CPU stop interrupts
IPI3:         0          0       CPU stop NMIs
IPI4:         0          0       Timer broadcast interrupts
IPI5:       496       1888       IRQ work interrupts
IPI6:         0          0       CPU backtrace interrupts
IPI7:         0          0       KGDB roundup interrupts
Err:          0

-Artur

Hi Daolin,

I've added the patch, but I still get the unbalanced IRQ issue.

-Artur

Hi Artur, 

Can you also perform the below tests

1. See if "unbalanced IRQ issue" appears when CPU0 is isolated (configure "isolcpus=0" in kernel command line)

2. Do the same as #1 but instead of CPU0 isolated, isolate CPU1

Artur Toews said:

Yes, we have promiscuous mode enabled. Why are you asking?

I was recently investigating a potential issue with promiscuous mode and XDP zero copy (still have yet to root cause if it is due to CPSW driver or some other issue yet). As a sanity check, can you test to see if you see the "unbalanced IRQ issue" when promiscuous mode is turned off on the XDP port?

Artur Toews said:

I'm wondering because the current code does not use the "atomic_set" function:
https://github.com/TexasInstruments/ti-linux-kernel/blob/c52c5589aab8c376f975b1feb7b8746b7b6624bd/drivers/net/ethernet/ti/am65-cpsw-nuss.c#L1972

But that means the 6.18.y kernel should not show the unbalanced issue?

6.18.y has the official fix for the issue, atomic_set was deemed not necessary to fix the issue at the time.

-Daolin

Hi Daolin,

with isolated CPU0 I don't see the unbalanced IRQ issues.
Even the communication seems to be a lot more stable. I'm testing PROFINET communication and had random connection drops. With core isolation I didn't see any drop so far.

-Artur