AM625: [CPSW2] Kernel panic happen when ETHTOOL change the TX queue number

Youngkyu Seo

Part Number: AM625

HI TI Expert

Could you check the kernel panic when ETHTOOL change the TX queue number?

root@am62xx-evm:~#
root@am62xx-evm:~# ifconfig eth0 down
[ 46.630377] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
root@am62xx-evm:~#
root@am62xx-evm:~#
root@am62xx-evm:~# ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:82 errors:0 dropped:0 overruns:0 frame:0
TX packets:82 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6220 (6.0 KiB) TX bytes:6220 (6.0 KiB)

root@am62xx-evm:~# ethtool -L eth0 rx 1 tx 4
[ 61.602924] ------------[ cut here ]------------
[ 61.607580] WARNING: CPU: 1 PID: 769 at kernel/irq/devres.c:143 devm_free_irq+0x8c/0x98
[ 61.615564] Modules linked in: iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_tables x_tables wlcore_sdio wl18xx wlcore mac80211 cfg80211 rfkill libarc4 rpmsg_char dwc3 udc_core usb_common cdns_csi2rx v4l2_fwnode crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils ti_k3_r5_remoteproc virtio_rpmsg_bus dwc3_am62 rti_wdt ti_k3_m4_remoteproc sa2ul sha512_generic authenc j721e_csi2rx videobuf2_dma_contig videobuf2_memops cdns_dphy videobuf2_v4l2 videobuf2_common tps6598x typec roles optee_rng rng_core sch_fq_codel cryptodev(O) ipv6
[ 61.666520] CPU: 1 PID: 769 Comm: ethtool Tainted: G O 5.10.153-g3eee621d16 #1
[ 61.675023] Hardware name: Texas Instruments AM625 SK (DT)
[ 61.680497] pstate: 40000005 (nZcv daif -PAN -UAO -TCO BTYPE=--)
[ 61.686487] pc : devm_free_irq+0x8c/0x98
[ 61.690398] lr : devm_free_irq+0x54/0x98
[ 61.694307] sp : ffff800011af3910
[ 61.697609] x29: ffff800011af3910 x28: 0000000000000000
[ 61.702911] x27: ffff800011648cc0 x26: ffff80001100a5f0
[ 61.708213] x25: ffff800011af3bb8 x24: ffff0000046fa580
[ 61.713513] x23: ffff000007a51d80 x22: ffff0000003ce410
[ 61.718812] x21: ffff000001a50120 x20: 000000000000007f
[ 61.724114] x19: ffff000007a51d80 x18: 0000000000000000
[ 61.729415] x17: 0000000000000000 x16: 0000000000000000
[ 61.734715] x15: 0000000001bed3a0 x14: ffffffffffffffff
[ 61.740016] x13: 0000000000000000 x12: 0101010101010101
[ 61.745317] x11: 0000000000000003 x10: 0101010101010101
[ 61.750618] x9 : ffff800010bfe470 x8 : 0000000000000004
[ 61.755918] x7 : 0000000000000008 x6 : ffff8000114ca000
[ 61.761219] x5 : ffff8000114ca178 x4 : ffff800011af38b0
[ 61.766520] x3 : 0000000000000000 x2 : ffff000007a51d80
[ 61.771820] x1 : ffff000007a51d80 x0 : 00000000fffffffe
[ 61.777121] Call trace:
[ 61.779559] devm_free_irq+0x8c/0x98
[ 61.783131] am65_cpsw_nuss_remove_tx_chns+0x54/0xc0
[ 61.788085] am65_cpsw_set_channels+0x44/0x70
[ 61.792435] ethnl_set_channels+0x1b4/0x3a0
[ 61.796606] genl_family_rcv_msg_doit.isra.0+0xd8/0x160
[ 61.801817] genl_rcv_msg+0xf4/0x1e0
[ 61.805381] netlink_rcv_skb+0x60/0x120
[ 61.809205] genl_rcv+0x40/0x58
[ 61.812336] netlink_unicast+0x1c4/0x280
[ 61.816247] netlink_sendmsg+0x1b4/0x3e8
[ 61.820162] __sys_sendto+0xe4/0x160
[ 61.823727] __arm64_sys_sendto+0x30/0x40
[ 61.827729] el0_svc_common.constprop.0+0x84/0x1d8
[ 61.832507] do_el0_svc+0x2c/0x98
[ 61.835815] el0_svc+0x20/0x30
[ 61.838860] el0_sync_handler+0xb0/0xb8
[ 61.842685] el0_sync+0x180/0x1c0
[ 61.845988] ---[ end trace 5315626349959256 ]---
[ 61.861004] ------------[ cut here ]------------
[ 61.865644] Trying to free already-free IRQ 127
[ 61.865699] WARNING: CPU: 1 PID: 769 at kernel/irq/manage.c:1774 free_irq+0x180/0x358
[ 61.878007] Modules linked in: iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_tables x_tables wlcore_sdio wl18xx wlcore mac80211 cfg80211 rfkill libarc4 rpmsg_char dwc3 udc_core usb_common cdns_csi2rx v4l2_fwnode crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils ti_k3_r5_remoteproc virtio_rpmsg_bus dwc3_am62 rti_wdt ti_k3_m4_remoteproc sa2ul sha512_generic authenc j721e_csi2rx videobuf2_dma_contig videobuf2_memops cdns_dphy videobuf2_v4l2 videobuf2_common tps6598x typec roles optee_rng rng_core sch_fq_codel cryptodev(O) ipv6
[ 61.878152] CPU: 1 PID: 769 Comm: ethtool Tainted: G W O 5.10.153-g3eee621d16 #1
[ 61.878155] Hardware name: Texas Instruments AM625 SK (DT)
[ 61.878161] pstate: 80000085 (Nzcv daIf -PAN -UAO -TCO BTYPE=--)
[ 61.878171] pc : free_irq+0x180/0x358
[ 61.952565] lr : free_irq+0x180/0x358
[ 61.952568] sp : ffff800011af38c0
[ 61.952571] x29: ffff800011af38c0 x28: 0000000000000000
[ 61.959525] x27: ffff800011648cc0 x26: ffff80001100a5f0
[ 61.959531] x25: 000000000000007f x24: ffff0000019ca4dc
[ 61.970128] x23: ffff0000019ca590 x22: ffff000001a50120
[ 61.970134] x21: 0000000000000000 x20: 000000000000007f
[ 61.980730] x19: ffff0000019ca400 x18: 0000000000000001
[ 61.980736] x17: 0000000000000000 x16: 0000000000000000
[ 61.991339] x15: ffff000007a522b0 x14: ffffffffffffffff
[ 61.991346] x13: ffff8000116aa688 x12: ffff8000116aa2c8
[ 61.991351] x11: ffff8000115555b0 x10: ffff80001153d570
[ 61.991357] x9 : ffff8000100c6154 x8 : 662d79646165726c
[ 61.991363] x7 : 612065657266206f x6 : 00000000405a78d0
[ 61.991368] x5 : 0000000000000001 x4 : 0000000000000000
[ 61.991374] x3 : 0000000000000027 x2 : 0000000000000023
[ 62.033735] x1 : 46092b1925ffea00 x0 : 0000000000000000
[ 62.033743] Call trace:
[ 62.033750] free_irq+0x180/0x358
[ 62.044780] devm_free_irq+0x68/0x98
[ 62.044793] am65_cpsw_nuss_remove_tx_chns+0x54/0xc0
[ 62.044801] am65_cpsw_set_channels+0x44/0x70
[ 62.057654] ethnl_set_channels+0x1b4/0x3a0
[ 62.057661] genl_family_rcv_msg_doit.isra.0+0xd8/0x160
[ 62.067038] genl_rcv_msg+0xf4/0x1e0
[ 62.067042] netlink_rcv_skb+0x60/0x120
[ 62.067047] genl_rcv+0x40/0x58
[ 62.077558] netlink_unicast+0x1c4/0x280
[ 62.077562] netlink_sendmsg+0x1b4/0x3e8
[ 62.077570] __sys_sendto+0xe4/0x160
[ 62.077574] __arm64_sys_sendto+0x30/0x40
[ 62.077583] el0_svc_common.constprop.0+0x84/0x1d8
[ 62.077590] do_el0_svc+0x2c/0x98
[ 62.101030] el0_svc+0x20/0x30
[ 62.101035] el0_sync_handler+0xb0/0xb8
[ 62.101042] el0_sync+0x180/0x1c0
[ 62.111202] ---[ end trace 5315626349959257 ]---
2023 Feb 22 08:26:38 am62xx-evm [ 62.118463] Unable to handle [ 62.118463] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
kernel NULL pointer dereference at virtual address 0000000000000008
[ 62.118475] Mem abort info:
[ 62.118481] ESR = 0x96000006
[ 62.118490] EC = 0x25: DABT (current EL), IL = 32 bits
[ 62.118496] SET = 0, FnV = 0
[ 62.118502] EA = 0, S1PTW = 0
[ 62.118508] Data abort info:
[ 62.118514] ISV = 0, ISS = 0x00000006
[ 62.118520] CM = 0, WnR = 0
[ 62.118529] user pgtable: 4k pages, 48-bit VAs, pgdp=000000008845b000
[ 62.118535] [0000000000000008] pgd=0000000086ba5003, p4d=0000000086ba5003, pud=000000008464e003, pmd=0000000000000000
[ 62.118573] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 62.118579] Modules linked in: iptable_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_tables x_tables wlcore_sdio wl18xx wlcore mac80211 cfg80211 rfkill libarc4 rpmsg_char dwc3 udc_core usb_common cdns_csi2rx v4l2_fwnode crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils ti_k3_r5_remoteproc virtio_rpmsg_bus dwc3_am62 rti_wdt ti_k3_m4_remoteproc sa2ul sha512_generic authenc j721e_csi2rx videobuf2_dma_contig videobuf2_memops cdns_dphy videobuf2_v4l2 videobuf2_common tps6598x typec roles optee_rng rng_core sch_fq_codel cryptodev(O) ipv6
[ 62.118703] CPU: 1 PID: 769 Comm: ethtool Tainted: G W O 5.10.153-g3eee621d16 #1
[ 62.118705] Hardware name: Texas Instruments AM625 SK (DT)
[ 62.118711] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
[ 62.118728] pc : gen_pool_size+0x20/0x68
[ 62.118732] lr : gen_pool_size+0x18/0x68
[ 62.118735] sp : ffff800011af3920
[ 62.118738] x29: ffff800011af3920 x28: 0000000000000000
[ 62.118744] x27: ffff800011648cc0 x26: ffff80001100a5f0
[ 62.118750] x25: ffff800011af3bb8 x24: ffff0000046fa580
[ 62.118756] x23: ffff000007a51d80 x22: ffff0000003ce410
[ 62.118762] x21: ffff000001a50080 x20: 0000000000000000
[ 62.118768] x19: 0000000000000000 x18: 0000000000000000
[ 62.118774] x17: 0000000000000000 x16: 0000000000000000
[ 62.118779] x15: 0000000000000000 x14: 0000000000000000
[ 62.118785] x13: 0000000000000000 x12: 0000000000000000
[ 62.118790] x11: 0000000000000000 x10: 0000000000000a50
[ 62.118796] x9 : ffff80001052f638 x8 : ffff000007a52830
[ 62.118802] x7 : 02902234d61603a8 x6 : ffff8000114ca000
[ 62.118808] x5 : ffff8000114ca178 x4 : 0000000000000000
[ 62.118813] x3 : 0000000000000000 x2 : 0000000000000000
[ 62.118819] x1 : ffff000007a51d80 x0 : 0000000000000001
[ 62.118826] Call trace:
[ 62.118831] gen_pool_size+0x20/0x68
[ 62.118839] k3_cppi_desc_pool_destroy+0x28/0xa0
[ 62.118849] am65_cpsw_nuss_remove_tx_chns+0x74/0xc0
[ 62.118854] am65_cpsw_set_channels+0x44/0x70
[ 62.118864] ethnl_set_channels+0x1b4/0x3a0
[ 62.118869] genl_family_rcv_msg_doit.isra.0+0xd8/0x160
[ 62.118873] genl_rcv_msg+0xf4/0x1e0
[ 62.118877] netlink_rcv_skb+0x60/0x120
[ 62.118882] genl_rcv+0x40/0x58
[ 62.118886] netlink_unicast+0x1c4/0x280
[ 62.118890] netlink_sendmsg+0x1b4/0x3e8
[ 62.118899] __sys_sendto+0xe4/0x160
[ 62.118903] __arm64_sys_sendto+0x30/0x40
[ 62.118914] el0_svc_common.constprop.0+0x84/0x1d8
[ 62.118918] do_el0_svc+0x2c/0x98
[ 62.118926] el0_svc+0x20/0x30
[ 62.118932] el0_sync_handler+0xb0/0xb8
[ 62.118936] el0_sync+0x180/0x1c0
[ 62.118947] Code: aa0003f3 97eec853 aa1303e3 d2800013 (f8408c61)
[ 62.118955] ---[ end trace 5315626349959258 ]---
2023 Feb 22 08:26:38 am62xx-evm [ 62.118475] Mem abort info:
2023 Feb 22 08:26:38 am62xx-evm Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
2023 Feb 2[ 131.446535] Initializing XFRM netlink socket

over 2 years ago

0 Youngkyu Seo over 2 years ago

Prodigy 30 points

DMA buffer pool was destroied when interface down then it seens that it was again when change the TX queues.

am65_cpsw_set_channels() function remove the tx channels (am65_cpsw_nuss_remove_tx_chns(common); ) even it was checked whether status was busy or not.
could you checked the this function?

static int am65_cpsw_set_channels(struct net_device *ndev,
struct ethtool_channels *chs)
{
struct am65_cpsw_common *common = am65_ndev_to_common(ndev);

if (!chs->rx_count || !chs->tx_count)
return -EINVAL;

/* Check if interface is up. Can change the num queues when
* the interface is down.
*/
if (common->usage_count)
return -EBUSY;

am65_cpsw_nuss_remove_tx_chns(common);

return am65_cpsw_nuss_update_tx_chns(common, chs->tx_count);
}

0 Ashwani Goel over 2 years ago in reply to Youngkyu Seo

TI__Mastermind 27310 points

Hi,

Thanks for your query.

We will check it and get back to you.

Best Regards

Ashwani

+1 Ashwani Goel over 2 years ago in reply to Ashwani Goel

TI__Mastermind 27310 points

Hi,

Can you please apply below patch and update the results ?

https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=e3f260a12e1462a9ab4a5846f936a46fa3a200db

Best Regards

Ashwani

0 Youngkyu Seo over 2 years ago in reply to Ashwani Goel

Prodigy 30 points

Hi Ashwani

After cherry-pick the commit e3f260a, it has compile errors with below

| /workspace/solid/ti/tisdk-default/build/arago-tmp-external-arm-glibc/work-shared/am62xx-evm/kernel-source/drivers/net/ethernet/ti/am65-cpsw-nuss.c:3205:12: error: 'struct am65_cpsw_common' has no member named 'rx_irq_disabled'
| 3205 | if (common->rx_irq_disabled)

So, I also cherry-pick the parents commit 44937fe that was added 'rx_irq_disabled' members in struct

https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/?h=ti-linux-5.10.y&id=44937fe59ccc661c4736fd55ef353cb653dbf1d6

Then I build successfully and verify the issuse was fixted with patches

- my head of build tree.
git log -3 --oneline
38536ea7f382 (HEAD -> ti-linux-5.10.y) Revert "net: ethernet: ti: am65-cpsw: Fix warning "Unbalanced enable IRQ""
a117072f538e net: ethernet: ti: am65-cpsw: Fix set channel operation
3eee621d164a (tag: cicd.2022.12.07.19.06.00, tag: cicd.2022.12.05.19.06.00, tag: cicd.2022.12.02.19.06.02, tag: cicd.2022.12.02.14.23.50, tag: 08.05.00.007) Merged TI feature connectivity into ti-linux-5.10.y-cicd

root@am62xx-evm:~# ifconfig eth0 down
[ 156.867569] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
root@am62xx-evm:~# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 1
TX: 8
Other: n/a
Combined: n/a
Current hardware settings:
RX: 1
TX: 1
Other: n/a
Combined: n/a
root@am62xx-evm:~# ethtool -L eth0 rx 1 tx 4
root@am62xx-evm:~# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 1
TX: 8
Other: n/a
Combined: n/a
Current hardware settings:
RX: 1
TX: 4
Other: n/a
Combined: n/a
root@am62xx-evm:~#

Thanks a lot
Young-kyu

0 Ashwani Goel over 2 years ago in reply to Youngkyu Seo

TI__Mastermind 27310 points

Thanks for update.

Best Regards

Ashwani

Processors

Processors forum

AM625: [CPSW2] Kernel panic happen when ETHTOOL change the TX queue number