AM625: IGH Ethercat stop error

Hello Mike, 

Mike Lee said:

I tested ethercat with kernel 6.6 with native driver, but got an error during ethercat stop

When you say "ethercat stop" are you stopping EtherCAT in software with some sort of Linux command or are you stopping by unplugging/bringing the link down on the Ethernet port connect to the EtherCAT subdevices?

-Daolin

igh ethercat command

/etc/init.d/ethercat stop 

Hello Mike, 

I was able to replicate the issue you are seeing on my setup (with a TI AM62x SKEVM). I will need some time to discuss internally. I plan to respond with an update by Tuesday next week. Please kindly ping this thread if you have not heard a response by then.

Log of my replicated results below:

root@am62xx-evm:~# uname -a
Linux am62xx-evm 6.6.32-g9ed4c5115030-dirty #1 SMP PREEMPT_RT Thu Dec 26 19:24:53 UTC 2024 aarch64 GNU/Linux
root@am62xx-evm:~# ethercatctl start
[   22.874139] EtherCAT: Master driver 1.6.1 1.6-alpha-176-gdb48d9b0
[   22.874772] EtherCAT: 1 master waiting for devices.
[   22.951770] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
[   23.247939] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
[   23.282437] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
[   23.311789] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
[   23.311847] davinci_mdio 8000f00.mdio: phy[1]: device 8000f00.mdio:01, driver TI DP83867
[   23.311941] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000002
[   23.312494] am65-cpsw-nuss 8000000.ethernet: Use random MAC address
[   23.312519] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
[   23.312526] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
[   23.314955] pps pps0: new PPS source ptp0
[   23.315492] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:1
[   23.580645] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
[   23.581485] EtherCAT: Accepting 1C:63:49:0F:61:14 as main device for master 0.
[   23.590780] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
[   23.590808] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): configuring for phy/rgmii-rxid link mode
[   23.590941] EtherCAT 0: Starting EtherCAT-IDLE thread.
root@am62xx-evm:~# [   23.700300] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:01] driver [TI DP83867] (irq=POLL)
[   23.702311] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
[   25.636161] EtherCAT 0: Link state of ecm0 changed to UP.
[   25.636215] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): Link is Up - 100Mbps/Full - flow control off
[   25.638480] EtherCAT 0: 11 slave(s) responding on main device.
[   25.638501] EtherCAT 0: Slave states on main device: PREOP.
[   25.638620] EtherCAT 0: Scanning bus.
[   27.051770] EtherCAT 0: Bus scanning completed in 1413 ms.
[   27.051822] EtherCAT 0: Using slave 0 as DC reference clock.

root@am62xx-evm:~# ethercatctl stop
rmmod: ERROR: Module ec_master is in use by: ec_ti_am65_cpsw_nuss
root@am62xx-evm:~# modprobe -r ec_ti_am65_cpsw_nuss                                                                                                                                       
[   51.414680] EtherCAT 0: Master thread exited.
[   51.415161] ------------[ cut here ]------------
[   51.415165] RTNL: assertion failed at drivers/net/phy/phylink.c (2138)
[   51.415256] WARNING: CPU: 2 PID: 475 at drivers/net/phy/phylink.c:2138 0xffff80007913fb6c
[   51.415265] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) snd_soc_hdmi_codec rpmsg_ctrl rpmsg_char irq_pruss_intc pru_rproc crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils pvrsrvk
m(O) display_connector ti_k3_r5_remoteproc k3_j72xx_bandgap rtc_ti_k3 cfg80211 tidss ti_k3_m4_remoteproc rti_wdt phylink sa2ul snd_soc_davinci_mcasp tps6598x phy_gmii_sel rfkill k3_cppi_desc_pool snd_soc_
ti_udma sii902x snd_soc_ti_edma drm_dma_helper authenc pruss mcrc64 snd_soc_ti_sdma typec drm_kms_helper snd_soc_tlv320aic3x_i2c at24 snd_soc_tlv320aic3x fuse drm drm_panel_orientation_quirks backlight ip
_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[   51.415393] CPU: 2 PID: 475 Comm: modprobe Tainted: G           O       6.6.32-g9ed4c5115030-dirty #1
[   51.415402] Hardware name: Texas Instruments AM625 SK (DT)
[   51.415407] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   51.415416] pc : 0xffff80007913fb6c
[   51.415419] lr : 0xffff80007913fb6c
[   51.415421] sp : ffff800081f73b90
[   51.415424] x29: ffff800081f73b90 x28: ffff000007729880 x27: 0000000000000000
[   51.415434] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[   51.415444] x23: ffff800079170100 x22: 0000000000000480 x21: ffff0000086c4080
[   51.415454] x20: ffff0000065ab000 x19: ffff0000074f7200 x18: ffffffffffffffff
[   51.415463] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800080efed6d
[   51.415472] x14: 0000000000000001 x13: ffff800080efed74 x12: 6166206e6f697472
[   51.415482] x11: ffff800080ddc398 x10: 00000000000001fe x9 : ffff800080e34398
[   51.415492] x8 : 0000000000017fe8 x7 : ffff800080e34398 x6 : 00000000fffff000
[   51.415502] x5 : ffff00007f675a10 x4 : 0000000000000000 x3 : 0000000000000027
[   51.415511] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000007729880
[   51.415521] Call trace:
[   51.415526]  0xffff80007913fb6c
[   51.415529]  0xffff8000791676c4
[   51.415532]  0xffff800079308690
[   51.415535]  0xffff800079308750
[   51.415538]  0xffff800079166550
[   51.415541]  0xffff8000791679ac
[   51.415544]  0xffff8000805c4a78
[   51.415547]  0xffff8000805c1c58
[   51.415550]  0xffff8000805c2e34
[   51.415553]  0xffff8000805c2f04
[   51.415556]  0xffff8000805c148c
[   51.415559]  0xffff8000805c34d0
[   51.415562]  0xffff8000805c47c0
[   51.415564]  0xffff80007916e580
[   51.415567]  0xffff8000800b8068
[   51.415570]  0xffff80008001e8d4
[   51.415574]  0xffff80008001e9b8
[   51.415576]  0xffff800080a32e14
[   51.415579]  0xffff800080a343c0
[   51.415582]  0xffff800080011508
[   51.415586] ---[ end trace 0000000000000000 ]---
[   51.415940] EtherCAT 0: Link state of ecm0 changed to DOWN.
[   51.415955] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): Link is Down
[   51.416347] ------------[ cut here ]------------
[   51.416352] RTNL: assertion failed at drivers/net/phy/phylink.c (1991)
[   51.416423] WARNING: CPU: 2 PID: 475 at drivers/net/phy/phylink.c:1991 0xffff80007913f4f8
[   51.416431] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) snd_soc_hdmi_codec rpmsg_ctrl rpmsg_char irq_pruss_intc pru_rproc crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils pvrsrvk
m(O) display_connector ti_k3_r5_remoteproc k3_j72xx_bandgap rtc_ti_k3 cfg80211 tidss ti_k3_m4_remoteproc rti_wdt phylink sa2ul snd_soc_davinci_mcasp tps6598x phy_gmii_sel rfkill k3_cppi_desc_pool snd_soc_
ti_udma sii902x snd_soc_ti_edma drm_dma_helper authenc pruss mcrc64 snd_soc_ti_sdma typec drm_kms_helper snd_soc_tlv320aic3x_i2c at24 snd_soc_tlv320aic3x fuse drm drm_panel_orientation_quirks backlight ip
_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[   51.416547] CPU: 2 PID: 475 Comm: modprobe Tainted: G        W  O       6.6.32-g9ed4c5115030-dirty #1
[   51.416555] Hardware name: Texas Instruments AM625 SK (DT)
[   51.416559] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   51.416567] pc : 0xffff80007913f4f8
[   51.416570] lr : 0xffff80007913f4f8
[   51.416573] sp : ffff800081f73b80
[   51.416576] x29: ffff800081f73b80 x28: ffff000007729880 x27: 0000000000000000
[   51.416586] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[   51.416595] x23: ffff800079170100 x22: 0000000000000480 x21: ffff0000086c4080
[   51.416604] x20: ffff0000065ab000 x19: ffff0000074f7200 x18: ffffffffffffffff
[   51.416614] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800080eff8bd
[   51.416623] x14: 0000000000000001 x13: ffff800080eff8c4 x12: 6166206e6f697472
[   51.416633] x11: ffff800080ddc398 x10: 000000000000022a x9 : ffff800080e34398
[   51.416643] x8 : 0000000000017fe8 x7 : ffff800080e34398 x6 : 00000000fffff000
[   51.416653] x5 : ffff00007f675a10 x4 : 0000000000000000 x3 : 0000000000000027
[   51.416662] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000007729880
[   51.416672] Call trace:
[   51.416676]  0xffff80007913f4f8
[   51.416680]  0xffff8000791676d4
[   51.416683]  0xffff800079308690
[   51.416686]  0xffff800079308750
[   51.416689]  0xffff800079166550
[   51.416692]  0xffff8000791679ac
[   51.416695]  0xffff8000805c4a78
[   51.416698]  0xffff8000805c1c58
[   51.416701]  0xffff8000805c2e34
[   51.416704]  0xffff8000805c2f04
[   51.416706]  0xffff8000805c148c
[   51.416710]  0xffff8000805c34d0
[   51.416713]  0xffff8000805c47c0
[   51.416715]  0xffff80007916e580
[   51.416718]  0xffff8000800b8068
[   51.416721]  0xffff80008001e8d4
[   51.416725]  0xffff80008001e9b8
[   51.416727]  0xffff800080a32e14
[   51.416731]  0xffff800080a343c0
[   51.416734]  0xffff800080011508
[   51.416737] ---[ end trace 0000000000000000 ]---
[   51.416804] EtherCAT 0: Releasing main device 1C:63:49:0F:61:14.
[   52.451433] am65-cpsw-nuss 8000000.ethernet: rx teardown timeout
[   52.511690] ------------[ cut here ]------------
[   52.511731] k3_knav_desc_pool size 64000 != avail 63872
[   52.512357] WARNING: CPU: 2 PID: 475 at drivers/net/ethernet/ti/k3-cppi-desc-pool.c:33 0xffff8000790b216c
[   52.512386] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) snd_soc_hdmi_codec rpmsg_ctrl rpmsg_char irq_pruss_intc pru_rproc crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils pvrsrvk
m(O) display_connector ti_k3_r5_remoteproc k3_j72xx_bandgap rtc_ti_k3 cfg80211 tidss ti_k3_m4_remoteproc rti_wdt phylink sa2ul snd_soc_davinci_mcasp tps6598x phy_gmii_sel rfkill k3_cppi_desc_pool snd_soc_
ti_udma sii902x snd_soc_ti_edma drm_dma_helper authenc pruss mcrc64 snd_soc_ti_sdma typec drm_kms_helper snd_soc_tlv320aic3x_i2c at24 snd_soc_tlv320aic3x fuse drm drm_panel_orientation_quirks backlight ip
_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[   52.512515] CPU: 2 PID: 475 Comm: modprobe Tainted: G        W  O       6.6.32-g9ed4c5115030-dirty #1
[   52.512525] Hardware name: Texas Instruments AM625 SK (DT)
[   52.512530] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   52.512539] pc : 0xffff8000790b216c
[   52.512542] lr : 0xffff8000790b216c
[   52.512545] sp : ffff800081f73c10
[   52.512548] x29: ffff800081f73c10 x28: ffff000007729880 x27: 0000000000000000
[   52.512559] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[   52.512569] x23: 000000000000002a x22: ffff800081f73c60 x21: ffff00000056c810
[   52.512579] x20: 000000000000fa00 x19: ffff000003825240 x18: 0000000000000000
Segmentation fault[   52.512588] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[   52.512597] x14: 0000000000000003 x13: 000000000000022d x12: 0000000000000002
[   52.512606] x11: 0000000000000000 x10: 0000000000000930 x9 : ffff800081f73a80

[   52.512616] x8 : ffff00000772a210 x7 : ffff000007729910 x6 : 0000000000000000
[   52.512625] x5 : 00000000035f9000 x4 : 0000000000000002 x3 : 000000000000020c
[   52.512634] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000007729880
[   52.512645] Call trace:
[   52.512649]  0xffff8000790b216c
[   52.512652]  0xffff80007916684c
[   52.512656]  0xffff8000805c6600
[   52.512658]  0xffff8000805c7f70
[   52.512661]  0xffff8000805c1a78
[   52.512664]  0xffff8000805c2e54
root@am62xx-evm:~# [   52.512667]  0xffff8000805c2f04
[   52.512670]  0xffff8000805c148c
[   52.512675]  0xffff8000805c34d0
[   52.512678]  0xffff8000805c47c0
[   52.512681]  0xffff80007916e580
[   52.512684]  0xffff8000800b8068
[   52.512687]  0xffff80008001e8d4
[   52.512690]  0xffff80008001e9b8
[   52.512693]  0xffff800080a32e14
[   52.512697]  0xffff800080a343c0
[   52.512701]  0xffff800080011508
[   52.512706] ---[ end trace 0000000000000000 ]---
[   52.513537] ------------[ cut here ]------------
[   52.513556] kernel BUG at lib/genalloc.c:255!
[   52.513567] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT_RT SMP
[   52.513574] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) snd_soc_hdmi_codec rpmsg_ctrl rpmsg_char irq_pruss_intc pru_rproc crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils pvrsrvk
m(O) display_connector ti_k3_r5_remoteproc k3_j72xx_bandgap rtc_ti_k3 cfg80211 tidss ti_k3_m4_remoteproc rti_wdt phylink sa2ul snd_soc_davinci_mcasp tps6598x phy_gmii_sel rfkill k3_cppi_desc_pool snd_soc_
ti_udma sii902x snd_soc_ti_edma drm_dma_helper authenc pruss mcrc64 snd_soc_ti_sdma typec drm_kms_helper snd_soc_tlv320aic3x_i2c at24 snd_soc_tlv320aic3x fuse drm drm_panel_orientation_quirks backlight ip
_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[   52.513706] CPU: 2 PID: 475 Comm: modprobe Tainted: G        W  O       6.6.32-g9ed4c5115030-dirty #1
[   52.513715] Hardware name: Texas Instruments AM625 SK (DT)
[   52.513720] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   52.513728] pc : 0xffff800080405c9c
[   52.513733] lr : 0xffff800080405c94
[   52.513736] sp : ffff800081f73bc0
[   52.513739] x29: ffff800081f73bc0 x28: ffff000007729880 x27: 0000000000000000
[   52.513749] x26: ffff000008796800 x25: ffff0000087967e0 x24: dead000000000122
[   52.513759] x23: 0000000000000007 x22: dead000000000100 x21: ffff000008796800
[   52.513769] x20: ffff8000814d9000 x19: 00000000000001f4 x18: 0000000000000000
[   52.513778] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[   52.513787] x14: 0000000000000003 x13: 00000000000002ec x12: 0000000000000002
[   52.513796] x11: 0000000000000001 x10: ffff00007f67f2d0 x9 : 0000000000000040
[   52.513806] x8 : ffff800081f73bc0 x7 : 0000000000000000 x6 : 0000000000000000
[   52.513815] x5 : ffff8000790b2124 x4 : 0000000000000147 x3 : 0000000000000140
[   52.513824] x2 : ffff8000814d9068 x1 : 00000000000001f4 x0 : 0000000000000147
[   52.513834] Call trace:
[   52.513838]  0xffff800080405c9c
[   52.513842]  0xffff8000790b212c
[   52.513845]  0xffff80007916684c
[   52.513848]  0xffff8000805c6600
[   52.513853]  0xffff8000805c7f70
[   52.513856]  0xffff8000805c1a78
[   52.513859]  0xffff8000805c2e54
[   52.513862]  0xffff8000805c2f04
[   52.513865]  0xffff8000805c148c
[   52.513868]  0xffff8000805c34d0
[   52.513871]  0xffff8000805c47c0
[   52.513874]  0xffff80007916e580
[   52.513877]  0xffff8000800b8068
[   52.513880]  0xffff80008001e8d4
[   52.513883]  0xffff80008001e9b8
[   52.513886]  0xffff800080a32e14
[   52.513890]  0xffff800080a343c0
[   52.513893]  0xffff800080011508
[   52.513902] Code: aa1303e1 97ff94d6 eb00027f 54fffda9 (d4210000) 
[   52.513909] ---[ end trace 0000000000000000 ]---
[   52.513915] note: modprobe[475] exited with irqs disabled
[   52.513919] note: modprobe[475] exited with preempt_count 1
[   52.521992] ------------[ cut here ]------------
[   52.522011] WARNING: CPU: 2 PID: 0 at kernel/context_tracking.c:128 0xffff800080a34e78
[   52.522022] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) snd_soc_hdmi_codec rpmsg_ctrl rpmsg_char irq_pruss_intc pru_rproc crct10dif_ce snd_soc_simple_card snd_soc_simple_card_utils pvrsrvk
m(O) display_connector ti_k3_r5_remoteproc k3_j72xx_bandgap rtc_ti_k3 cfg80211 tidss ti_k3_m4_remoteproc rti_wdt phylink sa2ul snd_soc_davinci_mcasp tps6598x phy_gmii_sel rfkill k3_cppi_desc_pool snd_soc_
ti_udma sii902x snd_soc_ti_edma drm_dma_helper authenc pruss mcrc64 snd_soc_ti_sdma typec drm_kms_helper snd_soc_tlv320aic3x_i2c at24 snd_soc_tlv320aic3x fuse drm drm_panel_orientation_quirks backlight ip
_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[   52.522158] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D W  O       6.6.32-g9ed4c5115030-dirty #1
[   52.522167] Hardware name: Texas Instruments AM625 SK (DT)
[   52.522172] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   52.522181] pc : 0xffff800080a34e78
[   52.522184] lr : 0xffff800080a34e90
[   52.522187] sp : ffff80008115be00
[   52.522189] x29: ffff80008115be00 x28: 0000000000000000 x27: 0000000000000000
[   52.522200] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[   52.522209] x23: 0000000000000000 x22: ffff0000005a24c0 x21: ffff800080dcba7c
[   52.522219] x20: 0000000000000005 x19: ffff00007f678dd8 x18: 0000000000000000
[   52.522228] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[   52.522237] x14: 0000000000000003 x13: 00000000000003eb x12: 0000000000000000
[   52.522248] x11: 0000000000000fc5 x10: 0000000000000930 x9 : 0000000000000000
[   52.522257] x8 : 0000000000000000 x7 : ffff800080dcb350 x6 : 00000002aded79c2
[   52.522266] x5 : 4000000000000002 x4 : ffff7ffffe94c000 x3 : ffff80008115be00
[   52.522276] x2 : ffff800080d2cdd8 x1 : ffff800080d2cdd8 x0 : 4000000000000000
[   52.522286] Call trace:
[   52.522291]  0xffff800080a34e78
[   52.522294]  0xffff800080a34e90
[   52.522297]  0xffff800080a35508
[   52.522300]  0xffff80008007ada0
[   52.522303]  0xffff80008007afe8
[   52.522306]  0xffff80008001de54
[   52.522309]  0xffff80008002486c
[   52.522314] ---[ end trace 0000000000000000 ]---

-Daolin

Hello Daolin,
Any new progress?

Thanks

-Mike

Hi Mike, 

Thanks for following up, I was not yet able to get into touch with the developer who worked on the ec_ti_am65_cpsw_nuss (which I believe to be the patched version of the am65_cpsw_nuss driver), I hope to get in touch in the next two days and will respond if I get an update.

In the meantime, I see that before the EtherCAT master is started, the "ti_am65_cpsw_nuss" is used by phylink and k3_cppi_desc_pool, and after the EtherCAT master is started "ec_ti_am65_cpsw_nuss" is used by phylink, k3_cppi_desc_pool, and ec_master. I noticed removing the "ti_am65_cpsw_nuss" module does not result in the same kernel panic as seen above, though it looks like phylink needs to be removed first.

Before EtherCAT master is started:

root@am62xx-evm:~# lsmod | grep ti_am65_cpsw_nuss
ti_am65_cpsw_nuss     81920 0
phylink               32768 1 ti_am65_cpsw_nuss
k3_cppi_desc_pool     12288 1 ti_am65_cpsw_nuss 

After EtherCAT master is started:

root@am62xx-evm:~# lsmod | grep ec_ti_am65_cpsw_nuss
ec_ti_am65_cpsw_nuss   61440 0
ec_master            241664 1 ec_ti_am65_cpsw_nuss
phylink               32768 1 ec_ti_am65_cpsw_nuss
k3_cppi_desc_pool     12288 1 ec_ti_am65_cpsw_nuss

Removing "ti_am65_cpsw_nuss" module:

root@am62xx-evm:~# lsmod | grep ti_am65_cpsw_nuss
ti_am65_cpsw_nuss     81920 0
phylink               32768 1 ti_am65_cpsw_nuss
k3_cppi_desc_pool     12288 1 ti_am65_cpsw_nuss
root@am62xx-evm:~# uname -a
Linux am62xx-evm 6.6.32-g9ed4c5115030-dirty #1 SMP PREEMPT_RT Thu Dec 26 19:24:53 UTC 2024 aarch64 GNU/Linux
root@am62xx-evm:~# modprobe -r ti_am65_cpsw_nuss
[ 192.793401] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
modprobe: ERROR: ../libkmod/libkmod-module.c:842 kmod_module_remove_module() could not remove 'phylink': Device or resource busy
root@am62xx-evm:~# modprobe -r phylink         
modprobe: ERROR: ../libkmod/libkmod-module.c:842 kmod_module_remove_module() could not remove 'phylink': Device or resource busy
root@am62xx-evm:~# modprobe -r ti_am65_cpsw_nuss
root@am62xx-evm:~# lsmod | grep ti_am65_cpsw_nuss
root@am62xx-evm:~# 

Looking at the steps in the FAQ (https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1409114/faq-how-do-i-build-the-native-igh-ethercat-stack-for-ti-microprocessors), it looks like the "ti-am65-cpsw-nuss_fixes.patch" applies some changes to the am65_cpsw_nuss driver. These changes could have resulted in not being able to cleanly remove the "ec_ti_am65_cpsw_nuss" module. One idea for you to try, while I'm getting in touch with the developer, is to try and revert these changes and see if the issue still occurs and if the EtherCAT native driver can still run.

-Daolin

Hello Daolin,

Did you install the patch when testing?

I currently think this patch is to allow the original driver(ti_am65_cpsw_nuss) to be removed properly. (/etc/init.d/ethercat start)

I will test it in a few days to see what happens if the patch is not installed.

-Mike

Hello Mike, 

Mike Lee said:

Did you install the patch when testing?

Yes, when I was able to replicate the ethercatctl stop issue shown in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1476965/am625-igh-ethercat-stop-error/5673926#5673926 this was with the ti-am65-cpsw-nuss_fixes.patch applied.

Mike Lee said:

I currently think this patch is to allow the original driver(ti_am65_cpsw_nuss) to be removed properly. (/etc/init.d/ethercat start)

I will test it in a few days to see what happens if the patch is not installed.

Yes you could be correct and it could be that the patch didn't handle the removal of ec_ti_am65_cpsw_nuss in a clean manner which lead to problems with ethercatctl stop. However, I'm not positive of this so I will need the developer who worked on this to explain. Once I receive an update, I will respond here, please kindly ping this thread if you have not received a response by next Tuesday.

-Daolin

Hello Daolin,
I originally tested this patch and it can stop normally when port1 is ethercat. 
However, if I use port2 as ethercat later, problems will still occur.

-Mike

Hello Mike, 

Have you modified the EtherCAT configuration file to specify the second port? (i.e. /usr/local/etc/ethercat.conf)

Looking at the conf file there is a description that appears to explain which port is actively configured for EtherCAT:

# the master to accept the first device offered by any Ethernet driver.
#
# The MASTER<X>_DEVICE variables also determine, how many masters will be
# created: A non-empty variable MASTER0_DEVICE will create one master, adding a
# non-empty variable MASTER1_DEVICE will create a second master, and so on.
#
# Examples:
# MASTER0_DEVICE="00:00:08:44:ab:66"
# MASTER0_DEVICE="eth0"
#
MASTER0_DEVICE="ff:ff:ff:ff:ff:ff"
#MASTER1_DEVICE=""

From my own tests, I've noticed the below behavior

• Setting "MASTER0_DEVICE" to all ffs will automatically assume eth0 (the first interface offered by the ethernet driver) is the master device interface
• This means that if your eth0 is connected to non-ethercat subdevice or not connected to anything activating ethercat will not work
• Even if eth1 is connected (eth0 is not connected or connected to a non-ethercat subdevice) activating ethercat will not work

Maybe you can try configuring "MASTER0_DEVICE" with the exact MAC address of the port you are trying to test?

Please note, my responses will be delayed next week as I will be out of office.

-Daolin

Hello Daolin,

Maybe I didn't make it clear enough

I suppose port1(eth0) mac address is AA:BB:AA:BB:AA:BB 
and port2(eth1) is CC:DD:CC:DD:CC:DD

When MASTER0_DEVICE="AA:BB:AA:BB:AA:BB", the module can be removed normally, 
but when MASTER0_DEVICE="CC:DD:CC:DD:CC:DD", the same error will occur.

(After the "[PATCH] Fix ec_ti_am65_cpsw_nuss module removal" has been applied)

-Mike

Hi Mike, 

Thanks for explaining these additional details. Just to clarify, are you using a full 6-byte MAC address for "MASTER0_DEVICE" or you are just using "AA:BB"/"CC:DD" which appears to be just 2 bytes?

-Daolin

Hi Daolin, 

using a full 6-byte MAC address
AA BB Just a simple explanation

-Mike

Hi Mike,

I will have to try testing it out myself and see if I can reproduce the same issue. I won’t be able to do this until first week of April since I will be out of office until then.

Do you only see this issue when using the IgH native CPSW EtherCAT or do you also see it when the generic IgH EtherCAT is used?

-Daolin

Hi Daolin,

Using the native CPSW driver, the generic driver works fine.

When using eth1 as EtherCAT and applying the patch, removing the module still causes issues.
using eth0 as EtherCAT works fine.

-Mike

Hi Mike,

Daolin is currently out of the office until the First week of April. Daolin will trying to reproduce the issue and will respond then.

Best Regards,

Schuyler

Hello Schuyler,
Any new progress?

Thanks

-Mike

Hello Mike, 

Thanks for reaching out again. I was working on reproducing this issue last week; however, on my setup, it looks like the previous provided patch sometimes causes an "Unable to handle kernel NULL pointer dereference" error when trying to stop the EtherCAT application. I was trying to enable the CONFIG_KALLSYMS and CONFIG_KALLSYMS_ALL kernel configuration in order debug the error. 

What I did notice was that when MASTER0_DEVICE is set to "ff:ff:ff:ff:ff:ff", sometimes stopping the EtherCAT application will work without issue, but sometimes it will result in an error. When MASTER0_DEVICE is set to a specific MAC address (including the MAC address for eth0), stopping the EtherCAT application will always result in an error.

Can you share again the exact error log you get from trying to set the eth1 MAC address? Also, are you able to try enabling CONFIG_KALLSYMS and CONFIG_KALLSYMS_ALL to see if you can make sense of the call trace?

-Daolin

Hi Daolin,
I'll try CONFIG_KALLSYMS and CONFIG_KALLSYMS_ALL next week.

-Mike

Hi Mike, 

Sounds good. I'm aiming to look into this further Friday. Let me know if you have any breakthroughs after enabling the configs next week.

-Daolin

Hi Daolin,

My kernel has been enabled CONFIG_KALLSYMS and CONFIG_KALLSYMS_ALL

This is setting eth1 mac address as MASTER0_DEVICE ethercat stop error log
MASTER0_DEVICE is set to "ff:ff:ff:ff:ff:ff" or eth0 mac address

dmesg:

[  144.570840] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): PHY [8000f00.mdio:01] driver [Qualcomm Atheros AR8035] (irq=POLL)
[  144.570913] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): configuring for phy/rgmii-rxid link mode
[  144.571043] EtherCAT 0: Starting EtherCAT-IDLE thread.
[  144.718813] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [Qualcomm Atheros AR8035] (irq=POLL)
[  144.718870] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
[  146.600272] EtherCAT 0: Link state of ecm0 changed to UP.
[  146.600336] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): Link is Up - 100Mbps/Full - flow control off
[  146.604646] EtherCAT WARNING 0: 1 datagram TIMED OUT!
[  146.606682] EtherCAT 0: 6 slave(s) responding on main device.
[  146.606719] EtherCAT 0: Slave states on main device: PREOP.
[  146.606821] EtherCAT 0: Scanning bus.
[  148.258366] EtherCAT 0: Bus scanning completed in 1651 ms.
[  148.258450] EtherCAT 0: Using slave 0 as DC reference clock.
[  151.126675] EtherCAT 0: Master thread exited.
[  151.127044] ------------[ cut here ]------------
[  151.127050] RTNL: assertion failed at drivers/net/phy/phylink.c (2138)
[  151.127150] WARNING: CPU: 2 PID: 837 at drivers/net/phy/phylink.c:2138 phylink_stop+0xfc/0x104 [phylink]
[  151.127200] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) cdc_ether usbnet r8152 rpmsg_ctrl rpmsg_char virtio_rpmsg_bus rpmsg_ns rpmsg_core xhci_plat_hcd dwc3 snd_soc_simple_card snd_soc_simple_card_utils crct10dif_ce panel_simple ti_k3_r5_remoteproc dwc3_am62 rtc_ti_k3 ti_k3_m4_remoteproc phy_gmii_sel tidss k3_cppi_desc_pool phylink drm_dma_helper tps6598x snd_soc_davinci_mcasp sa2ul mcrc64 snd_soc_ti_udma snd_soc_tlv320aic3x_i2c typec snd_soc_ti_edma drm_kms_helper snd_soc_tlv320aic3x snd_soc_ti_sdma dm_mod ipmi_devintf ipmi_msghandler fuse drm drm_panel_orientation_quirks ip_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[  151.127324] CPU: 2 PID: 837 Comm: rmmod Tainted: G        W  O       6.6.58-rt45 #5
[  151.127333] Hardware name: Texas Instruments AM625 SK (DT)
[  151.127338] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  151.127346] pc : phylink_stop+0xfc/0x104 [phylink]
[  151.127366] lr : phylink_stop+0xfc/0x104 [phylink]
[  151.127386] sp : ffff800082c7bb50
[  151.127389] x29: ffff800082c7bb50 x28: ffff0000086a5700 x27: 0000000000000000
[  151.127400] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[  151.127411] x23: ffff00000037b890 x22: 0000000000000480 x21: ffff00000c755500
[  151.127422] x20: ffff000003fec000 x19: ffff00000638b800 x18: ffffffffffffffff
[  151.127432] x17: 0000000000000001 x16: 00000afd6509963e x15: ffff8000813d186c
[  151.127442] x14: 0000000000000001 x13: ffff8000813d1865 x12: 6166206e6f697472
[  151.127452] x11: fffffffffffe0000 x10: 000000000000000a x9 : ffff80008129e580
[  151.127462] x8 : 00000000ffffefff x7 : ffff8000812f6580 x6 : 00000000000006f1
[  151.127473] x5 : ffff0000772ceb88 x4 : 0000000000000000 x3 : 0000000000000027
[  151.127482] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000086a5700
[  151.127494] Call trace:
[  151.127498]  phylink_stop+0xfc/0x104 [phylink]
[  151.127518]  am65_cpsw_nuss_ndo_slave_stop+0x28/0x300 [ec_ti_am65_cpsw_nuss]
[  151.127544]  ec_device_close+0x30/0x8c [ec_master]
[  151.127633]  ecdev_close+0x2c/0x6c [ec_master]
[  151.127687]  am65_cpsw_nuss_cleanup_ndev+0x6c/0xac [ec_ti_am65_cpsw_nuss]
[  151.127706]  am65_cpsw_nuss_remove+0x38/0x1c0 [ec_ti_am65_cpsw_nuss]
[  151.127723]  platform_remove+0x2c/0x6c
[  151.127741]  device_remove+0x4c/0x80
[  151.127749]  device_release_driver_internal+0x1cc/0x220
[  151.127758]  driver_detach+0x50/0x98
[  151.127766]  bus_remove_driver+0x6c/0xbc
[  151.127773]  driver_unregister+0x30/0x60
[  151.127782]  platform_driver_unregister+0x14/0x20
[  151.127790]  am65_cpsw_nuss_driver_exit+0x18/0x1620 [ec_ti_am65_cpsw_nuss]
[  151.127808]  __arm64_sys_delete_module+0x178/0x260
[  151.127822]  invoke_syscall+0x48/0x118
[  151.127833]  el0_svc_common.constprop.0+0x40/0xe8
[  151.127842]  do_el0_svc+0x20/0x2c
[  151.127851]  el0_svc+0x48/0x174
[  151.127863]  el0t_64_sync_handler+0x13c/0x158
[  151.127872]  el0t_64_sync+0x190/0x194
[  151.127882] ---[ end trace 0000000000000000 ]---
[  151.129763] EtherCAT 0: Link state of ecm0 changed to DOWN.
[  151.129845] am65-cpsw-nuss 8000000.ethernet ecm0 (uninitialized): Link is Down
[  151.129908] ------------[ cut here ]------------
[  151.129911] RTNL: assertion failed at drivers/net/phy/phylink.c (1991)
[  151.130017] WARNING: CPU: 0 PID: 837 at drivers/net/phy/phylink.c:1991 phylink_disconnect_phy+0xa8/0xb0 [phylink]
[  151.130065] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) cdc_ether usbnet r8152 rpmsg_ctrl rpmsg_char virtio_rpmsg_bus rpmsg_ns rpmsg_core xhci_plat_hcd dwc3 snd_soc_simple_card snd_soc_simple_card_utils crct10dif_ce panel_simple ti_k3_r5_remoteproc dwc3_am62 rtc_ti_k3 ti_k3_m4_remoteproc phy_gmii_sel tidss k3_cppi_desc_pool phylink drm_dma_helper tps6598x snd_soc_davinci_mcasp sa2ul mcrc64 snd_soc_ti_udma snd_soc_tlv320aic3x_i2c typec snd_soc_ti_edma drm_kms_helper snd_soc_tlv320aic3x snd_soc_ti_sdma dm_mod ipmi_devintf ipmi_msghandler fuse drm drm_panel_orientation_quirks ip_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[  151.130188] CPU: 0 PID: 837 Comm: rmmod Tainted: G        W  O       6.6.58-rt45 #5
[  151.130196] Hardware name: Texas Instruments AM625 SK (DT)
[  151.130202] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  151.130209] pc : phylink_disconnect_phy+0xa8/0xb0 [phylink]
[  151.130229] lr : phylink_disconnect_phy+0xa8/0xb0 [phylink]
[  151.130249] sp : ffff800082c7bb40
[  151.130252] x29: ffff800082c7bb40 x28: ffff0000086a5700 x27: 0000000000000000
[  151.130263] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[  151.130273] x23: ffff00000037b890 x22: 0000000000000480 x21: ffff00000c755500
[  151.130284] x20: ffff000003fec000 x19: ffff00000638b800 x18: ffffffffffffffff
[  151.130295] x17: 0000000000000001 x16: 0000000000000001 x15: ffff8000813d253c
[  151.130305] x14: 0000000000000001 x13: ffff8000813d2535 x12: 6166206e6f697472
[  151.130315] x11: fffffffffffe0000 x10: 000000000000000a x9 : ffff80008129e580
[  151.130325] x8 : 00000000ffffefff x7 : ffff8000812f6580 x6 : 0000000000000cd9
[  151.130335] x5 : ffff0000772a6b88 x4 : 0000000000000000 x3 : 0000000000000027
[  151.130345] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000086a5700
[  151.130356] Call trace:
[  151.130361]  phylink_disconnect_phy+0xa8/0xb0 [phylink]
[  151.130382]  am65_cpsw_nuss_ndo_slave_stop+0x38/0x300 [ec_ti_am65_cpsw_nuss]
[  151.130407]  ec_device_close+0x30/0x8c [ec_master]
[  151.130497]  ecdev_close+0x2c/0x6c [ec_master]
[  151.130550]  am65_cpsw_nuss_cleanup_ndev+0x6c/0xac [ec_ti_am65_cpsw_nuss]
[  151.130568]  am65_cpsw_nuss_remove+0x38/0x1c0 [ec_ti_am65_cpsw_nuss]
[  151.130586]  platform_remove+0x2c/0x6c
[  151.130603]  device_remove+0x4c/0x80
[  151.130611]  device_release_driver_internal+0x1cc/0x220
[  151.130621]  driver_detach+0x50/0x98
[  151.130630]  bus_remove_driver+0x6c/0xbc
[  151.130638]  driver_unregister+0x30/0x60
[  151.130646]  platform_driver_unregister+0x14/0x20
[  151.130655]  am65_cpsw_nuss_driver_exit+0x18/0x1620 [ec_ti_am65_cpsw_nuss]
[  151.130672]  __arm64_sys_delete_module+0x178/0x260
[  151.130687]  invoke_syscall+0x48/0x118
[  151.130699]  el0_svc_common.constprop.0+0x40/0xe8
[  151.130708]  do_el0_svc+0x20/0x2c
[  151.130717]  el0_svc+0x48/0x174
[  151.130729]  el0t_64_sync_handler+0x13c/0x158
[  151.130739]  el0t_64_sync+0x190/0x194
[  151.130748] ---[ end trace 0000000000000000 ]---
[  152.167622] am65-cpsw-nuss 8000000.ethernet: rx teardown timeout
[  152.169693] EtherCAT 0: Releasing main device AA:BB:CC:DD:EE:FF.
[  152.171501] ------------[ cut here ]------------
[  152.171514] k3_knav_desc_pool size 64000 != avail 63872
[  152.171611] WARNING: CPU: 0 PID: 837 at drivers/net/ethernet/ti/k3-cppi-desc-pool.c:33 k3_cppi_desc_pool_destroy+0xa0/0xa8 [k3_cppi_desc_pool]
[  152.171643] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) cdc_ether usbnet r8152 rpmsg_ctrl rpmsg_char virtio_rpmsg_bus rpmsg_ns rpmsg_core xhci_plat_hcd dwc3 snd_soc_simple_card snd_soc_simple_card_utils crct10dif_ce panel_simple ti_k3_r5_remoteproc dwc3_am62 rtc_ti_k3 ti_k3_m4_remoteproc phy_gmii_sel tidss k3_cppi_desc_pool phylink drm_dma_helper tps6598x snd_soc_davinci_mcasp sa2ul mcrc64 snd_soc_ti_udma snd_soc_tlv320aic3x_i2c typec snd_soc_ti_edma drm_kms_helper snd_soc_tlv320aic3x snd_soc_ti_sdma dm_mod ipmi_devintf ipmi_msghandler fuse drm drm_panel_orientation_quirks ip_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[  152.171765] CPU: 0 PID: 837 Comm: rmmod Tainted: G        W  O       6.6.58-rt45 #5
[  152.171773] Hardware name: Texas Instruments AM625 SK (DT)
[  152.171778] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  152.171786] pc : k3_cppi_desc_pool_destroy+0xa0/0xa8 [k3_cppi_desc_pool]
[  152.171799] lr : k3_cppi_desc_pool_destroy+0xa0/0xa8 [k3_cppi_desc_pool]
[  152.171812] sp : ffff800082c7bbe0
[  152.171816] x29: ffff800082c7bbe0 x28: ffff0000086a5700 x27: 0000000000000000
[  152.171827] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[  152.171837] x23: 000000000000002a x22: ffff800082c7bc78 x21: ffff00000030b810
[  152.171847] x20: 000000000000fa00 x19: ffff00000c65e400 x18: ffffffffffffffff
[  152.171857] x17: 000000017a11473c x16: 0000000000000001 x15: ffff8000813d3224
[  152.171867] x14: ffffffffffffffff x13: ffff8000813d321e x12: 6f705f637365645f
[  152.171878] x11: fffffffffffe0000 x10: 2d2d2d2d2d2d2d2d x9 : 000000000000f980
[  152.171888] x8 : ffff800082c7bbe0 x7 : ffff80008129e528 x6 : 0000000000000d83
[  152.171898] x5 : ffff0000772a6b88 x4 : 0000000000000000 x3 : 0000000000000027
[  152.171908] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000086a5700
[  152.171919] Call trace:
[  152.171924]  k3_cppi_desc_pool_destroy+0xa0/0xa8 [k3_cppi_desc_pool]
[  152.171938]  am65_cpsw_nuss_free_rx_chns+0x2c/0x54 [ec_ti_am65_cpsw_nuss]
[  152.171964]  devm_action_release+0x14/0x20
[  152.171978]  devres_release_all+0x9c/0xfc
[  152.171988]  device_unbind_cleanup+0x18/0x68
[  152.171997]  device_release_driver_internal+0x1ec/0x220
[  152.172006]  driver_detach+0x50/0x98
[  152.172014]  bus_remove_driver+0x6c/0xbc
[  152.172022]  driver_unregister+0x30/0x60
[  152.172030]  platform_driver_unregister+0x14/0x20
[  152.172039]  am65_cpsw_nuss_driver_exit+0x18/0x1620 [ec_ti_am65_cpsw_nuss]
[  152.172057]  __arm64_sys_delete_module+0x178/0x260
[  152.172070]  invoke_syscall+0x48/0x118
[  152.172082]  el0_svc_common.constprop.0+0x40/0xe8
[  152.172092]  do_el0_svc+0x20/0x2c
[  152.172100]  el0_svc+0x48/0x174
[  152.172113]  el0t_64_sync_handler+0x13c/0x158
[  152.172123]  el0t_64_sync+0x190/0x194
[  152.172132] ---[ end trace 0000000000000000 ]---
[  152.173677] ------------[ cut here ]------------
[  152.173695] kernel BUG at lib/genalloc.c:255!
[  152.173707] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT_RT SMP
[  152.173715] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) cdc_ether usbnet r8152 rpmsg_ctrl rpmsg_char virtio_rpmsg_bus rpmsg_ns rpmsg_core xhci_plat_hcd dwc3 snd_soc_simple_card snd_soc_simple_card_utils crct10dif_ce panel_simple ti_k3_r5_remoteproc dwc3_am62 rtc_ti_k3 ti_k3_m4_remoteproc phy_gmii_sel tidss k3_cppi_desc_pool phylink drm_dma_helper tps6598x snd_soc_davinci_mcasp sa2ul mcrc64 snd_soc_ti_udma snd_soc_tlv320aic3x_i2c typec snd_soc_ti_edma drm_kms_helper snd_soc_tlv320aic3x snd_soc_ti_sdma dm_mod ipmi_devintf ipmi_msghandler fuse drm drm_panel_orientation_quirks ip_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[  152.173836] CPU: 0 PID: 837 Comm: rmmod Tainted: G        W  O       6.6.58-rt45 #5
[  152.173845] Hardware name: Texas Instruments AM625 SK (DT)
[  152.173850] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  152.173858] pc : gen_pool_destroy+0x98/0xc8
[  152.173878] lr : gen_pool_destroy+0x90/0xc8
[  152.173887] sp : ffff800082c7bb90
[  152.173890] x29: ffff800082c7bb90 x28: ffff0000086a5700 x27: 0000000000000000
[  152.173901] x26: ffff000009ba89e0 x25: ffff000009ba89c0 x24: dead000000000100
[  152.173911] x23: dead000000000122 x22: 0000000000000007 x21: ffff000009ba89e0
[  152.173921] x20: 00000000000001f4 x19: ffff800081905000 x18: ffffffffffffffff
[  152.173932] x17: 000000017a11473c x16: 0000000000000001 x15: ffff8000813d3224
[  152.173942] x14: ffffffffffffffff x13: ffff8000813d321e x12: 6f705f637365645f
[  152.173952] x11: fffffffffffe0000 x10: ffff0000772b2e50 x9 : ffff0000772b2e80
[  152.173962] x8 : fffffc0001ddb788 x7 : 0000000000000010 x6 : 00000000ffffffe0
[  152.173973] x5 : 0000000041001000 x4 : 00000000000000f9 x3 : 00000000000000c0
[  152.173982] x2 : ffff800081905058 x1 : 00000000000001f4 x0 : 00000000000000f9
[  152.173993] Call trace:
[  152.173997]  gen_pool_destroy+0x98/0xc8
[  152.174007]  k3_cppi_desc_pool_destroy+0x5c/0xa8 [k3_cppi_desc_pool]
[  152.174027]  am65_cpsw_nuss_free_rx_chns+0x2c/0x54 [ec_ti_am65_cpsw_nuss]
[  152.174051]  devm_action_release+0x14/0x20
[  152.174059]  devres_release_all+0x9c/0xfc
[  152.174067]  device_unbind_cleanup+0x18/0x68
[  152.174076]  device_release_driver_internal+0x1ec/0x220
[  152.174085]  driver_detach+0x50/0x98
[  152.174092]  bus_remove_driver+0x6c/0xbc
[  152.174100]  driver_unregister+0x30/0x60
[  152.174108]  platform_driver_unregister+0x14/0x20
[  152.174117]  am65_cpsw_nuss_driver_exit+0x18/0x1620 [ec_ti_am65_cpsw_nuss]
[  152.174136]  __arm64_sys_delete_module+0x178/0x260
[  152.174148]  invoke_syscall+0x48/0x118
[  152.174159]  el0_svc_common.constprop.0+0x40/0xe8
[  152.174168]  do_el0_svc+0x20/0x2c
[  152.174176]  el0_svc+0x48/0x174
[  152.174188]  el0t_64_sync_handler+0x13c/0x158
[  152.174198]  el0t_64_sync+0x190/0x194
[  152.174212] Code: aa1403e1 97ff945e eb00029f 54fffdc9 (d4210000)
[  152.174218] ---[ end trace 0000000000000000 ]---
[  152.174224] note: rmmod[837] exited with irqs disabled
[  152.174228] note: rmmod[837] exited with preempt_count 1
[  152.224871] ------------[ cut here ]------------
[  152.224887] WARNING: CPU: 0 PID: 0 at kernel/context_tracking.c:128 ct_kernel_exit.constprop.0+0xa0/0xa8
[  152.224916] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) cdc_ether usbnet r8152 rpmsg_ctrl rpmsg_char virtio_rpmsg_bus rpmsg_ns rpmsg_core xhci_plat_hcd dwc3 snd_soc_simple_card snd_soc_simple_card_utils crct10dif_ce panel_simple ti_k3_r5_remoteproc dwc3_am62 rtc_ti_k3 ti_k3_m4_remoteproc phy_gmii_sel tidss k3_cppi_desc_pool phylink drm_dma_helper tps6598x snd_soc_davinci_mcasp sa2ul mcrc64 snd_soc_ti_udma snd_soc_tlv320aic3x_i2c typec snd_soc_ti_edma drm_kms_helper snd_soc_tlv320aic3x snd_soc_ti_sdma dm_mod ipmi_devintf ipmi_msghandler fuse drm drm_panel_orientation_quirks ip_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[  152.225050] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G      D W  O       6.6.58-rt45 #5
[  152.225059] Hardware name: Texas Instruments AM625 SK (DT)
[  152.225064] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  152.225072] pc : ct_kernel_exit.constprop.0+0xa0/0xa8
[  152.225082] lr : ct_idle_enter+0x14/0x20
[  152.225093] sp : ffff800081283d80
[  152.225096] x29: ffff800081283d80 x28: 000000008301607c x27: ffff800081291ee8
[  152.225107] x26: ffff800080f98918 x25: ffff0000772f7080 x24: 0000000000000000
[  152.225117] x23: ffff8000813c0000 x22: ffff8000812923c0 x21: ffff80008128bae4
[  152.225128] x20: 0000000000000005 x19: ffff0000772abd08 x18: 0000000000000000
[  152.225138] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[  152.225147] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[  152.225157] x11: ffff0000772d6100 x10: 0000000000000a10 x9 : ffff800081283d00
[  152.225167] x8 : ffff800081292e30 x7 : 0000000000000000 x6 : 0000000c84aadb55
[  152.225177] x5 : 4000000000000002 x4 : ffff7ffff61ce000 x3 : ffff800081283d80
[  152.225187] x2 : ffff8000810ddd08 x1 : ffff8000810ddd08 x0 : 4000000000000000
[  152.225198] Call trace:
[  152.225203]  ct_kernel_exit.constprop.0+0xa0/0xa8
[  152.225214]  ct_idle_enter+0x14/0x20
[  152.225223]  default_idle_call+0x1c/0x3c
[  152.225229]  do_idle+0xa8/0xfc
[  152.225243]  cpu_startup_entry+0x38/0x3c
[  152.225252]  kernel_init+0x0/0x1d8
[  152.225258]  arch_post_acpi_subsys_init+0x0/0x8
[  152.225271]  start_kernel+0x500/0x604
[  152.225281]  __primary_switched+0xbc/0xc4
[  152.225292] ---[ end trace 0000000000000000 ]---

sometimes appear Unable to handle kernel NULL pointer dereference

when MASTER0_DEVICE is set to "ff:ff:ff:ff:ff:ff" or eth0 mac address error log

dmesg:

[   92.655944] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[   92.655950] Mem abort info:
[   92.655953]   ESR = 0x0000000096000004
[   92.655956]   EC = 0x25: DABT (current EL), IL = 32 bits
[   92.655962]   SET = 0, FnV = 0
[   92.655965]   EA = 0, S1PTW = 0
[   92.655969]   FSC = 0x04: level 0 translation fault
[   92.655973] Data abort info:
[   92.655976]   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
[   92.655980]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[   92.655984]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[   92.655989] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000083991000
[   92.655995] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000
[   92.656009] Internal error: Oops: 0000000096000004 [#1] PREEMPT_RT SMP
[   92.656016] Modules linked in: ec_ti_am65_cpsw_nuss(O-) ec_master(O) cdc_ether usbnet r8152 rpmsg_ctrl rpmsg_char virtio_rpmsg_bus rpmsg_ns rpmsg_core xhci_plat_hcd dwc3 snd_soc_simple_card crct10dif_ce panel_simple snd_soc_simple_card_utils rtc_ti_k3 ti_k3_r5_remoteproc dwc3_am62 tidss drm_dma_helper k3_cppi_desc_pool ti_k3_m4_remoteproc phy_gmii_sel snd_soc_davinci_mcasp phylink sa2ul snd_soc_ti_udma tps6598x mcrc64 drm_kms_helper snd_soc_ti_edma snd_soc_ti_sdma snd_soc_tlv320aic3x_i2c typec snd_soc_tlv320aic3x dm_mod ipmi_devintf ipmi_msghandler drm fuse drm_panel_orientation_quirks ip_tables x_tables ipv6 [last unloaded: ti_am65_cpsw_nuss]
[   92.656146] CPU: 0 PID: 666 Comm: irq/208-8000000 Tainted: G        W  O       6.6.58-rt45 #5
[   92.656156] Hardware name: Texas Instruments AM625 SK (DT)
[   92.656160] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[   92.656169] pc : eth_type_trans+0x60/0x16c
[   92.656189] lr : am65_cpsw_nuss_rx_packets.constprop.0+0x3d4/0x8ec [ec_ti_am65_cpsw_nuss]
[   92.656223] sp : ffff800082833b00
[   92.656226] x29: ffff800082833b00 x28: ffff0000061af800 x27: ffff000009613688
[   92.656237] x26: ffff00000962f000 x25: ffff0000062c0000 x24: ffff000009613880
[   92.656248] x23: ffff00000a046080 x22: 000000000000003c x21: 000000000000017f
[   92.656258] x20: ffff000009612080 x19: 0000000000000000 x18: 0000000000000050
[   92.656268] x17: 6f44207369206b6e x16: 0000000000000001 x15: 0000272a7308411c
[   92.656278] x14: 009d025da4c5e6b4 x13: ffff800080c0be18 x12: ffff000003a9ad1c
[   92.656289] x11: ffff800081905000 x10: 0000000000000000 x9 : 0000000000000000
[   92.656299] x8 : ffff0000061af8b4 x7 : 00000000000007a8 x6 : 0000000000000000
[   92.656309] x5 : ffff0000062c0000 x4 : ffff0000062c0100 x3 : 0000000000000100
[   92.656319] x2 : ffff0000061af800 x1 : ffff00000962f000 x0 : 0000000000000000
[   92.656330] Call trace:
[   92.656335]  eth_type_trans+0x60/0x16c
[   92.656343]  am65_cpsw_nuss_rx_packets.constprop.0+0x3d4/0x8ec [ec_ti_am65_cpsw_nuss]
[   92.656361]  am65_cpsw_nuss_rx_poll+0x60/0xf4 [ec_ti_am65_cpsw_nuss]
[   92.656378]  __napi_poll.constprop.0+0x38/0x18c
[   92.656389]  net_rx_action+0x120/0x260
[   92.656396]  handle_softirqs.constprop.0+0x10c/0x1fc
[   92.656407]  __local_bh_enable_ip+0x94/0xec
[   92.656414]  irq_forced_thread_fn+0x80/0xc8
[   92.656426]  irq_thread+0x174/0x238
[   92.656433]  kthread+0x118/0x11c
[   92.656444]  ret_from_fork+0x10/0x20
[   92.656458] Code: 91003880 f9006440 f9418c20 92403c63 (f9400000)
[   92.656466] ---[ end trace 0000000000000000 ]---
[   92.656472] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[   92.966462] SMP: stopping secondary CPUs
[   92.970390] Kernel Offset: disabled
[   92.973867] CPU features: 0x0,00000008,00020000,1000420b
[   92.979167] Memory Limit: none
[   92.982218] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

-Mike

Hi Mike, 

Mike Lee said:

This is setting eth1 mac address as MASTER0_DEVICE ethercat stop error log
MASTER0_DEVICE is set to "ff:ff:ff:ff:ff:ff" or eth0 mac address

Is this error log from MASTER0_DEVICE set as eth0 mac address, eth1 mac address, or "ff:ff:ff:ff:ff:ff"?

Just to clarify, do you observe the "Unable to handle kernel NULL pointer dereference" issue when MASTER0_DEVICE is set as eth0 mac address, eth1 mac address, or "ff:ff:ff:ff:ff:ff"?

It appears that there could be a chance that the "Unable to handle kernel NULL pointer dereference" error is related to https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5f079290e591 

I've contacted the developer who worked on the ec_ti_am65_cpsw_nuss regarding the "assertion failed at drivers/net/phy/phylink.c" issue. To me it appears this issue that you see seems to be the same issue as you shared before applying the changes described in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1476965/am625-igh-ethercat-stop-error/5689134#5689134? 

-Daolin

Hi Daolin,
Unable to handle kernel NULL pointer dereference only occurs in MASTER0_DEVICE setting eth0 mac address 
and ff:ff:ff:ff:ff:ff

MASTER0_DEVICE setting eth1 mac address, which is the same as the initial error.

I will see what this changed

Daolin Qiu said:

It appears that there could be a chance that the "Unable to handle kernel NULL pointer dereference" error is related to https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5f079290e591 

-Mike

Hi Mike, 

Here is an update after I spoke internally about this issue:

So as far as the IgH Native driver (of which the FAQ in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1409114/faq-how-do-i-build-the-native-igh-ethercat-stack-for-ti-microprocessors describes how to set up) goes, this was simply meant as a starting point for those looking to explore the open source IgH EtherCAT master stack. The example is not a thoroughly tested solution and there may be problems as we see here in this thread in regards to stopping the EtherCAT master application from running. Speaking of which, is there a particular reason you are looking to explicitly stop EtherCAT master application from running?

Since the native driver is part of the IgH open-source project, any issues that have been discovered should be addressed in the open source IgH community and not directly with TI. You are welcome to debug this issue and share your fixes to the open source IgH community but at TI we cannot provide direct support on this issue. We can point you to third party software vendors who your company can contract to help solve the issue if that is something that you are interested in.

Please let me know if you have follow-up questions about what I explained.

-Daolin

Hi Daolin,

Currently, there are no plans to collaborate with other companies. Additionally, I have already fixed the two issues: "Unable to handle kernel NULL pointer dereference" and "MASTER0_DEVICE setting eth1 mac error."

To ensure proper execution of ethercat stop, it's necessary to close "ecdev" first.

static void am65_cpsw_nuss_cleanup_ndev(struct am65_cpsw_common *common)
{
	struct am65_cpsw_port *port;
	int i;

	/* Only clean two ports */
	int ecdev_idx = common->ports[0].ecdev ? 0 : 1;    // ecdev port
	int other_idx = ecdev_idx ? 0 : 1;                 // other port

	/* clean ecdev */
	port = &common->ports[ecdev_idx];
	rtnl_lock();
	ecdev_close(port->ecdev);
	rtnl_unlock();
	ecdev_withdraw(port->ecdev);
	port->ecdev = NULL;
	/* cleanup code relies on rx irq */
	common->rx_irq_disabled = false;
	enable_irq(common->rx_chns.irq);

	/* clean ndev */
	port = &common->ports[other_idx];
	if (port->ndev && port->ndev->reg_state == NETREG_REGISTERED) {
		unregister_netdev(port->ndev);
	}
	free_netdev(port->ndev);
	port->ndev = NULL;

}

Thanks

-Mike

Hi Mike,

Mike Lee said:

Additionally, I have already fixed the two issues: "Unable to handle kernel NULL pointer dereference" and "MASTER0_DEVICE setting eth1 mac error."

To ensure proper execution of ethercat stop, it's necessary to close "ecdev" first.

Thanks for sharing this finding. Was this change (close "ecdev") the only thing that was required to fix the two issues? Or did the patch in https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=5f079290e591 help fix the "Unable to handle kernel NULL pointer dereference" issue?

-Daolin

Thanks Mike, 

I appreciate the clarification.

-Daolin