Tool/software:
PROCESSOR-SDK-RTOS-J721E/07.02.00.06
PROCESSOR-SDK-LINUX-J721E/07.02.00.07
kernel crash with:"NETDEV WATCHDOG: eth0 (j721e-cpsw-virt-mac): transmit queue 0 timed out "
The vehicle that has already entered mass production has now been found to have this issue in only one car.
Aug 15 13:53:40 buildroot kernel: [ 3050.319721] el0_svc_handler+0x20/0x80
Aug 15 13:53:40 buildroot kernel: [ 3050.319723] el0_svc+0x8/0x204
Aug 20 13:54:08 buildroot kernel: [ 547.811888] ------------[ cut here ]------------
Aug 20 13:54:08 buildroot kernel: [ 547.811893] NETDEV WATCHDOG: eth0 (j721e-cpsw-virt-mac): transmit queue 0 timed out
Aug 20 13:54:08 buildroot kernel: [ 547.811925] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x2ec/0x2f8
Aug 20 13:54:08 buildroot kernel: [ 547.811926] Modules linked in: 8021q garp stp mrp llc xhci_plat_hcd xhci_hcd usbcore cdns3 roles udc_core usb_common rpmsg_char virtio_rpmsg_bus crct10dif_ce ti_j721e_cpsw_virt_mac acu_timesync ti_k3_r5_remoteproc pvrsrvkm(O) sa2ul sha512_generic authenc cdns3_ti ti_k3_dsp_remoteproc sch_fq_codel rpmsg_kdrv_switch ip_tables x_tables ipv6
Aug 20 13:54:08 buildroot kernel: [ 547.811950] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 5.4.74-ACU-10.0.65.1 #4
Aug 20 13:54:08 buildroot kernel: [ 547.811952] Hardware name: Texas Instruments K3 J721E SoC (DT)
Aug 20 13:54:08 buildroot kernel: [ 547.811954] pstate: 40000005 (nZcv daif -PAN -UAO)
Aug 20 13:54:08 buildroot kernel: [ 547.811957] pc : dev_watchdog+0x2ec/0x2f8
Aug 20 13:54:08 buildroot kernel: [ 547.811959] lr : dev_watchdog+0x2ec/0x2f8
Aug 20 13:54:08 buildroot kernel: [ 547.811960] sp : ffff80001002fdb0
Aug 20 13:54:08 buildroot kernel: [ 547.811962] x29: ffff80001002fdb0 x28: ffff00084466a280
Aug 20 13:54:08 buildroot kernel: [ 547.811964] x27: 0000000000000004 x26: 0000000000000140
Aug 20 13:54:08 buildroot kernel: [ 547.811967] x25: 00000000ffffffff x24: 0000000000000001
Aug 20 13:54:08 buildroot kernel: [ 547.811969] x23: ffff00085034041c x22: ffff000850340000
Aug 20 13:54:08 buildroot kernel: [ 547.811971] x21: ffff000850340440 x20: ffff800011111000
Aug 20 13:54:08 buildroot kernel: [ 547.811973] x19: 0000000000000000 x18: 0000000000000000
Aug 20 13:54:08 buildroot kernel: [ 547.811975] x17: 0000000000000000 x16: 0000000000000000
Aug 20 13:54:08 buildroot kernel: [ 547.811977] x15: ffff80001112a000 x14: ffff8000111e74e2
Aug 20 13:54:08 buildroot kernel: [ 547.811979] x13: 0000000000000000 x12: ffff8000111e6000
Aug 20 13:54:08 buildroot kernel: [ 547.811981] x11: ffff80001112a000 x10: ffff8000111e6b30
Aug 20 13:54:08 buildroot kernel: [ 547.811983] x9 : 0000000000000000 x8 : 0000000000000000
Aug 20 13:54:08 buildroot kernel: [ 547.811985] x7 : 0000000000000000 x6 : 00000000000002e1
Aug 20 13:54:08 buildroot kernel: [ 547.811987] x5 : ffff00087fa81848 x4 : 0000000000000001
Aug 20 13:54:08 buildroot kernel: [ 547.811989] x3 : ffff00087fa81848 x2 : 0000000000000007
Aug 20 13:54:08 buildroot kernel: [ 547.811991] x1 : 69d5dd1444ec3600 x0 : 0000000000000000
Aug 20 13:54:08 buildroot kernel: [ 547.811993] Call trace:
Aug 20 13:54:08 buildroot kernel: [ 547.811996] dev_watchdog+0x2ec/0x2f8
Aug 20 13:54:08 buildroot kernel: [ 547.812000] call_timer_fn.isra.0+0x20/0x78
Aug 20 13:54:08 buildroot kernel: [ 547.812002] run_timer_softirq+0x1a0/0x408
Aug 20 13:54:08 buildroot kernel: [ 547.812005] __do_softirq+0x120/0x25c
Aug 20 13:54:08 buildroot kernel: [ 547.812007] irq_exit+0xb8/0xd8
Aug 20 13:54:08 buildroot kernel: [ 547.812011] __handle_domain_irq+0x64/0xb8
Aug 20 13:54:08 buildroot kernel: [ 547.812012] gic_handle_irq+0x5c/0x148
Aug 20 13:54:08 buildroot kernel: [ 547.812014] el1_irq+0xb8/0x180
Aug 20 13:54:08 buildroot kernel: [ 547.812017] arch_cpu_idle+0x10/0x18
Aug 20 13:54:08 buildroot kernel: [ 547.812021] do_idle+0xc0/0x140
Aug 20 13:54:08 buildroot kernel: [ 547.812023] cpu_startup_entry+0x20/0x68
Aug 20 13:54:08 buildroot kernel: [ 547.812026] secondary_start_kernel+0x148/0x180
Aug 20 13:54:08 buildroot kernel: [ 547.812028] ---[ end trace 0bebd40e4029afd4 ]---
Aug 20 13:54:08 buildroot kernel: [ 547.812039] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth0: txq:0 DRV_XOFF:0 tmo:7796 dql_avail:-62 free_desc:453
Aug 20 13:54:13 buildroot kernel: [ 552.931899] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth0: txq:0 DRV_XOFF:0 tmo:12916 dql_avail:-62 free_desc:453
Aug 20 13:54:18 buildroot kernel: [ 558.051895] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth0: txq:0 DRV_XOFF:0 tmo:18036 dql_avail:-62 free_desc:453
Aug 20 13:54:21 buildroot kernel: [ 560.615884] rcu: INFO: rcu_preempt self-detected stall on CPU
Aug 20 13:54:21 buildroot kernel: [ 560.615891] rcu: 0-....: (1 GPs behind) idle=3ee/0/0x3 softirq=126043/126044 fqs=2624
Aug 20 13:54:21 buildroot kernel: [ 560.615893] (t=5250 jiffies g=232845 q=6256)
Aug 20 13:54:21 buildroot kernel: [ 560.615895] Task dump for CPU 0:
Aug 20 13:54:21 buildroot kernel: [ 560.615897] swapper/0 R running task 0 0 0 0x0000002a
Aug 20 13:54:21 buildroot kernel: [ 560.615901] Call trace:
Aug 20 13:54:21 buildroot kernel: [ 560.615908] dump_backtrace+0x0/0x140
Aug 20 13:54:21 buildroot kernel: [ 560.615910] show_stack+0x14/0x20
Aug 20 13:54:21 buildroot kernel: [ 560.615914] sched_show_task+0x108/0x138
Aug 20 13:54:21 buildroot kernel: [ 560.615917] dump_cpu_task+0x40/0x50
Aug 20 13:54:21 buildroot kernel: [ 560.615920] rcu_dump_cpu_stacks+0x94/0xd0
Aug 20 13:54:21 buildroot kernel: [ 560.615923] rcu_sched_clock_irq+0x75c/0x9c8
Aug 20 13:54:21 buildroot kernel: [ 560.615925] update_process_times+0x2c/0xb8
Aug 20 13:54:21 buildroot kernel: [ 560.615929] tick_sched_handle.isra.0+0x30/0x48
Aug 20 13:54:21 buildroot kernel: [ 560.615931] tick_sched_timer+0x48/0x98
Aug 20 13:54:21 buildroot kernel: [ 560.615933] __hrtimer_run_queues+0x110/0x1b0
Aug 20 13:54:21 buildroot kernel: [ 560.615935] hrtimer_interrupt+0xe4/0x240
Aug 20 13:54:21 buildroot kernel: [ 560.615940] arch_timer_handler_phys+0x30/0x40
Aug 20 13:54:21 buildroot kernel: [ 560.615942] handle_percpu_devid_irq+0x80/0x140
Aug 20 13:54:21 buildroot kernel: [ 560.615945] generic_handle_irq+0x24/0x38
Aug 20 13:54:21 buildroot kernel: [ 560.615947] __handle_domain_irq+0x60/0xb8
Aug 20 13:54:21 buildroot kernel: [ 560.615949] gic_handle_irq+0x5c/0x148
Aug 20 13:54:21 buildroot kernel: [ 560.615951] el1_irq+0xb8/0x180
Aug 20 13:54:21 buildroot kernel: [ 560.615953] __do_softirq+0xa4/0x25c
Aug 20 13:54:21 buildroot kernel: [ 560.615955] irq_exit+0xb8/0xd8
Aug 20 13:54:21 buildroot kernel: [ 560.615957] __handle_domain_irq+0x64/0xb8
Aug 20 13:54:21 buildroot kernel: [ 560.615959] gic_handle_irq+0x5c/0x148
Aug 20 13:54:21 buildroot kernel: [ 560.615961] el1_irq+0xb8/0x180
Aug 20 13:54:21 buildroot kernel: [ 560.615963] arch_cpu_idle+0x10/0x18
Aug 20 13:54:21 buildroot kernel: [ 560.615966] do_idle+0xc0/0x140
Aug 20 13:54:21 buildroot kernel: [ 560.615968] cpu_startup_entry+0x24/0x68
Aug 20 13:54:21 buildroot kernel: [ 560.615972] rest_init+0xd0/0xdc
Aug 20 13:54:21 buildroot kernel: [ 560.615976] arch_call_rest_init+0xc/0x14
Aug 20 13:54:21 buildroot kernel: [ 560.615978] start_kernel+0x408/0x43c
Aug 20 13:54:23 buildroot kernel: [ 562.915899] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth0: txq:0 DRV_XOFF:0 tmo:22900 dql_avail:-62 free_desc:453
Aug 20 13:54:28 buildroot kernel: [ 568.035897] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth0: txq:0 DRV_XOFF:0 tmo:28020 dql_avail:-62 free_desc:453
Is this a known issue with a new kernel patch available?