TDA4VM: kernel crashed with log NETDEV WATCHDOG: eth1 (j721e-cpsw-virt-mac): transmit queue 0 timed out

other wang

Part Number: TDA4VM

There are kernel crash issue with the follow log, the full log is attached to this case. could you please give us some suggestions about it

I found some similar error issue from tie2e, maybe it is useful.

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/979085/am4376-netdev-watchdog-does-not-reset-step-2?tisearch=e2e-sitesearch&keymatch=NETDEV%20WATCHDOG%3A%20eth1%20%28j721e-cpsw-virt-mac%29#

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/192365/am3874-netdev-watchdog-eth0-cpsw-transmit-queue-0-timed-out

Jun 21 15:11:21 sixi-slave kernel: [ 57.139014] am65-cpsw-nuss 46000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
Jun 21 15:11:21 sixi-slave kernel: [ 57.139039] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jun 21 15:11:30 sixi-slave kernel: [ 66.098083] ------------[ cut here ]------------
Jun 21 15:11:30 sixi-slave kernel: [ 66.098091] NETDEV WATCHDOG: eth1 (j721e-cpsw-virt-mac): transmit queue 0 timed out
Jun 21 15:11:30 sixi-slave kernel: [ 66.098136] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x2ec/0x2f8
Jun 21 15:11:30 sixi-slave kernel: [ 66.098137] Modules linked in: usb_storage xhci_plat_hcd xhci_hcd rpmsg_client_sample ti_am335x_adc usbcore omap_rng rpmsg_char kfifo_buf rng_core cdns3 virtio_rpmsg_bus roles udc_core pru_rproc usb_common irq_pruss_intc crct10dif_ce ti_j721e_cpsw_virt_mac icss_iep ti_k3_r5_remoteproc sa2ul ti_am335x_tscadc pruss sha512_generic authenc ti_k3_dsp_remoteproc cdns3_ti rti_wdt sch_fq_codel rpmsg_kdrv_switch ip_tables x_tables ipv6
Jun 21 15:11:30 sixi-slave kernel: [ 66.098172] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.74-ACU-2.1.1.10 #1
Jun 21 15:11:30 sixi-slave kernel: [ 66.098173] Hardware name: Texas Instruments K3 J721E SoC (DT)
Jun 21 15:11:30 sixi-slave kernel: [ 66.098176] pstate: 40000005 (nZcv daif -PAN -UAO)
Jun 21 15:11:30 sixi-slave kernel: [ 66.098179] pc : dev_watchdog+0x2ec/0x2f8
Jun 21 15:11:30 sixi-slave kernel: [ 66.098182] lr : dev_watchdog+0x2ec/0x2f8
Jun 21 15:11:30 sixi-slave kernel: [ 66.098183] sp : ffff80001002fdb0
Jun 21 15:11:30 sixi-slave kernel: [ 66.098185] x29: ffff80001002fdb0 x28: ffff000848713c80
Jun 21 15:11:30 sixi-slave kernel: [ 66.098187] x27: 0000000000000004 x26: 0000000000000140
Jun 21 15:11:30 sixi-slave kernel: [ 66.098190] x25: 00000000ffffffff x24: 0000000000000001
Jun 21 15:11:30 sixi-slave kernel: [ 66.098192] x23: ffff00085252c41c x22: ffff00085252c000
Jun 21 15:11:30 sixi-slave kernel: [ 66.098195] x21: ffff00085252c440 x20: ffff800010ff1000
Jun 21 15:11:30 sixi-slave kernel: [ 66.098197] x19: 0000000000000000 x18: 0000000000000000
Jun 21 15:11:30 sixi-slave kernel: [ 66.098199] x17: 0000000000000000 x16: 0000000000000000
Jun 21 15:11:30 sixi-slave kernel: [ 66.098201] x15: ffff80001100a000 x14: ffff8000110c74e2
Jun 21 15:11:30 sixi-slave kernel: [ 66.098203] x13: 00000000000073a8 x12: ffff8000110c6000
Jun 21 15:11:30 sixi-slave kernel: [ 66.098205] x11: ffff80001100a000 x10: ffff8000110c6b30
Jun 21 15:11:30 sixi-slave kernel: [ 66.098207] x9 : 0000000000000001 x8 : 0000000000000000
Jun 21 15:11:30 sixi-slave kernel: [ 66.098209] x7 : 0000000000000004 x6 : 00000000000004f6
Jun 21 15:11:30 sixi-slave kernel: [ 66.098211] x5 : ffff00087fa81848 x4 : 0000000000000001
Jun 21 15:11:30 sixi-slave kernel: [ 66.098213] x3 : ffff00087fa81848 x2 : 0000000000000007
Jun 21 15:11:30 sixi-slave kernel: [ 66.098215] x1 : 88b06faecedb5600 x0 : 0000000000000000
Jun 21 15:11:30 sixi-slave kernel: [ 66.098218] Call trace:
Jun 21 15:11:30 sixi-slave kernel: [ 66.098222] dev_watchdog+0x2ec/0x2f8
Jun 21 15:11:30 sixi-slave kernel: [ 66.098227] call_timer_fn.isra.0+0x20/0x78
Jun 21 15:11:30 sixi-slave kernel: [ 66.098229] run_timer_softirq+0x1a0/0x408
Jun 21 15:11:30 sixi-slave kernel: [ 66.098232] __do_softirq+0x120/0x25c
Jun 21 15:11:30 sixi-slave kernel: [ 66.098235] irq_exit+0xb8/0xd8
Jun 21 15:11:30 sixi-slave kernel: [ 66.098239] __handle_domain_irq+0x64/0xb8
Jun 21 15:11:30 sixi-slave kernel: [ 66.098241] gic_handle_irq+0x5c/0x148
Jun 21 15:11:30 sixi-slave kernel: [ 66.098243] el1_irq+0xb8/0x180
Jun 21 15:11:30 sixi-slave kernel: [ 66.098246] arch_cpu_idle+0x10/0x18
Jun 21 15:11:30 sixi-slave kernel: [ 66.098250] do_idle+0xc0/0x140
Jun 21 15:11:30 sixi-slave kernel: [ 66.098253] cpu_startup_entry+0x24/0x68
Jun 21 15:11:30 sixi-slave kernel: [ 66.098256] secondary_start_kernel+0x148/0x180
Jun 21 15:11:30 sixi-slave kernel: [ 66.098258] ---[ end trace e23048f2ad374935 ]---
Jun 21 15:11:30 sixi-slave kernel: [ 66.098267] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth1: txq:0 DRV_XOFF:0 tmo:8960 dql_avail:-350 free_desc:509
Jun 21 15:11:34 sixi-slave kernel: [ 70.962094] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth1: txq:0 DRV_XOFF:0 tmo:13824 dql_avail:-350 free_desc:509
Jun 21 15:11:40 sixi-slave kernel: [ 76.082092] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth1: txq:0 DRV_XOFF:0 tmo:18944 dql_avail:-350 free_desc:509
Jun 21 15:11:44 sixi-slave kernel: [ 80.946092] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth1: txq:0 DRV_XOFF:0 tmo:23808 dql_avail:-350 free_desc:509
Jun 21 15:11:50 sixi-slave kernel: [ 86.066092] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth1: txq:0 DRV_XOFF:0 tmo:28928 dql_avail:-350 free_desc:509
Jun 21 15:11:54 sixi-slave kernel: [ 90.930092] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth1: txq:0 DRV_XOFF:0 tmo:33792 dql_avail:-350 free_desc:509
Jun 21 15:12:00 sixi-slave kernel: [ 96.050094] j721e-cpsw-virt-mac main_r5fss_cpsw9g_virt_mac0 eth1: txq:0 DRV_XOFF:0 tmo:38912 dql_avail:-350 free_desc:509

over 4 years ago

0 Vineet Roy over 4 years ago

TI__Mastermind 37435 points

Hi,

Can you tell me under what condition you run into this issue ? I understand that it's caused by Tx queue transmit timing out but would like to understand why it's happening. It shouldn't occur in the first place.

Regards

Vineet

0 other wang over 4 years ago in reply to Vineet Roy

Intellectual 420 points

Hi,

It is very hard to reproduce this problem.

1. Currently it can be reproduce only one platform on one car.

2. we meet the problem a few times in two weeks.

3. Basically happens when the system starts

We are continuing to pay attention to this issue , is there any advice about it?

Regards

WangHongjie

0 Vineet Roy over 4 years ago in reply to other wang

TI__Mastermind 37435 points

Hi,

I am sorry I forgot to ask

1. Which SDK version is it ?

2. Have you made any changes to the driver ? If yes, then please post your patches.

3. Have you tried applying the patch you referred to ? Does it fix the problem ?

Regards

Vineet

0 other wang over 4 years ago in reply to Vineet Roy

Intellectual 420 points

Hi,

We are working on sdk7.2 .

Could you tell me the file location of the driver which you talk about

I didn't test the patch since it is different chip.

regards

Wanghongjie

0 Vineet Roy over 4 years ago in reply to other wang

TI__Mastermind 37435 points

Hi,

I looked at the CPSW 9G virtual driver and it looks like this kind of patch is already part of the driver. See function virt_cpsw_nuss_ndo_host_tx_timeout() in drivers/net/ethernet/ti/j721e-cpsw-virt-mac.c

I will consult internally and get back to you with next steps.

Can you tell me if this happens under any particular circumstance ? Any specific traffic ? What kind of traffic/bitrate is A72 trying to send when this occurs.

Regards

Vineet

0 Vineet Roy over 4 years ago in reply to Vineet Roy

TI__Mastermind 37435 points

Hi,

other wang said:
There are kernel crash issue with the follow log, the full log is attached to this case. could you please give us some suggestions about it

Can you provide the complete log when crash occurs ?

Also provide the boot log

Regards

Vineet

0 Vineet Roy over 4 years ago in reply to Vineet Roy

TI__Mastermind 37435 points

Also, please tell if you are able to recover from the issue by doing "ip link set ethX down" / "ip link set ethX up"?

Regards

Vineet

0 other wang over 4 years ago in reply to Vineet Roy

Intellectual 420 points

Hi,

The if up and down are used for the tow cases:

1. we will reset the mac addr during system boot.

2. there is a work around for the network unstable issue.

Both of the two network interface (cpsw2g & cpsw9g ) are unstable , it will disconnecte a few times and it Will resume after ifdown up, it is very hard to reproduce this problem . so there is a service to check the network status, and do "ifconfg down&up" when disconnected.

Regards.

0 Vineet Roy over 4 years ago in reply to other wang

TI__Mastermind 37435 points

Hi,

I didn't understand. Do you have to reset the system when kernel crashes due to this issue ? or can you recover ?

other wang said:
we will reset the mac addr during system boot.

If you don't reset the mac address does this issue disappear ?

other wang said:
there is a work around for the network unstable issue.

I am not aware of any network unstable issues so I don't know of any work around.

Please reach out via your TI representative and we can setup a call to discuss this

Regards

Vineet

0 other wang over 4 years ago in reply to Vineet Roy

Intellectual 420 points

We are trying to reproduce this problem on our platform and hope to find steps.

I will update this case when reproduced.

0 other wang over 4 years ago in reply to other wang

Intellectual 420 points

Hi,

we don't reproduce this issue , If we meet same problem, we will reopen this issue.

Thanks. please close it.

0 user4836286 over 4 years ago in reply to other wang

Expert 1300 points

Hi TI team,

I also experienced same problem.

PSDK 7.1 and foreground application is using socket CAN. No explicit ethernet operation.

Best regards,

Wilson.

0 Vineet Roy over 4 years ago in reply to user4836286

TI__Mastermind 37435 points

Hi,

Can you open another E2E ticket for your issue ? The behavior might be the same but your application scenario is different

Regards

Vineet

Processors

Processors forum

TDA4VM: kernel crashed with log NETDEV WATCHDOG: eth1 (j721e-cpsw-virt-mac): transmit queue 0 timed out