• State
  • Locked Locked
  • Replies 12 replies
  • Subscribers 97 subscribers
  • Views 3190 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/AM5728: UDP traffic leads to CPSW driver failure

cc_cc
Prodigy 80 points

Part Number: AM5728

Tool/software: Linux

Hi,

I am encountering an issue where under certain conditions, a watchdog timeout is observed and later all connectivity is lost. This is using Kernel 4.4.32-rt41 on a custom board. 

It looks like this thread is about the same issue however it is not clear how the issue was solved.

Here is the kernel trace that appears after a short time with sufficient UDP traffic:

[  670.004814] ------------[ cut here ]------------
[  670.012508] WARNING: CPU: 0 PID: 4 at [removed]/kernel-source/net/sched/sch_generic.c:306 dev_watchdog+0x26c/0x278()
[  670.041482] NETDEV WATCHDOG: eth1 (cpsw): transmit queue 0 timed out
[  670.050634] Modules linked in: pm4351(O) jupiter_pcie(O) cryptodev(O) cpu_comms_fifos(O) cmemk(O)
[  670.063589] CPU: 0 PID: 4 Comm: ktimersoftd/0 Tainted: G           O    4.4.32-rt41 #1
[  670.063592] Hardware name: Generic DRA74X (Flattened Device Tree)
[  670.063596] Backtrace:
[  670.063613] [<c0013b90>] (dump_backtrace) from [<c0013d8c>] (show_stack+0x18/0x1c)
[  670.063622]  r7:c0563098 r6:200f0013 r5:00000000 r4:c099c604
[  670.063634] [<c0013d74>] (show_stack) from [<c0298a44>] (dump_stack+0x8c/0xa0)
[  670.063645] [<c02989b8>] (dump_stack) from [<c00361f4>] (warn_slowpath_common+0x88/0xb8)
[  670.063653]  r7:c0563098 r6:00000132 r5:00000009 r4:ee4a1e00
[  670.063662] [<c003616c>] (warn_slowpath_common) from [<c003625c>] (warn_slowpath_fmt+0x38/0x40)
[  670.063671]  r8:edf68e00 r7:00000001 r6:c0968580 r5:ee512000 r4:c08bc250
[  670.063680] [<c0036228>] (warn_slowpath_fmt) from [<c0563098>] (dev_watchdog+0x26c/0x278)
[  670.063685]  r3:ee512000 r2:c08bc250
[  670.063688]  r4:00000000
[  670.063696] [<c0562e2c>] (dev_watchdog) from [<c0093850>] (call_timer_fn.constprop.3+0x30/0xa0)
[  670.063706]  r10:c0562e2c r9:ee512000 r8:eed305d4 r7:c0562e2c r6:00000000 r5:00000000
[  670.063709]  r4:ffffe000
[  670.063715] [<c0093820>] (call_timer_fn.constprop.3) from [<c0093a5c>] (run_timer_softirq+0x19c/0x228)
[  670.063722]  r7:00000200 r6:00000000 r5:00000000 r4:eed30580
[  670.063730] [<c00938c0>] (run_timer_softirq) from [<c0039138>] (do_current_softirqs+0x1b8/0x254)
[  670.063739]  r10:00000001 r9:ee4a1ed0 r8:00000000 r7:04208140 r6:ee4a0000 r5:00000004
[  670.063742]  r4:c09612b0
[  670.063748] [<c0038f80>] (do_current_softirqs) from [<c0039490>] (run_ksoftirqd+0x34/0x64)
[  670.063757]  r10:00000000 r9:00000000 r8:ffffe000 r7:c0980c70 r6:00000001 r5:ee443c40
[  670.063759]  r4:ffffe000
[  670.063768] [<c003945c>] (run_ksoftirqd) from [<c0054668>] (smpboot_thread_fn+0x164/0x2b8)
[  670.063772]  r5:ee443c40 r4:ee4a0000
[  670.063780] [<c0054504>] (smpboot_thread_fn) from [<c0051124>] (kthread+0xe4/0xfc)
[  670.063790]  r10:00000000 r9:00000000 r8:00000000 r7:c0054504 r6:ee443c40 r5:ee443cc0
[  670.063793]  r4:00000000 r3:ee4915c0
[  670.063801] [<c0051040>] (kthread) from [<c0010790>] (ret_from_fork+0x14/0x24)
[  670.063808]  r7:00000000 r6:00000000 r5:c0051040 r4:ee443cc0
[  670.063810] ---[ end trace 0000000000000002 ]---

Some additional information:

Looking at /proc/softirqs, we can see that there is a lack of symmetry between the TX and RX counters. I would have expected them to be close to 1:1

~# cat /proc/softirqs
                    CPU0       CPU1
          HI:          0          2
       TIMER:     142416     142438
      NET_TX:        526          4
      NET_RX:    2380381      20020
       BLOCK:          0          0
BLOCK_IOPOLL:          0          0
     TASKLET:          2         31
       SCHED:     140379      77417
     HRTIMER:        407        194
         RCU:          0          0

  • 0
    TI__Guru**** 393215 points
    Please follow this checklist and post the results here: processors.wiki.ti.com/.../5x_CPSW
  • 0 in reply to Biser Gatchev-XID
    Prodigy 80 points

    I've attached the requested info to this message. 

    One other interesting note is that I've observed changing the socket buffer sized (making them larger) results in it taking much longer for all network traffic to fail after seeing the initial kernel trace about watchdog failure.

    The following sysctl settings help but a few hours later I find the board cannot communicate over ethernet.

    net.core.netdev_max_backlog = 65536
    net.core.optmem_max = 65536
    net.core.somaxconn = 16384
    net.core.rmem_default = 1048576
    net.core.wmem_default = 1048576
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.udp_rmem_min = 16384
    net.ipv4.udp_wmem_min = 16384

    Fullscreen ti_triage1.txt Download 
    Triage report.

Kernel version and source, also include the results of this command: uname -a

	Linux myhostname 4.4.32-rt41 #1 SMP PREEMPT RT Tue Dec 5 12:57:39 EST 2017 armv7l GNU/Linux

File system, TI SDK or Arago/Yocto based filesytem

	Yocto on ext4

Custom board or TI board? Please include device tree source file.

	Custom Board based on AM5728 EVM. See below for device tree snippet

Console log of the boot process that includes U-Boot and the Kernel.

	See Below

ethtool, ethtool -S, ifconfig shown below...

##################################
#### From bootup (dmesg)
##################################

~# dmesg | grep cpsw
[    2.125443] cpsw 48484000.ethernet: Detected MACID = 00:0e:98:02:d9:4b
[    2.132095] cpsw 48484000.ethernet: cpts: overflow check period 800
[    2.139065] cpsw 48484000.ethernet: cpsw: Detected MACID = 00:0e:98:02:d9:4c
[   11.759090] net eth1: initializing cpsw version 1.15 (0)
[   11.764438] net eth0: initialized cpsw ale version 1.4
[   11.999086] net eth0: initializing cpsw version 1.15 (0)
[   15.883294] cpsw 48484000.ethernet eth1: Link is Up - 1Gbps/Full - flow control rx/tx
[   16.113132] cpsw 48484000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   16.113132] cpsw 48484000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx

##################################
#### ethtool
##################################

root@cut:~# ethtool eth0
Settings for eth0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 1
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes
root@cut:~# ethtool eth1
Settings for eth1:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  1000baseT/Full
        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 2
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes

##################################
#### ethtool -S 
##################################
ethtool -S eth0
NIC statistics:
     Good Rx Frames: 4073
     Broadcast Rx Frames: 1722
     Multicast Rx Frames: 407
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 1137305
     Good Tx Frames: 3017
     Broadcast Tx Frames: 2
     Multicast Tx Frames: 423
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 304972
     Rx + Tx 64 Octet Frames: 1689
     Rx + Tx 65-127 Octet Frames: 3975
     Rx + Tx 128-255 Octet Frames: 725
     Rx + Tx 256-511 Octet Frames: 67
     Rx + Tx 512-1023 Octet Frames: 10
     Rx + Tx 1024-Up Octet Frames: 624
     Net Octets: 1442277
     Rx Start of Frame Overruns: 0
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 0
     Rx DMA chan: head_enqueue: 1
     Rx DMA chan: tail_enqueue: 3831
     Rx DMA chan: pad_enqueue: 0
     Rx DMA chan: misqueued: 0
     Rx DMA chan: desc_alloc_fail: 0
     Rx DMA chan: pad_alloc_fail: 0
     Rx DMA chan: runt_receive_buf: 0
     Rx DMA chan: runt_transmit_buf: 0
     Rx DMA chan: empty_dequeue: 0
     Rx DMA chan: busy_dequeue: 3384
     Rx DMA chan: good_dequeue: 3704
     Rx DMA chan: requeue: 0
     Rx DMA chan: teardown_dequeue: 0
     Tx DMA chan: head_enqueue: 2227
     Tx DMA chan: tail_enqueue: 790
     Tx DMA chan: pad_enqueue: 0
     Tx DMA chan: misqueued: 790
     Tx DMA chan: desc_alloc_fail: 0
     Tx DMA chan: pad_alloc_fail: 0
     Tx DMA chan: runt_receive_buf: 0
     Tx DMA chan: runt_transmit_buf: 40
     Tx DMA chan: empty_dequeue: 2227
     Tx DMA chan: busy_dequeue: 0
     Tx DMA chan: good_dequeue: 3017
     Tx DMA chan: requeue: 0
     Tx DMA chan: teardown_dequeue: 0
	 
~# ethtool -S eth1
NIC statistics:
     Good Rx Frames: 4099
     Broadcast Rx Frames: 1722
     Multicast Rx Frames: 409
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 1149637
     Good Tx Frames: 3049
     Broadcast Tx Frames: 2
     Multicast Tx Frames: 423
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 307888
     Rx + Tx 64 Octet Frames: 1689
     Rx + Tx 65-127 Octet Frames: 4021
     Rx + Tx 128-255 Octet Frames: 729
     Rx + Tx 256-511 Octet Frames: 67
     Rx + Tx 512-1023 Octet Frames: 10
     Rx + Tx 1024-Up Octet Frames: 632
     Net Octets: 1457525
     Rx Start of Frame Overruns: 0
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 0
     Rx DMA chan: head_enqueue: 1
     Rx DMA chan: tail_enqueue: 3855
     Rx DMA chan: pad_enqueue: 0
     Rx DMA chan: misqueued: 0
     Rx DMA chan: desc_alloc_fail: 0
     Rx DMA chan: pad_alloc_fail: 0
     Rx DMA chan: runt_receive_buf: 0
     Rx DMA chan: runt_transmit_buf: 0
     Rx DMA chan: empty_dequeue: 0
     Rx DMA chan: busy_dequeue: 3404
     Rx DMA chan: good_dequeue: 3728
     Rx DMA chan: requeue: 0
     Rx DMA chan: teardown_dequeue: 0
     Tx DMA chan: head_enqueue: 2255
     Tx DMA chan: tail_enqueue: 794
     Tx DMA chan: pad_enqueue: 0
     Tx DMA chan: misqueued: 794
     Tx DMA chan: desc_alloc_fail: 0
     Tx DMA chan: pad_alloc_fail: 0
     Tx DMA chan: runt_receive_buf: 0
     Tx DMA chan: runt_transmit_buf: 40
     Tx DMA chan: empty_dequeue: 2255
     Tx DMA chan: busy_dequeue: 0
     Tx DMA chan: good_dequeue: 3049
     Tx DMA chan: requeue: 0
     Tx DMA chan: teardown_dequeue: 0

##################################
#### ifconfig
##################################

~# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0e:98:02:d9:4b
          inet addr:192.168.0.114  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20e:98ff:fe02:d94b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1615 errors:0 dropped:0 overruns:0 frame:0
          TX packets:311 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:101887 (99.4 KiB)  TX bytes:47648 (46.5 KiB)
          Interrupt:85

eth0:0    Link encap:Ethernet  HWaddr 00:0e:98:02:d9:4b
          inet addr:192.168.0.104  Bcast:192.168.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:85

eth1      Link encap:Ethernet  HWaddr 00:0e:98:02:d9:4c
          inet addr:10.0.0.2  Bcast:10.0.0.3  Mask:255.255.255.252
          inet6 addr: fe80::20e:98ff:fe02:d94c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1790 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2474 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:867850 (847.5 KiB)  TX bytes:224748 (219.4 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:121205 errors:0 dropped:0 overruns:0 frame:0
          TX packets:121205 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:6117938 (5.8 MiB)  TX bytes:6117938 (5.8 MiB)

sitara    Link encap:Ethernet  HWaddr 00:0e:98:02:d9:4b
          inet addr:10.0.0.5  Bcast:10.0.0.7  Mask:255.255.255.252
          inet6 addr: fe80::20e:98ff:fe02:d94b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:131 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:19371 (18.9 KiB)



##################################
#### Device Tree snippet
##################################

mac: ethernet@48484000 {
	compatible = "ti,dra7-cpsw","ti,cpsw";
	ti,hwmods = "gmac";
	clocks = <&gmac_main_clk>, <&gmac_rft_clk_mux>;
	clock-names = "fck", "cpts";
	cpdma_channels = <8>;
	ale_entries = <1024>;
	bd_ram_size = <0x2000>;
	mac_control = <0x20>;
	slaves = <2>;
	active_slave = <0>;
	cpts_clock_mult = <0x784CFE14>;
	cpts_clock_shift = <29>;
	reg = <0x48484000 0x1000
		   0x48485200 0x2E00>;
	#address-cells = <1>;
	#size-cells = <1>;

	/*
	 * Do not allow gating of cpsw clock as workaround
	 * for errata i877. Keeping internal clock disabled
	 * causes the device switching characteristics
	 * to degrade over time and eventually fail to meet
	 * the data manual delay time/skew specs.
	 */
	ti,no-idle;

	/*
	 * rx_thresh_pend
	 * rx_pend
	 * tx_pend
	 * misc_pend
	 */
	interrupts = <GIC_SPI 334 IRQ_TYPE_LEVEL_HIGH>,
			 <GIC_SPI 335 IRQ_TYPE_LEVEL_HIGH>,
			 <GIC_SPI 336 IRQ_TYPE_LEVEL_HIGH>,
			 <GIC_SPI 337 IRQ_TYPE_LEVEL_HIGH>;
	ranges;
	syscon = <&scm_conf>;
	status = "disabled";
	dual_emac;

	davinci_mdio: mdio@48485000 {
		compatible = "ti,cpsw-mdio";
		#address-cells = <1>;
		#size-cells = <0>;
		ti,hwmods = "davinci_mdio";
		bus_freq = <1000000>;
		reg = <0x48485000 0x100>;
	};

	cpsw_emac0: slave@48480200 {
		/* Filled in by U-Boot */
		mac-address = [ 00 00 00 00 00 00 ];
		phy_id = <&davinci_mdio>, <1>;
		phy-mode = "rgmii";
		dual_emac_res_vlan = <1>;
	};

	cpsw_emac1: slave@48480300 {
		/* Filled in by U-Boot */
		mac-address = [ 00 00 00 00 00 00 ];
		phy_id = <&davinci_mdio>, <2>;
		phy-mode = "rgmii";
		dual_emac_res_vlan = <2>;
	};

	phy_sel: cpsw-phy-sel@4a002554 {
		compatible = "ti,dra7xx-cpsw-phy-sel";
		reg= <0x4a002554 0x4>;
		reg-names = "gmii-sel";
	};
};



##################################
#### After Kernel trace (watchdog timeout)
##################################

ethtool -S eth0
NIC statistics:
     Good Rx Frames: 424598
     Broadcast Rx Frames: 1975
     Multicast Rx Frames: 492
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 55817774
     Good Tx Frames: 426810
     Broadcast Tx Frames: 2
     Multicast Tx Frames: 437
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 55373197
     Rx + Tx 64 Octet Frames: 2067
     Rx + Tx 65-127 Octet Frames: 25687
     Rx + Tx 128-255 Octet Frames: 822831
     Rx + Tx 256-511 Octet Frames: 69
     Rx + Tx 512-1023 Octet Frames: 10
     Rx + Tx 1024-Up Octet Frames: 744
     Net Octets: 111190971
     Rx Start of Frame Overruns: 698
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 698
     Rx DMA chan: head_enqueue: 1
     Rx DMA chan: tail_enqueue: 423580
     Rx DMA chan: pad_enqueue: 0
     Rx DMA chan: misqueued: 75
     Rx DMA chan: desc_alloc_fail: 0
     Rx DMA chan: pad_alloc_fail: 0
     Rx DMA chan: runt_receive_buf: 0
     Rx DMA chan: runt_transmit_buf: 0
     Rx DMA chan: empty_dequeue: 0
     Rx DMA chan: busy_dequeue: 123529
     Rx DMA chan: good_dequeue: 423453
     Rx DMA chan: requeue: 0
     Rx DMA chan: teardown_dequeue: 0
     Tx DMA chan: head_enqueue: 107758
     Tx DMA chan: tail_enqueue: 319052
     Tx DMA chan: pad_enqueue: 0
     Tx DMA chan: misqueued: 319052
     Tx DMA chan: desc_alloc_fail: 1
     Tx DMA chan: pad_alloc_fail: 0
     Tx DMA chan: runt_receive_buf: 0
     Tx DMA chan: runt_transmit_buf: 44
     Tx DMA chan: empty_dequeue: 106977
     Tx DMA chan: busy_dequeue: 126
     Tx DMA chan: good_dequeue: 426808
     Tx DMA chan: requeue: 0
     Tx DMA chan: teardown_dequeue: 0
~# ethtool -S eth1
NIC statistics:
     Good Rx Frames: 478539
     Broadcast Rx Frames: 1978
     Multicast Rx Frames: 492
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 62862593
     Good Tx Frames: 480941
     Broadcast Tx Frames: 2
     Multicast Tx Frames: 437
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 62447360
     Rx + Tx 64 Octet Frames: 2072
     Rx + Tx 65-127 Octet Frames: 26575
     Rx + Tx 128-255 Octet Frames: 930010
     Rx + Tx 256-511 Octet Frames: 69
     Rx + Tx 512-1023 Octet Frames: 10
     Rx + Tx 1024-Up Octet Frames: 744
     Net Octets: 125309953
     Rx Start of Frame Overruns: 764
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 764
     Rx DMA chan: head_enqueue: 1
     Rx DMA chan: tail_enqueue: 477460
     Rx DMA chan: pad_enqueue: 0
     Rx DMA chan: misqueued: 76
     Rx DMA chan: desc_alloc_fail: 0
     Rx DMA chan: pad_alloc_fail: 0
     Rx DMA chan: runt_receive_buf: 0
     Rx DMA chan: runt_transmit_buf: 0
     Rx DMA chan: empty_dequeue: 0
     Rx DMA chan: busy_dequeue: 143119
     Rx DMA chan: good_dequeue: 477333
     Rx DMA chan: requeue: 0
     Rx DMA chan: teardown_dequeue: 0
     Tx DMA chan: head_enqueue: 138455
     Tx DMA chan: tail_enqueue: 342486
     Tx DMA chan: pad_enqueue: 0
     Tx DMA chan: misqueued: 342486
     Tx DMA chan: desc_alloc_fail: 1
     Tx DMA chan: pad_alloc_fail: 0
     Tx DMA chan: runt_receive_buf: 0
     Tx DMA chan: runt_transmit_buf: 45
     Tx DMA chan: empty_dequeue: 137299
     Tx DMA chan: busy_dequeue: 178
     Tx DMA chan: good_dequeue: 480941
     Tx DMA chan: requeue: 0
     Tx DMA chan: teardown_dequeue: 0

  • 0 in reply to cc_cc
    TI__Mastermind 41370 points
    Hi,

    Thanks for posting triage results.

    The source of the kernel could you please explain the source of it? Is it from the TI SDK, mainline? The version in the uname -a command does not look like something TI released.

    The watchdog timeout occurs when the interface is receiving a extremely high data rate and the kernel network stack is timing out on trying a send a packet. This message is generated by the stack and it issues a reset to the driver. The message is only printed once but the watchdog can happen again and subsequently cause a reset on the driver without console indication.

    Setting the sysctl values will help with UDP packet drops. In the results of the ethtool -S command attached are two fields showing start of frame drops and DMA overruns. These are packet drops and are caused by the internal RX FIFOs being full and the CPDMA being out of RX descriptors.

    Could you please describe the application that you are doing and the requirements you need for your application?

    When the connectivity is stopping, what is stopping? Are the RX & TX octets count on the ifconfig or ethtool -S not increasing? Are pings not working either to or from? Is there any indication on the console output saying link down?

    Best Regards,
    Schuyler
  • 0 in reply to Schuyler Patton
    Prodigy 80 points
    Thanks for the insights, Schuyler!

    The kernel is from the TI SDK (processor-sdk-linux-rt-03.02.00) with some modifications (nothing related to cpsw ethernet).

    I guess it may be useful to add some printk to see if the driver is continuously being reset.

    In the case where the internal RX FIFOs are being full and the CPDMA is running out of descriptors, it should be able to recover after losing some packets, correct?

    The situation that I'm observing results in complete and permanent traffic loss. Ping does not work and counters stop increasing. Also interesting is that when running the command [netstat -c --udp -an] (before connectivity loss occurs) you can see the send and receive queues grow and shrink. When the loss occurs, ususally there is a value in one of the queues that remains the same from that point. This seems to indicate that the queues are no longer being serviced.

    Oh and there is never a link down message.

    I wonder if the link I posted above to the other forum post provides any clues. Something about adding more DMA channels perhaps?
  • 0 in reply to cc_cc
    TI__Mastermind 41370 points

    Fullscreen 5873.0001-net-ethernet-ti-cpdma-am437x-allow-descs-to-be-plase.txt Download 
    From a4e50455d7725c40f547b1395333cd923067f2f2 Mon Sep 17 00:00:00 2001
From: Grygorii Strashko <grygorii.strashko@ti.com>
Date: Thu, 1 Sep 2016 20:37:07 +0300
Subject: [PATCH v2 1/7] net: ethernet: ti: cpdma: am437x: allow descs to be
 plased in ddr

It's observed that cpsw/cpdma is not working properly when CPPI
descriptors are placed in DDR instead of internal CPPI RAM on am437x
SoC:
- rx/tx silently stops processing packets;
- or - after boot it's working for sometime, but stuck once Network
load is increased (ping is working, but iperf is not).
(The same issue has not been reproduced on am335x and am57xx).

It seems that write to HDP register processed faster by interconnect
than writing of descriptor memory buffer in DDR, which is probably
caused by store buffer / write buffer differences as these functions
are implemented differently across devices. So, to fix this i come up
with two minimal, required changes:

1) all accesses to the channel register HDP/CP/RXFREE registers should
be done using sync IO accessors readl()/writel(), because all previous
memory writes writes have to be completed before starting channel
(write to HDP) or completing desc processing.

2) the change 1 only doesn't work on am437x and additional reading of
desc's field is required right after the new descriptor was filled
with data and before pointer on it will be stored in
prev_desc->hw_next field or HDP register.

In addition, to above changes this patch eliminates all relaxed ordering
I/O accessors in this driver as suggested by David Miller to avoid such
kind of issues in the future, but with one exception - relaxed IO accessors
will still be used to fill desc in cpdma_chan_submit(), which is safe as
there is read barrier at the end of write sequence, and because sync IO
accessors usage here will affect on net performance.

Signed-off-by: Grygorii Strashko <grygorii.strashko@ti.com>
---
 drivers/net/ethernet/ti/davinci_cpdma.c | 40 ++++++++++++++++++---------------
 1 file changed, 22 insertions(+), 18 deletions(-)

diff --git a/drivers/net/ethernet/ti/davinci_cpdma.c b/drivers/net/ethernet/ti/davinci_cpdma.c
index 36518fc..d6f0ded 100644
--- a/drivers/net/ethernet/ti/davinci_cpdma.c
+++ b/drivers/net/ethernet/ti/davinci_cpdma.c
@@ -166,12 +166,12 @@ static struct cpdma_control_info controls[] = {
 #define num_chan	params.num_chan
 
 /* various accessors */
-#define dma_reg_read(ctlr, ofs)		__raw_readl((ctlr)->dmaregs + (ofs))
-#define chan_read(chan, fld)		__raw_readl((chan)->fld)
-#define desc_read(desc, fld)		__raw_readl(&(desc)->fld)
-#define dma_reg_write(ctlr, ofs, v)	__raw_writel(v, (ctlr)->dmaregs + (ofs))
-#define chan_write(chan, fld, v)	__raw_writel(v, (chan)->fld)
-#define desc_write(desc, fld, v)	__raw_writel((u32)(v), &(desc)->fld)
+#define dma_reg_read(ctlr, ofs)		readl((ctlr)->dmaregs + (ofs))
+#define chan_read(chan, fld)		readl((chan)->fld)
+#define desc_read(desc, fld)		readl(&(desc)->fld)
+#define dma_reg_write(ctlr, ofs, v)	writel(v, (ctlr)->dmaregs + (ofs))
+#define chan_write(chan, fld, v)	writel(v, (chan)->fld)
+#define desc_write(desc, fld, v)	writel((u32)(v), &(desc)->fld)
 
 #define cpdma_desc_to_port(chan, mode, directed)			\
 	do {								\
@@ -542,10 +542,10 @@ int cpdma_ctlr_start(struct cpdma_ctlr *ctlr)
 	}
 
 	for (i = 0; i < ctlr->num_chan; i++) {
-		__raw_writel(0, ctlr->params.txhdp + 4 * i);
-		__raw_writel(0, ctlr->params.rxhdp + 4 * i);
-		__raw_writel(0, ctlr->params.txcp + 4 * i);
-		__raw_writel(0, ctlr->params.rxcp + 4 * i);
+		writel(0, ctlr->params.txhdp + 4 * i);
+		writel(0, ctlr->params.rxhdp + 4 * i);
+		writel(0, ctlr->params.txcp + 4 * i);
+		writel(0, ctlr->params.rxcp + 4 * i);
 	}
 
 	dma_reg_write(ctlr, CPDMA_RXINTMASKCLEAR, 0xffffffff);
@@ -1061,13 +1061,17 @@ int cpdma_chan_submit(struct cpdma_chan *chan, void *token, void *data,
 	mode = CPDMA_DESC_OWNER | CPDMA_DESC_SOP | CPDMA_DESC_EOP;
 	cpdma_desc_to_port(chan, mode, directed);
 
-	desc_write(desc, hw_next,   0);
-	desc_write(desc, hw_buffer, buffer);
-	desc_write(desc, hw_len,    len);
-	desc_write(desc, hw_mode,   mode | len);
-	desc_write(desc, sw_token,  token);
-	desc_write(desc, sw_buffer, buffer);
-	desc_write(desc, sw_len,    len);
+	/* Relaxed IO accessors can be used here as there is read barrier
+	 * at the end of write sequence.
+	 */
+	writel_relaxed(0, &desc->hw_next);
+	writel_relaxed(buffer, &desc->hw_buffer);
+	writel_relaxed(len, &desc->hw_len);
+	writel_relaxed(mode | len, &desc->hw_mode);
+	writel_relaxed(token, &desc->sw_token);
+	writel_relaxed(buffer, &desc->sw_buffer);
+	writel_relaxed(len, &desc->sw_len);
+	desc_read(desc, sw_len);
 
 	__cpdma_chan_submit(chan, desc);
 
@@ -1136,7 +1140,7 @@ static int __cpdma_chan_process(struct cpdma_chan *chan)
 	}
 	desc_dma = desc_phys(pool, desc);
 
-	status	= __raw_readl(&desc->hw_mode);
+	status	= desc_read(desc, hw_mode);
 	outlen	= status & 0x7ff;
 	if (status & CPDMA_DESC_OWNER) {
 		chan->stats.busy_dequeue++;
-- 
2.10.1.dirty
    Hi,

    You might hitting an issue that was encountered a few SDKs ago when handling high bandwidth network usage. Attached is a patch that changes how the CPDMA registers are accessed. The patch may have to be hand applied as the looking at the code base you started with has some of these changes already. The patch has a good description of the problem it fixed on another processor which was a follow on to the AM335x.

    Concerning the driver reset, this is something the kernel does if it hits the network watchdog. The reset is intended to be seemless, though not without impact to througput. If you are looking at wireshark you will see a dip in traffic as the interface is reset.

    Best Regards,

    Schuyler

  • 0 in reply to Schuyler Patton
    Prodigy 80 points

    We tried the patch but unfortunately it did not resolve the issue. I would claim that our application is not very high bandwidth. Maybe 300kbps when the watchdog trace appears. 

  • 0 in reply to cc_cc
    TI__Mastermind 41370 points
    That is a low data rate. Seeing the watchdog timeout at that bit rate is pointing to something else as being the issue. Since this is an RT kernel, what other apps are running?

    Also to keep you informed on possible delayed responses we are about to enter our holiday break, I will be back the first week of Jan.
  • 0 in reply to Schuyler Patton
    Prodigy 80 points

    Hi Schuyler,

    We are able to reproduce the issue with essentially just busybox shell and a test application. The test application was made just for reproducing this issue (more quickly). All it does is open a socket and call sendto(...) with small packets (200 byte payload).

    The problem appears to be with TX only. If we send lots of packets from an other machine to it, we see the RX count go up but the issue never appears.

    In our investigation it seems that at some point CPDMA channels get filled up and we can see that cpdma_chan_submit() returns an error as it fails to allocate a DMA descriptor from its pool. The cpsw driver seems to never recover on either of its MACs.

    See the kernel source here.

    What do you think?

  • 0 in reply to cc_cc
    TI__Mastermind 41370 points
    Hi,

    While the error is possible the packet should be dropped and the driver should recover. When I have seen the timeout the TX is the channel that is usually timing out.

    Would it be possible to attach the application code that you are using? The driver should recover from rx desc exhaustion condition.

    Regards,
    Schuyler
  • 0 in reply to Schuyler Patton
    Prodigy 80 points

    We also would expect the driver to recover. If left alone, it never does.

    Running the following commands will restore network for some time but it quickly fails again in the same way:

    ip link set eth0 down

    ip link set eth0 up

    I've attached the super simple code for reproducing the problem. I'm aware that ignoring the return value of a send() function is not proper but this is just to reproduce the worst case. Userspace should not be able to break networking.

    Fullscreen pktgen.c Download 
    #include <errno.h>
#include <fcntl.h> 
#include <string.h>
#include <termios.h>
#include <unistd.h>
#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/socket.h>
#include <endian.h>
#include <arpa/inet.h>

#define DATA_SIZE 200

uint8_t data[DATA_SIZE];

int main(int argc, char* argv[])
{
    if(argc < 2)
    {
        printf("Usage:\n%s host_address\n", argv[0]);
        return 1;
    }

    struct sockaddr_in address;
    int s;

    address.sin_family = AF_INET;
    address.sin_port = htons(1104);
    inet_pton(AF_INET, argv[1], &address.sin_addr);

    s = socket(AF_INET, SOCK_DGRAM | SOCK_NONBLOCK, 0);
    if(s == -1)
    {
        perror("socket()");
        return 1;
    }

    while(1)
    {
        sendto(s, data, DATA_SIZE, 0, (struct sockaddr*)&address, sizeof(address));
    }
    return 0;
}

  • 0 in reply to cc_cc
    TI__Mastermind 41370 points

    Hi,

    After reviewing the issue that you are seeing there 2 patches we would like you to please try, the first is link concerning the watchdog timeout and the other is a proposed patch.

    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=75514b6654859e0130b512396dc964d2a9e84967

    Fullscreen 2438.0001-net-ethernet-cpsw-add-check-for-rgmii-phy-interface-patch.txt Download 
    From b1a8ca49cd275c94ab78228c4e7e3d0e9c362a96 Mon Sep 17 00:00:00 2001
From: Schuyler Patton <spatton@ti.com>
Date: Thu, 19 Feb 2015 16:01:27 -0600
Subject: [PATCH] net: ethernet: cpsw: add check for rgmii phy interface for in band mode

Only RGMII ethernet PHYs have the capability of supporting in-band mode
at 10Mbps. This patch adds a check of the PHY interface for the various
RGMII interface types when a link speed of 10Mbps has been detected
before setting in-band mode.

Signed-off-by: Schuyler Patton <spatton@ti.com>
---
 drivers/net/ethernet/ti/cpsw.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index 06c75bd..e9f77df 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -920,8 +920,12 @@ static void _cpsw_adjust_link(struct cpsw_slave *slave,
 		/* set speed_in input in case RMII mode is used in 100Mbps */
 		if (phy->speed == 100)
 			mac_control |= BIT(15);
-		else if (phy->speed == 10)
-			mac_control |= BIT(18); /* In Band mode */
+		else if ((phy->speed == 10) &&
+			((phy->interface == PHY_INTERFACE_MODE_RGMII) ||
+			 (phy->interface == PHY_INTERFACE_MODE_RGMII_ID) ||
+			 (phy->interface == PHY_INTERFACE_MODE_RGMII_RXID) ||
+			 (phy->interface == PHY_INTERFACE_MODE_RGMII_TXID)))
+				mac_control |= BIT(18); /* In Band mode */
 
 		*link = true;
 	} else {
-- 
1.7.0.4

    Best Regards,

    Schuyler

  • 0 in reply to Schuyler Patton
    Prodigy 80 points
    Hi,

    The first patch is interesting. It does have the effect of avoiding total traffic loss. Now, when the issue occurs there is a temporary calm where no packets are transmitted and then when the timeout function is called, a bunch of packets go out. Then there is nothing until the next timeout (1s period) happens. This is an improvement to the situation but the root cause is still unknown. Perhaps we are getting close though.

    In my own investigation I was suspecting an issue with starting and stopping the netdev queue. It seems that somehow nobody is calling wake_queue()... My understanding is that this should be done by:

    tx_poll() -> chan_process() -> chan_free() -> tx_handler()...


    As for the second patch; it seems to have no effect. We are mostly using a link speed of 100 (or 1000).