Linux/AM5728: CPSW driver failure at high bit rates

Triage report.

Kernel version and source, also include the results of this command: uname -a

	Linux am57xx-evm 4.4.41-rt50-g968d071ce9 #16 SMP PREEMPT RT Fri Nov 17 15:22:05 CST 2017 armv7l GNU/Linux

File system, TI SDK or Arago/Yocto based filesytem

	Yocto on ext4

Custom board or TI board? Please include device tree source file.

	Custom Board based on AM5728 EVM. See below for device tree snippet

Console log of the boot process that includes U-Boot and the Kernel.

	See Below

ethtool, ethtool -S, ifconfig shown below...

##################################
#### From bootup (dmesg)
##################################


[    1.587461] davinci_mdio 48485000.mdio: davinci mdio revision 1.6
[    1.593586] davinci_mdio 48485000.mdio: detected phy mask fffffffd
[    1.605834] davinci_mdio 48485000.mdio: phy[1]: device 48485000.mdio:01, driver Micrel KSZ9031 Gigabit PHY
[    1.616225] cpsw 48484000.ethernet: Detected MACID = a0:f6:fd:c2:33:8e
[    1.622875] cpsw 48484000.ethernet: cpts: overflow check period 800
[    1.629847] cpsw 48484000.ethernet: cpsw: Detected MACID = a0:f6:fd:c2:33:8f
[   10.297896] net eth1: initializing cpsw version 1.15 (0)
[   10.303245] net eth0: initialized cpsw ale version 1.4
[   10.511436] net eth0: initializing cpsw version 1.15 (0)
[   14.616152] cpsw 48484000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off


##################################
#### ethtool
##################################
~# ethtool eth0
Settings for eth0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 1
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes


##################################
#### ethtool -S 
##################################
~# ethtool -S eth0
NIC statistics:
     Good Rx Frames: 1513915
     Broadcast Rx Frames: 175
     Multicast Rx Frames: 58
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 289202958
     Good Tx Frames: 4325332
     Broadcast Tx Frames: 13
     Multicast Tx Frames: 52
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 2183162096
     Rx + Tx 64 Octet Frames: 1375778
     Rx + Tx 65-127 Octet Frames: 7907
     Rx + Tx 128-255 Octet Frames: 7988
     Rx + Tx 256-511 Octet Frames: 18355
     Rx + Tx 512-1023 Octet Frames: 47038
     Rx + Tx 1024-Up Octet Frames: 4382181
     Net Octets: 2472365054
     Rx Start of Frame Overruns: 121
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 121
     Rx DMA chan: head_enqueue: 1
     Rx DMA chan: tail_enqueue: 1513885
     Rx DMA chan: pad_enqueue: 0
     Rx DMA chan: misqueued: 55
     Rx DMA chan: desc_alloc_fail: 0
     Rx DMA chan: pad_alloc_fail: 0
     Rx DMA chan: runt_receive_buf: 0
     Rx DMA chan: runt_transmit_buf: 0
     Rx DMA chan: empty_dequeue: 0
     Rx DMA chan: busy_dequeue: 155099
     Rx DMA chan: good_dequeue: 1513758
     Rx DMA chan: requeue: 0
     Rx DMA chan: teardown_dequeue: 0
     Tx DMA chan: head_enqueue: 57118
     Tx DMA chan: tail_enqueue: 4268290
     Tx DMA chan: pad_enqueue: 0
     Tx DMA chan: misqueued: 160259
     Tx DMA chan: desc_alloc_fail: 1
     Tx DMA chan: pad_alloc_fail: 0
     Tx DMA chan: runt_receive_buf: 0
     Tx DMA chan: runt_transmit_buf: 290
     Tx DMA chan: empty_dequeue: 54140
     Tx DMA chan: busy_dequeue: 188771
     Tx DMA chan: good_dequeue: 4325331
     Tx DMA chan: requeue: 42943
     Tx DMA chan: teardown_dequeue: 0


##################################
#### ifconfig
##################################

~# ifconfig

eth0      Link encap:Ethernet  HWaddr A0:F6:FD:C2:33:8E
          inet addr:192.168.1.29  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::a2f6:fdff:fec2:338e%132688/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1697767 errors:0 dropped:3 overruns:0 frame:0
          TX packets:4758363 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:307428096 (293.1 MiB)  TX bytes:2816467865 (2.6 GiB)
          Interrupt:93

eth1      Link encap:Ethernet  HWaddr A0:F6:FD:C2:33:8F
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1%132688/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:80274 errors:0 dropped:0 overruns:0 frame:0
          TX packets:80274 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:212221644 (202.3 MiB)  TX bytes:212221644 (202.3 MiB)



##################################
#### Device Tree snippet
##################################

        mac: ethernet@48484000 {
            compatible = "ti,dra7-cpsw","ti,cpsw";
            ti,hwmods = "gmac";
            clocks = <&gmac_main_clk>, <&gmac_rft_clk_mux>;
            clock-names = "fck", "cpts";
            cpdma_channels = <8>;
            ale_entries = <1024>;
            bd_ram_size = <0x2000>;
            mac_control = <0x20>;
            slaves = <2>;
            active_slave = <0>;
            cpts_clock_mult = <0x784CFE14>;
            cpts_clock_shift = <29>;
            reg = <0x48484000 0x1000
                   0x48485200 0x2E00>;
            #address-cells = <1>;
            #size-cells = <1>;

            /*
             * Do not allow gating of cpsw clock as workaround
             * for errata i877. Keeping internal clock disabled
             * causes the device switching characteristics
             * to degrade over time and eventually fail to meet
             * the data manual delay time/skew specs.
             */
            ti,no-idle;
            /*
             * rx_thresh_pend
             * rx_pend
             * tx_pend
             * misc_pend
             */
            interrupts = <GIC_SPI 334 IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI 335 IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI 336 IRQ_TYPE_LEVEL_HIGH>,
                     <GIC_SPI 337 IRQ_TYPE_LEVEL_HIGH>;
            ranges;
            syscon = <&scm_conf>;
            status = "disabled";

            davinci_mdio: mdio@48485000 {
                compatible = "ti,cpsw-mdio";
                #address-cells = <1>;
                #size-cells = <0>;
                ti,hwmods = "davinci_mdio";
                bus_freq = <1000000>;
                reg = <0x48485000 0x100>;
            };

	    cpsw_emac0: slave@48480200 {
                /* Filled in by U-Boot */
                mac-address = [ 00 00 00 00 00 00 ];
            };

            cpsw_emac1: slave@48480300 {
                /* Filled in by U-Boot */
                mac-address = [ 00 00 00 00 00 00 ];
            };

            phy_sel: cpsw-phy-sel@4a002554 {
                compatible = "ti,dra7xx-cpsw-phy-sel";
                reg= <0x4a002554 0x4>;
                reg-names = "gmii-sel";
            };
        };

&mac {
    status = "okay";
    dual_emac;
};

&cpsw_emac0 {
    phy_id = <&davinci_mdio>, <1>;
    phy-mode = "rgmii";
    dual_emac_res_vlan = <1>;
};

&cpsw_emac1 {
    phy_id = <&davinci_mdio>, <2>;
    phy-mode = "rgmii";
    dual_emac_res_vlan = <2>;
};


##################################
#### Error message
##################################
root@am57xx-evm:~# [  820.007443] ------------[ cut here ]------------
[  820.012103] WARNING: CPU: 0 PID: 4 at net/sched/sch_generic.c:306 dev_watchdog+0x26c/0x278()
[  820.020587] NETDEV WATCHDOG: eth0 (cpsw): transmit queue 0 timed out
[  820.026968] Modules linked in: bc_example(O) xhci_plat_hcd xhci_hcd usbcore rpmsg_pru rpmsg_proto rpmsg_rpc dwc3 udc_core cbc bluetooth drbg xfrm_user xfrm4_tunnel ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo pru_rproc pruss_intc ecb sha512_generic sha512_arm sha256_generic hmac snd_soc_simple_card md5 sha1_generic sha1_arm_neon sha1_arm snd_soc_omap_hdmi_audio pvrsrvkm(O) omap_aes_driver pruss omap_sham ahci_platform libahci_platform libahci omap_wdt libata virtio_rpmsg_bus scsi_mod ti_vpe ti_sc ti_csc ti_vpdma dwc3_omap extcon_palmas rtc_palmas extcon palmas_pwrbutton rtc_omap snd_soc_tlv320aic3x omap_des omap_rng rng_core debugss_kmodule(O) omap_remoteproc remoteproc virtio virtio_ring sch_fq_codel uio_module_drv(O) uio gdbserverproxy(O) cryptodev(O) cmemk(O)
[  820.094852] CPU: 0 PID: 4 Comm: ktimersoftd/0 Tainted: G           O    4.4.41-rt50-g968d071ce9 #1
[  820.094854] Hardware name: Generic DRA74X (Flattened Device Tree)
[  820.094858] Backtrace:
[  820.094875] [<c00130f4>] (dump_backtrace) from [<c00132f0>] (show_stack+0x18/0x1c)
[  820.094884]  r7:c0575a2c r6:20010013 r5:00000000 r4:c0946374
[  820.094894] [<c00132d8>] (show_stack) from [<c02bb7c4>] (dump_stack+0x8c/0xa0)
[  820.094904] [<c02bb738>] (dump_stack) from [<c00346e4>] (warn_slowpath_common+0x88/0xb8)
[  820.094912]  r7:c0575a2c r6:00000132 r5:00000009 r4:ee4a1e00
[  820.094920] [<c003465c>] (warn_slowpath_common) from [<c003474c>] (warn_slowpath_fmt+0x38/0x40)
[  820.094928]  r8:edf49000 r7:00000001 r6:c0918580 r5:edcc1800 r4:c08780fc
[  820.094937] [<c0034718>] (warn_slowpath_fmt) from [<c0575a2c>] (dev_watchdog+0x26c/0x278)
[  820.094941]  r3:edcc1800 r2:c08780fc
[  820.094944]  r4:00000000
[  820.094952] [<c05757c0>] (dev_watchdog) from [<c0093320>] (call_timer_fn+0x30/0xa0)
[  820.094961]  r10:c05757c0 r9:edcc1800 r8:00000200 r7:c05757c0 r6:00000000 r5:eed255c0
[  820.094964]  r4:ffffe000
[  820.094970] [<c00932f0>] (call_timer_fn) from [<c0093540>] (run_timer_softirq+0x1b0/0x23c)
[  820.094977]  r7:00000000 r6:00000000 r5:eed255c0 r4:edcc1a78
[  820.094984] [<c0093390>] (run_timer_softirq) from [<c00383ec>] (do_current_softirqs+0x1b8/0x254)
[  820.094992]  r10:00000001 r9:ee4a1ed0 r8:00000000 r7:04208140 r6:ee4a0000 r5:00000004
[  820.094995]  r4:c09112fc
[  820.095001] [<c0038234>] (do_current_softirqs) from [<c0038904>] (run_ksoftirqd+0x34/0x64)
[  820.095009]  r10:00000000 r9:00000000 r8:ffffe000 r7:c092ae2c r6:00000001 r5:ee443c80
[  820.095011]  r4:ffffe000
[  820.095020] [<c00388d0>] (run_ksoftirqd) from [<c005395c>] (smpboot_thread_fn+0x164/0x2b8)
[  820.095024]  r5:ee443c80 r4:ee4a0000
[  820.095032] [<c00537f8>] (smpboot_thread_fn) from [<c0050574>] (kthread+0xe4/0xfc)
[  820.095040]  r10:00000000 r9:00000000 r8:00000000 r7:c00537f8 r6:ee443c80 r5:ee443d00
[  820.095044]  r4:00000000 r3:ee4915c0
[  820.095051] [<c0050490>] (kthread) from [<c000fad0>] (ret_from_fork+0x14/0x24)
[  820.095057]  r7:00000000 r6:00000000 r5:c0050490 r4:ee443d00
[  820.095060] ---[ end trace 0000000000000002 ]---



##################################
#### After Kernel trace (watchdog timeout)
##################################

##################################
#### ping ��Re-plug��
##################################
root@am57xx-evm:~# ping 192.168.1.201
PING 192.168.1.201 (192.168.1.201): 56 data bytes
^C
--- 192.168.1.201 ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss
root@am57xx-evm:~# [ 1400.665053] cpsw 48484000.ethernet eth0: Link is Down
[ 1403.676790] cpsw 48484000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

root@am57xx-evm:~# ping 192.168.1.201
PING 192.168.1.201 (192.168.1.201): 56 data bytes
64 bytes from 192.168.1.201: seq=0 ttl=64 time=0.603 ms
64 bytes from 192.168.1.201: seq=1 ttl=64 time=0.415 ms
64 bytes from 192.168.1.201: seq=2 ttl=64 time=0.433 ms
^C
--- 192.168.1.201 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.415/0.483/0.603 ms
root@am57xx-evm:~#


##################################
#### ethtool
##################################
~# ethtool eth0

Settings for eth0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 1
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000000 (0)

        Link detected: yes


##################################
#### ethtool -S 
##################################
~# ethtool -S eth0

NIC statistics:
     Good Rx Frames: 7177889
     Broadcast Rx Frames: 521
     Multicast Rx Frames: 76
     Pause Rx Frames: 0
     Rx CRC Errors: 0
     Rx Align/Code Errors: 0
     Oversize Rx Frames: 0
     Rx Jabbers: 0
     Undersize (Short) Rx Frames: 0
     Rx Fragments: 0
     Rx Octets: 1065301161
     Good Tx Frames: 16715815
     Broadcast Tx Frames: 19
     Multicast Tx Frames: 55
     Pause Tx Frames: 0
     Deferred Tx Frames: 0
     Collisions: 0
     Single Collision Tx Frames: 0
     Multiple Collision Tx Frames: 0
     Excessive Collisions: 0
     Late Collisions: 0
     Tx Underrun: 0
     Carrier Sense Errors: 0
     Tx Octets: 3696377664
     Rx + Tx 64 Octet Frames: 6743094
     Rx + Tx 65-127 Octet Frames: 18467
     Rx + Tx 128-255 Octet Frames: 19153
     Rx + Tx 256-511 Octet Frames: 44456
     Rx + Tx 512-1023 Octet Frames: 110568
     Rx + Tx 1024-Up Octet Frames: 16957966
     Net Octets: 466711529
     Rx Start of Frame Overruns: 196
     Rx Middle of Frame Overruns: 0
     Rx DMA Overruns: 196
     Rx DMA chan: head_enqueue: 1
     Rx DMA chan: tail_enqueue: 7177766
     Rx DMA chan: pad_enqueue: 0
     Rx DMA chan: misqueued: 56
     Rx DMA chan: desc_alloc_fail: 0
     Rx DMA chan: pad_alloc_fail: 0
     Rx DMA chan: runt_receive_buf: 0
     Rx DMA chan: runt_transmit_buf: 0
     Rx DMA chan: empty_dequeue: 0
     Rx DMA chan: busy_dequeue: 339660
     Rx DMA chan: good_dequeue: 7177639
     Rx DMA chan: requeue: 0
     Rx DMA chan: teardown_dequeue: 0
     Tx DMA chan: head_enqueue: 102634
     Tx DMA chan: tail_enqueue: 16613181
     Tx DMA chan: pad_enqueue: 0
     Tx DMA chan: misqueued: 361974
     Tx DMA chan: desc_alloc_fail: 3
     Tx DMA chan: pad_alloc_fail: 0
     Tx DMA chan: runt_receive_buf: 0
     Tx DMA chan: runt_transmit_buf: 815
     Tx DMA chan: empty_dequeue: 97176
     Tx DMA chan: busy_dequeue: 783242
     Tx DMA chan: good_dequeue: 16715815
     Tx DMA chan: requeue: 122281
     Tx DMA chan: teardown_dequeue: 0

root@am57xx-evm:~# iperf -c 192.168.1.22 -t 60 -w 8KB
------------------------------------------------------------
Client connecting to 192.168.1.22, TCP port 5001
TCP window size: 16.0 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
[  3] local 192.168.1.29 port 40316 connected with 192.168.1.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  1.31 GBytes   187 Mbits/sec


root@am57xx-evm:~# iperf -c 192.168.1.22 -t 60 -w 64KB
------------------------------------------------------------
Client connecting to 192.168.1.22, TCP port 5001
TCP window size:  128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[  3] local 192.168.1.29 port 40318 connected with 192.168.1.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  3.29 GBytes   471 Mbits/sec


root@am57xx-evm:~# iperf -c 192.168.1.22 -t 60 -w 128KB
------------------------------------------------------------
Client connecting to 192.168.1.22, TCP port 5001
TCP window size:  256 KByte (WARNING: requested  128 KByte)
------------------------------------------------------------
[  3] local 192.168.1.29 port 40320 connected with 192.168.1.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  3.45 GBytes   494 Mbits/sec


root@am57xx-evm:~# iperf -c 192.168.1.22 -t 60 -w 256KB
------------------------------------------------------------
Client connecting to 192.168.1.22, TCP port 5001
TCP window size:  320 KByte (WARNING: requested  256 KByte)
------------------------------------------------------------
[  3] local 192.168.1.29 port 40322 connected with 192.168.1.22 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  3.89 GBytes   556 Mbits/sec