This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM3352: Ethernet slows down

Part Number: AM3352
Other Parts Discussed in Thread: TLK105L

Hello,
my customer uses the AM3352. The first prototyps running ok last year. Now he build up 3 new prototyps with new AM3352 samples. All boards had sporadic problems with the Ethernet baudrate. Sometime the TX baudrate run at ~700Kbit (instead of 100Mbit) measured at Port1 RMII. After reconnection the network cable a couple of times it is ok again. RX signal with 100Mbit is ok. RX/TX 25MHz oscillator is ok.What could be the reason?

Regards, Holger

  • Hi Holger,
    I assume they need to do more debug. E.g. look at Ethernet statistics and error counters. Are there many re-transmits of packets on the line? How do the MII signals look like? Any RX errors indicated?
    If the issue goes away it may be marginal timing (layout, temperature effects) or other connectivity issues. So far no one can tell...
    3 boards is not a big sample size. There may even be soldering issues if a different process was applied than for the first board.
    Regards,
  • Hello Frank,the error counter is not countig up:

    ifconfig eth0

    eth0      Link encap:Ethernet  HWaddr 54:4A:16:A0:FF:45
           inet addr:192.168.1.66  Bcast:192.168.1.255  Mask:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:27793 errors:0 dropped:0 overruns:0 frame:0
          TX packets:55947 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:2191402 (2.0 MiB)  TX bytes:83949913 (80.0 MiB)
           Interrupt:170

    it looks like no pakets were re-transmitted

    ss -ti
    State       Recv-Q Send-Q                                                                       Local Address:Port                                                                                        Peer Address:Port
    ESTAB       0      0                                                                                127.0.0.1:44355                                                                                          127.0.0.1:4966
             cubic wscale:4,4 rto:210 rtt:0.368/0.202 mss:21848 cwnd:10 send 4749.6Mbps lastsnd:1099760 lastrcv:809360 lastack:1099760 pacing_rate 9499.1Mbps rcv_space:43690
    ESTAB       0      0                                                                                127.0.0.1:4966                                                                                           127.0.0.1:44355
             cubic wscale:4,4 rto:200 ato:40 mss:21848 cwnd:10 lastsnd:1099760 lastrcv:1099760 lastack:1099760 rcv_space:43690
    ESTAB       0      0                                                                      ::ffff:192.168.1.66:ssh                                                                                  ::ffff:192.168.1.77:50240
             cubic rto:440 rtt:48.446/74.593 ato:40 mss:1460 cwnd:3 ssthresh:2 send 723.3Kbps lastrcv:40 pacing_rate 1.4Mbps retrans:0/40 rcv_space:29200

    fyi TX is lower than RX:

    iperf -t 2 -c 192.168.1.77
    ------------------------------------------------------------
    Client connecting to 192.168.1.77, TCP port 5001'
    TCP window size: 43.8 KByte (default)
    ------------------------------------------------------------
    [  3] local 192.168.1.66 port 39681 connected with 192.168.1.77 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0- 9.4 sec   256 KBytes   224 Kbits/sec

    iperf -s
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 85.3 KByte (default)
    ------------------------------------------------------------
    [  4] local 192.168.1.66 port 5001 connected with 192.168.1.77 port 50420
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0- 6.2 sec  35.5 MBytes  47.9 Mbits/sec
    (only 47.9Mbits, because one further application is running with higher priority)

    MII signals are evaluated right now and will be send later
    The error also occurs with ifconfig down/up.

    Question:
    Could be power supply be one reason? May be the application is at a limit and eth0 slows down but eth1 stays fast?

    Regards, Holger

  • Hi Holger,
    so this is Linux... Actually the data is not clear. Where is the issue? Can they give clear comparison between good and bad boards? Differences in TCP direction performance may be normal...
    Measuring TCP performance is probably not the best here. TCP is a two-way protocol. Even if there are no errors in ifconfig there may still be packet re-transmits due to other issues (packets dropped on the line, buffer overflows,..). TCP does dynamically adjust packet size and fragmentation and depends on system configuration (MTU, Window size). So you are measuring also software performance here. So if another Linux process is using a lot of processing time the TCP performance may go down a lot and vary over time. UDP measured in two directions individually may provide better info. And then run a packet log at the same time using an external TAP that can record errored packets to see any issues on the line.

    I don't understand that:
    >>May be the application is at a limit and eth0 slows down but eth1 stays fast?
    Is there any data here comparing eth0 and eth1?
    I don't know the 'ss -ti' command. Maybe the BU experts can tell...

    If the power supply is not sufficient and instable anything can happen.

    Regards,
  • To be sure that the other Linux process do not disturb the iperf if have stopped the process.
    And I have changes to UDP.

    The result on eth0 (act as client):
    iperf -t 2000 -c 192.168.1.77 -u
    ------------------------------------------------------------
    Client connecting to 192.168.1.77, UDP port 5001
    Sending 1470 byte datagrams
    UDP buffer size: 160 KByte (default)
    ------------------------------------------------------------
    [ 3] local 192.168.1.66 port 45476 connected with 192.168.1.77 port 5001
    ^C[ ID] Interval Transfer Bandwidth
    [ 3] 0.0-11.1 sec 1.39 MBytes 1.05 Mbits/sec
    [ 3] Sent 988 datagrams
    [ 3] Server Report:
    [ 3] 0.0-11.1 sec 1.04 MBytes 787 Kbits/sec 0.256 ms 247/ 988 (25%)


    Act as server on eth0:
    iperf -s
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 85.3 KByte (default)
    ------------------------------------------------------------
    [ 4] local 192.168.1.66 port 5001 connected with 192.168.1.77 port 52047
    [ ID] Interval Transfer Bandwidth
    [ 4] 0.0- 7.1 sec 79.3 MBytes 94.1 Mbits/sec

    We are using the cpsw in switch mode, so we cannot send explicit on eth1. But I have connected another device on eth1 and start the test.
    The switch unit should direct the packets to eth1…
    iperf -t 2000 -c 192.168.1.124
    ------------------------------------------------------------
    Client connecting to 192.168.1.124, TCP port 5001
    TCP window size: 43.8 KByte (default)
    ------------------------------------------------------------
    [ 3] local 192.168.1.66 port 60721 connected with 192.168.1.124 port 5001
    ^C[ ID] Interval Transfer Bandwidth
    [ 3] 0.0- 7.8 sec 83.5 MBytes 89.4 Mbits/sec

    “ss” is a tool to get socket-statistics…

    We will check the power supply…
  • So you say that UDP performance is the issue here?

    According to some udp iperf docs:

    -u, --udp Use UDP rather than TCP. See also the -b option.

    -b, --bandwidth n[KM] Set target bandwidth to n bits/sec (default 1 Mbit/sec for UDP, unlimited for TCP). If there are multiple streams (-P flag), the bandwidth limit is applied separately to each stream. You can also add a ’/’ and a number to the bandwidth specifier. This is called "burst mode". It will send the given number of packets without pausing, even if that temporarily exceeds the specified bandwidth limit.


    Can you make sure there is no limit due to iperf here?

  • First I reproduced the problem:
    iperf -c 192.168.1.77 -t 2000
    ------------------------------------------------------------
    Client connecting to 192.168.1.77, TCP port 5001
    TCP window size: 43.8 KByte (default)
    ------------------------------------------------------------
    [ 3] local 192.168.1.113 port 59856 connected with 192.168.1.77 port 5001
    ^C[ ID] Interval Transfer Bandwidth
    [ 3] 0.0- 6.1 sec 256 KBytes 344 Kbits/sec

    And after that I tried again using UDP:
    iperf -c 192.168.1.77 -t 2000 -u -b 100M
    ------------------------------------------------------------
    Client connecting to 192.168.1.77, UDP port 5001
    Sending 1470 byte datagrams
    UDP buffer size: 160 KByte (default)
    ------------------------------------------------------------
    [ 3] local 192.168.1.113 port 32872 connected with 192.168.1.77 port 5001
    ^C[ ID] Interval Transfer Bandwidth
    [ 3] 0.0-17.4 sec 198 MBytes 95.7 Mbits/sec
    [ 3] Sent 141454 datagrams
    [ 3] Server Report:
    [ 3] 0.0-17.4 sec 147 MBytes 70.9 Mbits/sec 0.282 ms 36652/141448 (26%)
    [ 3] 0.0-17.4 sec 1 datagrams received out-of-order

    I’ve also attached a Wireshark-capture…
    I’m wondering that I get so different results. Why is UDP faster than TCP? Does the CPSW differentiate those protocols?
  • We had to take the discussion offline due to complex issue and exchange of proprietary data.

    Current analysis shows packet loss on TX on Eth0 only. We may need to involve Ethernet team as phy is TLK105L here.

    Error is not systematic as some boards are ok and others observe the issue only on a fraction of link up/down events.