• State
  • Locked Locked
  • Replies 16 replies
  • Subscribers 98 subscribers
  • Views 962 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM623: How to increase netperf (or iperf3) CPU utilization to improve ethernet TX/RX bandwidth.

Gibbs Shih
Expert 5335 points
Part Number: AM623
Other Parts Discussed in Thread: SK-AM62B

Tool/software:

Hi, Dear Expert

Customer's board has lower TX (client -> server) ethernet bandwidth, so I try to help them identify root cause.

First step, I try to base on software point of view to debug this issue.

 

Here is my netperf script,

#!/bin/bash
for i in 5
do
   netperf -H 192.168.10.99 -j -c -l 60 -p 1203 -t TCP_STREAM
     # -k DIRECTION,THROUGHPUT,MEAN_LATENCY,LOCAL_CPU_UTIL,REMOTE_CPU_UTIL,LOCAL_BYTES_SENT,REMOTE_BYTES_RECVD,LOCAL_SEND_SIZE &
   netperf -H 192.168.10.99 -j -c -l 60 -p 1203 -t TCP_MAERTS --
     # -k DIRECTION,THROUGHPUT,MEAN_LATENCY,LOCAL_CPU_UTIL,REMOTE_CPU_UTIL,LOCAL_BYTES_SENT,REMOTE_BYTES_RECVD,LOCAL_SEND_SIZE &
done

* [TEST 1] : Run netperf on customer board, result as below.

root@am62xx-evm:~# ./netperf-client.sh
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % ?      us/KB   us/KB

131072  16384  16384    60.03       862.70   28.28    -1.00    10.742  -1.000
MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Recv     Send     Recv    Send
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % ?      us/KB   us/KB

131072  16384  16384    60.00       941.41   34.34    -1.00    11.954  -1.000

We found the bandwidth from client to server (TX) only 862.70Gbps, and local CPU utilization around 28.28%

* Dump CPU utilization before netperf running on customer's board.

00:12:28     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
00:12:29     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
00:12:29       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
00:12:29       1    0.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00   99.00
00:12:29       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
00:12:29       3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

* Dump CPU utilization when netperf is running on customer's board..

00:22:13     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
00:22:14     all    0.51    0.00   13.37    0.00    2.83   18.51    0.00    0.00   64.78
00:22:14       0    0.00    0.00   17.17    0.00   10.10   72.73    0.00    0.00    0.00
00:22:14       1    0.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00   99.00
00:22:14       2    2.22    0.00   26.67    0.00    1.11    0.00    0.00    0.00   70.00
00:22:14       3    0.00    0.00    9.09    0.00    0.00    0.00    0.00    0.00   90.91

* [TEST 2] : Run netperf on AM62 EVB, result as below.

root@am62xx-evm:~# ./netperf-client.sh
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % ?      us/KB   us/KB

131072  16384  16384    60.02       934.94   42.89    -1.00    15.032  -1.000
MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Recv     Send     Recv    Send
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % ?      us/KB   us/KB

131072  16384  16384    60.00       941.45   51.28    -1.00    17.847  -1.000

We found AM62 EVB has better ethernet bandwidth performance (934~941Gbps) and higher CPU utilization (42%~51%)

* Dump CPU utilization before netperf running on EVB.

01:13:18     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:13:19     all    0.75    0.00    2.51    0.00    0.25    0.25    0.00    0.00   96.24
01:13:19       0    1.01    0.00    5.05    0.00    0.00    0.00    0.00    0.00   93.94
01:13:19       1    2.97    0.00    3.96    0.00    0.99    0.00    0.00    0.00   92.08
01:13:19       2    0.99    0.00    0.99    0.00    0.00    0.99    0.00    0.00   97.03
01:13:19       3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

* Dump CPU utilization when netperf is running on EVB.

01:17:10     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:17:11     all    0.80    0.00   18.50    0.00    6.43   24.66    0.00    0.00   49.60
01:17:11       0    1.15    0.00   36.78    0.00    4.60    0.00    0.00    0.00   57.47
01:17:11       1    0.00    0.00    0.00    0.00   11.96   88.04    0.00    0.00    0.00
01:17:11       2    1.01    0.00    9.09    0.00    6.06   10.10    0.00    0.00   73.74
01:17:11       3    1.06    0.00   29.79    0.00    3.19    0.00    0.00    0.00   65.96

Base on test result, I think CPU utilization and performance should be one of factor to effect ethernet bandwidth.

Our EVB has higher CPU utilization for netpref, so we get better performance on ethernet.

I try to adjust "Interrupt Pacing", but it does not work for me on customer board.

higher delay period get worst ethernet bandwidth, default value "20" is better.

ethtool -C eth0 rx-usecs <delayperiod> 
ethtool -C socnet0 tx-usecs 20
ethtool -C socnet0 rx-usecs 20

Do we have another software way to improve ethernet performance? 

Need more suggestion.

Many Thanks

Gibbs

  • 0
    TI__Genius 16435 points

    Hello Gibbs,

    Do you see the same throughput performance when running an iperf3 test on the customer's board and the EVM? 

    One suggestion I have is to run a UDP iperf3 test 940Mbps throughput rate and check if there are any lost datagrams reported (something like "iperf3 -c <ip address> -b940M -l1400 -u" on the client device). Additionally check the CPU utilization during this test to compare with the results you had from netperf. 

    I'm assuming that you have the DUT directly connected to a single link partner device/board?

    The CPU utilization you reported, I'm assuming that's from the client device?

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Expert 5335 points

    Hi, Daolin

    Thanks your replies.

    * Brief my test environment.

    AM62EVB or customer's DUT connect to one of gateway port. I also setup another PC for iperf3 server, server also connect to one of gateway port.

    gateway information as below : 

    https://www.genians.com/platform/ZyXEL_ZyWALL_5-N_Firewall/?infokey=RswMNQW2%2BGWq7vfOIlbp7A%3D%3D

    gateway ip : 192.168.10.99

    * Issues Background

    Customer's DUT has lower TX throughput , especially base on iperf3 bi-direction test. so I try to clarify the relationship between ethernet bandwidth (BW) and CPU utilization first.

    Test A. Iperf3 single direction test

    A.1 : EVB test result as below

    root@am62xx-evm:~# iperf3 -c 192.168.10.99 -i 4 -t 20
Connecting to host 192.168.10.99, port 5201
[  5] local 192.168.10.11 port 44426 connected to 192.168.10.99 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-4.00   sec   448 MBytes   938 Mbits/sec    0    833 KBytes
[  5]   4.00-8.00   sec   444 MBytes   931 Mbits/sec    0   1.14 MBytes
[  5]   8.00-12.00  sec   445 MBytes   932 Mbits/sec    0   1.14 MBytes
[  5]  12.00-16.00  sec   448 MBytes   939 Mbits/sec    0   1.14 MBytes
[  5]  16.00-20.00  sec   446 MBytes   936 Mbits/sec    0   1.79 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.18 GBytes   935 Mbits/sec    0             sender
[  5]   0.00-20.04  sec  2.18 GBytes   932 Mbits/sec                  receiver

    A.2 : DUT test result as below

    root@am62xx-evm:~# iperf3 -c 192.168.10.99 -i 4 -t 20 -b 0
Connecting to host 192.168.10.99, port 5201
[  5] local 192.168.10.3 port 49758 connected to 192.168.10.99 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-4.00   sec   412 MBytes   863 Mbits/sec    0    679 KBytes
[  5]   4.00-8.00   sec   409 MBytes   857 Mbits/sec    0    710 KBytes
[  5]   8.00-12.00  sec   409 MBytes   857 Mbits/sec    0    744 KBytes
[  5]  12.00-16.00  sec   410 MBytes   860 Mbits/sec    0    744 KBytes
[  5]  16.00-20.00  sec   409 MBytes   857 Mbits/sec    0    744 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.00 GBytes   859 Mbits/sec    0             sender
[  5]   0.00-20.04  sec  2.00 GBytes   856 Mbits/sec                  receiver

    root@am62xx-evm:~# iperf3 -c 192.168.10.99 -i 4 -t 20 -b 0 -R
Connecting to host 192.168.10.99, port 5201
Reverse mode, remote host 192.168.10.99 is sending
[  5] local 192.168.10.3 port 59090 connected to 192.168.10.99 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-4.00   sec   449 MBytes   941 Mbits/sec
[  5]   4.00-8.00   sec   449 MBytes   941 Mbits/sec
[  5]   8.00-12.00  sec   449 MBytes   941 Mbits/sec
[  5]  12.00-16.00  sec   449 MBytes   941 Mbits/sec
[  5]  16.00-20.00  sec   449 MBytes   941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.04  sec  2.19 GBytes   940 Mbits/sec   36             sender
[  5]   0.00-20.00  sec  2.19 GBytes   941 Mbits/sec                  receiver

    Conclusion : test result tell us, DUT has lower "TX" throughput (856~859 Mbps). But I think this bitrate should be also fine, even our EVB has better performance. 

    Test B. Iperf3 bir-direction test

    B.1 : EVB test result as below

    Usage: iperf3 [-s|-c host] [options]
Try `iperf3 --help' for more information.
root@am62xx-evm:~# iperf3 -c 192.168.10.99 -i 4 -t 20 --bidir
Connecting to host 192.168.10.99, port 5201
[  5] local 192.168.10.11 port 36190 connected to 192.168.10.99 port 5201
[  7] local 192.168.10.11 port 36202 connected to 192.168.10.99 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-4.00   sec   440 MBytes   923 Mbits/sec    0   1001 KBytes
[  7][RX-C]   0.00-4.00   sec   395 MBytes   827 Mbits/sec
[  5][TX-C]   4.00-8.00   sec   436 MBytes   915 Mbits/sec    0   1.03 MBytes
[  7][RX-C]   4.00-8.00   sec   416 MBytes   873 Mbits/sec
[  5][TX-C]   8.00-12.00  sec   436 MBytes   915 Mbits/sec    0   1.03 MBytes
[  7][RX-C]   8.00-12.00  sec   408 MBytes   855 Mbits/sec
[  5][TX-C]  12.00-16.00  sec   435 MBytes   912 Mbits/sec    0   1.03 MBytes
[  7][RX-C]  12.00-16.00  sec   405 MBytes   849 Mbits/sec
[  5][TX-C]  16.00-20.00  sec   435 MBytes   912 Mbits/sec    0   1.03 MBytes
[  7][RX-C]  16.00-20.00  sec   396 MBytes   831 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-20.00  sec  2.13 GBytes   915 Mbits/sec    0             sender
[  5][TX-C]   0.00-20.04  sec  2.13 GBytes   913 Mbits/sec                  receiver
[  7][RX-C]   0.00-20.00  sec  1.98 GBytes   848 Mbits/sec  378             sender
[  7][RX-C]   0.00-20.04  sec  1.97 GBytes   846 Mbits/sec                  receiver

    B.1 : DUT test result as below

    root@am62xx-evm:~# iperf3 -c 192.168.10.99 -i 4 -t 20 -b 0 --bidir
Connecting to host 192.168.10.99, port 5201
[  5] local 192.168.10.3 port 59868 connected to 192.168.10.99 port 5201
[  7] local 192.168.10.3 port 59876 connected to 192.168.10.99 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-4.00   sec   159 MBytes   333 Mbits/sec    9    287 KBytes
[  7][RX-C]   0.00-4.00   sec   444 MBytes   930 Mbits/sec
[  5][TX-C]   4.00-8.00   sec   170 MBytes   357 Mbits/sec    2    211 KBytes
[  7][RX-C]   4.00-8.00   sec   443 MBytes   929 Mbits/sec
[  5][TX-C]   8.00-12.00  sec   171 MBytes   359 Mbits/sec    0    331 KBytes
[  7][RX-C]   8.00-12.00  sec   444 MBytes   930 Mbits/sec
[  5][TX-C]  12.00-16.00  sec   167 MBytes   351 Mbits/sec    2    322 KBytes
[  7][RX-C]  12.00-16.00  sec   443 MBytes   930 Mbits/sec
[  5][TX-C]  16.00-20.00  sec   163 MBytes   342 Mbits/sec    5    318 KBytes
[  7][RX-C]  16.00-20.00  sec   444 MBytes   931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-20.00  sec   831 MBytes   348 Mbits/sec   18             sender
[  5][TX-C]   0.00-20.04  sec   829 MBytes   347 Mbits/sec                  receiver
[  7][RX-C]   0.00-20.00  sec  2.17 GBytes   931 Mbits/sec    5             sender
[  7][RX-C]   0.00-20.04  sec  2.17 GBytes   928 Mbits/sec                  receiver

    Conclusion : test result tell us, DUT has "more worst" TX performance (347Mbps). 

    Test C. Iperf3 localhost test (bi-direction), ethernet packet do not routing outside.

    C.1 : EVB test result as below

    root@am62xx-evm:~# iperf3 -c localhost -i 4 -t 20 --bidir
Connecting to host localhost, port 5201
Accepted connection from ::1, port 38232
[  5] local ::1 port 5201 connected to ::1 port 38234
[  5] local ::1 port 38234 connected to ::1 port 5201
[  8] local ::1 port 5201 connected to ::1 port 38238
[  7] local ::1 port 38238 connected to ::1 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][RX-S]   0.00-1.00   sec   231 MBytes  1.94 Gbits/sec
[  8][TX-S]   0.00-1.00   sec   237 MBytes  1.99 Gbits/sec    0   1.44 MBytes
[  5][RX-S]   1.00-2.00   sec   222 MBytes  1.86 Gbits/sec
[  8][TX-S]   1.00-2.00   sec   242 MBytes  2.03 Gbits/sec    0   1.44 MBytes
[  5][RX-S]   2.00-3.00   sec   228 MBytes  1.92 Gbits/sec
[  8][TX-S]   2.00-3.00   sec   240 MBytes  2.01 Gbits/sec    0   1.44 MBytes
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][RX-S]   3.00-4.00   sec   223 MBytes  1.87 Gbits/sec
[  5][TX-C]   0.00-4.00   sec   914 MBytes  1.92 Gbits/sec    0   1.31 MBytes
[  8][TX-S]   3.00-4.00   sec   246 MBytes  2.06 Gbits/sec    0   1.44 MBytes
[  7][RX-C]   0.00-4.00   sec   957 MBytes  2.01 Gbits/sec
[  5][RX-S]   4.00-5.00   sec   223 MBytes  1.87 Gbits/sec
[  8][TX-S]   4.00-5.00   sec   244 MBytes  2.05 Gbits/sec    0   1.44 MBytes
[  5][RX-S]   5.00-6.00   sec   230 MBytes  1.92 Gbits/sec
[  8][TX-S]   5.00-6.00   sec   241 MBytes  2.02 Gbits/sec    0   1.44 MBytes
[  5][RX-S]   6.00-7.00   sec   215 MBytes  1.80 Gbits/sec
[  8][TX-S]   6.00-7.00   sec   256 MBytes  2.15 Gbits/sec    0   1.44 MBytes
[  5][RX-S]   7.00-8.00   sec   224 MBytes  1.88 Gbits/sec
[  5][TX-C]   4.00-8.00   sec   891 MBytes  1.87 Gbits/sec    0   1.37 MBytes
[  8][TX-S]   7.00-8.00   sec   241 MBytes  2.03 Gbits/sec    0   1.44 MBytes
[  7][RX-C]   4.00-8.00   sec   983 MBytes  2.06 Gbits/sec
[  5][RX-S]   8.00-9.00   sec   222 MBytes  1.86 Gbits/sec
[  8][TX-S]   8.00-9.00   sec   242 MBytes  2.03 Gbits/sec    0   1.44 MBytes
[  5][RX-S]   9.00-10.00  sec   228 MBytes  1.92 Gbits/sec
[  8][TX-S]   9.00-10.00  sec   242 MBytes  2.04 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  10.00-11.00  sec   227 MBytes  1.90 Gbits/sec
[  8][TX-S]  10.00-11.00  sec   242 MBytes  2.03 Gbits/sec    0   1.44 MBytes
[  5][TX-C]   8.00-12.00  sec   898 MBytes  1.88 Gbits/sec    0   1.37 MBytes
[  7][RX-C]   8.00-12.00  sec   973 MBytes  2.04 Gbits/sec
[  5][RX-S]  11.00-12.00  sec   221 MBytes  1.85 Gbits/sec
[  8][TX-S]  11.00-12.00  sec   246 MBytes  2.06 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  12.00-13.00  sec   221 MBytes  1.86 Gbits/sec
[  8][TX-S]  12.00-13.00  sec   248 MBytes  2.08 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  13.00-14.00  sec   232 MBytes  1.95 Gbits/sec
[  8][TX-S]  13.00-14.00  sec   238 MBytes  1.99 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  14.00-15.00  sec   214 MBytes  1.80 Gbits/sec
[  8][TX-S]  14.00-15.00  sec   254 MBytes  2.13 Gbits/sec    0   1.44 MBytes
[  5][TX-C]  12.00-16.00  sec   886 MBytes  1.86 Gbits/sec    0   1.37 MBytes
[  7][RX-C]  12.00-16.00  sec   991 MBytes  2.08 Gbits/sec
[  5][RX-S]  15.00-16.00  sec   217 MBytes  1.82 Gbits/sec
[  8][TX-S]  15.00-16.00  sec   252 MBytes  2.12 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  16.00-17.00  sec   222 MBytes  1.86 Gbits/sec
[  8][TX-S]  16.00-17.00  sec   246 MBytes  2.06 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  17.00-18.00  sec   224 MBytes  1.88 Gbits/sec
[  8][TX-S]  17.00-18.00  sec   245 MBytes  2.06 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  18.00-19.00  sec   219 MBytes  1.84 Gbits/sec
[  8][TX-S]  18.00-19.00  sec   249 MBytes  2.09 Gbits/sec    0   1.44 MBytes
[  5][RX-S]  19.00-20.00  sec   225 MBytes  1.89 Gbits/sec
[  8][TX-S]  19.00-20.00  sec   241 MBytes  2.02 Gbits/sec    0   1.44 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][RX-S]   0.00-20.00  sec  4.36 GBytes  1.87 Gbits/sec                  receiver
[  8][TX-S]   0.00-20.00  sec  4.78 GBytes  2.05 Gbits/sec    0             sender
[  5][TX-C]  16.00-20.00  sec   889 MBytes  1.86 Gbits/sec    0   1.37 MBytes
[  7][RX-C]  16.00-20.00  sec   982 MBytes  2.06 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-20.00  sec  4.37 GBytes  1.88 Gbits/sec    0             sender
[  5][TX-C]   0.00-20.00  sec  4.36 GBytes  1.87 Gbits/sec                  receiver
[  7][RX-C]   0.00-20.00  sec  4.78 GBytes  2.05 Gbits/sec    0             sender
[  7][RX-C]   0.00-20.00  sec  4.77 GBytes  2.05 Gbits/sec                  receiver

    C.2 : DUT test result as below

    root@am62xx-evm:~# iperf3 -c localhost -i 4 -t 20 --bidir
Connecting to host localhost, port 5201
[  5] local ::1 port 34142 connected to ::1 port 5201
[  7] local ::1 port 34148 connected to ::1 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-4.00   sec   178 MBytes   372 Mbits/sec    0   4.74 MBytes
[  7][RX-C]   0.00-4.00   sec  1.66 GBytes  3.56 Gbits/sec
[  5][TX-C]   4.00-8.00   sec   168 MBytes   351 Mbits/sec    0   4.74 MBytes
[  7][RX-C]   4.00-8.00   sec  1.63 GBytes  3.51 Gbits/sec
[  5][TX-C]   8.00-12.00  sec   170 MBytes   357 Mbits/sec    0   4.74 MBytes
[  7][RX-C]   8.00-12.00  sec  1.66 GBytes  3.56 Gbits/sec
[  5][TX-C]  12.00-16.00  sec   170 MBytes   357 Mbits/sec    0   4.74 MBytes
[  7][RX-C]  12.00-16.00  sec  1.64 GBytes  3.51 Gbits/sec
[  5][TX-C]  16.00-20.00  sec   169 MBytes   354 Mbits/sec    0   4.74 MBytes
[  7][RX-C]  16.00-20.00  sec  1.65 GBytes  3.54 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-20.00  sec   854 MBytes   358 Mbits/sec    0             sender
[  5][TX-C]   0.00-20.00  sec   845 MBytes   354 Mbits/sec                  receiver
[  7][RX-C]   0.00-20.00  sec  8.23 GBytes  3.54 Gbits/sec    0             sender
[  7][RX-C]   0.00-20.00  sec  8.23 GBytes  3.54 Gbits/sec                  receiver

    Conclusion : DUT even local host test, the TX performance is only 358Mbps~354Mbps. 

    So that's way I suspect software should be have opportunity to fine tune. 

    I also suspect TX/RX has bandwidth race condition.

    total total BW about 3.6Gbps, client to server is 354~358Mbps, client to client is 3.54Gbps

    [ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-20.00  sec   854 MBytes   358 Mbits/sec    0             sender
[  5][TX-C]   0.00-20.00  sec   845 MBytes   354 Mbits/sec                  receiver
[  7][RX-C]   0.00-20.00  sec  8.23 GBytes  3.54 Gbits/sec    0             sender
[  7][RX-C]   0.00-20.00  sec  8.23 GBytes  3.54 Gbits/sec                  receiver

    So I curious want to know, beside RGMII / PHY chip HW reason, do we have any chance to improve this problem

    Many Thanks

    Gibbs 

  • 0 in reply to Gibbs Shih
    TI__Genius 16435 points

    Hi Gibbs,

    Thanks for sharing these detailed results.

    One thing I wanted to clarify, have you confirmed the transmitted packets from the DUT to the link partner PC are not corrupted or dropped at the PC's receive end? Similarly, are there any corrupted packets seen on the receive end of the DUT sent from the link partner PC? I just want to check this to rule out any possibility of the lower bandwidth due to dropped packets. You can check this with "ethtool -S <interface name>".

    Another question is if you are running RT-Linux (PREEMPT-RT) or regular Linux on the DUT? If running RT-Linux, sometimes increasing the priority of the ksoftirqs and iperf3 itself can increase throughput. This would be more noticeably running the "iperf -u" option to send UDP packets rather than the TCP packets you've been running with iperf3. 

    ps -ALo psr,policy,priority,pid,tid,cputime,comm | grep ksoft --> to grep priority of ksoftirqs
chrt -f -p <priority> <process id> ---> to change priority of process id

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Expert 5335 points

    Hi, Daolin

    Thanks you replies, let's me make some updates.

    Steps as below, 

    a) Test Environment : 

    iperf3 server : 192.168.10.99

    iperf3 DUT : 192.168.10.3

    DUT OS : regular Linux without RT

    b) DUT threads after boots : 

      PID USER       VSZ STAT COMMAND
    1 root      157m S    {systemd} /sbin/init
    2 root         0 SW   [kthreadd]
    3 root         0 IW<  [rcu_gp]
    4 root         0 IW<  [rcu_par_gp]
    5 root         0 IW<  [slub_flushwq]
    6 root         0 IW<  [netns]
    7 root         0 IW   [kworker/0:0-mm_]
    8 root         0 IW<  [kworker/0:0H-kb]
   10 root         0 IW<  [mm_percpu_wq]
   11 root         0 IW   [rcu_tasks_kthre]
   12 root         0 IW   [rcu_tasks_trace]
   13 root         0 RW   [ksoftirqd/0]
   14 root         0 SW   [ktimers/0]
   16 root         0 IW   [rcu_preempt]
   17 root         0 SW   [rcub/0]
   18 root         0 SW   [rcuc/0]
   19 root         0 SW   [migration/0]
   20 root         0 SW   [irq_work/0]
   21 root         0 SW   [cpuhp/0]
   22 root         0 SW   [cpuhp/1]
   23 root         0 SW   [irq_work/1]
   24 root         0 SW   [migration/1]
   25 root         0 SW   [rcuc/1]
   26 root         0 SW   [ktimers/1]
   27 root         0 SW   [ksoftirqd/1]
   29 root         0 IW<  [kworker/1:0H-ev]
   30 root         0 SW   [cpuhp/2]
   31 root         0 SW   [irq_work/2]
   32 root         0 SW   [migration/2]
   33 root         0 SW   [rcuc/2]
   34 root         0 SW   [ktimers/2]
   35 root         0 SW   [ksoftirqd/2]
   36 root         0 IW   [kworker/2:0-mm_]
   37 root         0 IW<  [kworker/2:0H-kb]
   38 root         0 SW   [cpuhp/3]
   39 root         0 SW   [irq_work/3]
   40 root         0 SW   [migration/3]
   41 root         0 SW   [rcuc/3]
   42 root         0 SW   [ktimers/3]
   43 root         0 SW   [ksoftirqd/3]
   44 root         0 IW   [kworker/3:0-pm]
   46 root         0 SW   [kdevtmpfs]
   47 root         0 IW<  [inet_frag_wq]
   50 root         0 SW   [kauditd]
   51 root         0 SW   [oom_reaper]
   52 root         0 IW<  [writeback]
   53 root         0 SW   [kcompactd0]
   54 root         0 SWN  [ksmd]
   55 root         0 IW<  [kintegrityd]
   56 root         0 IW<  [kblockd]
   57 root         0 IW<  [blkcg_punt_bio]
   58 root         0 IW<  [tpm_dev_wq]
   59 root         0 IW<  [ata_sff]
   60 root         0 IW<  [edac-poller]
   61 root         0 IW<  [devfreq_wq]
   62 root         0 SW   [watchdogd]
   64 root         0 IW<  [kworker/1:1H-kb]
   65 root         0 IW<  [rpciod]
   67 root         0 IW<  [xprtiod]
   68 root         0 SW   [kswapd0]
   69 root         0 IW<  [nfsiod]
   70 root         0 IW<  [kthrotld]
   74 root         0 IW<  [kpcitest]
   75 root         0 IW<  [kpcintb]
   76 root         0 IW<  [vfio-irqfd-clea]
  124 root         0 SW   [irq/17-4d000000]
  128 root         0 IW   [kworker/3:3-mm_]
  129 root         0 SW   [irq/26-20000000]
  130 root         0 SW   [irq/27-20010000]
  131 root         0 SW   [irq/28-20020000]
  132 root         0 SW   [irq/29-tps65219]
  135 root         0 SW   [pr/ttyS2]
  136 root         0 SW   [irq/291-2012000]
  137 root         0 SW   [spi1]
  139 root         0 SW   [ptp0]
  140 root         0 SW   [irq/292-8000000]
  141 root         0 RW   [irq/165-8000000]
  142 root         0 SW   [irq/255-8000000]
  146 root         0 IW<  [kworker/u9:0]
  147 root         0 IW<  [sdhci]
  148 root         0 SW   [irq/295-mmc0]
  149 root         0 SW   [irq/295-s-mmc0]
  151 root         0 IW<  [sdhci]
  152 root         0 SW   [irq/476-mmc1]
  153 root         0 SW   [irq/476-s-mmc1]
  156 root         0 SW   [irq/472-fa00000]
  157 root         0 IW<  [mmc_complete]
  159 root         0 IW<  [kworker/3:1H-kb]
  161 root         0 SW   [jbd2/mmcblk0p6-]
  162 root         0 IW<  [ext4-rsv-conver]
  164 root         0 IW<  [mld]
  165 root         0 IW<  [ipv6_addrconf]
  190 rpc       4376 S    /usr/sbin/rpcbind -w -f
  191 root     34052 S    /lib/systemd/systemd-journald
  194 root         0 IW<  [cryptodev_queue]
  211 root     18732 S    /lib/systemd/systemd-udevd
  228 systemd-  9280 S    /lib/systemd/systemd-resolved
  232 root         0 IW   [kworker/0:7-rcu]
  234 systemd- 83028 S    /lib/systemd/systemd-timesyncd
  236 root         0 SW   [irq/477-2b20000]
  237 root         0 SW   [irq/478-2b20000]
  242 root         0 SW   [irq/279-485c000]
  243 root         0 SW   [irq/107-485c010]
  244 root         0 SW   [irq/95-485c0100]
  245 root         0 SW   [irq/280-485c000]
  248 root         0 SW   [irq/143-485c010]
  249 root         0 SW   [irq/125-485c010]
  250 root         0 SW   [irq/144-485c010]
  251 root         0 SW   [irq/126-485c010]
  252 root         0 SW   [irq/237-485c000]
  255 root         0 SW   [irq/145-485c010]
  259 root         0 SW   [irq/127-485c010]
  260 root         0 SW   [irq/146-485c010]
  261 root         0 SW   [irq/128-485c010]
  262 root         0 SW   [irq/147-485c010]
  264 root         0 SW   [irq/129-485c010]
  274 root         0 SW   [irq/14-mbox-r5-]
  275 root         0 SW   [irq/494-2b1f000]
  280 messageb  6660 S    /usr/bin/dbus-broker-launch --scope system --audit
  281 1001      2816 S    /usr/sbin/rpc.statd -F
  349 messageb  3268 S    dbus-broker --log 4 --controller 9 --machine-id 5ce2
  353 avahi     5796 S    avahi-daemon: running [am62xx-evm.local]
  367 avahi     5612 S    avahi-daemon: chroot helper
  388 root      3052 S    /usr/sbin/netserver -D
  391 root      2892 S    /usr/sbin/phc2sys -w -s eth0 -m
  401 root      2952 S    /usr/sbin/ptp4l -f /etc/linuxptp/ptp4l.conf -i eth0
  412 root      9216 S    /lib/systemd/systemd-logind
  440 root      2708 S    /usr/sbin/tee-supplicant
  450 root     1463m S    /usr/bin/containerd
  454 root         0 IW<  [optee_bus_scan]
  476 root      3088 S    /usr/sbin/telnetd
  555 root         0 IW<  [kworker/2:2H-kb]
  556 root         0 IW<  [kworker/3:2H]
  560 root         0 IW<  [kworker/0:2H-kb]
  595 root         0 IW   [kworker/1:3-cgr]
  598 root         0 IW   [kworker/1:6-mm_]
  632 root         0 SW   [irq/496-xhci-hc]
  727 root         0 SW   [jbd2/mmcblk0p13]
  731 root         0 IW<  [ext4-rsv-conver]
  843 root     10448 S    /opt/apx/data/www/cgi/apx.cgi
  845 root      2244 S    /sbin/agetty -o -p -- \u --noclear - linux
  846 root      9620 S    nginx: master process /opt/apx/data/bin/nginx
  847 nobody   10072 S    nginx: worker process
  848 root      5284 S    /bin/login -p --
  850 root     10448 S    /opt/apx/data/www/cgi/apx.cgi
  852 root     10448 S    /opt/apx/data/www/cgi/apx.cgi
  853 root     10448 S    /opt/apx/data/www/cgi/apx.cgi
  854 root     10448 S    /opt/apx/data/www/cgi/apx.cgi
  855 root         0 SW   [irq/288-2800000]
  856 root         0 SW   [irq/287-2800000]
  864 root      8852 S    /lib/systemd/systemd-userdbd
  869 root     10972 S    /lib/systemd/systemd --user
  870 root      158m S    (sd-pam)
  876 root      5000 S    -sh
 1155 root         0 IW   [kworker/2:2-cgr]
 1158 root      2876 S    /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_
 1159 root      4896 S    -sh
 1164 root      2876 S    /usr/sbin/dropbear -i -r /etc/dropbear/dropbear_rsa_
 1165 root      4896 S    -sh
 1193 root         0 IW   [kworker/u8:3-ev]
 1289 root         0 IW   [kworker/u8:1-ev]
 1309 root         0 IW   [kworker/u8:2-ev]
 1313 root      8860 S    systemd-userwork
 1314 root      8860 S    systemd-userwork
 1315 root      8860 S    systemd-userwork
 1316 root      8200 R    iperf3 -c 192.168.10.99 -i 10 -t 200 -b 0 --bidir
 1321 root      3188 R    ps

    c) Before iperf3 test, check server/DUT frame status : 

    c.1) server.

    gibbs@gibbs-MS-7D17:~$ ethtool -S enp1s0
NIC statistics:
     tx_packets: 10313092
     rx_packets: 3824610
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 3823104
     broadcast: 1160
     multicast: 346
     tx_aborted: 0
     tx_underrun: 0

    c.2) DUT

    root@am62xx-evm:~# ethtool -S eth0
NIC statistics:
     p0_rx_good_frames: 3442577
     p0_rx_broadcast_frames: 13
     p0_rx_multicast_frames: 41
     p0_rx_crc_errors: 0
     p0_rx_oversized_frames: 0
     p0_rx_undersized_frames: 0
     p0_ale_drop: 0
     p0_ale_overrun_drop: 0
     p0_rx_octets: 456186582
     p0_tx_good_frames: 9567834
     p0_tx_broadcast_frames: 165
     p0_tx_multicast_frames: 160
     p0_tx_octets: 3717603911
     p0_tx_64B_frames: 1369
     p0_tx_65_to_127B_frames: 1856720
     p0_tx_128_to_255B_frames: 389
     p0_tx_256_to_511B_frames: 64
     p0_tx_512_to_1023B_frames: 24
     p0_tx_1024B_frames: 11151845
     p0_net_octets: 4173790493
     p0_rx_bottom_fifo_drop: 0
     p0_rx_port_mask_drop: 0
     p0_rx_top_fifo_drop: 0
     p0_ale_rate_limit_drop: 0
     p0_ale_vid_ingress_drop: 0
     p0_ale_da_eq_sa_drop: 0
     p0_ale_block_drop: 0
     p0_ale_secure_drop: 0
     p0_ale_auth_drop: 0
     p0_ale_unknown_ucast: 0
     p0_ale_unknown_ucast_bytes: 0
     p0_ale_unknown_mcast: 0
     p0_ale_unknown_mcast_bytes: 0
     p0_ale_unknown_bcast: 0
     p0_ale_unknown_bcast_bytes: 0
     p0_ale_pol_match: 0
     p0_ale_pol_match_red: 0
     p0_ale_pol_match_yellow: 0
     p0_ale_mcast_sa_drop: 0
     p0_ale_dual_vlan_drop: 0
     p0_ale_len_err_drop: 0
     p0_ale_ip_next_hdr_drop: 0
     p0_ale_ipv4_frag_drop: 0
     p0_tx_mem_protect_err: 0
     p0_tx_pri0: 9567834
     p0_tx_pri1: 0
     p0_tx_pri2: 0
     p0_tx_pri3: 0
     p0_tx_pri4: 0
     p0_tx_pri5: 0
     p0_tx_pri6: 0
     p0_tx_pri7: 0
     p0_tx_pri0_bcnt: 3717603911
     p0_tx_pri1_bcnt: 0
     p0_tx_pri2_bcnt: 0
     p0_tx_pri3_bcnt: 0
     p0_tx_pri4_bcnt: 0
     p0_tx_pri5_bcnt: 0
     p0_tx_pri6_bcnt: 0
     p0_tx_pri7_bcnt: 0
     p0_tx_pri0_drop: 0
     p0_tx_pri1_drop: 0
     p0_tx_pri2_drop: 0
     p0_tx_pri3_drop: 0
     p0_tx_pri4_drop: 0
     p0_tx_pri5_drop: 0
     p0_tx_pri6_drop: 0
     p0_tx_pri7_drop: 0
     p0_tx_pri0_drop_bcnt: 0
     p0_tx_pri1_drop_bcnt: 0
     p0_tx_pri2_drop_bcnt: 0
     p0_tx_pri3_drop_bcnt: 0
     p0_tx_pri4_drop_bcnt: 0
     p0_tx_pri5_drop_bcnt: 0
     p0_tx_pri6_drop_bcnt: 0
     p0_tx_pri7_drop_bcnt: 0
     rx_good_frames: 9567843
     rx_broadcast_frames: 165
     rx_multicast_frames: 169
     rx_pause_frames: 0
     rx_crc_errors: 0
     rx_align_code_errors: 0
     rx_oversized_frames: 0
     rx_jabber_frames: 0
     rx_undersized_frames: 0
     rx_fragments: 0
     ale_drop: 9
     ale_overrun_drop: 0
     rx_octets: 3717604577
     tx_good_frames: 3442577
     tx_broadcast_frames: 13
     tx_multicast_frames: 41
     tx_pause_frames: 0
     tx_deferred_frames: 0
     tx_collision_frames: 0
     tx_single_coll_frames: 0
     tx_mult_coll_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     rx_ipg_error: 0
     tx_carrier_sense_errors: 0
     tx_octets: 456186582
     tx_64B_frames: 1369
     tx_65_to_127B_frames: 1856729
     tx_128_to_255B_frames: 389
     tx_256_to_511B_frames: 64
     tx_512_to_1023B_frames: 24
     tx_1024B_frames: 11151845
     net_octets: 4173791159
     rx_bottom_fifo_drop: 0
     rx_port_mask_drop: 9
     rx_top_fifo_drop: 0
     ale_rate_limit_drop: 0
     ale_vid_ingress_drop: 0
     ale_da_eq_sa_drop: 0
     ale_block_drop: 0
     ale_secure_drop: 0
     ale_auth_drop: 0
     ale_unknown_ucast: 9567509
     ale_unknown_ucast_bytes: 3717574393
     ale_unknown_mcast: 169
     ale_unknown_mcast_bytes: 15772
     ale_unknown_bcast: 165
     ale_unknown_bcast_bytes: 14412
     ale_pol_match: 0
     ale_pol_match_red: 0
     ale_pol_match_yellow: 0
     ale_mcast_sa_drop: 0
     ale_dual_vlan_drop: 0
     ale_len_err_drop: 0
     ale_ip_next_hdr_drop: 0
     ale_ipv4_frag_drop: 0
     iet_rx_assembly_err: 0
     iet_rx_assembly_ok: 0
     iet_rx_smd_err: 0
     iet_rx_frag: 0
     iet_tx_hold: 0
     iet_tx_frag: 0
     tx_mem_protect_err: 0
     tx_pri0: 3442577
     tx_pri1: 0
     tx_pri2: 0
     tx_pri3: 0
     tx_pri4: 0
     tx_pri5: 0
     tx_pri6: 0
     tx_pri7: 0
     tx_pri0_bcnt: 456186582
     tx_pri1_bcnt: 0
     tx_pri2_bcnt: 0
     tx_pri3_bcnt: 0
     tx_pri4_bcnt: 0
     tx_pri5_bcnt: 0
     tx_pri6_bcnt: 0
     tx_pri7_bcnt: 0
     tx_pri0_drop: 0
     tx_pri1_drop: 0
     tx_pri2_drop: 0
     tx_pri3_drop: 0
     tx_pri4_drop: 0
     tx_pri5_drop: 0
     tx_pri6_drop: 0
     tx_pri7_drop: 0
     tx_pri0_drop_bcnt: 0
     tx_pri1_drop_bcnt: 0
     tx_pri2_drop_bcnt: 0
     tx_pri3_drop_bcnt: 0
     tx_pri4_drop_bcnt: 0
     tx_pri5_drop_bcnt: 0
     tx_pri6_drop_bcnt: 0
     tx_pri7_drop_bcnt: 0

    d) Runs iperf3 in DUT

    root@am62xx-evm:~# iperf3 -c 192.168.10.99 -i 4 -t 20 -b 0 --bidir
Connecting to host 192.168.10.99, port 5201
[  5] local 192.168.10.3 port 39906 connected to 192.168.10.99 port 5201
[  7] local 192.168.10.3 port 39916 connected to 192.168.10.99 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-4.00   sec   170 MBytes   356 Mbits/sec    0    455 KBytes
[  7][RX-C]   0.00-4.00   sec   443 MBytes   929 Mbits/sec
[  5][TX-C]   4.00-8.00   sec   171 MBytes   358 Mbits/sec    0    455 KBytes
[  7][RX-C]   4.00-8.00   sec   444 MBytes   931 Mbits/sec
[  5][TX-C]   8.00-12.00  sec   173 MBytes   362 Mbits/sec    0    455 KBytes
[  7][RX-C]   8.00-12.00  sec   444 MBytes   931 Mbits/sec
[  5][TX-C]  12.00-16.00  sec   172 MBytes   361 Mbits/sec    0    455 KBytes
[  7][RX-C]  12.00-16.00  sec   444 MBytes   931 Mbits/sec
[  5][TX-C]  16.00-20.00  sec   171 MBytes   359 Mbits/sec    0    455 KBytes
[  7][RX-C]  16.00-20.00  sec   444 MBytes   931 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-20.00  sec   857 MBytes   359 Mbits/sec    0             sender
[  5][TX-C]   0.00-20.04  sec   855 MBytes   358 Mbits/sec                  receiver
[  7][RX-C]   0.00-20.00  sec  2.17 GBytes   932 Mbits/sec    1             sender
[  7][RX-C]   0.00-20.04  sec  2.17 GBytes   929 Mbits/sec                  receiver

    e) Check server/DUT frame status again after iperf3 test.

    e.1) Server

    gibbs@gibbs-MS-7D17:~$ ethtool -S enp1s0
NIC statistics:
     tx_packets: 12226605
     rx_packets: 4508245
     tx_errors: 0
     rx_errors: 0
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 4506715
     broadcast: 1184
     multicast: 346
     tx_aborted: 0
     tx_underrun: 0
gibbs@gibbs-MS-7D17:~$

    e.2) DUT

    root@am62xx-evm:~# ethtool -S eth0
NIC statistics:
     p0_rx_good_frames: 4126229
     p0_rx_broadcast_frames: 13
     p0_rx_multicast_frames: 41
     p0_rx_crc_errors: 0
     p0_rx_oversized_frames: 0
     p0_rx_undersized_frames: 0
     p0_ale_drop: 0
     p0_ale_overrun_drop: 0
     p0_rx_octets: 1400595653
     p0_tx_good_frames: 11481487
     p0_tx_broadcast_frames: 197
     p0_tx_multicast_frames: 164
     p0_tx_octets: 1886654641
     p0_tx_64B_frames: 1656
     p0_tx_65_to_127B_frames: 2225390
     p0_tx_128_to_255B_frames: 403
     p0_tx_256_to_511B_frames: 77
     p0_tx_512_to_1023B_frames: 29
     p0_tx_1024B_frames: 13380161
     p0_net_octets: 3287250294
     p0_rx_bottom_fifo_drop: 0
     p0_rx_port_mask_drop: 0
     p0_rx_top_fifo_drop: 0
     p0_ale_rate_limit_drop: 0
     p0_ale_vid_ingress_drop: 0
     p0_ale_da_eq_sa_drop: 0
     p0_ale_block_drop: 0
     p0_ale_secure_drop: 0
     p0_ale_auth_drop: 0
     p0_ale_unknown_ucast: 0
     p0_ale_unknown_ucast_bytes: 0
     p0_ale_unknown_mcast: 0
     p0_ale_unknown_mcast_bytes: 0
     p0_ale_unknown_bcast: 0
     p0_ale_unknown_bcast_bytes: 0
     p0_ale_pol_match: 0
     p0_ale_pol_match_red: 0
     p0_ale_pol_match_yellow: 0
     p0_ale_mcast_sa_drop: 0
     p0_ale_dual_vlan_drop: 0
     p0_ale_len_err_drop: 0
     p0_ale_ip_next_hdr_drop: 0
     p0_ale_ipv4_frag_drop: 0
     p0_tx_mem_protect_err: 0
     p0_tx_pri0: 11481487
     p0_tx_pri1: 0
     p0_tx_pri2: 0
     p0_tx_pri3: 0
     p0_tx_pri4: 0
     p0_tx_pri5: 0
     p0_tx_pri6: 0
     p0_tx_pri7: 0
     p0_tx_pri0_bcnt: 1886654641
     p0_tx_pri1_bcnt: 0
     p0_tx_pri2_bcnt: 0
     p0_tx_pri3_bcnt: 0
     p0_tx_pri4_bcnt: 0
     p0_tx_pri5_bcnt: 0
     p0_tx_pri6_bcnt: 0
     p0_tx_pri7_bcnt: 0
     p0_tx_pri0_drop: 0
     p0_tx_pri1_drop: 0
     p0_tx_pri2_drop: 0
     p0_tx_pri3_drop: 0
     p0_tx_pri4_drop: 0
     p0_tx_pri5_drop: 0
     p0_tx_pri6_drop: 0
     p0_tx_pri7_drop: 0
     p0_tx_pri0_drop_bcnt: 0
     p0_tx_pri1_drop_bcnt: 0
     p0_tx_pri2_drop_bcnt: 0
     p0_tx_pri3_drop_bcnt: 0
     p0_tx_pri4_drop_bcnt: 0
     p0_tx_pri5_drop_bcnt: 0
     p0_tx_pri6_drop_bcnt: 0
     p0_tx_pri7_drop_bcnt: 0
     rx_good_frames: 11481496
     rx_broadcast_frames: 197
     rx_multicast_frames: 173
     rx_pause_frames: 0
     rx_crc_errors: 0
     rx_align_code_errors: 0
     rx_oversized_frames: 0
     rx_jabber_frames: 0
     rx_undersized_frames: 0
     rx_fragments: 0
     ale_drop: 9
     ale_overrun_drop: 0
     rx_octets: 1886655307
     tx_good_frames: 4126229
     tx_broadcast_frames: 13
     tx_multicast_frames: 41
     tx_pause_frames: 0
     tx_deferred_frames: 0
     tx_collision_frames: 0
     tx_single_coll_frames: 0
     tx_mult_coll_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     rx_ipg_error: 0
     tx_carrier_sense_errors: 0
     tx_octets: 1400595653
     tx_64B_frames: 1656
     tx_65_to_127B_frames: 2225399
     tx_128_to_255B_frames: 403
     tx_256_to_511B_frames: 77
     tx_512_to_1023B_frames: 29
     tx_1024B_frames: 13380161
     net_octets: 3287250960
     rx_bottom_fifo_drop: 0
     rx_port_mask_drop: 9
     rx_top_fifo_drop: 0
     ale_rate_limit_drop: 0
     ale_vid_ingress_drop: 0
     ale_da_eq_sa_drop: 0
     ale_block_drop: 0
     ale_secure_drop: 0
     ale_auth_drop: 0
     ale_unknown_ucast: 11481126
     ale_unknown_ucast_bytes: 1886621847
     ale_unknown_mcast: 173
     ale_unknown_mcast_bytes: 16300
     ale_unknown_bcast: 197
     ale_unknown_bcast_bytes: 17160
     ale_pol_match: 0
     ale_pol_match_red: 0
     ale_pol_match_yellow: 0
     ale_mcast_sa_drop: 0
     ale_dual_vlan_drop: 0
     ale_len_err_drop: 0
     ale_ip_next_hdr_drop: 0
     ale_ipv4_frag_drop: 0
     iet_rx_assembly_err: 0
     iet_rx_assembly_ok: 0
     iet_rx_smd_err: 0
     iet_rx_frag: 0
     iet_tx_hold: 0
     iet_tx_frag: 0
     tx_mem_protect_err: 0
     tx_pri0: 4126229
     tx_pri1: 0
     tx_pri2: 0
     tx_pri3: 0
     tx_pri4: 0
     tx_pri5: 0
     tx_pri6: 0
     tx_pri7: 0
     tx_pri0_bcnt: 1400595653
     tx_pri1_bcnt: 0
     tx_pri2_bcnt: 0
     tx_pri3_bcnt: 0
     tx_pri4_bcnt: 0
     tx_pri5_bcnt: 0
     tx_pri6_bcnt: 0
     tx_pri7_bcnt: 0
     tx_pri0_drop: 0
     tx_pri1_drop: 0
     tx_pri2_drop: 0
     tx_pri3_drop: 0
     tx_pri4_drop: 0
     tx_pri5_drop: 0
     tx_pri6_drop: 0
     tx_pri7_drop: 0
     tx_pri0_drop_bcnt: 0
     tx_pri1_drop_bcnt: 0
     tx_pri2_drop_bcnt: 0
     tx_pri3_drop_bcnt: 0
     tx_pri4_drop_bcnt: 0
     tx_pri5_drop_bcnt: 0
     tx_pri6_drop_bcnt: 0
     tx_pri7_drop_bcnt: 0

    Colclusion

    1. No frames drop&error on both DUT/server side.

    2. Try to change iperf3 PID priority (priority=99,SCHED_FIFO/SCHED_RR) on both DUT/server side, I think no obviously improve for sender/receiver bandwidth race condition.

    3. Does it have another factor to effect sender/receiver speed? (Mb/s)

    [ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-20.00  sec   857 MBytes   359 Mbits/sec    0             sender
[  5][TX-C]   0.00-20.04  sec   855 MBytes   358 Mbits/sec                  receiver
[  7][RX-C]   0.00-20.00  sec  2.17 GBytes   932 Mbits/sec    1             sender
[  7][RX-C]   0.00-20.04  sec  2.17 GBytes   929 Mbits/sec                  receiver

    Many Thanks

    Gibbs

  • 0 in reply to Gibbs Shih
    TI__Genius 16435 points

    Hello Gibbs,

    Thanks for sharing these details. Apologies for the delay in response as I was out of office the last few days.

    From my current understanding, this low throughput of around ~350Mbps only shows on DUT but not the EVM?

    Additionally, can you share what SDK version are you using? Did you use the defconfig from the TI SDK? Where did you obtain the SDK (Ti.com or mainline Linux kernel)?

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Expert 5335 points

    Hi, Daolin

    Thanks your replies, let's me update debug status.

    (1) Low throughput of around ~350Mbps only shows on customer's DUT.

    (2) Customer's SDK is : ti-processor-sdk-linux-rt-am62xx-evm-09.00.00.03

    After few days debug, we found key clue.

    It seems that if you enable "CONFIG_PREEMPT_RT" in kernel, it will happen low throughput (~350Mbps) on TX side. 

    config file as attachment. 

    Is it normal status when RT enables? Why does it happen? and How to solve it?

    Many Thanks

    Gibbs

  • 0 in reply to Gibbs Shih
    TI__Genius 16435 points

    Hi Gibbs,

    Thanks for sharing your findings.

    I'm assuming the config file containing the "CONFIG_PREEMPT_RT=y" you shared are generated from the default defconfig provided in the ti-processor-sdk-linux-rt-am62xx-evm-09.00.00.03 SDK? 

    Have you observed the low throughput ~350Mbps on TI EVM when the config contains "CONFIG_PREEMPT_RT" on TX side? 

    From my knowledge, there shouldn't be a lower throughput performance resulting from when RT is enabled, at least on an TI EVM. Below is the CPSW throughput I saw on SK-AM62B EVM running ti-processor-sdk-linux-rt-am62xx-evm-09.02.01.09. I currently do not have a ti-processor-sdk-linux-rt-am62xx-evm-09.00.00.03 set up. 

    First section is when Linux Host PC is acting as client/sending TCP packets to EVM, second section is when AM64x EVM console is acting as client/sending TCP packets to Linux Host PC.
 
 
 
// Linux Host PC console - with CPSW ethernet
a0500327@a0500327ws:~$ iperf3 -c 172.168.1.175
Connecting to host 172.168.1.175, port 5201
[  5] local 172.168.1.1 port 39170 connected to 172.168.1.175 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   953 Mbits/sec    0    467 KBytes       
[  5]   1.00-2.00   sec   109 MBytes   911 Mbits/sec    0    694 KBytes       
[  5]   2.00-3.00   sec   109 MBytes   912 Mbits/sec    0    740 KBytes       
[  5]   3.00-4.00   sec   108 MBytes   902 Mbits/sec  163    687 KBytes       
[  5]   4.00-5.00   sec   109 MBytes   912 Mbits/sec   34    542 KBytes       
[  5]   5.00-6.00   sec   108 MBytes   902 Mbits/sec    0    588 KBytes       
[  5]   6.00-7.00   sec   108 MBytes   902 Mbits/sec    0    625 KBytes       
[  5]   7.00-8.00   sec   108 MBytes   902 Mbits/sec    0    645 KBytes       
[  5]   8.00-9.00   sec   108 MBytes   902 Mbits/sec    0    653 KBytes       
[  5]   9.00-10.00  sec   109 MBytes   912 Mbits/sec    0    658 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.06 GBytes   911 Mbits/sec  197             sender
[  5]   0.00-10.00  sec  1.06 GBytes   908 Mbits/sec                  receiveriperf Done.
 
// AM64x EVM console - CPSW ethernet
root@am64xx-evm:~# iperf3 -c 172.168.1.1
Connecting to host 172.168.1.1, port 5201
[  5] local 172.168.1.175 port 40270 connected to 172.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   110 MBytes   925 Mbits/sec    0    410 KBytes       
[  5]   1.00-2.00   sec   108 MBytes   903 Mbits/sec    0    428 KBytes       
[  5]   2.00-3.00   sec   108 MBytes   909 Mbits/sec    0    428 KBytes       
[  5]   3.00-4.00   sec   108 MBytes   905 Mbits/sec    0    428 KBytes       
[  5]   4.00-5.00   sec   110 MBytes   926 Mbits/sec    0    428 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   939 Mbits/sec    0    428 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   935 Mbits/sec    0    428 KBytes       
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec    0    484 KBytes       
[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec    0    484 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   944 Mbits/sec    0    625 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   925 Mbits/sec    0             sender
[  5]   0.00-10.04  sec  1.07 GBytes   919 Mbits/sec                  receiveriperf Done.

    Is it possible to try out ti-processor-sdk-linux-rt-am62xx-evm-09.02.01.09 on the customer's DUT, just to see if a newer SDK might show improvements in the performance?

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Expert 5335 points

    Hi, Daolin

    Thanks your comment.

    Few questions & status update

    (1) I found your test is single direction, not bi-direction. single direction are fine on DUT, issue only happens on bi-direction. sample command like below

     iperf3 -c 192.168.10.99 -i 4 -t 20 -b 0 --bidir

    (2) Share you some test result from customer's feedback. Because of authorize concern, I will send this data to your mail. It seems that "RT enable" issue only happens on DUT, not on EVM.

    Gibbs

  • 0 in reply to Gibbs Shih
    TI__Genius 16435 points

    Hello Gibbs,

    Thanks for sharing the test results from the customer, I have a couple of questions/comments regarding those results that I will ask in email.

    You mentioned "DUT OS : regular Linux without RT" in one of your earlier responses but in more recent responses you mentioned "(2) Customer's SDK is : ti-processor-sdk-linux-rt-am62xx-evm-09.00.00.03". Just to clarify, the target OS is RT-Linux not "regular" Linux (no PREEMPT_RT config)? What is the customer's use-case (is there a particular reason they might require RT-Linux)?

    Yes, the results I showed previously were single direction tests in both RX and TX path (from EVM point of view), these were tests I ran several months back so I did not have a readily available bidirectional test result to share).

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Expert 5335 points

    Hi, Daolin

    Problem should be solved.

    This is could be IRQ over loading for a specific A-core. try to check IRQ status for 8000000.ethernet-tx0 & 8000000.ethernet when iperf running.

    Ref this thread, try to modify IRQ & A-Core service mapping.

    (1) https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1342656/sk-am62a-lp-frame-drop-often-happens-on-am62ax-platform/5161884?tisearch=e2e-sitesearch&keymatch=irqbalance#5161884

    (2) https://blog.csdn.net/ye1223/article/details/85050498

    Gibbs

  • 0 in reply to Gibbs Shih
    TI__Genius 16435 points

    Hello Gibbs,

    Thanks for sharing this update. Just to clarify the solution for this issue, making sure the two IRQs (8000000.ethernet-tx0 and 8000000.ethernet) are running on the same A53 core, which seems to be the default setting on the default SDK on the EVM, based on your statement below from your email, will not result in the lower throughput rate you indicated before?

    "Root cause :

    They try to separate “8000000.ethernet-tx0” & “8000000.ethernet” irq to different core

    Default SDK on EVM already separate IRQ (117/207), but I am not sure why they use the same core."

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Expert 5335 points

    Hi, Daolin

    Question 1 : 

    I though “8000000.ethernet-tx0” & “8000000.ethernet” use different irq, one is 117, another is 207. Am I miss understanding?

    Question 2 :

    How to know IRQs (8000000.ethernet-tx0 and 8000000.ethernet) works on the same core? And how to separate IRQs to different core?

    Many Thanks

    Gibbs

  • 0 in reply to Gibbs Shih
    TI__Genius 16435 points

    Hello Gibbs,

    For these questions I will need to check with the development team and a colleague. I hope to get back with an answer tomorrow.

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Genius 16435 points

    Hi Gibbs,

    Additionally, after speaking to a colleague we want to double check the following:

    Can you clarify the below question? Did you/customer manually enable the CONFIG_PREEMPT_RT=y? The reason why we are asking is because CONFIG_PREEMPT_RT=y should already be set with the ti-processor-sdk-linux-rt-am62xx-evm-09.00.00.03 SDK. The defconfig provided in this SDK has already been tuned for the best performance so changing the config beyond it may introduce worse performance.

    Daolin Qiu said:
    I'm assuming the config file containing the "CONFIG_PREEMPT_RT=y" you shared are generated from the default defconfig provided in the ti-processor-sdk-linux-rt-am62xx-evm-09.00.00.03 SDK? 

    In general, we don't change the IRQs to specific cores to improve throughput performance, that is why we want to check again on the defconfig.

    -Daolin

  • 0 in reply to Daolin Qiu
    TI__Expert 5335 points

    Hi, Daolin

    1)  "CONFIG_PREEMPT_RT=y" is not root cause, because they try to porting "ti-processor-sdk-linux-am62xx-evm-09.00.00.03-Linux-x86-Install.bin" and "ti-processor-sdk-linux-rt-am62xx-evm-09.00.00.03-Linux-x86-Install.bin" kernel on DUT & use default kernel config, issue is still happens. No matter RT enable or not, issue is still happen.

    2) I try to use  "ti-processor-sdk-linux-am62xx-evm-09.00.00.03-Linux-x86-Install.bin" and "processor-sdk-linux-rt-am62xx-evm-09.00.00.03-Linux-x86-Install.bin" on EVB with default kernel setting & rootfs, Issue does not happen.

    2) Because item 1&2 conclusion, they try to debug rootfs . I share IRQ related suggestion from Varis,Pekka, I believe they have some idea, so they try to study IRQ.

    Thanks

    Gibbs

  • 0 in reply to Gibbs Shih
    TI__Genius 16435 points

    Hello Gibbs,

    Thanks for clarifying the CONFIG_PREEMPT_RT configuration. I've spoken to the CPSW ethernet developer and here is the feedback:

    Gibbs Shih said:

    Question 1 : 

    I though “8000000.ethernet-tx0” & “8000000.ethernet” use different irq, one is 117, another is 207. Am I miss understanding?

    These are indeed two separate IRQ, your understanding is correct.

    • 8000000.ethernet-tx0 is the ethernet transmit IRQ (software places packets to be transmitted out on the DMA channels to get passed to the host port)
    • 8000000.ethernet is the ethernet receive IRQ (software receives packets from host port)
    Gibbs Shih said:

    Question 2 :

    How to know IRQs (8000000.ethernet-tx0 and 8000000.ethernet) works on the same core? And how to separate IRQs to different core?

    This question is a bit confusing as which cores the two ethernet IRQs are run on are already indicated by your screenshot results from /proc/interrupts. For that specific screenshot case, there were 14 8000000.ethernet-tx0 interrupts on core 0 and 419 8000000.ethernet-tx0 interrupts on core 2, there were 6 8000000.ethernet interrupts on core 0 and 149 8000000.ethernet interrupts on core 1.

    Which core each IRQ runs on are typically dependent on the irqbalance daemon which tries to keep the number of interrupts equal on all cores. If you wanted to have full control over setting the cpu affinity of the interrupts, you would have to disable the irqbalance daemon and mask the interrupts to different cores.

    Daolin Qiu said:

    "Root cause :

    They try to separate “8000000.ethernet-tx0” & “8000000.ethernet” irq to different core

    Default SDK on EVM already separate IRQ (117/207), but I am not sure why they use the same core."

    I want to clarify my understanding of your response previously about the root cause of the lower throughput rate. You are saying you are not sure why the customer's platform is setting these two IRQs to run on the same core and that when they are on the same core, the throughput rate is lower?

    Can you check if they have an irqbalance daemon running in their root filesystem? The suspicion is if irqbalance is disabled, perhaps that is why both IRQs are running on the same core.

    -Daolin