Other Parts Discussed in Thread: SK-AM62B
Tool/software:
Hi, Dear Expert
Customer's board has lower TX (client -> server) ethernet bandwidth, so I try to help them identify root cause.
First step, I try to base on software point of view to debug this issue.
Here is my netperf script,
#!/bin/bash for i in 5 do netperf -H 192.168.10.99 -j -c -l 60 -p 1203 -t TCP_STREAM # -k DIRECTION,THROUGHPUT,MEAN_LATENCY,LOCAL_CPU_UTIL,REMOTE_CPU_UTIL,LOCAL_BYTES_SENT,REMOTE_BYTES_RECVD,LOCAL_SEND_SIZE & netperf -H 192.168.10.99 -j -c -l 60 -p 1203 -t TCP_MAERTS -- # -k DIRECTION,THROUGHPUT,MEAN_LATENCY,LOCAL_CPU_UTIL,REMOTE_CPU_UTIL,LOCAL_BYTES_SENT,REMOTE_BYTES_RECVD,LOCAL_SEND_SIZE & done
* [TEST 1] : Run netperf on customer board, result as below.
root@am62xx-evm:~# ./netperf-client.sh
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % ? us/KB us/KB
131072 16384 16384 60.03 862.70 28.28 -1.00 10.742 -1.000
MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Recv Send Recv Send
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % ? us/KB us/KB
131072 16384 16384 60.00 941.41 34.34 -1.00 11.954 -1.000
We found the bandwidth from client to server (TX) only 862.70Gbps, and local CPU utilization around 28.28%
* Dump CPU utilization before netperf running on customer's board.
00:12:28 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
00:12:29 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
00:12:29 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
00:12:29 1 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
00:12:29 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
00:12:29 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
* Dump CPU utilization when netperf is running on customer's board..
00:22:13 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
00:22:14 all 0.51 0.00 13.37 0.00 2.83 18.51 0.00 0.00 64.78
00:22:14 0 0.00 0.00 17.17 0.00 10.10 72.73 0.00 0.00 0.00
00:22:14 1 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 99.00
00:22:14 2 2.22 0.00 26.67 0.00 1.11 0.00 0.00 0.00 70.00
00:22:14 3 0.00 0.00 9.09 0.00 0.00 0.00 0.00 0.00 90.91
* [TEST 2] : Run netperf on AM62 EVB, result as below.
root@am62xx-evm:~# ./netperf-client.sh
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % ? us/KB us/KB
131072 16384 16384 60.02 934.94 42.89 -1.00 15.032 -1.000
MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.10.99 () port 0 AF_INET : histogram : interval
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Recv Send Recv Send
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % ? us/KB us/KB
131072 16384 16384 60.00 941.45 51.28 -1.00 17.847 -1.000
We found AM62 EVB has better ethernet bandwidth performance (934~941Gbps) and higher CPU utilization (42%~51%)
* Dump CPU utilization before netperf running on EVB.
01:13:18 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
01:13:19 all 0.75 0.00 2.51 0.00 0.25 0.25 0.00 0.00 96.24
01:13:19 0 1.01 0.00 5.05 0.00 0.00 0.00 0.00 0.00 93.94
01:13:19 1 2.97 0.00 3.96 0.00 0.99 0.00 0.00 0.00 92.08
01:13:19 2 0.99 0.00 0.99 0.00 0.00 0.99 0.00 0.00 97.03
01:13:19 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
* Dump CPU utilization when netperf is running on EVB.
01:17:10 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
01:17:11 all 0.80 0.00 18.50 0.00 6.43 24.66 0.00 0.00 49.60
01:17:11 0 1.15 0.00 36.78 0.00 4.60 0.00 0.00 0.00 57.47
01:17:11 1 0.00 0.00 0.00 0.00 11.96 88.04 0.00 0.00 0.00
01:17:11 2 1.01 0.00 9.09 0.00 6.06 10.10 0.00 0.00 73.74
01:17:11 3 1.06 0.00 29.79 0.00 3.19 0.00 0.00 0.00 65.96
Base on test result, I think CPU utilization and performance should be one of factor to effect ethernet bandwidth.
Our EVB has higher CPU utilization for netpref, so we get better performance on ethernet.
I try to adjust "Interrupt Pacing", but it does not work for me on customer board.
higher delay period get worst ethernet bandwidth, default value "20" is better.
ethtool -C eth0 rx-usecs <delayperiod>
ethtool -C socnet0 tx-usecs 20
ethtool -C socnet0 rx-usecs 20
Do we have another software way to improve ethernet performance?
Need more suggestion.
Many Thanks
Gibbs