TDA4VM: TDA4VM CPSW9G: UDP large-package loss

Dear Experts,

     This is similiar to this:https://e2e.ti.com/support/switches-multiplexers-group/switches-multiplexers/f/switches-multiplexers-forum/1169821/tda4-vm-switch-udp-package-loss 

     After talking with customer they find this case will happen in both QNX7.3 and Linux 8.2. When trying to reproduce it we need to add -l 6400(large packet) and the two iperf3 process should run at almost the same time.

     I am trying to reproduce it on our EVM board tomorrow, if you have encountered such case before please help give us some advice thanks.

BR

Sikai

tcpdump.log.zip

The packages from PC and ECU1,ECU2

Dear Experts and Zhongxin,

     I have tried to set this enviroment on our side with 2 TDA4VM EVM boards in SDK 8.2. Since customer is using SDK7.3, I think swtich port or MAC-only port have not been added in SDK yet.

     Here is my test environment, I connect TDA4_1 and TDA4_2, TDA4_1 is connected to PC through an external swtich:

     Before doing the same test I have verified that TDA4_1 and TDA4_2 can ping PC individually. Also they can ping each other successfully.

     Then I do the iperf3 test and the two process run within an interval of 1s:

     Here are the logs and I think I do not see great packet loss here. Zhongxin could you help check if I set the environment correctly?

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/TDA4_5F00_1https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/TDA4_5F00_2

BR

Sikai

Hi Sikai,

The ENV is ok, maybe you can try changing the option to "-l 64000" and "-t 60".

if you still don't see great packet loss, you can do the following:

1. keep the iperf3 stream of TDA4_1

2. start the iperf3 stream of TDA4_2 several seconds, and then stop the  iperf3 stream of TDA4_2

3. Retry step 2 several times

Dear Zhongxin,

     1. I tried "-l 64000" and "-t 60", still not packet loss:

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/2110.TDA4_5F00_1  https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/6378.TDA4_5F00_2

    2. I tried to repeat step 2 for over 10 times and the whole process twice. I can see a few packet loss on server side, however it only exits for several seconds and then the communication remain well. I think this is not the same case as yours right?

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/TDA4_5F00_1_5F00_1https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/TDA4_5F00_2_5F00_2

BR

Sikai     

Dear Experts,

     I have reproduced customer's case on our EVM board with SDK 8.2.

     On TDA4_1: iperf3 -c -u PC_IP -p 5201 -b 100M -t 6000 -l 64000 PC: iperf3 -s -p 5201

     On TDA4_2: iperf3 -c -u PC_IP -p 5301 -b 100M -t 60 -l 64000 PC: iperf3 -s -p 5301

     From the log you can see I have kept TDA4_1 stream and it has high possiblity that when I start another stream of TDA4_2, we lose the UDP transition.

     Could you help anaylze it for us? Thanks.

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/1207.TDA4_5F00_1_5F00_1https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/1738.TDA4_5F00_2_5F00_2

BR

Sikai

udp.issue.log.tar.gz

Hi Sikai,

The attachment is the log info of ECU1 and ECU2

testcase1_ALE.log: the ALE info before iperf3 test

testcase1_PORT.log: the CPSW switch port stats info before iperf3 test

ecux.iperf3.test.log: the iperf3 test result

testcase2_ALE.log: the ALE info after iperf3 test

testcase2_PORT.log: the CPSW switch port stats info after iperf3 test

Dear Zhongxin,

     I see many ports on ECU1 and ECU2, could you help explain how do these ports work and how many ports show packet loss when this case reproduces?

     I know that your script only print registers that not equals 0, however I also see may "#" character in your log. What does it mean? 

BR

Sikai

Sikai,

The test env is: 

On ECU2, the port 6 is not connected(the ECU2 code was ported from ECU1, some configuration is same as ECU1).

From the CPSW port stats, i don't find great packet loss information when this case reproduces.

The code of this port stats was ported from  cpsw_stats_print_regs.gel;

string "#15" means "\n" (The log output converts "\n" to "#15")

udp.issue.all.regs.iperf3.log.tar.gz

Hi Sikai,

The attachment is the log info of ECU1 and ECU2 with all cpsw stats register value

testcase1_ALE.log: the ALE info before iperf3 test

testcase1_PORT.log: the CPSW switch port stats info before iperf3 test

ecux.iperf3.log: the iperf3 test result

testcase2_ALE.log: the ALE info after iperf3 test

testcase2_PORT.log: the CPSW switch port stats info after iperf3 test

Dear Zhongxin,

     1. Could you try to use iperf rather than iperf3?

     2. As we talked before, this only happens while transferring large package. The TX FIFO Queue size is 2K Bytes. Therefore, when large number of packets need to be forwarded, it is possible that the packets are dropped due to exhaustion of the Queue. This might be a possible cause for the issue. An alternative would be to try using VLAN to assign packet priority to the traffic from ECU2. With this, ECU1's traffic to PC will be sent on TX Queue 0 while ECU2's traffic will be forwarded to PC on TX Queue 1 (If VLAN priority is 1). Or you can try to enlarge the fifo size to see if this could be improved. Please pay attention that when you enlarge the fifo size here you need to reduce the memory size somewhere else.

     3. I still see "#" characters in your log. Could you directly read the value of 

udp.issue.all.regs.iperf3.log.new.tar.gz

Hi Sikai,

1. I will upload the iperf test result after finished the iperf test. 

2. I will try using VLAN to assign packet priority. if you have examples, can you provide them to us ?

3. Update the cpsw stats information, RX_BOTTOM_OF_FIFO_DROP/RX_TOP_OF_FIFO_DROP/ALE_OVERRUN_DROP value can be found in files testcaseX_PORT.log

Dear Zhongxin,

     We have tried some tests:

     1. PC be the server and ECU1/ECU2 be the client. Able to reproduce your issue.

     2. ECU2 be the server and PC/ECU1 be the client. Could see packet loss immediately.

     3. ECU1 be the server and PC/ECU2 be the client. Did not reproduce and not see packet loss.

         3.1 Increase the traffic from PC to ECU1, we add one more port 5401 and increase the bandwidth to 60M( with port 5201 30M, the total bandwidth from PC would be 90M). We still not see packet loss.

     In conclusion, the packet loss only be observed when packet is forwarded by ECU1. In this case, something wrong with ECU1:

     1. It may be caused by forwarding packet overflows the CPSW TX FIFO queue in hardware.

     2. Or it may be caused by ethfw not able to process the forward packet fast enough. Then ethfw may drop the packet .

     Will confirm internally and get back to you later.

BR

Sikai 

Dear Zhongxin,

     Could you do a test that when running the iperf test on ECU1 and ECU2, please use the command 

     iperf3 -c PC_IP -p 5301 -b 30M -t 60 -u -l 6400 -S 0xB8

     on ECU2 and keep the ECU1 command as before.

     And could you help make a comparison with the new test result with the former one to see if it is improved?

BR

Sikai

udp.issue.iperf3.log.20230203.171003.tar.gz

Hi Sikai,

The attachment is the iperf3 test result.

From the test result, We didn't see a significant improvement.

Dear Zhongxin,

    We run a test with PC as server and ECU1/ECU2 as client.

    You can see the PC is dropping the packets.

    Could you help check on your side with the same environment:

    Running "tcpdump -i if_name" before dropping the packets and stop it after you see the huge packet drop. 

    After that please check the amount of "packets dropped by kernel". 

BR

Sikai

iperf3.tcpdump.case.tar.gz

Hi Sikai,

We run 2 test with PC as server and ECU1/ECU2 as client.

Case 1: reboot and prepare

Case 2: PC as iperf3 server, ECU1 as iperf3 client, run tcpdump on PC

               pc.case2.tcpdump.log: tcpdump log on PC

               ecu1.case2.iperf3.test.log: ECU1 iperf3 client log

Case 3: PC as iperf3 server, ECU1 and ECU2 as iperf3 client, run tcpdump on PC

               pc.case3.tcpdump.log: tcpdump log on PC

               ecu1.case3.iperf3.test.log: ECU1 iperf3 client log

               ecu2.case3.iperf3.test.log:  ECu2 iperf3 client log

From the case3 iperf3 client test log, we can see packet loss:

but the log of tcpdump on PC show no kernel drop:

Dear Experts,

     It is very strange that:

    1. In case 2, they use only ECU1 and PC, they could see packet drop in TCPDUMP while no loss in iperf.

     But the number is quite small and as customer said even they connect two Laptop together and run iperf test they could also see small packet drop. So I think we can ignore this.

    2. While in case 3, they use ECU1 and ECU2. They could see drop on iperf but no in TCPDUMP.

   This is different from our test. It seems our board is dropping packet right?

BR

Sikai

Dear Siddharth,

     After confirming with customer here are some updates:

     1. They could run SDK 8.2 Linux to test.

     2. For ARP Flood, other cores may also running some processes. It is hard to cancel all those tasks but they will try.

     3. For gateway, they also cannot make sure gateway does not drop packet. They employ gateway to transfer the hardware line of ethernet from industrial to automotive. But they told me that they can find a single transceiver to avoid that.

     4.  They also run same test between to laptop, here is their test environment

         We can see from the test vedio that there are huge packet drops in tcpdump while iperf is running well:

BR

Sikai

Dear Siddharth,

    Here is customer's test today, they use SDK 8.2 Linux, PC ---- Transceiver(100M) ----- ECU1 (TC397) ------ ECU2

    1. TC397 has some background noise, to test that we run a test at the beginning of the vedio. No iperf3 process was running and we can see 600+ packets captured by  TCPDUMP with 0 packet drop. So we think this background task will not affect the number of packet drop.

    2. After reproducing the error and check the TCPDUMP, we can see 700+ packets dropped out of 65000+ captured.

iperf3 -c 192.168.3.99 -p 5201 -b 30M -t 20 -u -l 64000 -T Linux-ECU1 --get-server-output
Linux-ECU1:  Connecting to host 192.168.3.99, port 5201
Linux-ECU1:  [  5] local 192.168.3.4 port 41636 connected to 192.168.3.99 port 5201
Linux-ECU1:  [ ID] Interval           Transfer     Bitrate         Total Datagrams
Linux-ECU1:  [  5]   0.00-1.81   sec  62.5 KBytes   284 Kbits/sec  1  
Linux-ECU1:  [  5]   1.81-2.00   sec  2.26 MBytes  97.6 Mbits/sec  37  
Linux-ECU1:  [  5]   2.00-3.00   sec  8.42 MBytes  70.7 Mbits/sec  138  
Linux-ECU1:  [  5]   3.00-4.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]   4.00-5.00   sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU1:  [  5]   5.00-6.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]   6.00-7.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]   7.00-8.00   sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU1:  [  5]   8.00-9.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]   9.00-10.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU1:  [  5]  10.00-11.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]  11.00-12.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]  12.00-13.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU1:  [  5]  13.00-14.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]  14.00-15.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU1:  [  5]  15.00-16.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]  16.00-17.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]  17.00-18.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU1:  [  5]  18.00-19.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU1:  [  5]  19.00-20.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU1:  - - - - - - - - - - - - - - - - - - - - - - - - -
Linux-ECU1:  [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
Linux-ECU1:  [  5]   0.00-20.00  sec  71.5 MBytes  30.0 Mbits/sec  0.000 ms  0/1172 (0%)  sender
Linux-ECU1:  [  5]   0.00-20.00  sec  18.9 MBytes  7.94 Mbits/sec  228.794 ms  19/329 (5.8%)  receiver
Linux-ECU1:  
Server output:
Accepted connection from 192.168.3.4, port 54494
[  5] local 192.168.3.99 port 5201 connected to 192.168.3.4 port 41636
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)  
[  5]   1.00-2.00   sec  2.14 MBytes  17.9 Mbits/sec  11672251871.845 ms  1/36 (2.8%)  
[  5]   2.00-3.00   sec  7.45 MBytes  62.5 Mbits/sec  4442734.811 ms  18/140 (13%)  
[  5]   3.00-4.00   sec  3.54 MBytes  29.7 Mbits/sec  105192.526 ms  0/58 (0%)  
[  5]   4.00-5.00   sec  3.60 MBytes  30.2 Mbits/sec  2335.135 ms  0/59 (0%)  
[  5]   5.00-6.00   sec  2.20 MBytes  18.4 Mbits/sec  228.794 ms  0/36 (0%)  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  11.00-12.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  15.00-16.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  17.00-18.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  18.00-19.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  19.00-20.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
[  5]  20.00-20.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  0/0 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-20.00  sec  0.00 Bytes  0.00 bits/sec  228.794 ms  19/329 (5.8%)  

Linux-ECU1:  
Linux-ECU1:  iperf Done.

iperf3 -c 192.168.3.99 -p 5301 -b 30M -t 20 -u -l 64000 -T Linux-ECU2 --get-server-output
Linux-ECU2:  Connecting to host 192.168.3.99, port 5301
Linux-ECU2:  [  5] local 192.168.3.5 port 39688 connected to 192.168.3.99 port 5301
Linux-ECU2:  [ ID] Interval           Transfer     Bitrate         Total Datagrams
Linux-ECU2:  [  5]   0.00-1.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]   1.00-2.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]   2.00-3.00   sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  [  5]   3.00-4.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]   4.00-5.00   sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  [  5]   5.00-6.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]   6.00-7.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]   7.00-8.00   sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  [  5]   8.00-9.00   sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]   9.00-10.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  [  5]  10.00-11.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]  11.00-12.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]  12.00-13.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  [  5]  13.00-14.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]  14.00-15.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  [  5]  15.00-16.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]  16.00-17.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]  17.00-18.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  [  5]  18.00-19.00  sec  3.60 MBytes  30.2 Mbits/sec  59  
Linux-ECU2:  [  5]  19.00-20.00  sec  3.54 MBytes  29.7 Mbits/sec  58  
Linux-ECU2:  - - - - - - - - - - - - - - - - - - - - - - - - -
Linux-ECU2:  [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
Linux-ECU2:  [  5]   0.00-20.00  sec  71.5 MBytes  30.0 Mbits/sec  0.000 ms  0/1172 (0%)  sender
Linux-ECU2:  [  5]   0.00-20.00  sec  6.47 MBytes  2.71 Mbits/sec  119430506.756 ms  0/106 (0%)  receiver
Linux-ECU2:  
Server output:
Accepted connection from 192.168.3.5, port 51714
[  5] local 192.168.3.99 port 5301 connected to 192.168.3.5 port 39688
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  3.60 MBytes  30.2 Mbits/sec  2480077680.504 ms  0/59 (0%)  
[  5]   1.00-2.00   sec  2.87 MBytes  24.1 Mbits/sec  119430506.756 ms  0/47 (0%)  
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  10.00-11.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  11.00-12.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  14.00-15.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  15.00-16.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  17.00-18.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  18.00-19.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  19.00-20.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
[  5]  20.00-20.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/0 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-20.00  sec  0.00 Bytes  0.00 bits/sec  119430506.756 ms  0/106 (0%)  

Linux-ECU2:  
Linux-ECU2:  iperf Done.

BR

Sikai

Hi Sikai,

I see that there were email threads on this as well. I am still catching-up to things. Can you let me know the current status for this. We can also have a call for this Tuesday morning if you prefer.

Regards,
Tanmay