This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PROCESSOR-SDK-AM64X: Processor SDK RT-Linux for AM64x v08.06.00.42: dropped packets on SW HSR on CPSW with iperf3

Part Number: PROCESSOR-SDK-AM64X

Hi,

 

My setup is next:

  • Board is AM64x-SK
  • Processor SDK RT-Linux for AM64x v08.06.00.42:
  • kernel is - Linux am64xx-evm 5.10.168-rt83-gc1a1291911 #1 SMP PREEMPT_RT Mon Feb 27 14:03:20 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
  • Image is from RT SDK release AM64x v08.06.00.42 all default from SDK but as board is GP board, tiboot3.bin is used for GP board  - Image name is tisdk-default-image-am64xx-evm.wic.xz
  • Two AM64x-SK are connected lwith etherent cables ike this

 

eth0 ------- eth0

eth1 ------- eth1

 

I have some questions about iperf results with SW HSR from linux kernel on CPSW switch.

 

Tests without HSR:

Client board setup and commands:

 

# ip addr add 192.168.2.20/24 dev eth0

# iperf3 -c 192.168.2.30  -i 10 -b 50M -t 50000 --bidir

[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd

[  5][TX-C]   0.00-10.00  sec  59.6 MBytes  50.0 Mbits/sec    0    314 KBytes      

[  7][RX-C]   0.00-10.00  sec  59.8 MBytes  50.2 Mbits/sec                 

               

 

Server board setupt and commands:

# ip addr add 192.168.2.30/24 dev eth0

# iperf3 -s

[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd

[  5][RX-S]   9.00-10.00  sec  6.00 MBytes  50.3 Mbits/sec                 

[  8][TX-S]   9.00-10.00  sec  6.00 MBytes  50.3 Mbits/sec    0    134 KBytes 

 

There is no Retr packges on 50M in this test on server and client side

 

Tests with HSR:

Commands used for setup:

BOTH boards:

ip link set eth0 down

ip link set eth1 down

ON SERVER:

ip link set dev eth0 address 70:FF:76:1C:0E:8F

ip link set dev eth1 address 70:FF:76:1C:0E:8F

ON CLIENT:

ip link set dev eth1 address 70:FF:76:1C:0E:8E

ip link set dev eth0 address 70:FF:76:1C:0E:8E

BOTH boards:

ip link set eth0 up

ip link set eth1 up

ip link add name hsr0 type hsr slave1 eth0 slave2 eth1 supervision 45 version 1

ON SERVER:

ip addr add 192.168.2.30/24 dev hsr0

ON CLIENT:

ip addr add 192.168.2.20/24 dev hsr0

BOTH boards:

ip link set hsr0 up

 

 

results on 25M

Client

# iperf3 -c 192.168.2.30  -i 10 -b 10M -t 50000 --bidir

[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd

[  5][TX-C]  20.00-30.00  sec  29.9 MBytes  25.1 Mbits/sec   26   94.3 KBytes      

[  7][RX-C]  20.00-30.00  sec  29.8 MBytes  25.0 Mbits/sec                 

Server

# iperf3 -s

[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd

[  8][TX-S]  21.00-22.00  sec  3.00 MBytes  25.2 Mbits/sec   29   81.7 KBytes      

[  5][RX-S]  22.00-23.00  sec  3.00 MBytes  25.2 Mbits/sec                 

[  8][TX-S]  22.00-23.00  sec  3.00 MBytes  25.2 Mbits/sec    1   76.0 KBytes      

[  5][RX-S]  23.00-24.00  sec  3.00 MBytes  25.2 Mbits/sec                 

[  8][TX-S]  23.00-24.00  sec  3.00 MBytes  25.2 Mbits/sec   26   76.0 KBytes 

 

As you can see, there are Retr packages... In attachment I am sending ethtool statistics  for all ports.

SLAVE ETH0 STAT:

slave-eth0-hsr-statistics.txt
root@am64xx-evm:~# ethtool -S eth0            
NIC statistics:
     p0_rx_good_frames: 849664
     p0_rx_broadcast_frames: 53
     p0_rx_multicast_frames: 1370
     p0_rx_crc_errors: 0
     p0_rx_oversized_frames: 0
     p0_rx_undersized_frames: 0
     p0_ale_drop: 0
     p0_ale_overrun_drop: 0
     p0_rx_octets: 1158013608
     p0_tx_good_frames: 852462
     p0_tx_broadcast_frames: 52
     p0_tx_multicast_frames: 1337
     p0_tx_octets: 1160616633
     p0_tx_64B_frames: 0
     p0_tx_65_to_127B_frames: 184328
     p0_tx_128_to_255B_frames: 153
     p0_tx_256_to_511B_frames: 101
     p0_tx_512_to_1023B_frames: 8
     p0_tx_1024B_frames: 1517536
     p0_net_octets: 2318630241
     p0_rx_bottom_fifo_drop: 0
     p0_rx_port_mask_drop: 0
     p0_rx_top_fifo_drop: 0
     p0_ale_rate_limit_drop: 0
     p0_ale_vid_ingress_drop: 0
     p0_ale_da_eq_sa_drop: 0
     p0_ale_block_drop: 0
     p0_ale_secure_drop: 0
     p0_ale_auth_drop: 0
     p0_ale_unknown_ucast: 0
     p0_ale_unknown_ucast_bytes: 0
     p0_ale_unknown_mcast: 598
     p0_ale_unknown_mcast_bytes: 44312
     p0_ale_unknown_bcast: 0
     p0_ale_unknown_bcast_bytes: 0
     p0_ale_pol_match: 0
     p0_ale_pol_match_red: 0
     p0_ale_pol_match_yellow: 0
     p0_ale_mcast_sa_drop: 0
     p0_ale_dual_vlan_drop: 0
     p0_ale_len_err_drop: 0
     p0_ale_ip_next_hdr_drop: 0
     p0_ale_ipv4_frag_drop: 0
     p0_tx_mem_protect_err: 0
     p0_tx_pri0: 852462
     p0_tx_pri1: 0
     p0_tx_pri2: 0
     p0_tx_pri3: 0
     p0_tx_pri4: 0
     p0_tx_pri5: 0
     p0_tx_pri6: 0
     p0_tx_pri7: 0
     p0_tx_pri0_bcnt: 1157207105
     p0_tx_pri1_bcnt: 0
     p0_tx_pri2_bcnt: 0
     p0_tx_pri3_bcnt: 0
     p0_tx_pri4_bcnt: 0
     p0_tx_pri5_bcnt: 0
     p0_tx_pri6_bcnt: 0
     p0_tx_pri7_bcnt: 0
     p0_tx_pri0_drop: 0
     p0_tx_pri1_drop: 0
     p0_tx_pri2_drop: 0
     p0_tx_pri3_drop: 0
     p0_tx_pri4_drop: 0
     p0_tx_pri5_drop: 0
     p0_tx_pri6_drop: 0
     p0_tx_pri7_drop: 0
     p0_tx_pri0_drop_bcnt: 0
     p0_tx_pri1_drop_bcnt: 0
     p0_tx_pri2_drop_bcnt: 0
     p0_tx_pri3_drop_bcnt: 0
     p0_tx_pri4_drop_bcnt: 0
     p0_tx_pri5_drop_bcnt: 0
     p0_tx_pri6_drop_bcnt: 0
     p0_tx_pri7_drop_bcnt: 0
     rx_good_frames: 426736
     rx_broadcast_frames: 25
     rx_multicast_frames: 750
     rx_pause_frames: 116814
     rx_crc_errors: 0
     rx_align_code_errors: 0
     rx_oversized_frames: 0
     rx_jabber_frames: 0
     rx_undersized_frames: 0
     rx_fragments: 0
     ale_drop: 84
     ale_overrun_drop: 0
     rx_octets: 579254889
     tx_good_frames: 540476
     tx_broadcast_frames: 26
     tx_multicast_frames: 116328
     tx_pause_frames: 115641
     tx_deferred_frames: 0
     tx_collision_frames: 0
     tx_single_coll_frames: 0
     tx_mult_coll_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     rx_ipg_error: 0
     tx_carrier_sense_errors: 0
     tx_octets: 586406908
     tx_64B_frames: 115641
     tx_65_to_127B_frames: 92235
     tx_128_to_255B_frames: 91
     tx_256_to_511B_frames: 49
     tx_512_to_1023B_frames: 4
     tx_1024B_frames: 759192
     net_octets: 1173137893
     rx_bottom_fifo_drop: 410
     rx_port_mask_drop: 116898
     rx_top_fifo_drop: 0
     ale_rate_limit_drop: 0
     ale_vid_ingress_drop: 0
     ale_da_eq_sa_drop: 0
     ale_block_drop: 0
     ale_secure_drop: 332
     ale_auth_drop: 0
     ale_unknown_ucast: 425551
     ale_unknown_ucast_bytes: 578563846
     ale_unknown_mcast: 419
     ale_unknown_mcast_bytes: 34663
     ale_unknown_bcast: 24
     ale_unknown_bcast_bytes: 8112
     ale_pol_match: 0
     ale_pol_match_red: 0
     ale_pol_match_yellow: 0
     ale_mcast_sa_drop: 0
     ale_dual_vlan_drop: 0
     ale_len_err_drop: 0
     ale_ip_next_hdr_drop: 0
     ale_ipv4_frag_drop: 0
     iet_rx_assembly_err: 0
     iet_rx_assembly_ok: 0
     iet_rx_smd_err: 0
     iet_rx_frag: 0
     iet_tx_hold: 0
     iet_tx_frag: 0
     tx_mem_protect_err: 0
     tx_pri0: 424835
     tx_pri1: 0
     tx_pri2: 0
     tx_pri3: 0
     tx_pri4: 0
     tx_pri5: 0
     tx_pri6: 0
     tx_pri7: 0
     tx_pri0_bcnt: 579007080
     tx_pri1_bcnt: 0
     tx_pri2_bcnt: 0
     tx_pri3_bcnt: 0
     tx_pri4_bcnt: 0
     tx_pri5_bcnt: 0
     tx_pri6_bcnt: 0
     tx_pri7_bcnt: 0
     tx_pri0_drop: 0
     tx_pri1_drop: 0
     tx_pri2_drop: 0
     tx_pri3_drop: 0
     tx_pri4_drop: 0
     tx_pri5_drop: 0
     tx_pri6_drop: 0
     tx_pri7_drop: 0
     tx_pri0_drop_bcnt: 0
     tx_pri1_drop_bcnt: 0
     tx_pri2_drop_bcnt: 0
     tx_pri3_drop_bcnt: 0
     tx_pri4_drop_bcnt: 0
     tx_pri5_drop_bcnt: 0
     tx_pri6_drop_bcnt: 0
     tx_pri7_drop_bcnt: 0

SLAVE ETH1 STAT:

slave-eth1-hsr-statistics.txt
root@am64xx-evm:~# ethtool -S eth1
NIC statistics:
     p0_rx_good_frames: 849718
     p0_rx_broadcast_frames: 53
     p0_rx_multicast_frames: 1424
     p0_rx_crc_errors: 0
     p0_rx_oversized_frames: 0
     p0_rx_undersized_frames: 0
     p0_ale_drop: 0
     p0_ale_overrun_drop: 0
     p0_rx_octets: 1158017512
     p0_tx_good_frames: 852516
     p0_tx_broadcast_frames: 52
     p0_tx_multicast_frames: 1391
     p0_tx_octets: 1160620649
     p0_tx_64B_frames: 0
     p0_tx_65_to_127B_frames: 184436
     p0_tx_128_to_255B_frames: 153
     p0_tx_256_to_511B_frames: 101
     p0_tx_512_to_1023B_frames: 8
     p0_tx_1024B_frames: 1517536
     p0_net_octets: 2318638161
     p0_rx_bottom_fifo_drop: 0
     p0_rx_port_mask_drop: 0
     p0_rx_top_fifo_drop: 0
     p0_ale_rate_limit_drop: 0
     p0_ale_vid_ingress_drop: 0
     p0_ale_da_eq_sa_drop: 0
     p0_ale_block_drop: 0
     p0_ale_secure_drop: 0
     p0_ale_auth_drop: 0
     p0_ale_unknown_ucast: 0
     p0_ale_unknown_ucast_bytes: 0
     p0_ale_unknown_mcast: 624
     p0_ale_unknown_mcast_bytes: 46236
     p0_ale_unknown_bcast: 0
     p0_ale_unknown_bcast_bytes: 0
     p0_ale_pol_match: 0
     p0_ale_pol_match_red: 0
     p0_ale_pol_match_yellow: 0
     p0_ale_mcast_sa_drop: 0
     p0_ale_dual_vlan_drop: 0
     p0_ale_len_err_drop: 0
     p0_ale_ip_next_hdr_drop: 0
     p0_ale_ipv4_frag_drop: 0
     p0_tx_mem_protect_err: 0
     p0_tx_pri0: 852516
     p0_tx_pri1: 0
     p0_tx_pri2: 0
     p0_tx_pri3: 0
     p0_tx_pri4: 0
     p0_tx_pri5: 0
     p0_tx_pri6: 0
     p0_tx_pri7: 0
     p0_tx_pri0_bcnt: 1157210905
     p0_tx_pri1_bcnt: 0
     p0_tx_pri2_bcnt: 0
     p0_tx_pri3_bcnt: 0
     p0_tx_pri4_bcnt: 0
     p0_tx_pri5_bcnt: 0
     p0_tx_pri6_bcnt: 0
     p0_tx_pri7_bcnt: 0
     p0_tx_pri0_drop: 0
     p0_tx_pri1_drop: 0
     p0_tx_pri2_drop: 0
     p0_tx_pri3_drop: 0
     p0_tx_pri4_drop: 0
     p0_tx_pri5_drop: 0
     p0_tx_pri6_drop: 0
     p0_tx_pri7_drop: 0
     p0_tx_pri0_drop_bcnt: 0
     p0_tx_pri1_drop_bcnt: 0
     p0_tx_pri2_drop_bcnt: 0
     p0_tx_pri3_drop_bcnt: 0
     p0_tx_pri4_drop_bcnt: 0
     p0_tx_pri5_drop_bcnt: 0
     p0_tx_pri6_drop_bcnt: 0
     p0_tx_pri7_drop_bcnt: 0
     rx_good_frames: 426757
     rx_broadcast_frames: 27
     rx_multicast_frames: 770
     rx_pause_frames: 117067
     rx_crc_errors: 0
     rx_align_code_errors: 0
     rx_oversized_frames: 0
     rx_jabber_frames: 0
     rx_undersized_frames: 0
     rx_fragments: 0
     ale_drop: 72
     ale_overrun_drop: 0
     rx_octets: 579256735
     tx_good_frames: 538863
     tx_broadcast_frames: 27
     tx_multicast_frames: 114717
     tx_pause_frames: 114007
     tx_deferred_frames: 0
     tx_collision_frames: 0
     tx_single_coll_frames: 0
     tx_mult_coll_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     rx_ipg_error: 0
     tx_carrier_sense_errors: 0
     tx_octets: 586303680
     tx_64B_frames: 114007
     tx_65_to_127B_frames: 92275
     tx_128_to_255B_frames: 90
     tx_256_to_511B_frames: 52
     tx_512_to_1023B_frames: 4
     tx_1024B_frames: 759192
     net_octets: 1173052703
     rx_bottom_fifo_drop: 438
     rx_port_mask_drop: 117139
     rx_top_fifo_drop: 0
     ale_rate_limit_drop: 0
     ale_vid_ingress_drop: 0
     ale_da_eq_sa_drop: 0
     ale_block_drop: 0
     ale_secure_drop: 346
     ale_auth_drop: 0
     ale_unknown_ucast: 425522
     ale_unknown_ucast_bytes: 578521220
     ale_unknown_mcast: 425
     ale_unknown_mcast_bytes: 34965
     ale_unknown_bcast: 26
     ale_unknown_bcast_bytes: 8788
     ale_pol_match: 0
     ale_pol_match_red: 0
     ale_pol_match_yellow: 0
     ale_mcast_sa_drop: 0
     ale_dual_vlan_drop: 0
     ale_len_err_drop: 0
     ale_ip_next_hdr_drop: 0
     ale_ipv4_frag_drop: 0
     iet_rx_assembly_err: 0
     iet_rx_assembly_ok: 0
     iet_rx_smd_err: 0
     iet_rx_frag: 0
     iet_tx_hold: 0
     iet_tx_frag: 0
     tx_mem_protect_err: 0
     tx_pri0: 424856
     tx_pri1: 0
     tx_pri2: 0
     tx_pri3: 0
     tx_pri4: 0
     tx_pri5: 0
     tx_pri6: 0
     tx_pri7: 0
     tx_pri0_bcnt: 579008480
     tx_pri1_bcnt: 0
     tx_pri2_bcnt: 0
     tx_pri3_bcnt: 0
     tx_pri4_bcnt: 0
     tx_pri5_bcnt: 0
     tx_pri6_bcnt: 0
     tx_pri7_bcnt: 0
     tx_pri0_drop: 0
     tx_pri1_drop: 0
     tx_pri2_drop: 0
     tx_pri3_drop: 0
     tx_pri4_drop: 0
     tx_pri5_drop: 0
     tx_pri6_drop: 0
     tx_pri7_drop: 0
     tx_pri0_drop_bcnt: 0
     tx_pri1_drop_bcnt: 0
     tx_pri2_drop_bcnt: 0
     tx_pri3_drop_bcnt: 0
     tx_pri4_drop_bcnt: 0
     tx_pri5_drop_bcnt: 0
     tx_pri6_drop_bcnt: 0
     tx_pri7_drop_bcnt: 0

MASTER ETH0 STAT:

master-eth0-hsr-statistics.txt
root@am64xx-evm:~# ethtool -S eth0
NIC statistics:
     p0_rx_good_frames: 853655
     p0_rx_broadcast_frames: 55
     p0_rx_multicast_frames: 1679
     p0_rx_crc_errors: 0
     p0_rx_oversized_frames: 0
     p0_rx_undersized_frames: 0
     p0_ale_drop: 0
     p0_ale_overrun_drop: 0
     p0_rx_octets: 1158527022
     p0_tx_good_frames: 849010
     p0_tx_broadcast_frames: 50
     p0_tx_multicast_frames: 1507
     p0_tx_octets: 1160219790
     p0_tx_64B_frames: 0
     p0_tx_65_to_127B_frames: 184781
     p0_tx_128_to_255B_frames: 178
     p0_tx_256_to_511B_frames: 101
     p0_tx_512_to_1023B_frames: 8
     p0_tx_1024B_frames: 1517597
     p0_net_octets: 2318746812
     p0_rx_bottom_fifo_drop: 0
     p0_rx_port_mask_drop: 0
     p0_rx_top_fifo_drop: 0
     p0_ale_rate_limit_drop: 0
     p0_ale_vid_ingress_drop: 0
     p0_ale_da_eq_sa_drop: 0
     p0_ale_block_drop: 0
     p0_ale_secure_drop: 0
     p0_ale_auth_drop: 0
     p0_ale_unknown_ucast: 0
     p0_ale_unknown_ucast_bytes: 0
     p0_ale_unknown_mcast: 752
     p0_ale_unknown_mcast_bytes: 60964
     p0_ale_unknown_bcast: 2
     p0_ale_unknown_bcast_bytes: 148
     p0_ale_pol_match: 0
     p0_ale_pol_match_red: 0
     p0_ale_pol_match_yellow: 0
     p0_ale_mcast_sa_drop: 0
     p0_ale_dual_vlan_drop: 0
     p0_ale_len_err_drop: 0
     p0_ale_ip_next_hdr_drop: 0
     p0_ale_ipv4_frag_drop: 0
     p0_tx_mem_protect_err: 0
     p0_tx_pri0: 849010
     p0_tx_pri1: 0
     p0_tx_pri2: 0
     p0_tx_pri3: 0
     p0_tx_pri4: 0
     p0_tx_pri5: 0
     p0_tx_pri6: 0
     p0_tx_pri7: 0
     p0_tx_pri0_bcnt: 1156823874
     p0_tx_pri1_bcnt: 0
     p0_tx_pri2_bcnt: 0
     p0_tx_pri3_bcnt: 0
     p0_tx_pri4_bcnt: 0
     p0_tx_pri5_bcnt: 0
     p0_tx_pri6_bcnt: 0
     p0_tx_pri7_bcnt: 0
     p0_tx_pri0_drop: 0
     p0_tx_pri1_drop: 0
     p0_tx_pri2_drop: 0
     p0_tx_pri3_drop: 0
     p0_tx_pri4_drop: 0
     p0_tx_pri5_drop: 0
     p0_tx_pri6_drop: 0
     p0_tx_pri7_drop: 0
     p0_tx_pri0_drop_bcnt: 0
     p0_tx_pri1_drop_bcnt: 0
     p0_tx_pri2_drop_bcnt: 0
     p0_tx_pri3_drop_bcnt: 0
     p0_tx_pri4_drop_bcnt: 0
     p0_tx_pri5_drop_bcnt: 0
     p0_tx_pri6_drop_bcnt: 0
     p0_tx_pri7_drop_bcnt: 0
     rx_good_frames: 424919
     rx_broadcast_frames: 25
     rx_multicast_frames: 773
     rx_pause_frames: 115641
     rx_crc_errors: 0
     rx_align_code_errors: 0
     rx_oversized_frames: 0
     rx_jabber_frames: 0
     rx_undersized_frames: 0
     rx_fragments: 0
     ale_drop: 18
     ale_overrun_drop: 0
     rx_octets: 579011354
     tx_good_frames: 543648
     tx_broadcast_frames: 27
     tx_multicast_frames: 117660
     tx_pause_frames: 116814
     tx_deferred_frames: 0
     tx_collision_frames: 0
     tx_single_coll_frames: 0
     tx_mult_coll_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     rx_ipg_error: 0
     tx_carrier_sense_errors: 0
     tx_octets: 586738619
     tx_64B_frames: 116814
     tx_65_to_127B_frames: 92415
     tx_128_to_255B_frames: 92
     tx_256_to_511B_frames: 50
     tx_512_to_1023B_frames: 4
     tx_1024B_frames: 759192
     net_octets: 1173150997
     rx_bottom_fifo_drop: 361
     rx_port_mask_drop: 115659
     rx_top_fifo_drop: 0
     ale_rate_limit_drop: 0
     ale_vid_ingress_drop: 0
     ale_da_eq_sa_drop: 0
     ale_block_drop: 0
     ale_secure_drop: 343
     ale_auth_drop: 0
     ale_unknown_ucast: 423760
     ale_unknown_ucast_bytes: 578395552
     ale_unknown_mcast: 430
     ale_unknown_mcast_bytes: 35582
     ale_unknown_bcast: 25
     ale_unknown_bcast_bytes: 8182
     ale_pol_match: 0
     ale_pol_match_red: 0
     ale_pol_match_yellow: 0
     ale_mcast_sa_drop: 0
     ale_dual_vlan_drop: 0
     ale_len_err_drop: 0
     ale_ip_next_hdr_drop: 0
     ale_ipv4_frag_drop: 0
     iet_rx_assembly_err: 0
     iet_rx_assembly_ok: 0
     iet_rx_smd_err: 0
     iet_rx_frag: 0
     iet_tx_hold: 0
     iet_tx_frag: 0
     tx_mem_protect_err: 0
     tx_pri0: 426834
     tx_pri1: 0
     tx_pri2: 0
     tx_pri3: 0
     tx_pri4: 0
     tx_pri5: 0
     tx_pri6: 0
     tx_pri7: 0
     tx_pri0_bcnt: 579264031
     tx_pri1_bcnt: 0
     tx_pri2_bcnt: 0
     tx_pri3_bcnt: 0
     tx_pri4_bcnt: 0
     tx_pri5_bcnt: 0
     tx_pri6_bcnt: 0
     tx_pri7_bcnt: 0
     tx_pri0_drop: 0
     tx_pri1_drop: 0
     tx_pri2_drop: 0
     tx_pri3_drop: 0
     tx_pri4_drop: 0
     tx_pri5_drop: 0
     tx_pri6_drop: 0
     tx_pri7_drop: 0
     tx_pri0_drop_bcnt: 0
     tx_pri1_drop_bcnt: 0
     tx_pri2_drop_bcnt: 0
     tx_pri3_drop_bcnt: 0
     tx_pri4_drop_bcnt: 0
     tx_pri5_drop_bcnt: 0
     tx_pri6_drop_bcnt: 0
     tx_pri7_drop_bcnt: 0

MASTER ETH1 STAT

master-eth1-hsr-statistics.txt
root@am64xx-evm:~# ethtool -S eth1
NIC statistics:
     p0_rx_good_frames: 853741
     p0_rx_broadcast_frames: 57
     p0_rx_multicast_frames: 1763
     p0_rx_crc_errors: 0
     p0_rx_oversized_frames: 0
     p0_rx_undersized_frames: 0
     p0_ale_drop: 0
     p0_ale_overrun_drop: 0
     p0_rx_octets: 1158533746
     p0_tx_good_frames: 849096
     p0_tx_broadcast_frames: 52
     p0_tx_multicast_frames: 1591
     p0_tx_octets: 1160226690
     p0_tx_64B_frames: 0
     p0_tx_65_to_127B_frames: 184949
     p0_tx_128_to_255B_frames: 178
     p0_tx_256_to_511B_frames: 105
     p0_tx_512_to_1023B_frames: 8
     p0_tx_1024B_frames: 1517597
     p0_net_octets: 2318760436
     p0_rx_bottom_fifo_drop: 0
     p0_rx_port_mask_drop: 0
     p0_rx_top_fifo_drop: 0
     p0_ale_rate_limit_drop: 0
     p0_ale_vid_ingress_drop: 0
     p0_ale_da_eq_sa_drop: 0
     p0_ale_block_drop: 0
     p0_ale_secure_drop: 0
     p0_ale_auth_drop: 0
     p0_ale_unknown_ucast: 0
     p0_ale_unknown_ucast_bytes: 0
     p0_ale_unknown_mcast: 794
     p0_ale_unknown_mcast_bytes: 64072
     p0_ale_unknown_bcast: 2
     p0_ale_unknown_bcast_bytes: 148
     p0_ale_pol_match: 0
     p0_ale_pol_match_red: 0
     p0_ale_pol_match_yellow: 0
     p0_ale_mcast_sa_drop: 0
     p0_ale_dual_vlan_drop: 0
     p0_ale_len_err_drop: 0
     p0_ale_ip_next_hdr_drop: 0
     p0_ale_ipv4_frag_drop: 0
     p0_tx_mem_protect_err: 0
     p0_tx_pri0: 849096
     p0_tx_pri1: 0
     p0_tx_pri2: 0
     p0_tx_pri3: 0
     p0_tx_pri4: 0
     p0_tx_pri5: 0
     p0_tx_pri6: 0
     p0_tx_pri7: 0
     p0_tx_pri0_bcnt: 1156830430
     p0_tx_pri1_bcnt: 0
     p0_tx_pri2_bcnt: 0
     p0_tx_pri3_bcnt: 0
     p0_tx_pri4_bcnt: 0
     p0_tx_pri5_bcnt: 0
     p0_tx_pri6_bcnt: 0
     p0_tx_pri7_bcnt: 0
     p0_tx_pri0_drop: 0
     p0_tx_pri1_drop: 0
     p0_tx_pri2_drop: 0
     p0_tx_pri3_drop: 0
     p0_tx_pri4_drop: 0
     p0_tx_pri5_drop: 0
     p0_tx_pri6_drop: 0
     p0_tx_pri7_drop: 0
     p0_tx_pri0_drop_bcnt: 0
     p0_tx_pri1_drop_bcnt: 0
     p0_tx_pri2_drop_bcnt: 0
     p0_tx_pri3_drop_bcnt: 0
     p0_tx_pri4_drop_bcnt: 0
     p0_tx_pri5_drop_bcnt: 0
     p0_tx_pri6_drop_bcnt: 0
     p0_tx_pri7_drop_bcnt: 0
     rx_good_frames: 424954
     rx_broadcast_frames: 26
     rx_multicast_frames: 809
     rx_pause_frames: 114007
     rx_crc_errors: 0
     rx_align_code_errors: 0
     rx_oversized_frames: 0
     rx_jabber_frames: 0
     rx_undersized_frames: 0
     rx_fragments: 0
     ale_drop: 15
     ale_overrun_drop: 0
     rx_octets: 579013644
     tx_good_frames: 543931
     tx_broadcast_frames: 29
     tx_multicast_frames: 117942
     tx_pause_frames: 117067
     tx_deferred_frames: 0
     tx_collision_frames: 0
     tx_single_coll_frames: 0
     tx_mult_coll_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     rx_ipg_error: 0
     tx_carrier_sense_errors: 0
     tx_octets: 586757049
     tx_64B_frames: 117067
     tx_65_to_127B_frames: 92480
     tx_128_to_255B_frames: 89
     tx_256_to_511B_frames: 53
     tx_512_to_1023B_frames: 4
     tx_1024B_frames: 759192
     net_octets: 1173067141
     rx_bottom_fifo_drop: 426
     rx_port_mask_drop: 114022
     rx_top_fifo_drop: 0
     ale_rate_limit_drop: 0
     ale_vid_ingress_drop: 0
     ale_da_eq_sa_drop: 0
     ale_block_drop: 0
     ale_secure_drop: 364
     ale_auth_drop: 0
     ale_unknown_ucast: 423693
     ale_unknown_ucast_bytes: 578296670
     ale_unknown_mcast: 445
     ale_unknown_mcast_bytes: 36276
     ale_unknown_bcast: 26
     ale_unknown_bcast_bytes: 8520
     ale_pol_match: 0
     ale_pol_match_red: 0
     ale_pol_match_yellow: 0
     ale_mcast_sa_drop: 0
     ale_dual_vlan_drop: 0
     ale_len_err_drop: 0
     ale_ip_next_hdr_drop: 0
     ale_ipv4_frag_drop: 0
     iet_rx_assembly_err: 0
     iet_rx_assembly_ok: 0
     iet_rx_smd_err: 0
     iet_rx_frag: 0
     iet_tx_hold: 0
     iet_tx_frag: 0
     tx_mem_protect_err: 0
     tx_pri0: 426864
     tx_pri1: 0
     tx_pri2: 0
     tx_pri3: 0
     tx_pri4: 0
     tx_pri5: 0
     tx_pri6: 0
     tx_pri7: 0
     tx_pri0_bcnt: 579266353
     tx_pri1_bcnt: 0
     tx_pri2_bcnt: 0
     tx_pri3_bcnt: 0
     tx_pri4_bcnt: 0
     tx_pri5_bcnt: 0
     tx_pri6_bcnt: 0
     tx_pri7_bcnt: 0
     tx_pri0_drop: 0
     tx_pri1_drop: 0
     tx_pri2_drop: 0
     tx_pri3_drop: 0
     tx_pri4_drop: 0
     tx_pri5_drop: 0
     tx_pri6_drop: 0
     tx_pri7_drop: 0
     tx_pri0_drop_bcnt: 0
     tx_pri1_drop_bcnt: 0
     tx_pri2_drop_bcnt: 0
     tx_pri3_drop_bcnt: 0
     tx_pri4_drop_bcnt: 0
     tx_pri5_drop_bcnt: 0
     tx_pri6_drop_bcnt: 0
     tx_pri7_drop_bcnt: 0

I am interesting to understand what are next

     ale_drop: 84

     rx_bottom_fifo_drop: 410

     ale_secure_drop: 257

Could you help me to understand why there a dropped packets on HSR even on 25M... I did same test on 10M too and still I have dropped packets...

 

Thanks a lot

Best regards

Milan

  • One item to check is setting the Ethernet priority in PREEMPT_RT

    #in RT linux to make the networking related kernel services run at higher priority, increases throughput in no packet loss case
    ps aux | grep ksoftirq
    #get the per CPU ksoftirq PIDs, 10 and 21 in this case
    chrt -f -p 10 10 
    chrt -f -p 10 21 

    Edited, the second comment I had here on MAC addresses is not relevant.

      Pekka

  • One item to check is setting the Ethernet priority in PREEMPT_RT

    #in RT linux to make the networking related kernel services run at higher priority, increases throughput in no packet loss case
    ps aux | grep ksoftirq
    #get the per CPU ksoftirq PIDs, 10 and 21 in this case
    chrt -f -p 10 10 
    chrt -f -p 10 21 

    Edited, the second comment I had here on MAC addresses is not relevant.

    Did you check the amount of drops and if it is related to the iperf3 TCP test or the bandwidth of that test, or if there are drops anyway from background services.

      Pekka

  • On slave side with 5min iperf3 test I have drops on ale_secure_drop but not on ale_drop

    This is statistics that I have

    root@am64xx-evm:~# ethtool -S eth0 | grep drop
    ale_secure_drop: 247

    root@am64xx-evm:~# ethtool -S eth1 | grep drop
    ale_secure_drop: 271

    [ ID][Role] Interval Transfer Bitrate Retr
    [ 5][TX-C] 0.00-300.00 sec 715 MBytes 20.0 Mbits/sec 243 sender
    [ 5][TX-C] 0.00-300.03 sec 715 MBytes 20.0 Mbits/sec receiver
    [ 7][RX-C] 0.00-300.00 sec 715 MBytes 20.0 Mbits/sec 159 sender
    [ 7][RX-C] 0.00-300.03 sec 715 MBytes 20.0 Mbits/sec receiver

    I will check your commands too

  • setting the Ethernet priority in PREEMPT_RT with chrt did not change anything

  • rx_bottom_fifo_drop: 410

    is something I thought the priority increase could have helped. Do you still have the rx_bottom_fifo_drop with the priority tweak?

    ale_secure_drop: 247

    Looking in the TRM https://www.ti.com/lit/pdf/spruim2 this is described as:

    2.1.4.6.18.1.19.3 ALE Secure Drop (Offset = 3A0A0h)

    The total number of frames received on a port that were dropped (zero port_mask) due to a secure violation (the source address is owned by a different receive port).
    • was any data or MAC control frame which matched a unicast, broadcast or multicast address, or matched due to promiscuous mode, and
    • Any length (including less than 64 bytes and greater than RX_MAXLEN bytes)
    • had no CRC error, alignment error, or code error, and
    • the source address is an entry in the table with the SECURE bit set and a port number for a different receive port.

    So likely something to do with MAC address, I suspect it is a looped packet with source MAC being the destination MAC. Are you able to get a trace of the traffic (like profishark in the middle)? Alternatively if you disconnect on cable, and run the test do you still see drops? 

  • Hi Pekka,

    I am still doing investigation on this topic... current status

    • when I removed some cable there is NO dropped packages
    • I am not sure now that issue is coming from ALE secure drop only.. I noticed that even there is no any trafic for testing ale_secure_drop is increasing. Also when I start test during long time, I see there is much more packets dropped in compare what  ale_secure_drop show... with more investigation I come to statistics from 12.2.1.4.6.18.1.16 Portmask Drop (Offset = 3A088h)

    it seems Portmask Drop is more responsible for drop in compare with  ALE secure drop... But I cannot exlude any of these two

    I will need to do tests with profishark in the middle.. 

  • Hi Pekka

    By doing chrt and setup priority, I have much better results and it is sure coming with rx_bottom_fifo_drop...

    Now, there is in avarage 1 packet lost per second and this is a case only in HSR setup with both cable attached... if we remove one cable no more lost packages

    I started with  profishark and for now I noticed next

    • There are mac-specific-ctrl-proto-01 pause frames MAC CTRL

    I notcied one more thing with Sequence numbers but this looks OK as it is bidirectional test

  • By adding ipef3 windows size results are even better. I think we can close this ticket

        -w, --window n[KMGT]
                  set socket buffer size / window size.  This value gets  sent  to
                  the  server and used on that side too; on both sides this option
                  sets both the sending and receiving socket buffer  sizes.   This
                  option  can  be  used to set (indirectly) the maximum TCP window
                  size.  Note that on Linux systems, the effective maximum  window
                  size  is  approximately  double what is specified by this option
                  (this behavior is not a bug in iperf3 but  a  "feature"  of  the
                  Linux kernel, as documented by tcp(7) and socket(7)).
  • Great, thanks for your analysis, I agree it makes sense.

      Pekka