PROCESSOR-SDK-AM64X: Packet loss occurs when using the Packets HSR Offload solution for forwarding

Hi Tianyi,

Thanks for bringing this to my attention. I have not yet tested on the public release 09.02.01.10 yet but will do so today and get back to you by Monday. 

-Daolin

Update:

I tested basic ping and iperf3 using the same command as you indicated and I don't see any packet loss. See below for my results. These tests were ran with the network disable commands you shared previously (another thread).

EVM1 (192.168.1.10) eth1  eth2 EVM2 (192.168.1.30) eth1  eth2 EVM3 (192.168.1.50) eth1  eth2 EVM1 (disconnecting cable between EVM3 and EVM1)

EVM1:

root@am64xx-evm:~# iperf3 -c 192.168.1.50 -t40 -b20M -l400 -u
Connecting to host 192.168.1.50, port 5201
[  5] local 192.168.1.10 port 37642 connected to 192.168.1.50 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  6245  
[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  6251  
[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  253.923246] icssg-prueth icssg1-eth eth2: Link is Down
[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  
[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  6251  
[  5]  10.00-11.00  sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]  11.00-12.00  sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]  12.00-13.00  sec  2.38 MBytes  20.0 Mbits/sec  6249  
[  5]  13.00-14.00  sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]  14.00-15.00  sec  2.38 MBytes  20.0 Mbits/sec  6251  
[  5]  15.00-16.00  sec  2.38 MBytes  20.0 Mbits/sec  6249  
^C[  5]  16.00-16.71  sec  1.70 MBytes  20.0 Mbits/sec  4445  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-16.71  sec  39.8 MBytes  20.0 Mbits/sec  0.000 ms  0/104440 (0%)  sender
[  5]   0.00-16.71  sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)  receiver
iperf3: interrupt - the client has terminated

EVM3:

root@am64xx-evm:~# iperf3 -s -B 192.168.1.50                                                                                                                                                   
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.1.10, port 51750
[  5] local 192.168.1.50 port 5201 connected to 192.168.1.10 port 37642
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6245 (0%)  
[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6250 (0%)  
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/6250 (0%)  
[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  0.010 ms  0/6251 (0%)  
[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  0.035 ms  0/6250 (0%)  
[  181.283935] icssg-prueth icssg1-eth eth1: Link is Down
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6250 (0%)  
[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6249 (0%)  
[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/6250 (0%)  
[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6250 (0%)  
[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6251 (0%)  
[  5]  10.00-11.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/6250 (0%)  
[  5]  11.00-12.00  sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6250 (0%)  
[  5]  12.00-13.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/6249 (0%)  
[  5]  13.00-14.00  sec  2.38 MBytes  20.0 Mbits/sec  0.007 ms  0/6250 (0%)  
[  5]  14.00-15.00  sec  2.38 MBytes  20.0 Mbits/sec  0.004 ms  0/6251 (0%)  
[  5]  15.00-16.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/6249 (0%)  
[  5]  15.00-16.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/6249 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-16.00  sec  39.8 MBytes  20.9 Mbits/sec  0.005 ms  0/104440 (0%)  receiver
iperf3: the client has terminated
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
root@am64xx-evm:~# dmesg | grep -i dual
[    8.435610] icssg-prueth icssg1-eth: TI PRU ethernet driver initialized: dual EMAC mode
root@am64xx-evm:~# 

Network disable commands:

systemctl disable dhcpcd.service
systemctl disable NetworkManager.service
systemctl disable systemd-resolved.service
systemctl disable systemd-networkd.service
sync
reboot

Since, I cannot reproduce the issue, can you share the entire console log that results when setting up the hsr offload? Additionally does ping result in packet loss?

-Daolin

Hi Daolin,

Thank you for the fast update !

It seems the test that you shared, you didn't disconnect the cable C which links between EVM3 (192.168.1.50) eth1 <->  eth2 EVM1

Could you confirm that ? If so, could you redo the test by disconnecting the cable C ?

The final set up should be like :

In our side, we replaced the EVM2 Image or (the DUT Image) by the TI latest one.And we found the packet lost !

Our next step is to replace the EVM1 and EVM3 Image by TI latest one and do the same test !

Tianyi

We redo the test with all the 3 boards with the latest release TI Image, with the following set up : (cable C which links between EVM3 (192.168.1.50) eth1 <->  eth2 EVM1 is disconnected)

We still have this among of packets lost, here are the logs

EVM1/Testboard1 : (Iperf Server)

        qtbase-qmlplugins

        qtbase

        qtconnectivity-plugins

root@am64xx-evm:~# 

root@am64xx-evm:~# 

root@am64xx-evm:~# ifconfig

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 413  bytes 31394 (30.6 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 413  bytes 31394 (30.6 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# ip link set dev eth1 up

[  158.722297] remoteproc remoteproc13: powering up 300b4000.pru

[  158.734285] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, siz6

[  158.734330] remoteproc remoteproc13: unsupported resource 5

[  158.734360] remoteproc remoteproc13: remote processor 300b4000.pru is now up

[  158.734543] remoteproc remoteproc14: powering up 30084000.rtu

root@am64xx-evm:~# [  158.736556] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu08

ip link set dev eth2 up[  158.736615] remoteproc remoteproc14: remote processor 30084000.rtu is nowp

[  158.736663] remoteproc remoteproc7: powering up 3008a000.txpru

[  158.738727] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, si2

[  158.738791] remoteproc remoteproc7: remote processor 3008a000.txpru is now up

[  158.739076] remoteproc remoteproc15: powering up 300b8000.pru

[  158.741267] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, siz2

[  158.741306] remoteproc remoteproc15: unsupported resource 5

[  158.741334] remoteproc remoteproc15: remote processor 300b8000.pru is now up

[  158.741376] remoteproc remoteproc16: powering up 30086000.rtu

[  158.743460] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, siz4

[  158.743513] remoteproc remoteproc16: remote processor 30086000.rtu is now up

[  158.743553] remoteproc remoteproc8: powering up 3008c000.txpru

[  158.745509] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, si4

[  158.745564] remoteproc remoteproc8: remote processor 3008c000.txpru is now up

[  158.748350] pps pps1: new PPS source ptp2



root@am64xx-evm:~# [  162.851025] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[  162.851755] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[  163.747036] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

[  163.747114] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

[  172.067022] icssg-prueth icssg1-eth eth1: Link is Down

[  175.139549] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[  188.450828] icssg-prueth icssg1-eth eth1: Link is Down

[  195.619546] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off



root@am64xx-evm:~# ./hsr_set_up.sh hsr_hw eth1 eth2 192.168.200.1 

ip=192.168.200.1

if=hsr0

mac=70:ff:76:1e:e7:8c

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

[  213.055481] icssg-prueth icssg1-eth eth1: Link is Down

root@am64xx-evm:~# ifconfig

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 11  bytes 1308 (1.2 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 87  bytes 11058 (10.7 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 39  bytes 7308 (7.1 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 34  bytes 4464 (4.3 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494

        inet 192.168.200.1  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 48  bytes 7708 (7.5 KiB)

        RX errors 0  dropped 4  overruns 0  frame 0

        TX packets 46  bytes 5302 (5.1 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 580  bytes 44826 (43.7 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 580  bytes 44826 (43.7 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# [  258.275220] icssg-prueth icssg1-eth eth2: Link is Down



(failed reverse-i-search)`ping': ./hsr_set_u^Csh hsr_hw eth1 eth2 192.168.200.1                     

root@am64xx-evm:~# ping 192.168.200.2

PING 192.168.200.2 (192.168.200.2): 56 data bytes

64 bytes from 192.168.200.2: seq=0 ttl=64 time=0.734 ms

64 bytes from 192.168.200.2: seq=1 ttl=64 time=0.422 ms

^C

--- 192.168.200.2 ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max = 0.422/0.578/0.734 ms

root@am64xx-evm:~# iperf3 -s -B 192.168.200.1

-----------------------------------------------------------

Server listening on 5201 (test #1)

-----------------------------------------------------------

Accepted connection from 192.168.200.2, port 42798

[  5] local 192.168.200.1 port 5201 connected to 192.168.200.2 port 39883

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-1.00   sec  1.24 MBytes  10.4 Mbits/sec  0.009 ms  2986/6246 (48%)  

[  5]   1.00-2.00   sec  1.22 MBytes  10.2 Mbits/sec  0.007 ms  3057/6250 (49%)  

[  5]   2.00-3.00   sec  1.25 MBytes  10.5 Mbits/sec  0.005 ms  2980/6249 (48%)  

[  5]   3.00-4.00   sec  1.23 MBytes  10.3 Mbits/sec  0.005 ms  3037/6249 (49%)  

[  5]   4.00-5.00   sec  1.25 MBytes  10.4 Mbits/sec  0.006 ms  2986/6251 (48%)  

[  5]   5.00-6.00   sec  1.28 MBytes  10.7 Mbits/sec  0.008 ms  2907/6250 (47%)  

[  5]   6.00-7.00   sec  1.27 MBytes  10.6 Mbits/sec  0.007 ms  2933/6250 (47%)  

[  5]   7.00-8.00   sec  1.24 MBytes  10.4 Mbits/sec  0.008 ms  2984/6247 (48%)  

[  5]   8.00-9.00   sec  1.26 MBytes  10.6 Mbits/sec  0.007 ms  2941/6253 (47%)  

[  5]   9.00-10.00  sec  1.24 MBytes  10.4 Mbits/sec  0.005 ms  2995/6250 (48%)  

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-10.00  sec  12.5 MBytes  10.5 Mbits/sec  0.005 ms  29806/62495 (48%)  receiver

-----------------------------------------------------------

Server listening on 5201 (test #2)

-----------------------------------------------------------

[  5]   0.00-10.00  sec  12.5 MBytes  10.5 Mbits/sec  0.005 ms  29806/62495 (48%)  receiver

DUT/EVM2:

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

[  156.650347] icssg-prueth icssg1-eth: Enabling HSR offload mode

^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A[  159.696139] remoteproc remoteproc14: powering up 3

00b4000.pru

[  159.698672] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 

37976

[  159.698711] remoteproc remoteproc14: unsupported resource 5

[  159.698739] remoteproc remoteproc14: remote processor 300b4000.pru is now up

[  159.698785] remoteproc remoteproc15: powering up 30084000.rtu

[  159.703184] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 

34632

[  159.703241] remoteproc remoteproc15: remote processor 30084000.rtu is now up

[  159.703286] remoteproc remoteproc7: powering up 3008a000.txpru

[  159.707230] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size

 36128

[  159.707290] remoteproc remoteproc7: remote processor 3008a000.txpru is now up

[  159.707575] remoteproc remoteproc16: powering up 300b8000.pru

[  159.709230] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 

38208

[  159.709266] remoteproc remoteproc16: unsupported resource 5

[  159.709292] remoteproc remoteproc16: remote processor 300b8000.pru is now up

[  159.709336] remoteproc remoteproc13: powering up 30086000.rtu

[  159.710742] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 

33844

[  159.710794] remoteproc remoteproc13: remote processor 30086000.rtu is now up

[  159.710834] remoteproc remoteproc8: powering up 3008c000.txpru

[  159.712378] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size

 34636

[  159.712439] remoteproc remoteproc8: remote processor 3008c000.txpru is now up

[  159.714818] pps pps1: new PPS source ptp2

^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^

[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[

[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[

A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A[  162.788279] icssg-prueth icssg1-eth eth1:

 Link is Up - 1Gbps/Full - flow control off

[  162.788355] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[  163.812106] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

[  163.812209] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

[  164.809221] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready

root@am64xx-evm:~# systemctl disable dhcpcd.service^C

root@am64xx-evm:~# [  181.220017] icssg-prueth icssg1-eth eth2: Link is Down

[  188.387950] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

[  205.795593] icssg-prueth icssg1-eth eth1: Link is Down

[  208.867955] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off



root@am64xx-evm:~# systemctl disable dhcpcd.service

Tesboard3/EVM3: (Client )

        qtmultimedia

root@am64xx-evm:~# ./hsr_set_up.sh hsr_hw eth1 eth2 192.168.200.2

ip=192.168.200.2

if=hsr0

mac=70:ff:76:1e:9f:54

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

[  204.887646] icssg-prueth icssg1-eth: Enabling HSR offload mode

[  207.954625] remoteproc remoteproc14: powering up 300b4000.pru

[  207.960653] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 37976

[  207.960695] remoteproc remoteproc14: unsupported resource 5

[  207.960723] remoteproc remoteproc14: remote processor 300b4000.pru is now up

[  207.960768] remoteproc remoteproc15: powering up 30084000.rtu

[  207.966623] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 34632

[  207.966687] remoteproc remoteproc15: remote processor 30084000.rtu is now up

[  207.966731] remoteproc remoteproc7: powering up 3008a000.txpru

[  207.968269] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 36128

[  207.968334] remoteproc remoteproc7: remote processor 3008a000.txpru is now up

[  207.968626] remoteproc remoteproc16: powering up 300b8000.pru

[  207.971315] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 38208

[  207.971358] remoteproc remoteproc16: unsupported resource 5

[  207.971386] remoteproc remoteproc16: remote processor 300b8000.pru is now up

[  207.971430] remoteproc remoteproc9: powering up 30086000.rtu

[  207.973024] remoteproc remoteproc9: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 33844

[  207.973084] remoteproc remoteproc9: remote processor 30086000.rtu is now up

[  207.973127] remoteproc remoteproc8: powering up 3008c000.txpru

[  207.974549] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34636

[  207.974608] remoteproc remoteproc8: remote processor 3008c000.txpru is now up

[  207.980055] pps pps1: new PPS source ptp2

[  211.108006] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[  211.108080] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[  212.131730] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

[  212.132760] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

root@am64xx-evm:~# [  213.092740] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready



root@am64xx-evm:~# [  230.563692] icssg-prueth icssg1-eth eth1: Link is Down



(failed reverse-i-search)`iper': v^Chsr_set_up.sh                                                                                                                                                           

root@am64xx-evm:~# iperf3 -c 192.18.200.1 -t 10 -b 20M -l 400 -u

iperf3: error - unable to connect to server - server may have stopped running or use a different port, firewall issue, etc.: Network is unreachable

root@am64xx-evm:~# ifconfig

eth1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:9f54  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:9f:54  txqueuelen 1000  (Ethernet)

        RX packets 2  bytes 236 (236.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 62  bytes 8088 (7.8 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:9f54  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:9f:54  txqueuelen 1000  (Ethernet)

        RX packets 6  bytes 548 (548.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 58  bytes 4886 (4.7 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494

        inet 192.168.200.2  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::72ff:76ff:fe1e:9f54  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:9f:54  txqueuelen 1000  (Ethernet)

        RX packets 8  bytes 672 (672.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 82  bytes 7582 (7.4 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 832  bytes 64168 (62.6 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 832  bytes 64168 (62.6 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# iperf3 -c 192.168.200.1 -t 10 -b 20M -l 400 -u                                                                                                                                           

[  317.968694] icssg-prueth icssg1-eth eth1: hsr_addr_subst_dest: Unknown node

Connecting to host 192.168.200.1, port 5201[  317.968724] icssg-prueth icssg1-eth eth2: hsr_addr_subst_dest: Unknown node



[  5] local 192.168.200.2 port 39883 connected to 192.168.200.1 port 5201

[ ID] Interval           Transfer     Bitrate         Total Datagrams

[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  6246  

[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  

[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  6251  

[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  

[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  6251  

[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  

[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  6251  

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-10.00  sec  23.8 MBytes  20.0 Mbits/sec  0.000 ms  0/62496 (0%)  sender

[  5]   0.00-10.00  sec  12.5 MBytes  10.5 Mbits/sec  0.005 ms  29806/62495 (48%)  receiver



iperf Done.

root@am64xx-evm:~# 

Tianyi

Hi Tianyi,

My results from my previous response were actually with the cable between EVM1 and EVM3 disconnected (apologies as I didn't make that clear before).

However, my previous response shared the results when EVM1-EVM3 cable disconnect during the iperf3 test. I remember that when I disconnected EVM1-EVM3 before the iperf3 test, I didn't see any packet loss either but I can share with you the results later today.

Have you checked that this packet loss occurs on Testboard1 or Testboard3 as the middle board/DUT? This is just to make sure this issue isn't only on the current DUT board.

I see that the transfer size and bitrate reported by the iperf3 server is ~half of what is reported by iperf3 client. Do you see the same results when testing iperf3 between just two devices? That is, DUT connected to just one other testboard, one being iperf3 client and the other iperf3 server and see if you also see ~half the tranfer size/bitrate on the iperf3 server.

-Daolin

Results from EVM1-EVM3 cable disconnected before iperf3 test. The issue is still not reproducible on my end

EVM1:

root@am64xx-evm:~# iperf3 -c 192.168.1.50 -t40 -b20M -l400 -u                                                                                                                                        
Connecting to host 192.168.1.50, port 5201
[  5] local 192.168.1.10 port 49196 connected to 192.168.1.50 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  2.37 MBytes  19.9 Mbits/sec  6223  
[  5]   1.00-2.00   sec  2.39 MBytes  20.1 Mbits/sec  6273  
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  
[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  6251  
[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  
[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  6251  
[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  
[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]  10.00-11.00  sec  2.38 MBytes  20.0 Mbits/sec  6251  
[  5]  11.00-12.00  sec  2.38 MBytes  20.0 Mbits/sec  6250  
[  5]  12.00-13.00  sec  2.38 MBytes  20.0 Mbits/sec  6249  
^C[  5]  13.00-13.29  sec   720 KBytes  20.0 Mbits/sec  1844  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-13.29  sec  31.7 MBytes  20.0 Mbits/sec  0.000 ms  0/83089 (0%)  sender
[  5]   0.00-13.29  sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)  receiver
iperf3: interrupt - the client has terminated

EVM3:

root@am64xx-evm:~# iperf3 -s -B 192.168.1.50                                                                                                                                                         
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.1.10, port 34334
[  5] local 192.168.1.50 port 5201 connected to 192.168.1.10 port 49196
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  2.37 MBytes  19.9 Mbits/sec  0.083 ms  0/6223 (0%)  
[  5]   1.00-2.00   sec  2.39 MBytes  20.1 Mbits/sec  0.005 ms  0/6273 (0%)  
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  0.007 ms  0/6249 (0%)  
[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6251 (0%)  
[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6250 (0%)  
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/6249 (0%)  
[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  0.007 ms  0/6250 (0%)  
[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  0.012 ms  0/6251 (0%)  
[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  0.007 ms  0/6249 (0%)  
[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  0.141 ms  0/6250 (0%)  
[  5]  10.00-11.00  sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6251 (0%)  
[  5]  11.00-12.00  sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6250 (0%)  
[  5]  12.00-13.00  sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6249 (0%)  
[  5]  12.00-13.00  sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6249 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-13.00  sec  31.7 MBytes  20.5 Mbits/sec  0.006 ms  0/83089 (0%)  receiver
iperf3: the client has terminated
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

EVM2:

hsr-tag-ins-offload: on
hsr-tag-rm-offload: on
hsr-fwd-offload: on
hsr-dup-offload: on
[   87.438457] icssg-prueth icssg1-eth: Enabling HSR offload mode
[   89.490640] remoteproc remoteproc12: powering up 300b4000.pru
[   89.495183] remoteproc remoteproc12: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size 37976
[   89.495225] remoteproc remoteproc12: unsupported resource 5
[   89.495253] remoteproc remoteproc12: remote processor 300b4000.pru is now up
[   89.495294] remoteproc remoteproc13: powering up 30084000.rtu
[   89.499778] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size 34632
[   89.500937] remoteproc remoteproc13: remote processor 30084000.rtu is now up
[   89.500995] remoteproc remoteproc7: powering up 3008a000.txpru
[   89.502993] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, size 36128
[   89.503052] remoteproc remoteproc7: remote processor 3008a000.txpru is now up
[   89.503336] remoteproc remoteproc14: powering up 300b8000.pru
[   89.507154] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size 38208
[   89.507194] remoteproc remoteproc14: unsupported resource 5
[   89.507222] remoteproc remoteproc14: remote processor 300b8000.pru is now up
[   89.507263] remoteproc remoteproc15: powering up 30086000.rtu
[   89.508521] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size 33844
[   89.508579] remoteproc remoteproc15: remote processor 30086000.rtu is now up
[   89.508621] remoteproc remoteproc8: powering up 3008c000.txpru
[   89.509885] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, size 34636
[   89.509939] remoteproc remoteproc8: remote processor 3008c000.txpru is now up
[   89.511371] pps pps1: new PPS source ptp2
[   93.603930] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off
[   93.604012] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   94.615095] hsr0: Slave B (eth2) is not up; please bring it up to get a fully working HSR network
root@am64xx-evm:~# [   94.616686] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready
[   94.628484] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off
[   94.628587] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

root@am64xx-evm:~# [  119.203468] icssg-prueth icssg1-eth eth1: Link is Down
[  122.275639] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

Are there any other configurations you might have made to the DUT/middle board?

-Daolin

Hi Daolin,

First on all boards we are disabling network

systemctl disable dhcpcd.service
systemctl disable NetworkManager.service
systemctl disable systemd-resolved.service
systemctl disable systemd-networkd.service
sync
reboot

Script to enable HSR is next

#!/bin/sh

#For non offload - sh hsr_setup.sh hsr_sw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR>
#For offload - sh hsr_setup.sh hsr_hw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR>

if [ "$#" != "4" ]
then
        echo "$0 <hsr_sw/hsr_hw> <intf1> <intf2> <ip addr>"
        exit
fi

if [ "$1" != "hsr_sw" ] && [ "$1" != "hsr_hw" ]
then
        echo "$0 <hsr_sw|hsr_hw>"
        exit
fi

if=hsr0

ifa=$2
ifb=$3

ip=$4
mac=`ifconfig $ifa | grep ether | cut -d " " -f 10`
# Hardcode the device name
device="platform/icssg1-eth"
#device+=`dmesg | grep $ifa | grep icssg-prueth | grep -m 1 "Link is Up" | awk '{print $4}'`

echo "ip=$ip"
echo "if=$if"
echo "mac=$mac"
echo "slave-a=$ifa"
echo "slave-b=$ifb"
echo "device=$device"

ip link delete hsr0  2> /dev/null

ip link set $ifa down
ip link set $ifb down
sleep 1

if [ "$1" == "hsr_hw" ]
then
        ethtool -k "$ifa" | grep hsr
        ethtool -K "$ifa" hsr-fwd-offload on
        ethtool -K "$ifa" hsr-dup-offload on
        ethtool -K "$ifa" hsr-tag-ins-offload on
        ethtool -K "$ifa" hsr-tag-rm-offload on
        ethtool -k "$ifa" | grep hsr

        ethtool -k "$ifb" | grep hsr
        ethtool -K "$ifb" hsr-fwd-offload on
        ethtool -K "$ifb" hsr-dup-offload on
        ethtool -K "$ifb" hsr-tag-ins-offload on
        ethtool -K "$ifb" hsr-tag-rm-offload on
        ethtool -k "$ifb" | grep hsr

        devlink dev param set "$device" name hsr_offload_mode value true cmode runtime

        sleep 3 

        # Enable Cut-Through : 
        devlink dev param set "$device" name cut_thru value 65535 cmode runtime
fi

ip link set dev $ifa address $mac
ip link set dev $ifb address $mac

ip link set $ifa up
sleep 1

ip link set $ifb up
sleep 1

ip link add name $if type hsr slave1 $ifa slave2 $ifb supervision 45 version 1

sleep 3

ip addr add $ip/24 dev $if
ip link set $if up

command is

sh ./<script_filename.sh> hsr_hw eth1 eth2 192.168.200.10

Best regards

Milan

Hi Milan,

Just to clarify, my results were also with disabling the network as mentioned in one of my earlier responses!

I see that you are setting the forwarding mode to cut-through in your script. That is a difference from my setup where I'm just using the default store-and-forward forwarding. Are you using the same script for all 3 boards? That is, is cut-through forwarding used on all 3 boards? I will give cut-through iperf3 test a try on my setup. 

Additional questions:

1. Have you checked that this packet loss occurs on Testboard1 or Testboard3 as the middle board/DUT? This is just to make sure this issue isn't only on the current DUT board.

2. I see that the transfer size and bitrate reported by the iperf3 server is ~half of what is reported by iperf3 client. Do you see the same results when testing iperf3 between just two devices? That is, DUT connected to just one other testboard, one being iperf3 client and the other iperf3 server and see if you also see ~half the tranfer size/bitrate on the iperf3 server.

3. When you mentioned "We don't have cable C that is on the image so when sending packet from Testboard1 to 2, it goes only through DUT which should forward the packets." does that mean you just don't have an ethernet cable at all in the setup or are you disconnecting the cable while testing (to test redundancy)?

4. Can you check the "ethtool -S eth1" and "ethtool -S eth2" statistics of the DUT/middle board to see if there are any tx or rx errors/drops?

5. Can you check if you see the same packet loss when DUT is in HSR nonoffload mode?

-Daolin

Hi Daolin,

we see this issue only if we are using cut-through. We do not see any errors or drops with ethtool statistics Also, we noticed if we have bigger packet size we do not have packet lost. Example packet size 1300 - lost is 13/19230 (0.068%). 

Can you check if we are enabling  cut-through properly. We are following TI doc for this.

 We see that the transfer size and bitrate reported by the iperf3 server is ~half of what is reported by iperf3 client. We do not see in case of test with two devices.

Point 3: We removed cable from the beggining just for this test. We are not sure is it coming on Testboard1 or Testboard3... Looks like issyue with cut-through 

Hi Daolin,

Thank you for your fast response !

You are right, the error comes from the number of queues enabled ! With the command you provided :

devlink dev param set platform/icssg1-eth name cut_thru value 1 cmode runtime

We don't see any packet lost !

I followed a doc earlier maybe in release 08.XX ....

Tianyi

Hi Daolin,

I come back to you because I still have some packet lost base on direction of sending.

Base on the set up:

TestBoard1/EVM1 <-> DUT/EVM2 <-> TestBoard2/EVM3

and no cable between EVM1 and EVM3.

Three boards has only 1 tx queue enable for the cut-through mode

If we create iperf server on EVM1, and we send packet from EVM3 to EVM1, we don't have packet lost.

However, if we create iperf server on EVM3, and we send packet from EVM1 to EVM3, we have packet lost ...

Here you can find the logs of all the three boards :

For EVM1 :

_____                    _____           _         _                                               

|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_                                             

|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|                                            

|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|                                              

              |___|                    |___|                                                        

                                                                                                    

Arago Project am64xx-evm -                                                                          

                                                                                                    

Arago 2023.10 am64xx-evm -                                                                          

                                                                                                    

am64xx-evm login: root                                                                              

[   34.971076] audit: type=1006 audit(1651229920.243:2): pid=985 uid=0 old-auid=4294967295 auid=0 t1

[   34.971161] audit: type=1300 audit(1651229920.243:2): arch=c00000b7 syscall=64 success=yes exit=)

[   34.971212] audit: type=1327 audit(1651229920.243:2): proctitle="(systemd)"                      

root@am64xx-evm:~# ip link set dev eth1 up                                                          

[   42.500668] remoteproc remoteproc15: powering up 300b4000.pru                                    

[   42.504538] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, siz6

[   42.504580] remoteproc remoteproc15: unsupported resource 5                                      

[   42.504608] remoteproc remoteproc15: remote processor 300b4000.pru is now up                     

[   42.504649] remoteproc remoteproc16: powering up 30084000.rtu                                    

root@am64xx-evm:~# [   42.509213] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu08

ip link set dev eth2 up[   42.509271] remoteproc remoteproc16: remote processor 30084000.rtu is nowp

[   42.509313] remoteproc remoteproc7: powering up 3008a000.txpru                                   

[   42.513010] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, si2

[   42.513070] remoteproc remoteproc7: remote processor 3008a000.txpru is now up                    

[   42.513352] remoteproc remoteproc13: powering up 300b8000.pru                                    

[   42.516057] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, siz2

[   42.516098] remoteproc remoteproc13: unsupported resource 5                                      

[   42.516128] remoteproc remoteproc13: remote processor 300b8000.pru is now up                     

[   42.516174] remoteproc remoteproc14: powering up 30086000.rtu                                    

[   42.517484] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, siz4

[   42.517532] remoteproc remoteproc14: remote processor 30086000.rtu is now up                     

[   42.517573] remoteproc remoteproc8: powering up 3008c000.txpru                                   

[   42.519083] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, si4

[   42.519138] remoteproc remoteproc8: remote processor 3008c000.txpru is now up                    

[   42.521430] pps pps1: new PPS source ptp2                                                        

                                                                                                    

root@am64xx-evm:~# [   45.603496] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow contf

[   45.603578] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready                              

[   47.587503] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off             

[   47.587586] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready                              

[   47.651300] icssg-prueth icssg1-eth eth1: Link is Down                                           

[   48.611515] icssg-prueth icssg1-eth eth2: Link is Down                                           

[   50.723487] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off             

[   51.683553] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off             

                                                                                                    

root@am64xx-evm:~# ./hsr_set_up.sh hsr_hw eth1 eth2 192.168.200.1                                   

ip=192.168.200.1                                                                                    

if=hsr0                                                                                             

mac=70:ff:76:1e:e7:8c                                                                               

slave-a=eth1                                                                                        

slave-b=eth2                                                                                        

device=platform/icssg1-eth                                                                          

[   62.094981] icssg-prueth icssg1-eth eth1: Link is Down                                           

[   62.119946] icssg-prueth icssg1-eth eth2: Link is Down                                           

[   62.124124] remoteproc remoteproc7: stopped remote processor 3008a000.txpru                      

[   62.124146] remoteproc remoteproc16: stopped remote processor 30084000.rtu                       

[   62.124156] remoteproc remoteproc15: stopped remote processor 300b4000.pru                       

[   62.124164] remoteproc remoteproc8: stopped remote processor 3008c000.txpru                      

[   62.124173] remoteproc remoteproc14: stopped remote processor 30086000.rtu                       

[   62.124181] remoteproc remoteproc13: stopped remote processor 300b8000.pru                       

hsr-tag-ins-offload: off                                                                            

hsr-tag-rm-offload: off                                                                             

hsr-fwd-offload: off                                                                                

hsr-dup-offload: off                                                                                

hsr-tag-ins-offload: on                                                                             

hsr-tag-rm-offload: on                                                                              

hsr-fwd-offload: on                                                                                 

hsr-dup-offload: on                                                                                 

hsr-tag-ins-offload: off                                                                            

hsr-tag-rm-offload: off                                                                             

hsr-fwd-offload: off                                                                                

hsr-dup-offload: off                                                                                

hsr-tag-ins-offload: on                                                                             

hsr-tag-rm-offload: on                                                                              

hsr-fwd-offload: on                                                                                 

hsr-dup-offload: on                                                                                 

[   63.254568] icssg-prueth icssg1-eth: Enabling HSR offload mode                                   

[   66.300726] remoteproc remoteproc15: powering up 300b4000.pru                                    

[   66.302800] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, siz6

[   66.302842] remoteproc remoteproc15: unsupported resource 5                                      

[   66.304816] remoteproc remoteproc15: remote processor 300b4000.pru is now up                     

[   66.305794] remoteproc remoteproc16: powering up 30084000.rtu                                    

[   66.307973] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, siz2

[   66.308033] remoteproc remoteproc16: remote processor 30084000.rtu is now up                     

[   66.308073] remoteproc remoteproc7: powering up 3008a000.txpru                                   

[   66.311698] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, si8

[   66.311757] remoteproc remoteproc7: remote processor 3008a000.txpru is now up                    

[   66.313977] remoteproc remoteproc13: powering up 300b8000.pru                                    

[   66.315799] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, siz8

[   66.315836] remoteproc remoteproc13: unsupported resource 5                                      

[   66.315939] remoteproc remoteproc13: remote processor 300b8000.pru is now up                     

[   66.315986] remoteproc remoteproc14: powering up 30086000.rtu                                    

[   66.317382] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, siz4

[   66.317442] remoteproc remoteproc14: remote processor 30086000.rtu is now up                     

[   66.317486] remoteproc remoteproc8: powering up 3008c000.txpru                                   

[   66.319102] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, si6

[   66.319161] remoteproc remoteproc8: remote processor 3008c000.txpru is now up                    

[   66.325378] pps pps1: new PPS source ptp2                                                        

[   69.411989] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off             

[   69.412065] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready                              

[   70.435688] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off             

[   70.435762] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready                              

[   71.417778] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready                              

root@am64xx-evm:~# [   75.555683] icssg-prueth icssg1-eth eth1: Link is Down                        

[   83.747740] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off             

                                                                                                    

root@am64xx-evm:~# [   90.915523] icssg-prueth icssg1-eth eth2: Link is Down                        

[   98.083772] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off             

                                                                                                    

root@am64xx-evm:~# iperf3 -s -B 192.168.200.1                                                       

-----------------------------------------------------------                                         

Server listening on 5201 (test #1)                                                                  

-----------------------------------------------------------                                         

[  115.491676] icssg-prueth icssg1-eth eth1: Link is Down                                           

Accepted connection from 192.168.200.2, port 56134                                                  

[  5] local 192.168.200.1 port 5201 connected to 192.168.200.2 port 56321                           

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams                

[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/6245 (0%)                         

[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  0.013 ms  0/6250 (0%)                         

[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6251 (0%)                         

[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  0.007 ms  0/6250 (0%)                         

[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  0.005 ms  0/6250 (0%)                         

[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6250 (0%)                         

[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  0.006 ms  0/6250 (0%)                         

- - - - - - - - - - - - - - - - - - - - - - - - -                                                   

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams                

[  5]   0.00-6.00   sec  16.1 MBytes  22.5 Mbits/sec  0.009 ms  0/42134 (0%)  receiver              

iperf3: the client has terminated                                                                   

-----------------------------------------------------------                                         

Server listening on 5201 (test #2)                                                                  

-----------------------------------------------------------                                         

^Ciperf3: interrupt - the server has terminated                                                     

root@am64xx-evm:~# iperf3 -c 192.168.200.2 -t 10 -b 20M -l 400 -u                                   

Connecting to host 192.168.200.2, port 5201                                                         

[  5] local 192.168.200.1 port 43009 connected to 192.168.200.2 port 5201                           

[ ID] Interval           Transfer     Bitrate         Total Datagrams                               

[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  6246                                          

[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  6249                                          

[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  6252                                          

[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  6248                                          

[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  6252                                          

[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  6248                                          

[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

- - - - - - - - - - - - - - - - - - - - - - - - -                                                   

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams                

[  5]   0.00-10.00  sec  23.8 MBytes  20.0 Mbits/sec  0.000 ms  0/62495 (0%)  sender                

[  5]   0.00-10.21  sec  12.5 MBytes  10.3 Mbits/sec  0.008 ms  29698/62495 (48%)  receiver         

                                                                                                    

iperf Done.                                                                                         

root@am64xx-evm:~# devlink dev param show platform/icssg1-eth name cut_thru                         

platform/icssg1-eth:                                                                                

  name cut_thru type driver-specific                                                                

    values:                                                                                         

      cmode runtime value 1                                                                         

root@am64xx-evm:~# 

For EVM2 :

# EVM2 or DUT : 

 _____                    _____           _         _   

|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_ 

|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|

|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  

              |___|                    |___|            



Arago Project am64xx-evm -



Arago 2023.10 am64xx-evm -



am64xx-evm login: root

[   34.401705] audit: type=1006 audit(1651168793.956:2): pid=987 uid=0 old-auid=4294967295 auid=0 tt1

[   34.401793] audit: type=1300 audit(1651168793.956:2): arch=c00000b7 syscall=64 success=yes exit=1)

[   34.401854] audit: type=1327 audit(1651168793.956:2): proctitle="(systemd)"

root@am64xx-evm:~# ip link set dev eth1 up

[   44.660022] remoteproc remoteproc15: powering up 300b4000.pru

[   44.663563] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, size6

[   44.663606] remoteproc remoteproc15: unsupported resource 5

[   44.663635] remoteproc remoteproc15: remote processor 300b4000.pru is now up

[   44.663675] remoteproc remoteproc16: powering up 30084000.rtu

[   44.666344] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-prueth-fw.elf, size8

[   44.666397] remoteproc remoteproc16: remote processor 30084000.rtu is now up

[   44.666442] remoteproc remoteproc7: powering up 3008a000.txpru

[   44.667800] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, siz2

root@am64xx-evm:~# [   44.667857] remoteproc remoteproc7: remote processor 3008a000.txpru is now up

[   44.668140] remoteproc remoteproc13: powering up 300b8000.pru

[   44.669807] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, size2

[   44.669845] remoteproc remoteproc13: unsupported resource 5

[   44.669874] remoteproc remoteproc13: remote processor 300b8000.pru is now up

[   44.669916] remoteproc remoteproc14: powering up 30086000.rtu

[   44.671541] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, size4

[   44.671594] remoteproc remoteproc14: remote processor 30086000.rtu is now up

[   44.671633] remoteproc remoteproc8: powering up 3008c000.txpru

[   44.672889] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, siz4

[   44.672942] remoteproc remoteproc8: remote processor 3008c000.txpru is now up

[   44.675852] pps pps1: new PPS source ptp2

ip link set dev eth2 up

root@am64xx-evm:~# [   47.779818] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[   47.780842] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[   49.763825] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

[   49.763902] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready



root@am64xx-evm:~# [   59.043623] icssg-prueth icssg1-eth eth1: Link is Down

[   66.211803] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[   71.267824] icssg-prueth icssg1-eth eth2: Link is Down

root@am64xx-evm:~# ./hsr_set_up.sh hsr_hw eth1 eth2 192.168.200.10[   78.436319] icssg-prueth icssg1f



ip=192.168.200.10

if=hsr0

mac=70:ff:76:1e:e6:f3

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

[   86.577294] icssg-prueth icssg1-eth eth1: Link is Down

[   86.612286] icssg-prueth icssg1-eth eth2: Link is Down

[   86.617942] remoteproc remoteproc7: stopped remote processor 3008a000.txpru

[   86.617966] remoteproc remoteproc16: stopped remote processor 30084000.rtu

[   86.617976] remoteproc remoteproc15: stopped remote processor 300b4000.pru

[   86.617985] remoteproc remoteproc8: stopped remote processor 3008c000.txpru

[   86.617994] remoteproc remoteproc14: stopped remote processor 30086000.rtu

[   86.618003] remoteproc remoteproc13: stopped remote processor 300b8000.pru

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

[   87.753877] icssg-prueth icssg1-eth: Enabling HSR offload mode

[   90.800998] remoteproc remoteproc15: powering up 300b4000.pru

[   90.803977] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, size6

[   90.804015] remoteproc remoteproc15: unsupported resource 5

[   90.804041] remoteproc remoteproc15: remote processor 300b4000.pru is now up

[   90.804083] remoteproc remoteproc16: powering up 30084000.rtu

[   90.808810] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, size2

[   90.808873] remoteproc remoteproc16: remote processor 30084000.rtu is now up

[   90.808919] remoteproc remoteproc7: powering up 3008a000.txpru

[   90.810410] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, siz8

[   90.810470] remoteproc remoteproc7: remote processor 3008a000.txpru is now up

[   90.810754] remoteproc remoteproc13: powering up 300b8000.pru

[   90.812621] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, size8

[   90.812659] remoteproc remoteproc13: unsupported resource 5

[   90.812687] remoteproc remoteproc13: remote processor 300b8000.pru is now up

[   90.812730] remoteproc remoteproc14: powering up 30086000.rtu

[   90.814022] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, size4

[   90.814078] remoteproc remoteproc14: remote processor 30086000.rtu is now up

[   90.814120] remoteproc remoteproc8: powering up 3008c000.txpru

[   90.815481] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, siz6

[   90.815535] remoteproc remoteproc8: remote processor 3008c000.txpru is now up

[   90.816971] pps pps1: new PPS source ptp2

[   93.923837] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off

[   93.924269] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

[   94.948071] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off

[   94.948149] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready

[   95.913986] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready

root@am64xx-evm:~# 

root@am64xx-evm:~# devlink dev param show platform/icssg1-eth name cut_thru

platform/icssg1-eth:

  name cut_thru type driver-specific

    values:

      cmode runtime value 1

root@am64xx-evm:~# 

EVM3/Testboard2 :

# EVM3 or Testboard 2 

|  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_                                             

|     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|                                            

|__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|                                              

              |___|                    |___|                                                        

                                                                                                    

Arago Project am64xx-evm -                                                                          

                                                                                                    

Arago 2023.10 am64xx-evm -                                                                          

                                                                                                    

am64xx-evm login: root                                                                              

[   34.856529] audit: type=1006 audit(1651283043.550:2): pid=985 uid=0 old-auid=4294967295 auid=0 t1

[   34.856558] audit: type=1300 audit(1651283043.550:2): arch=c00000b7 syscall=64 success=yes exit=)

[   34.856572] audit: type=1327 audit(1651283043.550:2): proctitle="(systemd)"                      

root@am64xx-evm:~# ip link set dev eth1 up                                                          

[   43.333576] remoteproc remoteproc13: powering up 300b4000.pru                                    

[   43.338013] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-prueth-fw.elf, siz6

[   43.338054] remoteproc remoteproc13: unsupported resource 5                                      

[   43.338085] remoteproc remoteproc13: remote processor 300b4000.pru is now up                     

root@am64xx-evm:~# [   43.338128] remoteproc remoteproc14: powering up 30084000.rtu                 

ip link set dev eth2 up[   43.342688] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-8

[   43.342746] remoteproc remoteproc14: remote processor 30084000.rtu is now up                     

[   43.342787] remoteproc remoteproc7: powering up 3008a000.txpru                                   

[   43.347688] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-prueth-fw.elf, si2

[   43.347748] remoteproc remoteproc7: remote processor 3008a000.txpru is now up                    

[   43.350049] remoteproc remoteproc15: powering up 300b8000.pru                                    

[   43.352164] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-prueth-fw.elf, siz2

[   43.352202] remoteproc remoteproc15: unsupported resource 5                                      

[   43.352231] remoteproc remoteproc15: remote processor 300b8000.pru is now up                     

[   43.352273] remoteproc remoteproc16: powering up 30086000.rtu                                    

[   43.354273] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-prueth-fw.elf, siz4

[   43.354326] remoteproc remoteproc16: remote processor 30086000.rtu is now up                     

[   43.354366] remoteproc remoteproc8: powering up 3008c000.txpru                                   

[   43.356225] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-prueth-fw.elf, si4

[   43.356283] remoteproc remoteproc8: remote processor 3008c000.txpru is now up                    

[   43.358649] pps pps1: new PPS source ptp2                                                        

                                                                                                    

root@am64xx-evm:~# [   46.435494] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready           

[   46.436844] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off             

[   48.227525] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off             

[   48.227614] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready                              

[   48.483328] icssg-prueth icssg1-eth eth1: Link is Down                                           

[   51.555509] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off             

                                                                                                    

root@am64xx-evm:~# [   60.515486] icssg-prueth icssg1-eth eth2: Link is Down                        

root@am64xx-evm:~# ./hsr_set_up.sh hsr_hw eth1 eth2 192.168.200.2[   67.683495] icssg-prueth icssg1f

                                                                                                    

ip=192.168.200.2                                                                                    

if=hsr0                                                                                             

mac=70:ff:76:1e:9f:54                                                                               

slave-a=eth1                                                                                        

slave-b=eth2                                                                                        

device=platform/icssg1-eth                                                                          

[   72.972956] icssg-prueth icssg1-eth eth1: Link is Down                                           

[   73.003958] icssg-prueth icssg1-eth eth2: Link is Down                                           

[   73.010643] remoteproc remoteproc7: stopped remote processor 3008a000.txpru                      

[   73.010668] remoteproc remoteproc14: stopped remote processor 30084000.rtu                       

[   73.010678] remoteproc remoteproc13: stopped remote processor 300b4000.pru                       

[   73.010687] remoteproc remoteproc8: stopped remote processor 3008c000.txpru                      

[   73.010698] remoteproc remoteproc16: stopped remote processor 30086000.rtu                       

[   73.010707] remoteproc remoteproc15: stopped remote processor 300b8000.pru                       

hsr-tag-ins-offload: off                                                                            

hsr-tag-rm-offload: off                                                                             

hsr-fwd-offload: off                                                                                

hsr-dup-offload: off                                                                                

hsr-tag-ins-offload: on                                                                             

hsr-tag-rm-offload: on                                                                              

hsr-fwd-offload: on                                                                                 

hsr-dup-offload: on                                                                                 

hsr-tag-ins-offload: off                                                                            

hsr-tag-rm-offload: off                                                                             

hsr-fwd-offload: off                                                                                

hsr-dup-offload: off                                                                                

hsr-tag-ins-offload: on                                                                             

hsr-tag-rm-offload: on                                                                              

hsr-fwd-offload: on                                                                                 

hsr-dup-offload: on                                                                                 

[   74.152806] icssg-prueth icssg1-eth: Enabling HSR offload mode                                   

[   77.201672] remoteproc remoteproc13: powering up 300b4000.pru                                    

[   77.204229] remoteproc remoteproc13: Booting fw image ti-pruss/am65x-sr2-pru0-pruhsr-fw.elf, siz6

[   77.204271] remoteproc remoteproc13: unsupported resource 5                                      

[   77.204298] remoteproc remoteproc13: remote processor 300b4000.pru is now up                     

[   77.204343] remoteproc remoteproc14: powering up 30084000.rtu                                    

[   77.208667] remoteproc remoteproc14: Booting fw image ti-pruss/am65x-sr2-rtu0-pruhsr-fw.elf, siz2

[   77.208726] remoteproc remoteproc14: remote processor 30084000.rtu is now up                     

[   77.208770] remoteproc remoteproc7: powering up 3008a000.txpru                                   

[   77.212793] remoteproc remoteproc7: Booting fw image ti-pruss/am65x-sr2-txpru0-pruhsr-fw.elf, si8

[   77.212857] remoteproc remoteproc7: remote processor 3008a000.txpru is now up                    

[   77.213222] remoteproc remoteproc15: powering up 300b8000.pru                                    

[   77.215736] remoteproc remoteproc15: Booting fw image ti-pruss/am65x-sr2-pru1-pruhsr-fw.elf, siz8

[   77.215780] remoteproc remoteproc15: unsupported resource 5                                      

[   77.215807] remoteproc remoteproc15: remote processor 300b8000.pru is now up                     

[   77.215851] remoteproc remoteproc16: powering up 30086000.rtu                                    

[   77.217979] remoteproc remoteproc16: Booting fw image ti-pruss/am65x-sr2-rtu1-pruhsr-fw.elf, siz4

[   77.218036] remoteproc remoteproc16: remote processor 30086000.rtu is now up                     

[   77.218080] remoteproc remoteproc8: powering up 3008c000.txpru                                   

[   77.219969] remoteproc remoteproc8: Booting fw image ti-pruss/am65x-sr2-txpru1-pruhsr-fw.elf, si6

[   77.220027] remoteproc remoteproc8: remote processor 3008c000.txpru is now up                    

[   77.222985] pps pps1: new PPS source ptp2                                                        

[   80.291995] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off             

[   80.292069] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready                              

[   81.315506] icssg-prueth icssg1-eth eth2: Link is Up - 1Gbps/Full - flow control off             

[   81.315789] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready                              

[   82.326142] IPv6: ADDRCONF(NETDEV_CHANGE): hsr0: link becomes ready                              

root@am64xx-evm:~# [   89.507530] icssg-prueth icssg1-eth eth1: Link is Down                        

[   96.675692] icssg-prueth icssg1-eth eth1: Link is Up - 1Gbps/Full - flow control off             

                                                                                                    

(reverse-i-search)`ipe': [  114.083498] icssg-prueth icssg1-eth eth2: Link is Down                  

root@am64xx-evm:~#                                                                                  

Connecting to host 192.168.200.1, port 5201                                                         

[  5] local 192.168.200.2 port 56321 connected to 192.168.200.1 port 5201                           

[ ID] Interval           Transfer     Bitrate         Total Datagrams                               

[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  6245                                          

[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  6251                                          

[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  6250                                          

^C[  5]   6.00-6.74   sec  1.77 MBytes  20.0 Mbits/sec  4638                                        

- - - - - - - - - - - - - - - - - - - - - - - - -                                                   

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams                

[  5]   0.00-6.74   sec  16.1 MBytes  20.0 Mbits/sec  0.000 ms  0/42134 (0%)  sender                

[  5]   0.00-6.74   sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (0%)  receiver                    

iperf3: interrupt - the client has terminated                                                       

root@am64xx-evm:~# iperf3 -s -B 192.168.200.2                                                       

-----------------------------------------------------------                                         

Server listening on 5201 (test #1)                                                                  

-----------------------------------------------------------                                         

^Ciperf3: interrupt - the server has terminated                                                     

root@am64xx-evm:~# iperf3 -s -B 192.168.200.2                                                       

-----------------------------------------------------------                                         

Server listening on 5201 (test #1)                                                                  

-----------------------------------------------------------                                         

Accepted connection from 192.168.200.1, port 57244                                                  

[  5] local 192.168.200.2 port 5201 connected to 192.168.200.1 port 43009                           

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams                

[  5]   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec  0.006 ms  2969/6243 (48%)                     

[  5]   1.00-2.00   sec  1.26 MBytes  10.6 Mbits/sec  0.007 ms  2945/6252 (47%)                     

[  5]   2.00-3.00   sec  1.26 MBytes  10.6 Mbits/sec  0.005 ms  2949/6250 (47%)                     

[  5]   3.00-4.00   sec  1.28 MBytes  10.7 Mbits/sec  0.004 ms  2907/6252 (46%)                     

[  5]   4.00-5.00   sec  1.26 MBytes  10.6 Mbits/sec  0.006 ms  2939/6248 (47%)                     

[  5]   5.00-6.00   sec  1.24 MBytes  10.4 Mbits/sec  0.004 ms  2986/6248 (48%)                     

[  5]   6.00-7.00   sec  1.24 MBytes  10.4 Mbits/sec  0.005 ms  3004/6251 (48%)                     

[  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec  0.005 ms  2971/6252 (48%)                     

[  5]   8.00-9.00   sec  1.23 MBytes  10.3 Mbits/sec  0.006 ms  3025/6247 (48%)                     

[  5]   9.00-10.00  sec  1.24 MBytes  10.4 Mbits/sec  0.008 ms  3003/6252 (48%)                     

[  5]  10.00-10.21  sec  0.00 Bytes  0.00 bits/sec  0.008 ms  0/0 (0%)                              

- - - - - - - - - - - - - - - - - - - - - - - - -                                                   

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams                

[  5]   0.00-10.21  sec  12.5 MBytes  10.3 Mbits/sec  0.008 ms  29698/62495 (48%)  receiver         

-----------------------------------------------------------                                         

Server listening on 5201 (test #2)                                                                  

-----------------------------------------------------------                                         

^Ciperf3: interrupt - the server has terminated                                                     

root@am64xx-evm:~# devlink dev param show platform/icssg1-eth name cut_thru                         

platform/icssg1-eth:                                                                                

  name cut_thru type driver-specific                                                                

    values:                                                                                         

      cmode runtime value 1                                                                         

root@am64xx-evm:~# 

We use also Profisharks to sniff packets between EVM1 <-> EVM2 and EVM2 <-> EVM3. We have observed that the sender sends correctly the total number of packets but the forwarder didn't forward all the packets ...

Is it possible to try to reproduce this setup on your side ?  (to do iperf server on EVM1 and EVM3)

Tianyi

Hi Tianyi,

Thanks for sharing your findings, in my previous response I had EVM1 as iperf3 client and EVM3 as iperf3 server (same case as where you see packet loss currently) and I showed no packet loss for configuring only one tx queue for cut-through forwarding on the middle board/DUT/EVM2. 

However, my results for EVM1 as iperf3 server and EVM3 as iperf3 client actually shows packet loss for configuring only one tx queue for cut-through forwarding on the middle board/DUT/EVM2. See below:

root@am64xx-evm:~# iperf3 -s -B 192.168.1.10
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.1.50, port 48842
[ 5] local 192.168.1.10 port 5201 connected to 192.168.1.50 port 53694
[ ID] Interval          Transfer    Bitrate        Jitter   Lost/Total Datagrams
[ 5]  0.00-1.00  sec 1.31 MBytes 11.0 Mbits/sec 0.022 ms 2819/6245 (45%) 
[ 5]  1.00-2.00  sec 1.34 MBytes 11.2 Mbits/sec 0.005 ms 2744/6251 (44%) 
[ 5]  2.00-3.00  sec 1.30 MBytes 10.9 Mbits/sec 0.010 ms 2831/6250 (45%) 
[ 5]  3.00-4.00  sec 1.30 MBytes 10.9 Mbits/sec 0.021 ms 2830/6238 (45%) 
[ 5]  4.00-5.00  sec 1.31 MBytes 11.0 Mbits/sec 0.011 ms 2832/6260 (45%) 
[ 5]  5.00-6.00  sec 1.30 MBytes 10.9 Mbits/sec 0.011 ms 2840/6251 (45%) 
[ 5]  6.00-7.00  sec 1.30 MBytes 10.9 Mbits/sec 0.008 ms 2849/6251 (46%) 
^C[ 5]  7.00-7.49  sec  638 KBytes 10.7 Mbits/sec 0.008 ms 1429/3061 (47%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval          Transfer    Bitrate        Jitter   Lost/Total Datagrams
[ 5]  0.00-7.49  sec 9.78 MBytes 11.0 Mbits/sec 0.008 ms 21174/46807 (45%) receiver
iperf3: interrupt - the server has terminated 

What I noticed is that switching the cable connection on the middle board/DUT/EVM2 will result in no packet loss for the same configuration (So from EVM2 eth1 <--> EVM3 and EVM2 eth2 <--> EVM1, switch the cable such that EVM2 eth1 <--> EVM1 and EVM2 eth2 <--> EVM3).

This seems to imply that forwarding from eth2 to eth1 on EVM1 will achieve no packet loss but for some reason eth1 to eth2 will result in packet loss.

In your setup that resulted in the packet loss, are packets forwarded from eth1 to eth2?

root@am64xx-evm:~# iperf3 -s -B 192.168.1.10
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.1.50, port 42484
[ 5] local 192.168.1.10 port 5201 connected to 192.168.1.50 port 45543
[ ID] Interval          Transfer    Bitrate        Jitter   Lost/Total Datagrams
[ 5]  0.00-1.00  sec 2.38 MBytes 20.0 Mbits/sec 0.008 ms 0/6245 (0%) 
[ 5]  1.00-2.00  sec 2.38 MBytes 20.0 Mbits/sec 0.006 ms 0/6251 (0%) 
[ 5]  2.00-3.00  sec 2.38 MBytes 20.0 Mbits/sec 0.008 ms 0/6249 (0%) 
[ 5]  3.00-4.00  sec 2.38 MBytes 20.0 Mbits/sec 0.007 ms 0/6252 (0%) 
[ 5]  4.00-5.00  sec 2.38 MBytes 20.0 Mbits/sec 0.005 ms 0/6249 (0%) 
[ 5]  5.00-6.00  sec 2.38 MBytes 20.0 Mbits/sec 0.005 ms 0/6251 (0%) 
[ 5]  6.00-7.00  sec 2.38 MBytes 20.0 Mbits/sec 0.007 ms 0/6248 (0%) 
[ 5]  7.00-8.00  sec 2.38 MBytes 20.0 Mbits/sec 0.004 ms 0/6252 (0%) 
[ 5]  8.00-9.00  sec 2.38 MBytes 20.0 Mbits/sec 0.006 ms 0/6249 (0%) 
[ 5]  9.00-10.00 sec 2.38 MBytes 20.0 Mbits/sec 0.004 ms 0/6251 (0%) 
[ 5] 10.00-11.00 sec 2.38 MBytes 20.0 Mbits/sec 0.005 ms 0/6249 (0%) 
^C[ 5] 11.00-11.51 sec 1.22 MBytes 20.0 Mbits/sec 0.006 ms 0/3206 (0%) 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval          Transfer    Bitrate        Jitter   Lost/Total Datagrams
[ 5]  0.00-11.51 sec 27.4 MBytes 20.0 Mbits/sec 0.006 ms 0/71952 (0%) receiver
iperf3: interrupt - the server has terminated 

Could you try switching the cable connection on the DUT?  It is odd that the packet loss only appears in one direction so I just want to check that it's not a specific interface issue.

-Daolin

Hi Daolin,

Yes I've tried to switch cable connection in the EVM2. And there is one direction that leads to some packets lost.

Based on the tests, when EVM2(forwarding) has the following configuration :

- eth2 : Receiving from EVM1 / Rx

- eth1 : forwarding to EVM3 /Tx

There is no packet lost.

When EVM2 has the following configuration :

- eth1 : Receiving from EVM1 / Rx

- eth2 : forwarding to EVM3 / Tx

we have the packets losts.

Another thing I've observed (maybe that could help you) is that using ethtool :

ethtool -S ethx

even if there are some packets lost during the forwarding action(confirmed also by using 2 Profisharks between EVM1 <-> EVM2 and EVM2 <-> EVM3 ), the values displayed on EVM2:

ethtool -S ethx | rx_good_frames

or

ethtool -S ethx | tx_good_frames

doesn't show any packet lost ...

Tianyi

Hi Tianyi,

The fact that ethtool -S ethX, the MAC hardware statistics, are not showing lost frames seems to imply that higher in the network stack on EVM2, about half of the packets are just not being forwarded and reach the exit ethernet port. 

Additionally, the issue of cut-through forward from eth2 to eth1 won't result in packet loss but from eth1 to eth2 will, I will check with the internal team to see what potentially causes this.

I'll respond with an update by tomorrow.

-Daolin

Hi Daolin,

Thanks for the quick update. Feel free to reach out if you need any support from us!

This feature cut-through is very important for our customer ! They need this by the middle of June ...

Tianyi

Update:

I've filed an internal bug report on this as this seems like a bug.

Quick comment from development team is to print out the packet aggregator (PA) stats 0x300a7000 - 0x300a7300, could you print out the content of these register addresses?

From Linux userspace you can do this with devmem2, here is the bash script I used but feel free to use whatever tool you want to print out the PA contents.

#!/bin/sh

startaddr=$1
endaddr=$2
inc=$3
range=$(($endaddr-$startaddr))

echo "startaddr=$startaddr"
echo "endaddr=$endaddr"
echo "range="
printf '%x\n' "$range"

addr=$startaddr
for ((i=1;i<=$range;i++));
do
	hex_number=$(printf "%x" "$addr")
	echo "readaddr=$hex_number"
	devmem2 $addr | grep Read | cut -d " " -f 7 | tee -a PA_stats.txt
	echo " "
	addr=$(($addr+$inc)) 
done

-Daolin

Hello Daolin,

You cand find the PA stats on the three board when packets lost occur :

EVM1 : sender :

0x020AD191
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000123
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000000E0
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000003E3
0x0000035D
0x0000001F
0x0000001F
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000028A
0x0000028A
0x00006E00
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000003A
0x0000036E
0x0000036C
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000115
0x00000243
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000003A9
0x000003A9
0x00007FFF
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000317
0x00000315
0x0000001E
0x0000001E
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x020AD191
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0325C587
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000005EC
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00012B44
0x0001E5D1
0x00000000
0x00000000
0x0000000E
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000128
0x00000118
0x0008799D
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000005
0x00000004
0x00000123
0x00000114
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000000E
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00004032
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000007
0x00000012
0x00000015
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00010688
0x00000000
0x00000489
0x000004E4
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000

EVM2 : forwarder :

0x00001EA0
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000066
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000000E0
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000003E3
0x0000035D
0x0000001F
0x0000001F
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000028A
0x0000028A
0x00006E00
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000003A
0x0000036E
0x0000036C
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000115
0x00000243
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000003A9
0x000003A9
0x00007FFF
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000317
0x00000315
0x0000001E
0x0000001E
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00001EA0
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00007850
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000186
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000730
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000033
0x0000011B
0x00000000
0x00000000
0x0000002A
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000067
0x0000006B
0x0004BCAD
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000001
0x00000000
0x00000066
0x0000006B
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000001
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000071CA
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000003
0x0000000E
0x00000010
0x00000010
0x00000000
0x00000000
0x00000000
0x00000000
0x000160F2
0x00000000
0x0000000B
0x00000000
0x00000339
0x00000161
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000

EVM3 : receiver :

0x02456790
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x02193A9B
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000005EC
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00017056
0x00012D0A
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000129
0x00000122
0x00087ACF
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000000F
0x00000007
0x0000011A
0x0000011B
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000000D
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00014800
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000007
0x00000000
0x00000011
0x00000015
0x00000000
0x00000000
0x00000000
0x00000000
0x00007D2D
0x0000008D
0x00000000
0x0000CF71
0x000004FE
0x00000476
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000

Any estimate on when this bug might be resolved?

Tianyi

Hi Tianyi,

Thanks for sharing these values. I noticed that the PA_Stats printout for EVM1 and EVM2 are a lot longer than the expected 768 number of values. I think you might have dumped the contents of those registers twice and appended to the same file. Are you able to dump the contents once into a file for each EVM again? Looking for 768 number of values specifically.

Unfortunately, I currently don't have a timeline on when the bug might be resolved. I understand the urgency on this issue and we are still working on understanding the results of the PA stats. I'll reply with an update next week.

-Daolin

Hi Daolin,

Here you can find the logs when there are packets lost :

Sender :

0x00002040
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x01B4F9F8
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000005EC
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000A44
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000033
0x0000F497
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000000A
0x0000000B
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000001
0x00000001
0x00000009
0x0000000A
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000007
0x0000000C
0x00000016
0x00000014
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000008B
0x00000062
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000

Forwarder :

0x00000134
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00003105
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00001C2E
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000003
0x0000005F
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000045
0x00000049
0x00000000
0x00000002
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000003
0x0000000A
0x00000042
0x0000003F
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000021
0x00000013
0x00000016
0x00000000
0x00000000
0x00000000
0x00000000
0x0000F339
0x00000000
0x00000000
0x00000000
0x0000016E
0x00000079
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000

Receiver :

0x00F19C54
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x0000382F
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000005EC
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00008AE3
0x00000079
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000010
0x00000014
0x00000001
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000006
0x00000009
0x0000000A
0x0000000B
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000007
0x00000000
0x00000010
0x00000010
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x000087A2
0x00000000
0x00000000
0x00000096
0x00000084
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000
0x00000000

Tianyi

Hi Tianyi,

Just as an update, the development team is still analyzing the root cause of this issue (with high priority).

We've found that the issue is observed with the standard cut-through forwarding (even without HSR setup) which may be what is causing this issue observed with HSR setup with cut-through forwarding.

I'm still trying to get an estimated timeframe from the team for a fix.

Thanks for your patience.

-Daolin

Hi Daolin,

I come back to you to check if there is any update about this issue ! Also did you have any idea if the bug comes fro the PRU firmware or the Linux Kernel code ?

Tianyi

Hi Tianyi, 

Apologies for the delayed response! I was away at training this week and I'm trying to catch up on the current the debug status. From what I'm understanding from earlier this week:

1. No packet is being dropped by firmware (checked during rtu eof when we inform tx_pru to add crc error or not for cut-through frames)

2. Did a wireshark capture, and looks like receiving EVM (EVM3) is not able to receive some packets 

3. At least from the counters we have in firmware there are no drops on forwarding path

I cannot confirm if this means the bug is in PRU firmware or the Linux kernel. I will check with the software team on the current debug and get back to you next week.

-Daolin

Tianyi/Milan,

This week's update from the PRU firmware expert:

I attached Profishark on both sides of forwarding DUT, all the packets are being forwarded but some of them gets corrupted during forwarding so that is causing the issue.
I had made some changes in firmware and was able to get TCP working but with a lot of retries each second

There appears to be a promising change to firmware that may resolve the issue (no 32B block corruptions, no Preamble corruption), but the firmware expert is still saying that there are two issues with this change (extra 1 byte preamble 0x55 captured by Profishark, Tx Underflow issues).

This issue remains a top priority for us but the debug has been difficult for the firmware team (at this point appears to be issue in the PRU firmware), so I don't have a timeline for when the fix will be determined.

-Daolin

Hi Daolin,

I was testing the latency that forwarding can introduce with cut-through mode  (HSR) and observe that the latency is increasing following the packets size. This behaviour seems to match store and forward mode and not cut-through mode ... 

We did the test with 3 board EVMs

EVM1 <-> Profishark1(Profitap1) <-> EVM2 <-> Profishark2(Profitap2) <-> EVM3

We send packet from EVM1 to EVM3 and Profishark 1 and 2 timestamp the packets

For 400B packets :

Mean = 5.231 us
Max = 5.350 us

For 1300B packets :

Mean = 12.398 us
Max = 12.526 us

Here you can find the script :

#!/bin/sh

#ffload - sh hsr_setup.sh hsr_sw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR> 
#For non offload - sh hsr_setup.sh hsr_hw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR> 

echo "$@"

if [ "$#" -ne "4" ]
then
    echo "usage:"
    echo "$0 <hsr_sw/hsr_hw/cut-thru> <intf1> <intf2> <IPaddress>"
    exit
fi

if [ "$1" != "hsr_hw" ] && [ "$1" != "hsr_sw" ] && [ "$1" != "cut-thru" ]
then
    echo "the first argument must be 'hsr_hw', 'hsr_sw' or 'cut-thru'."
    exit
fi

if=hsr0 

ifa=$2 
ifb=$3 

ip=$4 
mac=`ifconfig $ifa | grep ether | cut -d " " -f 10` 
# Hardcode the device name
device="platform/icssg1-eth"
#device+=`dmesg | grep $ifa | grep icssg-prueth | grep -m 1 "Link is Up" | awk '{print $4}'` 

echo "ip=$ip" 
echo "if=$if" 
echo "mac=$mac" 
echo "slave-a=$ifa" 
echo "slave-b=$ifb" 
echo "device=$device" 

ip link delete hsr0  2> /dev/null 

ip link set $ifa down 
ip link set $ifb down 
sleep 1 

if [ "$1" == "hsr_hw" ] 
then 
        ethtool -k "$ifa" | grep hsr
        ethtool -K "$ifa" hsr-fwd-offload on
        ethtool -K "$ifa" hsr-dup-offload on
        ethtool -K "$ifa" hsr-tag-ins-offload on
        ethtool -K "$ifa" hsr-tag-rm-offload on
        ethtool -k "$ifa" | grep hsr

        ethtool -k "$ifb" | grep hsr
        ethtool -K "$ifb" hsr-fwd-offload on
        ethtool -K "$ifb" hsr-dup-offload on
        ethtool -K "$ifb" hsr-tag-ins-offload on
        ethtool -K "$ifb" hsr-tag-rm-offload on
        ethtool -k "$ifb" | grep hsr

        devlink dev param set "$device" name hsr_offload_mode value true cmode runtime

        sleep 3 

        # Enable Cut-Through : 
        devlink dev param set "$device" name cut_thru value 1 cmode runtime
fi

ip link set dev $ifa address $mac 
ip link set dev $ifb address $mac 

ip link set $ifa up 
sleep 1 

ip link set $ifb up 
sleep 1 

ip link add name $if type hsr slave1 $ifa slave2 $ifb supervision 45 version 1 

sleep 3 

ip addr add $ip/24 dev $if 
ip link set $if up 

Tianyi

Hi Tianyi,

Just to clarify, with this setup to test for latency, were you testing in the forwarding direction without packet loss that we were discussing before (eth2-->eth1)? From what I recall, eth1-->eth2 was the problematic forwarding path.

Actually yesterday while I was testing several combinations of the following

1. 100Mbps vs 1Gbps

2. HSR offload with cut-through vs No HSR just cut-through

3. Iperf3 UDP 20M vs 90M vs 200M

4. eth1--> eth2 direction vs eth2--> eth1 direction

I saw that with the Iperf 3 IDP 20M (iperf3 -c 192.168.200.2 -t 10 -b 20M -l 400 -u - same command you used), 1Gbps, HSR offload with cut through, both forwarding directions, I no longer see lost packets that we were seeing originally. However, at larger bit rates (iperf3 -c 192.168.200.2 -t 10 -b 200M -l 400 -u), I see lost datagrams in both directions. I want double check this today in case there is something wrong with my setup as this appears different from the issue we were seeing before. I did not change the PRU firmware nor the Linux kernel since my last tests a few weeks ago.

As a sanity check for your report here about the cut-through showing larger latency for larger packets, can you check if the same shows up for the just a cut-through switch setup without HSR? Below is the script I use to setup cut-through no HSR.

#!/bin/sh

#Setup Cut through switching (without HSR)

if [ "$#" != "4" ]
then
        echo "$0 <portA> <portB> <ethernet type> <ip addr/16>"
        exit
fi

portA=$1
portB=$2
enettype=$3
ipaddr=$4

ip link set br0 down
sleep 1
ip link delete br0 2> /dev/null
sleep 1

if [ "$3" == "icssg" ]
then
	ip link set dev $portA down
	ip link set dev $portB down
	devlink dev param set platform/icssg1-eth name switch_mode value 1 cmode runtime
	ip link add name br0 type bridge
	ip link set dev $portA master br0
	ip link set dev $portB master br0
	ip link set dev br0 up
	devlink dev param set platform/icssg1-eth name cut_thru value 1 cmode runtime
	ip link set dev $portA up
	sleep 1
	ip link set dev $portB up
	sleep 1
	ip addr add $ipaddr dev br0
	bridge vlan add dev br0 vid 1 pvid untagged self
elif [ "$3" == "cpsw" ]
then
	ip link set dev $portA down 
	ip link set dev $portB down
	devlink dev param set platform/8000000.ethernet name switch_mode value true cmode runtime
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port1/cut_thru_rx_pri_mask
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port1/cut_thru_tx_pri_mask
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port2/cut_thru_rx_pri_mask
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port2/cut_thru_tx_pri_mask
	ethtool --set-priv-flags $portA cut-thru on
	ethtool --set-priv-flags $portB cut-thru on
	ip link add name br0 type bridge
	ip link set dev br0 type bridge ageing_time 1000
	ip link set dev $portA up
	sleep 1
	ip link set dev $portB up
	sleep 1
	ip link set dev $portA master br0
	ip link set dev $portB master br0
	ip link set dev br0 type bridge stp_state 1
	ip link set dev br0 up
	ip addr add $ipaddr dev br0
	bridge vlan add dev br0 vid 1 self
	bridge vlan add dev br0 vid 1 pvid untagged self
fi
	

-Daolin

Tianyi,

>>>I want double check this today in case there is something wrong with my setup as this appears different from the issue we were seeing before. I did not change the PRU firmware nor the Linux kernel since my last tests a few weeks ago.

I just double checked this and it seems that I can no longer reproduce the issue with the packet loss occurring only in one direction for HSR offload with cut-through. What I see instead is this:

Test 1: Check current one direction drop frames with cut-through still occurs
Setup: HSR offload with cut through on all 3 EVMs, 1Gbps line rate, iperf3 udp at 20M bit rate
Result: No packet loss in both directions! Strange because I did not change setup, PRU firmware, nor Linux kernel since last when I saw the issue

Test 2: Same as test 1 except at 200M bit rate
Setup: HSR offload with cut through all EVMs, 1Gbps line rate, iperf3 udp at 200M bit rate
Result: Packet loss in both directions! 

Is it possible for you to check on your setup if the original issue still occurs?

This is the script I used for my HSR offload with cut through setup (very similar to yours): 

#!/bin/sh

#For non offload - sh hsr_setup.sh hsr_sw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR>
#For non offload - sh hsr_setup.sh hsr_hw <INTF_A> <INTF_B> <HSR_INTF_IP_ADDR>

if [ "$#" != "5" ]
then
        echo "$0 <hsr_sw/hsr_hw> <intf1> <intf2> <ip addr> <portforward>"
        exit
fi

if [ "$1" != "hsr_sw" ] && [ "$1" != "hsr_hw" ]
then
        echo "$0 <hsr_sw|hsr_hw>"
        exit
fi

if=hsr0

ifa=$2
ifb=$3
        
ip=$4
mac=`ifconfig $ifa | grep ether | cut -d " " -f 10`

portforward=$5

device="platform/icssg1-eth"
#device+=`dmesg | grep $ifa | grep icssg-prueth | grep -m 1 "Link is Up" | awk '{print $4}'`

echo "ip=$ip"
echo "if=$if"
echo "mac=$mac"
echo "slave-a=$ifa"
echo "slave-b=$ifb"
echo "device=$device"

ip link set hsr0 down
sleep 1
ip link delete hsr0  2> /dev/null
sleep 1

ip link set $ifa down
ip link set $ifb down
sleep 2

if [ "$1" == "hsr_hw" ]
then
        ethtool -k $ifa | grep hsr
        ethtool -K $ifa hsr-fwd-offload off
        ethtool -K $ifa hsr-dup-offload off
        ethtool -K $ifa hsr-tag-ins-offload off
        ethtool -K $ifa hsr-tag-rm-offload off

        ethtool -K $ifa hsr-fwd-offload on
        ethtool -K $ifa hsr-dup-offload on
        ethtool -K $ifa hsr-tag-ins-offload on
        ethtool -K $ifa hsr-tag-rm-offload on
        ethtool -k $ifa | grep hsr

        ethtool -k $ifb | grep hsr
        ethtool -K $ifb hsr-fwd-offload off
        ethtool -K $ifb hsr-dup-offload off
        ethtool -K $ifb hsr-tag-ins-offload off
        ethtool -K $ifb hsr-tag-rm-offload off

        ethtool -K $ifb hsr-fwd-offload on
        ethtool -K $ifb hsr-dup-offload on
        ethtool -K $ifb hsr-tag-ins-offload on
        ethtool -K $ifb hsr-tag-rm-offload on
        ethtool -k $ifb | grep hsr

        devlink dev param set $device name hsr_offload_mode value true cmode runtime

	if [ "$portforward" == "cutthru" ]                                                   
        then                                        
                devlink dev param set platform/icssg1-eth name cut_thru value 1 cmode runtime
        else                                   
                devlink dev param set platform/icssg1-eth name cut_thru value 0 cmode runtime   
        fi 


else
        ethtool -k $ifa | grep hsr
        ethtool -K $ifa hsr-fwd-offload off
        ethtool -K $ifa hsr-dup-offload off
        ethtool -K $ifa hsr-tag-ins-offload off
        ethtool -K $ifa hsr-tag-rm-offload off
        ethtool -k $ifa | grep hsr

        ethtool -k $ifb | grep hsr
        ethtool -K $ifb hsr-fwd-offload off
        ethtool -K $ifb hsr-dup-offload off
        ethtool -K $ifb hsr-tag-ins-offload off
        ethtool -K $ifb hsr-tag-rm-offload off
        ethtool -k $ifb | grep hsr
        devlink dev param set $device name hsr_offload_mode value false cmode runtime        
fi

sleep 2

ip link set dev $ifa address $mac
ip link set dev $ifb address $mac

ip link set $ifa up
sleep 1

ip link set $ifb up
sleep 1

ip link add name $if type hsr slave1 $ifa slave2 $ifb supervision 45 version 1

sleep 3

ip addr add $ip/24 dev $if
ip link set $if up

-Daolin

Hi Daolin,

I did the test in the direction when there is no packet lost !
I just redid the test with the script you provided And there is still packet lost  ~50% : 

Test 1: Check current one direction drop frames with cut-through still occurs
Setup: HSR offload with cut through on all 3 EVMs, 1Gbps line rate, iperf3 udp at 20M bit rate

#EVM2 : the one who forward : 

root@am64xx-evm:~# device^C

root@am64xx-evm:~# ./ti_hsr_off_script.sh hsr_hw eth1 eth2 192.168.200.10 cutthru                                                                                                                           

ip=192.168.200.10

if=hsr0

mac=70:ff:76:1e:e6:f3

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

Cannot find device "hsr0"

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

[  450.840062] remoteproc remoteproc13: unsupported resource 5

[  450.849451] remoteproc remoteproc15: unsupported resource 5
                                                                                                                                

root@am64xx-evm:~#

EVM2 receiver :

root@am64xx-evm:~# ./ti_hsr_off_script.sh 

./ti_hsr_off_script.sh <hsr_sw/hsr_hw> <intf1> <intf2> <ip addr> <portforward>

root@am64xx-evm:~# ./ti_hsr_off_script.sh hsr_hw eth1 eth2 192.168.200.2 cutthru

ip=192.168.200.2

if=hsr0

mac=70:ff:76:1e:e7:8c

slave-a=eth1

slave-b=eth2

device=platform/icssg1-eth

Cannot find device "hsr0"

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

hsr-tag-ins-offload: off

hsr-tag-rm-offload: off

hsr-fwd-offload: off

hsr-dup-offload: off

hsr-tag-ins-offload: on

hsr-tag-rm-offload: on

hsr-fwd-offload: on

hsr-dup-offload: on

[  388.200215] remoteproc remoteproc13: unsupported resource 5

[  388.209683] remoteproc remoteproc15: unsupported resource 5

root@am64xx-evm:~# ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.100.2  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::3608:e1ff:fe80:a425  prefixlen 64  scopeid 0x20<link>

        ether 34:08:e1:80:a4:25  txqueuelen 1000  (Ethernet)

        RX packets 88  bytes 31102 (30.3 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 67  bytes 14532 (14.1 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 32  bytes 2518 (2.4 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 19  bytes 1562 (1.5 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494

        inet 192.168.200.2  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 13  bytes 928 (928.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 1168  bytes 98890 (96.5 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 1168  bytes 98890 (96.5 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# ifconfig

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.100.2  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::3608:e1ff:fe80:a425  prefixlen 64  scopeid 0x20<link>

        ether 34:08:e1:80:a4:25  txqueuelen 1000  (Ethernet)

        RX packets 93  bytes 34052 (33.2 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 67  bytes 14532 (14.1 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 50  bytes 3638 (3.5 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 23  bytes 1882 (1.8 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



hsr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1494

        inet 192.168.200.2  netmask 255.255.255.0  broadcast 0.0.0.0

        inet6 fe80::72ff:76ff:fe1e:e78c  prefixlen 64  scopeid 0x20<link>

        ether 70:ff:76:1e:e7:8c  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 29  bytes 1908 (1.8 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 1248  bytes 105210 (102.7 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 1248  bytes 105210 (102.7 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@am64xx-evm:~# ifoc

-sh: ifoc: command not found

root@am64xx-evm:~# iperf3 -s -B 192.168.200.2

-----------------------------------------------------------

Server listening on 5201 (test #1)

-----------------------------------------------------------

Accepted connection from 192.168.200.1, port 59836

[  5] local 192.168.200.2 port 5201 connected to 192.168.200.1 port 38465

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-1.00   sec  1.25 MBytes  10.4 Mbits/sec  0.012 ms  2981/6246 (48%)  

[  5]   1.00-2.00   sec  1.22 MBytes  10.2 Mbits/sec  0.008 ms  3050/6249 (49%)  

[  5]   2.00-3.00   sec  1.24 MBytes  10.4 Mbits/sec  0.016 ms  2994/6250 (48%)  

[  5]   3.00-4.00   sec  1.21 MBytes  10.1 Mbits/sec  0.003 ms  3086/6251 (49%)  

[  5]   4.00-5.00   sec  1.22 MBytes  10.2 Mbits/sec  0.007 ms  3050/6250 (49%)  

[  5]   5.00-6.00   sec  1.23 MBytes  10.3 Mbits/sec  0.007 ms  3027/6247 (48%)  

[  5]   6.00-7.00   sec  1.23 MBytes  10.3 Mbits/sec  0.005 ms  3021/6253 (48%)  

[  5]   7.00-8.00   sec  1.22 MBytes  10.2 Mbits/sec  0.004 ms  3049/6250 (49%)  

[  5]   8.00-9.00   sec  1.22 MBytes  10.2 Mbits/sec  0.011 ms  3048/6249 (49%)  

[  5]   9.00-10.00  sec  1.21 MBytes  10.1 Mbits/sec  0.011 ms  3084/6251 (49%)  

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-10.00  sec  12.2 MBytes  10.3 Mbits/sec  0.011 ms  30390/62496 (49%)  receiver

-----------------------------------------------------------

Server listening on 5201 (test #2)

-----------------------------------------------------------

# EVM1 : sender :

# EVM1 hsr off load cut-through enable: 

root@am64xx-evm:~# iperf3 -c 192.168.200.2 -t 10 -b 20M -l 400 -u                                    

Connecting to host 192.168.200.2, port 5201

[  5] local 192.168.200.1 port 38465 connected to 192.168.200.2 port 5201

[ ID] Interval           Transfer     Bitrate         Total Datagrams

[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  6246  

[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  6250  

[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  6249  

[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  6251  

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams

[  5]   0.00-10.00  sec  23.8 MBytes  20.0 Mbits/sec  0.000 ms  0/62496 (0%)  sender

[  5]   0.00-10.00  sec  12.2 MBytes  10.3 Mbits/sec  0.011 ms  30390/62496 (49%)  receiver



iperf Done.

root@am64xx-evm:~# 

Did you remove the cable between EVM1 and EVM3 when redoing the test ?

Tianyi

Could you provide the command you did for the test cut-through switch without hsr ?

Tianyi

Hi Tianyi, 

>>>Did you remove the cable between EVM1 and EVM3 when redoing the test ? 

Good point, I forgot to remove that direct cable connection. I will give it a test again to make sure. 

>>>Could you provide the command you did for the test cut-through switch without hsr ?

This is the script I used for the case of just cut-through with HSR. Just make sure you select "icssg", NOT "cpsw".

#!/bin/sh

#Setup Cut through switching (without HSR)

if [ "$#" != "4" ]
then
        echo "$0 <portA> <portB> <ethernet type> <ip addr/16>"
        exit
fi

portA=$1
portB=$2
enettype=$3
ipaddr=$4

ip link set br0 down
sleep 1
ip link delete br0 2> /dev/null
sleep 1

if [ "$3" == "icssg" ]
then
	ip link set dev $portA down
	ip link set dev $portB down
	devlink dev param set platform/icssg1-eth name switch_mode value 1 cmode runtime
	ip link add name br0 type bridge
	ip link set dev $portA master br0
	ip link set dev $portB master br0
	ip link set dev br0 up
	devlink dev param set platform/icssg1-eth name cut_thru value 1 cmode runtime
	ip link set dev $portA up
	sleep 1
	ip link set dev $portB up
	sleep 1
	ip addr add $ipaddr dev br0
	bridge vlan add dev br0 vid 1 pvid untagged self
elif [ "$3" == "cpsw" ]
then
	ip link set dev $portA down 
	ip link set dev $portB down
	devlink dev param set platform/8000000.ethernet name switch_mode value true cmode runtime
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port1/cut_thru_rx_pri_mask
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port1/cut_thru_tx_pri_mask
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port2/cut_thru_rx_pri_mask
	echo 1 > /sys/kernel/debug/8000000.ethernet/Port2/cut_thru_tx_pri_mask
	ethtool --set-priv-flags $portA cut-thru on
	ethtool --set-priv-flags $portB cut-thru on
	ip link add name br0 type bridge
	ip link set dev br0 type bridge ageing_time 1000
	ip link set dev $portA up
	sleep 1
	ip link set dev $portB up
	sleep 1
	ip link set dev $portA master br0
	ip link set dev $portB master br0
	ip link set dev br0 type bridge stp_state 1
	ip link set dev br0 up
	ip addr add $ipaddr dev br0
	bridge vlan add dev br0 vid 1 self
	bridge vlan add dev br0 vid 1 pvid untagged self
fi

-Daolin

Update:

I tested once again ensuring the direct cable connection between EVM1 and EVM3 was disconnected and I see the below results. Currently no change in PRU firmware or Linux kernel on my setup with these tests.

Test 1: Check current one direction drop frames with cut-through still occurs  
Setup: HSR offload with cut through on all 3 EVMs, 1Gbps line rate, iperf3 udp at 20M bit rate
Result: Packet loss only on eth1-->eth2 path, though I don't observe ~50% loss with half transfer size of 10-11 Mbits/sec anymore, now see 20Mbits/sec but lost datagrams, no change in PRU firmware or Linux kernel from before

EVM3 (Receiver) Iperf3 results
[  5] local 192.168.1.50 port 5201 connected to 192.168.1.10 port 60797
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  2.37 MBytes  19.9 Mbits/sec  0.011 ms  8/1785 (0.45%)  
[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/1785 (0%)  
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/1786 (0%)  
[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/1786 (0%)  
[  5]   4.00-5.00   sec  2.38 MBytes  19.9 Mbits/sec  0.009 ms  4/1785 (0.22%)  
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  2/1786 (0.11%)  
[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  2/1786 (0.11%)  
[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  0.010 ms  0/1786 (0%)  
[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  0.010 ms  3/1785 (0.17%)  
[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/1786 (0%)  
[  5]  10.00-11.00  sec  2.38 MBytes  20.0 Mbits/sec  0.011 ms  1/1786 (0.056%)  
[  5]  11.00-12.00  sec  2.35 MBytes  19.7 Mbits/sec  0.010 ms  27/1785 (1.5%)  
[  5]  12.00-13.00  sec  2.30 MBytes  19.3 Mbits/sec  0.008 ms  62/1786 (3.5%)  
[  5]  13.00-14.00  sec  2.34 MBytes  19.6 Mbits/sec  0.009 ms  35/1786 (2%)  
[  5]  14.00-15.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  4/1786 (0.22%)  
[  5]  15.00-16.00  sec  2.36 MBytes  19.8 Mbits/sec  0.010 ms  21/1785 (1.2%)  
[  5]  16.00-17.00  sec  2.36 MBytes  19.8 Mbits/sec  0.012 ms  19/1786 (1.1%)  
[  5]  17.00-18.00  sec  2.37 MBytes  19.8 Mbits/sec  0.009 ms  14/1786 (0.78%)  
[  5]  18.00-19.00  sec  2.38 MBytes  19.9 Mbits/sec  0.010 ms  5/1785 (0.28%)  
[  5]  19.00-20.00  sec  2.35 MBytes  19.7 Mbits/sec  0.010 ms  24/1786 (1.3%)  
[  5]  20.00-21.00  sec  2.36 MBytes  19.8 Mbits/sec  0.027 ms  22/1786 (1.2%)  
[  5]  21.00-22.00  sec  2.37 MBytes  19.9 Mbits/sec  0.035 ms  11/1786 (0.62%)  
[  5]  22.00-23.00  sec  2.37 MBytes  19.9 Mbits/sec  0.016 ms  9/1785 (0.5%)  
[  5]  23.00-24.00  sec  2.38 MBytes  20.0 Mbits/sec  0.011 ms  0/1786 (0%)  
[  5]  24.00-25.00  sec  2.38 MBytes  20.0 Mbits/sec  0.011 ms  0/1786 (0%)  
[  5]  25.00-26.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/1785 (0%)  
[  5]  26.00-27.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  3/1786 (0.17%)  
[  5]  27.00-28.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/1786 (0%)  
[  5]  28.00-29.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  2/1786 (0.11%)  
[  5]  29.00-30.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/1785 (0%)  
[  5]  30.00-31.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  0/1786 (0%)  
[  5]  31.00-32.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  3/1786 (0.17%)  
[  5]  32.00-33.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  0/1785 (0%)  
[  5]  33.00-34.00  sec  2.38 MBytes  20.0 Mbits/sec  0.010 ms  2/1786 (0.11%)  
[  5]  34.00-35.00  sec  2.38 MBytes  20.0 Mbits/sec  0.010 ms  0/1786 (0%)  
[  5]  35.00-36.00  sec  2.38 MBytes  19.9 Mbits/sec  0.013 ms  7/1786 (0.39%)  
[  5]  36.00-37.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  1/1785 (0.056%)  
[  5]  37.00-38.00  sec  2.38 MBytes  20.0 Mbits/sec  0.008 ms  4/1786 (0.22%)  
[  5]  38.00-39.00  sec  2.38 MBytes  20.0 Mbits/sec  0.009 ms  2/1786 (0.11%)  
[  5]  39.00-40.00  sec  2.37 MBytes  19.9 Mbits/sec  0.010 ms  8/1785 (0.45%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-40.00  sec  95.0 MBytes  19.9 Mbits/sec  0.010 ms  305/71427 (0.43%)  receiver

Test 2: Same as test 1 except at 200M bit rate
Setup: HSR offload with cut through all EVMs, 1Gbps line rate, iperf3 udp at 200M bit rate
Result: Packet loss in both directions! 

Test 3: Same as test 1 except at 100Mbps
Setup: HSR offload with cut through all EVMs, 100 Mbps line rate, iperf3 udp at 20M bit rate
Result: 0 packet loss both directions

Test 4: Same as test 3 except 95M bit rate
Setup: HSR offload with cut through all EVMs, 100 Mbps line rate, iperf3 udp at 95M bit rate
Result: Packet loss only in eth1--> eth2 direction 

So, for some reason I can no longer reproduce the exact same results as you are seeing despite being able to see it several weeks ago, although I still see the issue with eth1-->eth2 path losing packets.

The PRU firmware team is close to getting a version of the firmware without lost packets when in HSR offload cut through mode. Once I receive that firmware, I will run these tests once again to see if the problems I observe have been fixed.

-Daolin

Hi Daolin,

Thank you for sharing the details !

As we are targeting the 100Mbps line rate, and based on your Test 3 and our Tests, we can say that there is no packets lost ! However, we have observed that when enabling cut-through at 100Mbps line rate, we don't have the cut-through behaviour but store and forward behaviour.

We did the test with 3 board EVMs

EVM1 <-> Profishark1(Profitap1) <-> EVM2 <-> Profishark2(Profitap2) <-> EVM3

We send packet from EVM1 to EVM3 and Profishark 1 and 2 timestamp the packets

Here are the results that we have observed :

For 400B

Mean = 39.256 us

Max = 39.433 us

For 1300B

Mean = 111.211 us

Max = 111.424 us

The behaviour seems not to be cut-through ... Is it possible on your side to reproduce this ? This is more urgent than 1Gbips speed with packet lost issue ...

Tianyi

Hi Tianyi,

I should emphasize the "Test 4" results I shared in my previous response. With a 95 mbps UDP throughput at 100Mbps line rate, there is still packet loss in the eth1-->eth2 forwarding path.

I have just received a version of the PRU firmware internally that may fix the original issue in this thread (dropped packets on a single forwarding path for HSR offload with cut-through), so I will work on testing that version of the firmware first, including 100Mbps line rate.

Once we have ensured the cut-through feature is working correctly, then I can try and work with our development team (they are based in India) on testing if we see the same behavior with the increased packet size, resulting in increased latency. I cannot directly replicate this on my side as my setup (in the U.S.) as we don't profitaps/profisharks available.

I'll respond with an update later today.

-Daolin

Update:

I ran several tests with the new PRU firmware shared with me internally and I'm seeing the following results

HSR offload with cut-through testing with new firmware:

Test 1: 1Gbps line rate, 20Mbps UDP iperf, no continuous lost packets both directions

Test 2: 1Gbps line rate, 200Mbps UDP iperf, both directions have continuous lost packets

Test 3: 100Mbps line rate, 20Mbps UDP iperf, no continuous lost packets both directions

Test 4: 100Mbps line rate, 95Mbps UDP iperf, no continuous lost packets both directions

As you can see, with a higher throughput rate, we see lost packets, but in both directions instead of one direction.

However, knowing that your team is mostly concerned with 100Mbps for HSR offload with cut-through, I think this version of the firmware will be sufficient for your use case. Can I confirm if this is the case for your team or do you anticipate needing higher throughput rate?

-Daolin

P.S. My responses will be delayed until July 10th as I will be out of office during the U.S. July 4th holiday

Hi Daolin,

Thank you for your fast response and the updates. This is a good news for us !

Yes you are right our current target is 100Mbps line rate cut-through mode.

If I understand well, we had 2 main concerns:

- packet lost at 100Mbps

- forwarding time at 100Mbps (400B ~39.256us and 1300B ~111.211us)

From the previous comments, packet lost at 100Mbps seems to be fixed !

We still may have this forwarding time (cut-through mode ) at 100Mbps which has the behavior of store and forward.

You can find above the detailed test !

"<-> "= Ethernet cable

EVM1 <->Profishark1 <-> EVM2 <-> Profishark2 <-> EVM3

The Profisharks are synchronized and has the same time base.

we disconnect the cable between EVM1 and EVM3 so packets go only through EVM2 (DUT )

1 We start the capture on Profisharks

2 we start iperf server on EVM3

3 we start iperf client udp on EVM1 :

Like Test 3 100Mbps line rate, 20Mbps UDP iperf during 10s

4 We stop the capture of Profisharks (both)

5 We have a script that retrieves the target packets and does the difference of timestamp between the pcap files generated by Profishark2 and Profishark1

6 We get the max and the mean of the difference of the timestamp !

If you need a meeting to help to set up the testbench feel free to ask !

And again thank you for the updates !

Tianyi

Hi Tianyi,

Thanks for confirming your requirements and the details of the cut-through latency setup.

According to the email sent to you and your team, our firmware expert is asking for the following test result from your team. Please give that a try and let us know if you see the same issue as before or not. 

• Residence time at 100Mbps (400B ~39.256us and 1300B ~111.211us):
  • The interface for enabling cut-through mode is separate for the 2 directions eth1->eth2 and eth2->eth1.
  • Please try with below command to enable cut-through mode: (sets bit 0 and bit 8)
    devlink dev param set platform/icssg1-eth name cut_thru value 257 cmode runtime

The firmware version with the fix for the packet loss at 100Mbps was provided in the same email by our firmware expert.

Just to clarify, the forwarding time at 100Mbps (400B ~39.256us and 1300B ~111.211us) is observed for the case of regular cut-through forwarding (no HSR setup)?

-Daolin

Hi Tianyi,

As discussed in the meeting today, I went asked the development team about whether the cut through configuration with queue1 and queue8 is a required configuration for cut-through functionality. Here is the feedback:

• To enable cut-through forwarding in both directions, cut-through will need to be enabled on both slice0 and slice1
• The devlink command to enable cut-thru takes a u16 value. all 16 bits of it represents queues. 8 for slice0, 8 for slice1.
• BIT 0 to BIT7 are slice0 queues, BIT 8 to BIT 15 are slice1 queues. This is why bit0 and bit8 need to be configured for cut-through with devlink command to configure queue1. I think the confusion might be that setting bit0 and bit8 does not mean setting queue1 AND queue8. It just means queue1 is set for slice 0 and queue1 is set for slice 2 to enable cut-through in both directions.
• Example, if you need to enable cut-thru on q0 and q1 for both slice 0 and slice1, you need to pass u16 value with BIT 0,1, 8,9 set i.e. you have to pass "0000 0011 0000 0011" = 771 in decimal.

We will update this in https://software-dl.ti.com/processor-sdk-linux-rt/esd/AM64X/latest/exports/docs/linux/Foundational_Components/PRU-ICSS/Linux_Drivers/PRU_ICSSG_Ethernet_Switch.html in the future.

What changed in the most recently shared firmware and the original firmware?

"We recently shifted to 333MHz for ICSSG, so that was creating issues for cut-throguh mode - fixed it in firmware. (This will result in increased residence time for 1G cut-through, 100M/10M remains same as before)."

-Daolin

Hi Daolin,

Thank you for the update. It's more clear now !

Tianyi LIU

Hi Tianyi,

I just want to ask, is there a specific cut-through forwarding latency requirement your team is targeting? 

From your previous response on E2E, you stated at 400B you observed ~40us latency but verbally during biweekly call you mentioned seeing 6us latency. I'm assuming this 6us latency is from the newer firmware replacement shared with you via email? Is my understanding correct?

Additionally from the call, you mentioned that the new firmware shared with you also seemed to fix the issue with increasing cut-through forwarding latency with increasing packet size. I just want to confirm on here this is still true?

-Daolin

Hello Tianyi,

Can you give a response on my previous questions? We want to make sure we are aligned on what cut-through performance your project is aiming for.

-Daolin

Hi Daolin,

This is correct. When I did the test on 100Mbps cut-through, we had ~6us latency to forward a packet for 400B and for 1300B !

-Tianyi

I remembered for 100Mbps link speed, there were not packets lost when forwarding the packets.

But for 1Gbps, there were some packets losts when forwarding packets in one direction

example :

           (EVM1 <-> EVM2 <-> EVM3 ).

           with EVM2 eth1 IN eth2 OUT or with EVM2 eth1 OUT eth2 IN

Hi Tianyi,

Thanks for confirming!

As mentioned via email, we will be looking at improvements in the PRU firmware for 1Gbps use-case in future releases. You mentioned in email that you are targeting a fix for this issue by the end of the year, is this correct?

Daolin Qiu said:

is there a specific cut-through forwarding latency requirement your team is targeting? 

Additionally, do you have a specific requirement to the above question?

-Daolin

Daolin:

 I am likewise now keeping an eye on this given I am working with PRU_ICSSG on AM6442 (in fact using all 3 RJ45 ports in a target application). 

Are we looking at AM64x Industrial Communications SDK v10  for "improvements in the PRU firmware for 1Gbps use-case" ? 

I hope both

(+) Thomas Leyrer's Profile - Thomas Leyrer - TI E2E support forums and (+) Pekka Varis's Profile - Pekka Varis - TI E2E support forums will be testing that.

thanks

Jim

Hello Jim,

Thanks for reaching out

Daolin Qiu said:

we will be looking at improvements in the PRU firmware for 1Gbps use-case in future releases

To clarify on this, specifically I am talking about the issue of HSR PRU firmware to decrease packet loss seen when the ICSSG ethernet ports are configured with HSR offloaded and using cut-through forwarding in one direction (the other direction doesn't have packet loss). This issue was observed on RT-Linux: https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-RT-AM64X. 

Jim Mrowca said:

Are we looking at AM64x Industrial Communications SDK v10  for "improvements in the PRU firmware for 1Gbps use-case" ? 

Is the industrial communications SDK this one? INDUSTRIAL-COMMUNICATIONS-SDK-AM64X — Industrial Communications SDK for AM64x - RTOS

If so, please note the issue observed in this thread was observed on Linux/RT-Linux. I do not know if Thomas or Pekka have tested this feature on the Industrial Communication SDK and seen similar issues.

-Daolin

Daolin - my application does require RT-Linux; and I am using the A53s for those high speed comm paths.

It is interesting that the " tirex" path lists Industrial Communications SDK for AM64x for the description without stating it's for RTOS only per se (a different question arises then wrt Linux support of said Industrial Communications SDK (I will put in a different e2e for that).

Hi Jim,

Thanks for clarifying. Below is what I found by following the tirex link you shared and going into the User Guide section. But please do create a separate e2e for this inquiry to keep this thread focused on HSR offload with cut-through forwarding.

"Welcome to Industrial Communications SDK for AM64X. This SDK contains examples, libraries and tools to develop RTOS and no-RTOS based Industrial Communication, Fieldbus and networking applications on PRU-ICSS"

-Daolin