AM6548: Dual ethernet ports issue on same ICSSG

Ilya Lukyanov

Part Number: AM6548

In our custom board and on the PHYTEC phycore-am65x board, RX packets are lost on one of the network interfaces. This happens if both interfaces with the same ICSSG eth0 and eth1 are connected to the same switch on the same subnet.

This happens within a few minutes with low traffic and within seconds with a high network load. At the same time, the "rx_good_frames" RX packet counters (which can be viewed through "ethtool -S ethX") continue to increase for both PRUs.
However, RX interrupts are no longer generated from one of the interfaces and packets do not get to Linux. If you execute ifconfig ethX down, ifconfig ethX up (reset the firmware), then the reception of packets is resumed.

If interfaces from different ICSSG are used to connect to the switch, then this is not observed, and RX packets go around the ring and are received on both network interfaces in Linux (as it should be).
In our development, we use firmware version 07.03.00.005 from the repository git.ti.com/.../
and linux kernel version 5.10.41 with patches based on the repository git.ti.com/.../

Please provide recommendations on how to fix this problem.
Do you plan to share the source codes of the firmware to be able to analyze this problem?

over 4 years ago

0 Ilya Lukyanov over 4 years ago

Prodigy 25 points

We are using processor revision SR2

0 Vineet Roy over 4 years ago in reply to Ilya Lukyanov

TI__Mastermind 37435 points

Hi Ilya,

1. Can you provide wireshark logs for the traffic that you are using ? Does this happen with any specific traffic ?

2. Can you provide a picture of your network configuration and what you are trying to do ?

3. Do you have an IDK where you see the same issue or is this on your board only ?

Regards

Vineet

0 Ilya Lukyanov over 4 years ago in reply to Vineet Roy

Prodigy 25 points

Hi, Vineet.

1. The error is easily reproducible with iperf3.

2. Schematic diagram of our test network:

On the board, we use 2 network interfaces from the same ICSSG (named eth0 and eth1).

Set up IP addresses from different subnets:
eth0: 192.168.101.100
eth1: 192.168.102.100

Next, we connect the board to a PC with two gigabit network cards directly with a cable (without using a switch).
PC network interfaces have the following IP addresses:
enp3s0: 192.168.101.254
enp5s0: 192.168.102.254

Run two separate iperf3 servers on the board and bind it's with specific interfaces:

iperf3 -s -B 192.168.101.100 &
iperf3 -s -B 192.168.102.100 &

On a PC, run two iperf3 clients on each network interface with the following parameters:

iperf3 -u -c 192.168.101.100 -B 192.168.101.254 -t 28800 -b 1G
iperf3 -u -c 192.168.102.100 -B 192.168.102.254 -t 28800 -b 1G

After a few seconds or minutes, one of the PRU-ICSSG network interfaces stops transmitting RX packets in Linux. At the same time, through ethtool -S ethX, you can see that the packets are received by the PRU, but not transmitted to the icssg-prueh driver.

3. This behavior is reproduced on our custom board and on the PHYTEC phycore-am65x board.

0 Vineet Roy over 4 years ago in reply to Ilya Lukyanov

TI__Mastermind 37435 points

Hi Ilya,

I can confirm the issue at my end. I have filed an internal bug for the same.

Will try and provide some initial diagnosis even if issue isn't completely fixed.

Regards

Vineet

0 Ilya Lukyanov over 4 years ago in reply to Vineet Roy

Prodigy 25 points

Hi, Vineet

Please tell me, changes

git.ti.com/.../

and

git.ti.com/.../ti

fix this problem?

0 Vineet Roy over 4 years ago in reply to Ilya Lukyanov

TI__Mastermind 37435 points

Hi Ilya,

There is no resolution yet. We are looking into this issue

Regards

Vineet

0 Vineet Roy over 4 years ago in reply to Ilya Lukyanov

TI__Mastermind 37435 points

Hi Ilya,

This was due to a firmware issue. We didn't root cause it but the latest firmware seems to fix the issue. This will be released with SDK 8.0 (Aug end)

If you need it sooner then please reach out to me over email. You will get my contact with PHYTEC

Regards

Vineet

0 Ilya Lukyanov over 4 years ago in reply to Vineet Roy

Prodigy 25 points

Hi Vineet.

Thanks for the help.

I would like to write to you by e-mail, but I do not see the address in your profile. How to contact you?

0 Vineet Roy over 4 years ago in reply to Ilya Lukyanov

TI__Mastermind 37435 points

Hi Ilya,

You can send me a private message by clicking on my name

Regards

Vineet

0 Vineet Roy over 4 years ago in reply to Vineet Roy

TI__Mastermind 37435 points

Hi Ilya,

We are unable to enable a private drop at the moment. SDK 8.0 (with the fix) will anyways be out in 2 weeks time, so suggest to wait until then.

Regards

Vineet

Processors

Processors forum

AM6548: Dual ethernet ports issue on same ICSSG