AM5726: PRU Ethernet dropping packets

Hello David,

Let me take a look at this. I will get back to you in a day or so.

Regards,

Nick

Hello David,

What are the iperf outputs you see on the server side for one iperf instance, two iperf instances, three iperf instances? I am curious about what kind of throughput you are seeing, and whether flow control packets are actually getting generated.

Regards,

Nick

Hi Nick, 
With 1 iperf instance, I am getting 94-96 Mbps.
With 2 iperf instance, I am getting around 47-48 Mbps for each, so the total is always around 94-96 Mbps.

Even with 3 instances, I still get roughly 95 Mbps total, but just a few packet drops:

[ 3] Server Report:
[ 3] 0.0-600.0 sec 2.24 GBytes 32.1 Mbits/sec 0.243 ms 5937/1643220 (0.36%)
[ 3] 0.0-600.0 sec 1 datagrams received out-of-order

[ 3] Server Report:
[ 3] Server Report:
[ 3] 0.0-600.0 sec 2.05 GBytes 29.4 Mbits/sec 0.857 ms 4984/1503751 (0.33%)
[ 3] 0.0-600.0 sec 1 datagrams received out-of-order
[ 3] 0.0-600.0 sec 2.16 GBytes 30.9 Mbits/sec 0.340 ms 5476/1583087 (0.35%)
[ 3] 0.0-600.0 sec 1 datagrams received out-of-order

Edited 2019-08-29

Hello David,

We do not implement flow control in our PRU Ethernet firmware or drivers. What is your terminal output when you establish the link? I would expect to see a message like "Link is Up - flow control off" This message does not actually indicate whether the MAC layer supports flow control. Even if you see something like "Link is Up - ... - flow control rx/tx", PRU Ethernet does not implement flow control.

tx

Do you have any indication that the server is actually sending flow control packets out, and that the PRU Ethernet is respecting those flow control packets?

Regards,

Nick

Hello Nick, 

I am seeing "Link is Up - ... - flow control rx/tx".

But I don't think the problem is with the flow control. If I use a single instance of iperf and specify bandwidth > 100M, I do not get any packet drops. It looks like there is a problem when multiple processes try to enqueue transmit packets concurrently.

Regards,
David

Hello David,

Let's back up. What are you trying to test here?

I thought you were trying to test flow control, but PRU Ethernet does not support flow control. So the throughput you see is probably limited by the PRU Ethernet, not by the server sending flow control packets back to the PRU port.

Regards,

Nick

Hi Nick,

I noticed this issue when trying to test flow control, but this is not an issue with flow control. From your previous comment, I understand/agree that PRU Ethernet ignores flow control packets sent from the remote side, but the PRU Ethernet should still limit the throughput to 100M.

It looks like there is an issue with the PRU Ethernet drivers. When there is more than 2 concurrent processes trying to transmit packets, the drivers do not always return -EAGAIN or -ENOBUFFS to indicate that the packet was not transmitted.

Are you able to reproduce the same behavior?

Regards,

David

Hi Nick, 

Were you able to reproduce this issue?

David

Hello David,

I am sorry for the delayed response. We will get back to you next week.

Regards,

Nick

Hello David,

We are able to observe that 2 iperf instances over VLAN has 0 loss, while 3 instances introduces some loss. However, this does not mean that the loss is happening on the transmit side - e.g., the iperf server on the receiver side might not handle 3 streams at once very well.

Are you able to observe that the packets are getting dropped at the transmit side rather than the receiver side?

Regards,

Nick

Hi Nick, 

Yes, if I run this test between 2 linux Desktops, I can run at least 5 instances of iperf without errors. 

I.e:

Test 1:
TI IDK transmits to Linux desktop - Packet loss with more than 2 streams.

Test 2:
Another Linux desktop transmit transmit to same Linux desktop as in test 1 - no packet loss with 5 streams (did not test with more than that)

>  the iperf server on the receiver side might not handle 3 streams at once very well.
Each iperf instances is running on a different VLAN, and the iperf server is listening on the VLAN interfaces, so each receiving iperf server only receives data from a single iperf client.

Hello David,

I am also taking a look at this. We've reproduced the issue but need to take some more time to identify the root cause and see if there is a fix or workaround. While I unfortunately don't have an update right now, we'll try to get back to you next week with some more information!

Best,

Aaron

Hello Aaron, 

Thank you for looking into this. Let me know if you need me to test anything.

David