K2HK: UDP packet loss issue

Hello,

No, I have not done this test with two K2H EVMs

Thanks

Rams

Hello,

I switched my kernel version from 3.10.72  which uses NetCP PA firmware rev 3.0.1.3 to an older kernel  version which uses NetCP PA firmware rev 3.0.1.2 and repeated iperf test with different bandwidth.  

The test results with NetCP PA firmware rev 3.0.1.2 showed better results than 3.0.1.3 in terms of percentage drops from 15% to 0.6%.  But then, this issue is different from what I have reported earlier.  Here the packet drops are in Linux kernel with RcvbufErrors.  The RcvbufErrors increases if the UDP packet are smaller.

 iperf3 -c 192.168.0.104 -u -b 800M -l 600

reports 30% loss whereas

 iperf3 -c 192.168.0.104 -u -b 800M

reports 0.22% loss.

These packet drops are in kernel with rcvBuf errors.

The idea was to test older NetCP PA firmware rev 3.0.1.2 to rule out any issue with NetCP firmware.  But before that, kindly clarify whether any wrong packet streaming interface configuration can introduce packet drops.  If so, how can we check the packet drop counts at PA?

Could you please clarify my previous posts also?

Thanks

Rams

We are running our custom application on our custom boards. We have some PA configuration to forward some traffic to the DSP cores.
I switched my kernel version from 3.10.72 which uses NetCP PA firmware rev 3.0.1.3 to an older kernel version which uses NetCP PA firmware rev 3.0.1.2 and repeated iperf test with different bandwidth.

Rams,

See the device tree binding documentation in /Documentation/devicetree/bindings/keystone-net.txt of your Linux source. This document explains that these device tree settings are configuring the the number of descriptors in each receive queue as well as the size of the descriptors in each queue. Your new settings are increasing the number of descriptors in the two receive queues of interface 0 from 128 descriptors to 512 descriptors. Since this alleviates your dropping issue I'm led to believe that the packets were getting dropped (after being passed up all the way to the NETCP PKTDMA) due to your host core being too slow to service the incoming packets without overwhelming the 128 descriptor buffer.

The path of a packet coming into the device is:

RX at external Ethernet port -> Forwarded to host port 0 -> sent to the PA for classification -> sent to the NETCP PKTDMA -> data copied into a descriptor buffer and the buffer is placed on the RX queue -> OS or host core consumes the buffer from the RX queue -> OS or host core returns the buffer to be used again. 

It appears that your packets were making it all the way up to the 'copy into a descriptor buffer' step and then when a descriptor/buffer wasn't available the packet was dropped. This is why the switch statistics weren't showing any dropped packets (only the first two steps above are handled by the switch) because from its perspective the packets were passed up the chain  to the PA with no errors.

Jason Reeder

Hi Jason Reeder,

Thanks a lot. 

You indicated that the core/OS is slow in consuming packets. Here, the Linux network stack has to handle the packets and the network drivers are running in kernel context which has higher priority.  If the applications are slow in handling the packets, it will be dropped anyway by the kernel.  The issue is that the packets did not reach kernel.  Our assumption is that with default NETCP configuration, the packets should arrive network stack and should not be dropped in PA even during full load.   Dont you think this will be a general issue for all running linux on K2H when there are burst traffic with RX FDQ queue size as 128 in PA.   Our system was running with appox 10% load.  There were burst of packets in our system.  We enabled flow control and checked for pause frame counters.  But the bandwidth was around 300M.  and we have  MAC-MAC link (No MDIO).  When switch is able to handle the burst traffic which was less than half of the maximum bandwidth of 1G, I guess, the PA RX FDQ  size should be large enough to handle the maximum bandwidth of the link. I had raised this question in this post already which was not answered. If the RX queue size is 128, is that large enough to handle 1G of bandwidth with burst packet inflow?

I have also asked on how to dump PA statistics.  Are there any counters that would point to us if there are packets drops in PA for what so ever reason.  We tried reading C1 and C2 statistics using the PA API paSysStats_t..

None of these counters classify1.nIpDepthOverflow, classify1.nSilentDiscard, classify2.nSilentDiscard indicated drops at PA

Can you point to me to the register which will indicate the packet drops in PA for this issue.

It will be very useful if TI clarifies all the queries.

Thanks
Rams

Rams,

Please see Figure 1-1 in section 1.4 of the NetCP User Guide (http://www.ti.com/lit/ug/sprugz6/sprugz6.pdf). The descriptors/buffers that you are modifying using the device tree would be to the left of the PKTDMA, completely outside of the NetCP itself. These buffers/descriptors are actually part of the Multicore Navigator system in the SoC (http://www.ti.com/lit/ug/sprugr9h/sprugr9h.pdf).

So, from the perspective of the GbE Switch (Layer 2 hardware switch) there are no packets dropped because they all get passed along to the PA. From the perspective of the PA there are no packets dropped because they were all classified and forwarded on to the PKTDMA. This is why those statistics are both showing no packets drops.

When there are no descriptors remaining in a free descriptor queue (we call this buffer starvation) then the PKTDMA has two choices: either drop the packet immediately, or retry the packet after a timeout period. This behavior is controlled by the RX_ERROR_HANDLING bit in the 'RX Flow N Configuration Register A' (section 4.2.4.1 http://www.ti.com/lit/ug/sprugr9h/sprugr9h.pdf). If this bit is set to 0 then the PKTDMA will drop packets as they arrive on the RX flow until a free descriptor becomes available. So, even if flow control is enabled, if RX_ERROR_HANDLING is set to 0 then packets will be dropped by the PKTDMA and no flow control frames would be sent. This is my theory on what is happening in your case with 128 descriptors in the FDQ. 

I'll ask someone else on my team to comment on your Linux specific questions.

Jason Reeder

Hi, Rams,

The descriptors are used by PA to send to CPPI and PA would not have stats for it. To PA, the packet is received and considered delivered. 

The current setting should be able to support 1K, may be up to 2K descriptors. Anything higher than that, the region-12 and pool-net may need to increase as well.

Increasing descriptors seems to hide the issue, not to solve the issue. You should find out why your hardware/software can not meet TI's performance. You mentioned that you are only be able to get 300Mbps out of 1GbE. Our iperf performance can reach 960Mbps. What MCSDK release are you using and are you using TI's driver?

Rex

Thanks Jason Reeder for giving more insight into the issue.

Hello Rex,

I completely agree that increasing the descriptors seems to hide the core issue and is not the solution.  I would like to clarify that we are able to get full bandwidth capacity  with iperf without losses when no applications are running.  I will repeat the tests and post the results.

We are seeing issues when system is loaded while running custom applications.

I am not sure how to simulate same conditions on Eval kit with iperf or other tool to generate burst traffic when there is normal traffic of 200M.

I am trying to understand your statement "You should find out why your hardware/software can not meet TI's performance".  When packets are getting dropped inside K2H, how could be this be attributed to specific hardware issue or software application issue.

Our tests show that there are no issues when we pump in upto 1G traffic and all that could be delivered without any issues till the K2 Switch. Even if you consider the situation during this issue,  the switch does not drop any packets.  There were no pause frames when flow control was enabled.  The packets are able to pass through the keystone switch without issues.  The packets are getting dropped in multicore navigator as pointed out by Jason Reeder and there seems to be buffer starvation.  At the consumption side, it is kernel driver (Using only TI driver, no custom drivers) which has to consume and pass it on to linux network stack and make up free descriptor for Multicore navigator.  There should not be any specific hardware issues or application software issues.  If the custom applications are slow and not grabbing packets, network stack will naturally drop the packets.  We do not see any packets getting dropped by linux kernel. Please correct me if I am wrong

We use TI driver and MCSDK releases.

We use 3_01_03_06 MCSDK  for MPM. 

 Our kernel version is from MCSDK 3.1.4 (K2_LINUX_03.10.72_15.08)

Our U-Boot version is from K2_UBOOT_2013_01_15.02_01

Rams,

What is the average size of the packets through ethernet port?

Rex

Hi Rex,

The average size of the packet is 512 bytes.

Rams

Hi, Rams,

There are several things we'd like to get clarification:

1) Without any software running, just iperf, are you able to get close to 1Gbps performance on your boards?

2) When you observed the UDP dropped, were there other ethernet port connected to the network? or just eth0?

3) Can you explain why in your netstat stats that IP packets are 400K more than UDP? ICMP and TCP are only around 70 pakcets total. Does it mean that the other ethernet port was also connected to the network? If that is the case, then the UDP counts will be the combination of all ethernet ports and will be off from what FPGA expects to receive.

4) Is the received packet coutns at FPGA close to the number of RX packets in ifconfig for that port?

5) How does the FPGA receive packets from kernel?

6) Kernel handles 64 packets (NAPI weight) per each Rx interrupt then returns these 64 descriptors back to the free queue. For each packet, it calls into application Rx function and this is in kernel context. Is it possible that the FPGA receive function is not returning the control back to kernel driver soon enough?

Rex

Hi Rex,

Please find answers inline.

1) Without any software running, just iperf, are you able to get close to 1Gbps performance on your boards?

Yes, you are right.  We are able to get close to 1Gbps when no application software is running.

2) When you observed the UDP dropped, were there other ethernet port connected to the network? or just eth0?

We are using only one ethernet port, port 0 which is directly interfaced to FPGA(PHYless configuration), No MDIO

3) Can you explain why in your netstat stats that IP packets are 400K more than UDP? ICMP and TCP are only around 70 pakcets total. Does it mean that the other ethernet port was also connected to the network? If that is the case, then the UDP counts will be the combination of all ethernet ports and will be off from what FPGA expects to receive.

We have enabled only one ethernet port which is interfaced to FPGA and configured in SGMII_LINK_MAC_MAC_FORCED mode.  

4) Is the received packet coutns at FPGA close to the number of RX packets in ifconfig for that port?

We tried match number of packets at FPGA close to the number of RX packets in the Switch statistics (ethtool -S eth0)

5) How does the FPGA receive packets from kernel?

FPGA is directly interfaced to eth0 and FPGA has unique ipadress

6) Kernel handles 64 packets (NAPI weight) per each Rx interrupt then returns these 64 descriptors back to the free queue. For each packet, it calls into application Rx function and this is in kernel context. Is it possible that the FPGA receive function is not returning the control back to kernel driver soon enough?

There is no FPGA driver component separately to handle receive/send.

Thanks

Rams

P.S

I am travelling and will be back to work on 8th Oct.  I will be able to send you a detailed block diagram if it is not clear.

Hi, Rams,

We get even more confused with your answers.

1) FPGA has unique IP address - is it not the same IP address assigned to eth0? If it is a different IP address, then how does that work? We have one MAC address on eth0 with 2 different IP addresses?

2) FPGA is directly interfaced with eth0 - You will need to describe more details on how this work. Does it mean that FPGA gets packets directly from eth0? If that is the case, where is the Linux Kernel in the picture?

3) There is no FPGA driver component separately to hanel receive/send - we will need to know how FPGA receive the packets from SoC. The way we understand the packet flow in Linux is that the UDP application opens a socket of a specific UDP port on a specific ethernet interface (IP address) which it listens to. Take iperf as an example which sits on top of the UDP stack listens to a UDP port of the eth0 IP address, and when UDP delivers the packets, iperf acts on them (bookkeeping, etc.). As I described in previous post, the kernel driver delivers packets in descriptors, 64 of them per interrupt. We'll need to know if the FPGA does not fit in this picture, how does it receive packets? If it receives directly (steals under IP stack) from eth0, then how is this done?

Please note the ethtool stats in K2H are the combination of all NetCP ports including host port which should not be taken as a whole being the stats on the specific interface.  The Rx Packets from ifconfig is more accurate on the packets count on that interface.

Rex

Hi Rex,

Sorry for late response.  I was out of office.

Please find the block diagram which shows how the FPGA is interfaced to CPU.

Note that the FPGA and CPU have the same IP address, and the FPGA acts simply as a bridge. (The FPGA logic will receive packets on specific ports and everything else is forwarded directly to the CPU.)

Our packet counter is right before the PCS. 

Thanks

Rams

Hi, Rams,

If I understand the diagram correctly, the FPGA replaces the PHY on TI EVM, and if I recall correctly that there are 2 devices connected to the Backplane, So your set up should look like: 

   Keystone-II-A  ==SGMII==> FPGA-A ==> BackPlane ==> FPGA-B ==SGMII==> Keystone-II-B

1) was iperf run between KS-II-A and KS-II-B which shows no packet loss? If not, is it from some other device send to BackPlane then to receiving KS-II-B?

2) If packet loss is in FPGA-B, then we should look at the transmit side because the packets are not even reached the CPSW and Kernel in Keystone-II-B.

3) What is the transmit packet count shown at the source application which generates the packets? in ifconfig of KS-II-A? and in FPGA-A?

Rex

Rams,

We understand your system better now. What we don't understand is why on TI EVM it can reach 960Mbps UDP throughput with default number of descriptors, but it can't on your system. Without reproducing it on TI EVM, it is impossible for us to work on the issue. If increasing the descriptor number can solve the issue, we suggest to adapter it as a workaround.

Rex