This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Packet loss when sending packet to Ethernet port (C6678)

Hello,

We have an application that follows the example of  multicoreExample to send out packets to Ethernet.

Below is the SendPacket() function in cppi_qmss_mgmt.c.

Int32 SendPacket (Void)
{
    Cppi_HostDesc*  pCppiDesc;
    UInt32          dataBufferSize;
    uint32_t        key;
    ...
...
...
/* Send the packet out the mac. It will loop back to PA if the mac/switch * have been configured properly */ Qmss_queuePushDescSize (gPaTxQHnd[8], pCppiDesc, SIZE_HOST_DESC); /* Increment the application transmit counter */ gTxCounter ++; /* Give some time for the PA to process the packet */ CycleDelay (10000); return 0; }

At the end it has a delay of 10000 cycles, and the comment says "Give some time for the PA to process the packet"

In our application, we have to increase the delay to 80000 cycles to avoid packet loss. This is unacceptable, as a lot of processing power is wasted in this long wait.

Why do we need to have this wait?  As I understand, gPaTxQHnd[8] is opened as CPSW queue. Ideally the queue should serve as a buffer and the CPU should not have to wait for the CPSW to put the packets out  the MAC.

In our code, we implemented a mechanism to have a free queue of size 1, and once a packet is pushed into CPSW queue, we wait for the buffer to show up in the free queue before another push. Even with this change, the packet loss still happens if we remove the delay.

The CPSW statistics do not indicate any dropped TX packets.

Is it possible that the CPSW queue may get into some overflow condition when packets are pushed too quickly into the queue?   Is there anything we can try to further locate the problem?

Urgently need your help. 

Thanks,

Daniel






  • Daniel,

    Are you using TI EVM or your own custom board?
    Yes, when sending packets to the PA from the Host, packets must be sent through the queue.
    The purpose of PA in the NETCP is to perform packet processing operations such as packet header classification, checksum generation, and multi-queue routing.
    This delay is used for PA processing only. Packets are added to a packet queue by writing the 32-bit descriptor address to a particular memory mapped location in the queue manager module. Packets are de-queued by reading the same location for that particular queue.
    Please go through the section "2.2 Packet Accelerator Transmit Queue Interface" in the Keystone PA User Guide to get more detailed information. And also refer the KeyStone Multicore Navigator User Guide.

    In addition that, I have tested the PA emac example to increase the MAX_NUM_PACKETS.

  • Hi Pubesh,

    I am using TI 6678  EVM board. 

    Our application follows the example of multicoreExample, and the gPaTxQHnd queues are opened in the following way.

    /** Number of PA Tx queues available */
    #define         NUM_PA_TX_QUEUES            9

    for (i = 0; i < NUM_PA_TX_QUEUES; i ++)
     {
                
                if ((gPaTxQHnd[i] = Qmss_queueOpen (Qmss_QueueType_PASS_QUEUE, QMSS_PARAM_NOT_SPECIFIED, &isAllocated)) < 0)
                {
                    #ifdef EVM
                    System_printf ("Error opening PA Tx queue \n");
                    #endif
                    return -1;
                }            
      }

    Judging from your code comments, I believe Tx queue #8 (0 to 8) is directly connected to CPSW, and does not go through PA, even though the queue is a PA queue.Nothing about this is mentioned in the PA guide or the Multicore Navigator guide.


    If the packets do not go through PA, do we still need the wait? Any idea why I have to wait 80000 cycles after sending each packet?

    Daniel

  • Daniel,

    I do not have the answer exactly for your question about delay. But the PA developer may used for PA processing and some task sync. I will share you the details, if I get more details from MCSDK team.

  • Thanks,  Pubesh.

    More information:

    I checked the value of RXGOODFRAMES (0x02090b00) of STASA module, and the value of TXGOODFRAMES (0x02090c34) of STASB modulet , and they all match the number of packets that the application has sent.

    It means that all the packets were successfully forwarded by the Gigabit Ethernet Switch Subsystem to SGMII module, but some of the packets did not show up on the physical network.

    I suspect that when you push a packet onto the CPSW queue, it gets sent out right away to the physical network regardless of the the bandwidth capacity of the network. If the bandwidth is 1Gb/s, then when you subtract the Ethernet/IP overhead, the effective bandwidth may be 0.6 Gb/s, i.e. 75M bytes per second. When you push more than 5 packets of 1500 bytes into the queue within 0.1ms, the instantaneous rate is higher than 75M bytes per second, and some packets will be lost. The reason that we need the delay is that the delay basically sets a limit on the instantaneous rate of putting data into transmission queue.

    This, if true,  would put a huge limitation on the application, because the application has to engage in flow control to make sure the instantaneous rate that it puts data into the transmission queue does not exceed 75M bytes per second. Previously,  we only need to care about the average rate. Because of this new limitation, the capacity of our system would be greatly reduced.

    Daniel

  • Daniel,

    I have checked with the author of this example, and was no particular reason for the addition of the delay.
    It can be left out if needed.

  • Hi Pubesh,

    Thank you for your response, but I don't think my questions were answered.

    Why do I need to apply a delay in sending to avoid packet loss?  It is hard to believe the delay is there for no reason, because clearly in my application, I need this delay to avoid packet loss. 

    Is it possible that the FIFO in the Gibabit Ethernet Switch Subsystem may overflow if the application pushes data into Tx queue too quickly? 

    Is it possible that Tx queue itself may overflow?

    Anything I can do to further locate the problem?

    Daniel

  • Hi,

    Who is the destination? Maybe a Windows PC?

    I have a similar problem but I discover that, at least in my case (UDP), it depends on the target PC and its Ethernet board.

    First, to improve the number of packets the PC can send/receive I increase the windows socket rx and tx buffer (SO_RCVBUF). In this way it works better but I continue to have problems.

    Then a moved on a power full PC with a better Ethernet card. no more packet lost (well, except for some very loaded cases).

    I have no explicit delay between send.

  • The destination is a Linux machine, but the packet loss was observed on the sending side. The EVM board was connected to a switch, and on the switch a mirror port was configured, and a packet capture was done on the mirror port. From the packet capture it can be seen that some packets were lost. The packet loss is not related to network conditions.

    The average bit rate is below 2M bps, but the sending is done in bursts. Every 10ms, we need to send out about 30 packets, and those packets can be pushed to the Tx queue one after another. With a delay put between each packet sending, the traffic is less bursty while the average bit rate is still the same, then the packet loss is gone.

  • Correction:

    Every 30ms (instead of 10ms)  we need to send out about 30 packets, each about 1400 bytes in length; if those packets are pushed into the Tx queue one after another with out delay, then packet loss is observed.

  • Daniel,

    Please make sure that you are using directed packets during transmit in order to bypass the ALE transmit port decision process. This can be set by using the Cppi_setPSFlags function in the SendPacket function in the cppi_qmss_mgmt.c file of the original example. Since you are using the 6678 EVM this should look something like this:

    Cppi_setPSFlags(Cppi_DescType_HOST, (Cppi_Desc *)pCppiDesc, 2);

    The '2' above tells the switch to send this packet directly out of switch port 2 which is the one that is eventually connected to the RJ-45 plug on the EVM. 

    I think that this might be your issue because you say that your Stats A RXGOODFRAMES matches your Stats B TXGOODFRAMES even though you do not see all the frames show up on the wire. This could be due to the fact that Stats B is the combined statistics for both Ethernet Port 1 and Ethernet Port 2. This might mean that some packets are being erroneously sent out of port 1 (due to not using a directed packet) and the statistics would still increment. 

    In order to only see the statistics for Ethernet port 2 (in Stats B) you can disable counting for Ethernet port 1 statistics by setting portStatCfg.p1StatEnable = 0; in the Init_Switch function in the cpsw_mgmt.c file of the example. 

    The GbE switch in the 6678 device can handle full line rate (1 Gbps) data transmission all the way down to 64 bytes packets (1.488 Mpps). If data is placed onto the switch TX queue (queue 648) faster than 1 Gbps this data will move out of the switch at 1 Gbps and the back up will remain on the TX queue, no dropping should occur unless you run out of free descriptors or buffers to place more data on. 

    Let me know if this doesn't help and we can move on to other possibilities.

    Thanks,

    Jason Reeder

  • Hi Jason,

    We had been doing the following before each push.

    Cppi_setPSFlags(Cppi_DescType_HOST, (Cppi_Desc *)pCppiDesc, 2);

    So this can't be the cause.

    We noticed that the switch has Rx/Tx FIFOs for port 1 and port 2. If the switch forwards the data to port 2 at 1Gbps, and if the network condition is such that it can just do 700Mbps,  then would the FIFO gets overflowed?  Our application follows the example of multicoreExample project, and does not have an Ethernet layer that deals with flow control like pause frame etc. Could this be the source of the problem?

    Daniel

  • Daniel,

    How are you performing your packet capture at the mirrored port on your external switch?

    I am under the impression that if the 6678 and its link partner (an external switch in your case?) negotiate to 1 Gbps speeds then any data placed on the TX queue will be clocked out of the 6678 device at 1 Gbps. If you place all of your packets on the GbE switch TX queue (queue 648) at the same time then the packets will move out of the device at 1 Gbps with only the Preamble, SOF delimiter, and the IPG between them.

    Like you said, if flow control is not used then the 6678 switch will have no indication that the network can only handle 700 Mbps and it will continue to send packets at the negotiated rate (1 Gbps here) with no pause between them. 

    You mentioned that the Stats A RXGOODFRAMES (host port 0 switch ingress) matches the Stats B TXGOODFRAMES (Ethernet port 2 switch egress) correct? And also that both of those stats match the number of packets that you are attempting to send? If this is the case then I think that the packets are leaving the 6678 device headed toward your external switch without dropping any packets. One way to verify this would be to connect two 6678 devices together and send packets from device 1 to device 2 and make sure that device 1 Stats B TXGOODFRAMES matches device 2 Stats B RXGOODFRAMES. 

    Thanks,

    Jason Reeder

  • Hi Jason,

    The EVM is connected to an external switch, and on the switch a mirror port is configured. The mirror port is connected to a PC, where wireshark is run to capture the packets. Capture is also done at the destination, which is a Linux machine using tcpdump. They all show that some packets are missing. 

    You mentioned that the Stats A RXGOODFRAMES (host port 0 switch ingress) matches the Stats B TXGOODFRAMES (Ethernet port 2 switch egress) correct?

     Yes.

    And also that both of those stats match the number of packets that you are attempting to send?

    Yes.

    One way to verify this would be to connect two 6678 devices together and send packets from device 1 to device 2 and make sure that device 1 Stats B TXGOODFRAMES matches device 2 Stats B RXGOODFRAMES. 

    Ok, I will do a test like that and let you know.

     

     

    Meanwhile, is there a way to slow down the rate of moving data out of the transmission queue, and consequently slow down the rate of switch forwarding the data to the Ethernet port? 

    Can I enable flow control for the 2 Ethernet ports, but don't enable flow control for the host port? If yes, then with flow control enabled, the switch will send out data to Ethernet at say 700Mbps, but it still fetch data from host transmission queue at 1Gbps; so even if the average bit rate is 2Mbps, if the sending from host is in bursts, FIFO on the switch may have overruns, right?  

    If, to avoid packet loss, we have to enable flow control on the host port, will the switch slow down the rate of moving data out of the transmission queue when FIFO is near full?  I believe the CPU will have no role in this business because the host CPU has no control over the speed at which data exit the transmission queue, right?

     

    Daniel

     

     

     

  • A related question:

    If the EVM is doing 1Gbps, while destination machine on the same external switch is doing 100Mbps instead of 1Gbps, then flow control has to be engaged to avoid packet loss.  But if data always exit transmission queue at 1Gbps, then how is flow control going to work?

    Daniel

  • Daniel,

    Yes, flow control will need to be enabled in order to avoid packet loss in this situation. Either your 100 Mbps machine or your external switch will need to send Pause Frames to the EVM to let it know that the receiver is being overwhelmed. If flow control/pause frames are not used the EVM will have no way to know that packet loss is occurring. 

    Data will exit the transmission queue at whatever the negotiated rate is. In this case that happens to be 1 Gbps. If you connected the EVM directly to the 100 Mbps machine then the negotiated rate would be 100 Mbps and data will exit the transmission queue at 100 Mbps. 

    Jason Reeder

  • Hi Jason,

    I still don't understand how flow control is going to work if the data always exit the transmission queue at negotiated rate.

    If negotiated rate is 100Mbps, but if due to congestion a pause frame is sent from the remote side, then ideally the switch shall slow down the rate of moving data out of transmission queue. If not, packet loss will happen.

    Daniel

  • Daniel,

    When I say negotiated rate I only mean the rate that the data travels on the wire. I am not talking about the throughput of the data. 

    Pause frames will introduce longer delays between packet transmission which will lower the throughput of data leaving the switch.

    The GbE switch in the 6678 device is designed to not drop packets that originate at the host port. If a pause frame causes the transmission on an Ethernet frame to be delayed then the TX buffer of that Ethernet port will begin to fill. Once that TX buffer is full then the RX buffer at the host port of the switch will begin to fill. Once that RX buffer is full the NetCP PKTDMA will no longer push data to the host port RX buffer and the packets will remain in the NetCP PKTDMA TX queue. 

    Eventually, if the host core continues to generate data while the Ethernet port is not transmitting then the problem will manifest itself by the program running out of free descriptors in the TX free descriptor queue as all of the descriptors will be sitting on the NetCP PKTDMA TX queue waiting for transmission.

    Here is a simplified image of the switch and its buffer that goes with the previous paragraph. Your data path for packet transmission is shown by the red arrows. 

  • Thanks Jason. It is now much clearer for me. I would like to enable flow control on the EVM to see if that is causing the packet loss that I am seeing. 

    Flow control by default is disabled on the switch. What shall I do to fully enable it?

    I noticed that there are quite a few registers that are related to flow control:

    For this register (FLOW_CONROL), what is the difference in the behavior of the switch when port 0 (p0_FLOW_EN)  receive flow control is enabled or disabled?

    For this register (P2_MAX_BLKS), the description is unclear to me as to what I should do to configure it properly.

    I guess for this one (MAC_CONTROL), I only need to enable the TX_FLOW_EN, right?

    Daniel

  • Daniel,

    P0_FLOW_EN defaults to 1 in the FLOW_CONTROL register and this enables RX flow control at the host port. This is what causes the pushback behavior that I described in my previous post in which the PKTDMA will stop sending packets to the Host Port RX buffer when it becomes full.

    For your current purposes I think the simplest way for you to test if flow control is your problem is just to enable TX flow control on the Ethernet port that is connected to the external switch:

    • TX_FLOW_EN = 1 in the MAC_CONTROL register for the Ethernet port that you are using
    • FULLDUPLEX = 1 in the MAC_CONTROL register for the Ethernet port that you are using

    By default, when TX flow control is enabled, pause frames that are received at the MAC module will be acted upon and then dropped. If you would like the pause frames to be transferred into the memory of the device then you will need to set the RX_CMF_EN bit to 1 in the MAC_CONTROL register. 

    To see if data transmission is currently halted due to TX flow control you can check the TX_FLOW_ACT bit in the MACSTATUS register.

    To see the current value of the transmit pause timer you can check the TX_PAUSE register. 

    The RXPAUSEFRAMES statistic for your Ethernet port will show you how many pause frames have been received. This will be able to give you confirmation of whether or not the other machine in your system has sent pause frames to the 6678 device. 

    Let me know if this changes the behavior you are seeing,

    Jason Reeder

  • Hi Jason,

    RXPAUSEFRAMES is always 0, and the problem still exists after the flow control is enabled. 

    So it is probably not related to flow control.

    Will do an EVM to EVM test (direct link without external switch). It will take some time for me to set it up, because my current test is driven by a PC and an external server. 

    Daniel

  • Hi Jason,

    I modified our code and did a EVM to EVM test with direct link, and did not observe any packet loss.

    Then I used the same code to stream packets directly to my PC without involvement from external switch or server, and again did not observe any packet loss. But packet loss is observed when external switch is involved.

    So looks like the packet loss is due to our network after all. The capture on the mirror port does not reflect what is actually on the wire.

    Thank you very much for your explanation, which gives me much better understanding of the NETCP system.

    Daniel

  • It is finally confirmed that the problem is due to a faulty switch that we have. After changing the switch, the problem goes away.

    Daniel

  • Daniel,

    I'm glad you were able to get to the bottom of the issue. 

    Out of curiosity, when you did the 6678 direct to PC test did your RXPAUSEFRAMES in Stats B remain 0? Or did the PC need to send pause frames to keep up with the 6678?

    Jason Reeder

  • Hi Jason,

    RXPAUSEFRAMES  in Stats B remains 0. Haven't observed any non-zero value for this register so far.

    Daniel