AM6422: ethnet DMA memory size

Hi,

wanglili 王 said:

we found that the DMA channel can only be configured with 16 for transmission and 32 for reception

Can you please elaborate which field you are trying to configure in the DMA? From sysconfig, we configure 16 packet buffers for Tx and 32 Rx buffers for Rx. And the ethernet packet size supported by each of the buffer is 1536 bytes. 

wanglili 王 said:

If we try to modify these configuration numbers, the configuration software reports an error

If you are trying to change the number of buffers allocated itself, then it will be not supported out of box, as the maximum number of packet buffers supported out of box is 196 buffers, Rx and Tx combined. You can increase them such that 128 are allocated for Rx and 64 to TX.If more packet buffers are needed, Please let us know so we can suggest any alternatives, or would enable you to configure higher limits. 

It remains same for gigabit as well, but gigabit speeds to host port with LwIP stack are not supported. We can reach gigabit receive to host(R5F) only in Layer 2 examples. Can you please provide more details on the requirement so that we can suggest possible solutions.

Regards,
Teja.

Hello,

    Currently, we are using Ethernet for data reception, with a data reception interval of 2ms per frame. During the testing process, we found that there were packet loss issues in the received data. We are considering whether the current buffer size of 32 can be increased to verify the packet loss.

    The way we use it is through the LWIP protocol stack. If we were to use Gigabit Ethernet, why wouldn't it be supported? And where exactly would the errors occur? Could you provide a detailed explanation on this?    

One more question: When using the FreeRTOS system in R5F, how can we test the bandwidth of the Ethernet? What is the maximum bandwidth that TI's Ethernet can achieve at a 100M rate?

According to your suggestion, the receive buffer was modified to 128. However, when modifying the transmit buffer from 16 to 32, the program enters an infinite loop during initialization at the specific location shown in the figure below. What is this issue related to?

Hello，

    The issue I mentioned earlier regarding the failure to modify the number of transmit buffers has been resolved, and it can now be set to 64. Additionally, I haven’t received your feedback yet on the bandwidth testing under the real-time core—I hope to get your input on this.

    We are adding our application processing based on the enet_lwip_cpsw project. Currently, we can confirm that when the transmission interval is relatively short, a pointer to an empty address is obtained in the function Lwip2Enet_sendTxPackets. The reason for this behavior is unclear, and we would appreciate your technical support on this matter.

    The value obtained from struct pbuf *hPbufPkt = pbufQ_deQ(&hTx->readyPbufQ); is NULL, which causes the program to enter the void Lwip2Enet_assertPriv(bool cond, const int32_t line, const char* file) function within Lwip2Enet_setSGList(pCurrDmaPacket, hPbufPkt, false);, ultimately resulting in an infinite loop.

void Lwip2Enet_sendTxPackets(Lwip2Enet_netif_t* pInterface, const Enet_MacPort macPort)
{
    /* If link is not up, simply return */
    if (pInterface->isLinkUp)
    {
        EnetDma_PktQ txSubmitQ;
        Lwip2Enet_TxHandle hTx = pInterface->hTx[0];
        EnetQueue_initQ(&txSubmitQ);

        if (pbufQ_count(&hTx->unusedPbufQ))
        {
            /* send any pending TX Q's */
            Lwip2Enet_pbufQ2PktInfoQ(hTx, &hTx->unusedPbufQ, &txSubmitQ, macPort);
        }

        /* Check if there is anything to transmit, else simply return */
        while (pbufQ_count(&hTx->readyPbufQ) != 0U)
        {
            /* Dequeue one free TX Eth packet */
            EnetDma_Pkt *pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&hTx->freePktInfoQ);

            if (pCurrDmaPacket == NULL)
            {
                /* If we run out of packet info Q, retrieve packets from HW
                * and try to dequeue free packet again */
                Lwip2Enet_retrieveTxPkts(hTx);
                pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&hTx->freePktInfoQ);
            }

            if (NULL != pCurrDmaPacket)
            {
                struct pbuf *hPbufPkt = pbufQ_deQ(&hTx->readyPbufQ);
                EnetDma_initPktInfo(pCurrDmaPacket);

                Lwip2Enet_setSGList(pCurrDmaPacket, hPbufPkt, false);
                pCurrDmaPacket->appPriv    = hPbufPkt;
                pCurrDmaPacket->txPortNum  = macPort;
                pCurrDmaPacket->node.next  = NULL;
                pCurrDmaPacket->chkSumInfo = LWIPIF_LWIP_getChkSumInfo(hPbufPkt);

                ENET_UTILS_COMPILETIME_ASSERT(offsetof(EnetDma_Pkt, node) == 0U);
                EnetQueue_enq(&txSubmitQ, &(pCurrDmaPacket->node));

                LWIP2ENETSTATS_ADDONE(&hTx->stats.freeAppPktDeq);
                LWIP2ENETSTATS_ADDONE(&hTx->stats.readyPbufPktDeq);
            }
            else
            {
                break;
            }
        }

        /* Submit the accumulated packets to the hardware for transmission */
        Lwip2Enet_submitTxPackets(hTx, &txSubmitQ);
    }
}
void Lwip2Enet_assertPriv(bool cond, const int32_t line, const char* file)
{
    volatile static bool waitInLoop = TRUE;

    if (!(cond))
    {
        void EnetAppUtils_print(const char *pcString,...);
        EnetAppUtils_print("Assertion @ Line: %d in %s : failed !!!\r\n", line, file);
        while (waitInLoop)
        {

        }
    }
}

Hello,

The issue that you are seeing is because the LwIP application that you are using is trying to enqueue a pBuf with a NULL pointer to be sent out. It is not mishandling in the interface layer that is causing the issue, but the application level itself. netif->linkoutput is being called for a NULL pointer. Can you please verify this by adding a check before you are sending the pBuf out?

Additionally, please give more details on

1. How many packets are being sent successfully?
2. Is there any case that you were able to send packets continuously? (under various Tx speeds, packet lengths, etc)
3. If you are observing the failure when you are not running your custom lwip task, and running other linux utilities like Ping or iperf?
4. What is the base example used to build this example?

These details will help us understand the issue better. I suggest to carry this conversation in a separate thread if you are fine with it. If there is already a thread opened on this topic, please share us the link.

Regards,
Teja.

Hello，

    The scenario we are using involves sending one frame of data every 4 ms. After running for approximately 4–5 minutes, this error occurs. Each packet sent is 178 bytes in size, at a speed of 100 Mbps. No other functions, such as ping or xperf, were used during operation. We aim to verify whether the Ethernet would crash under extreme conditions.

    According to your suggestion, if the issue is not caused by the driver, how does the driver handle the situation when new data arrives before the data in the DMA transmit buffer has been fully sent, especially with such short transmission intervals?

Hi,

The reason of my suspicion is, the failure is occurring where the pBuf from the stack is being unpacked. There is a check to ensure we are operating on valid pBuf pointer. Since the pBuf pointer is seen as NULL, the application is going into an infinite loop.

The enet driver and the lwip interface layer use buffer queues to transmit packets. In instances where a free buffer is not available, it will bail out from the transmit loop, and try again without any need of application interaction. Our lwip layer is tested with even higher packet loads, beyond 10,000 packets per second. 

Please add checks in your periodic packet send function to ensure the pBuf being sent is having a valid pointer. If the issue is still seen, we shall try to investigate further into the issue.

Regards,
Teja.

Hello，

    Following your suggestion, I traced the location where the application sends data and found that in the code below, there are cases where buf obtained from buf = pbuf_alloc(PBUF_RAW, ulSize, PBUF_RAM); is NULL. However, in this situation, the program execution enters the undef handler instead of the previously mentioned Lwip2Enet_assertPriv function.

	buf = pbuf_alloc(PBUF_RAW, ulSize, PBUF_RAM);
	goosebuf = buf;

	if(buf==NULL)
	{
	    DebugP_log("## goose pbuf is NULL!!\r\n");
	}
	/*if(ulSize==179)
	{
	    DebugP_log("## send size is 179!!\r\n");
	}*/
	buf->payload = tptr;
	buf->len = ulSize;
	LWIPIF_LWIP_send(g_pNetif[1], buf);	

Hello，

    Yes, we have processed this part and are currently trying to determine the reason for entering this function Lwip2Enet_assertPriv. At this point, it does not appear to be an application layer issue.

Hello,

wanglili 王 said:

At this point, it does not appear to be an application layer issue.

Can you please clarify if the issue is still seen even if NULL pointers are not passed as pBufs to the LwIP interface layer? What are the current symptoms of failure?

Regards,
Teja.

Hello，

    Currently, there are two issues. One is that when sending data by calling LWIPIF_LWIP_send, the corresponding pbuf allocation returns NULL, which leads to an error entering HwiP_undefined_handler. The other issue is that when the pbuf is not NULL, the system enters the Lwip2Enet_assertPriv function. Through simulation, it was found that while executing the Lwip2Enet_sendTxPackets function and calling the function struct pbuf *hPbufPkt = pbufQ_deQ(&hTx->readyPbufQ), hPbufPkt returns NULL.

Hello，

    I carried out the test as you instructed. Thank you for your explanation.

Hello,

I am glad that this helped you here. If there are any further queries, please reach out to us.

Regards,
Teja.

Hello，

    I currently have another question. When sending data and using pbuf_alloc to request a buffer, is this buffer size fixed? My current test results show that it can only allocate up to 18 buffers. If I request more than 18, it fails to allocate. Where is this size configured?

    Another question is, in the lwipopts.h file, what are #define MEM_SIZE 10240 and #define MEMP_NUM_PBUF 128 related to?

Hello,

    Thanks for your reply.

    I think we have identified the reason for the issue we mentioned earlier regarding struct pbuf *hPbufPkt = pbufQ_deQ(&hTx->readyPbufQ) returning NULL, which is related to how we are using the system. Let me explain this to you in detail.

    We are using the FreeRTOS operating system implementation, where there are two tasks related to Ethernet operations. One task has a relatively higher priority and calls the LWIPIF_LWIP_send function to send data. The other task has a relatively lower priority and calls the sendto(sntpSocket, (char *)&sntpRequest, sizeof(sntpRequest), 0, (struct sockaddr *)&dstAddr, sizeof(dstAddr)) function. However, when we trace this function, we find that it ultimately also calls the LWIPIF_LWIP_send function.

    In this scenario, if the lower-priority task is executing the while (pbufQ_count(&hTx->readyPbufQ) != 0U) condition check within the Lwip2Enet_sendTxPackets function (which is called by LWIPIF_LWIP_send), and the higher-priority task preempts the lower-priority task to execute the LWIPIF_LWIP_send function, the higher-priority task will detect and transmit all the data previously queued in the DMA queue by the lower-priority task to the driver layer. At this point, the hTx->readyPbufQ->count value corresponding to the lower-priority task has already become 0. When the execution returns to the lower-priority task and it attempts to execute struct pbuf *hPbufPkt = pbufQ_deQ(&hTx->readyPbufQ);, since the count value is already 0, the retrieved address value will be NULL. This subsequently causes the Lwip2Enet_setSGList function to enter the Lwip2Enet_assertPriv infinite loop.

    Is this scenario I described actually possible? If so, how should we avoid it?

err_t LWIPIF_LWIP_send(struct netif *netif, struct pbuf *p)
{
    /* Get the pointer to the private data */
    Lwip2Enet_netif_t* pInterface  = (Lwip2Enet_netif_t*) netif->state;
    const Enet_MacPort macPort = pInterface->macPort;
    Lwip2Enet_TxHandle hTx = pInterface->hTx[0];

    Lwip2Enet_assert(pInterface != NULL);
    Lwip2Enet_assert(hTx != NULL);
    /*
     * When transmitting a packet, the buffer may be deleted before transmission by the
     * stack. The stack implements a 'ref' feature within the buffers. The following happens
     * internally:
     *  If p->ref > 1, ref--;
     *  If p->ref == 1, free(p);
     * pbuf_ref(p) increments the ref.
     */
    pbuf_ref(p);
    /* Enqueue the packet */
    pbufQ_enQ(&hTx->readyPbufQ, p);
    LWIP2ENETSTATS_ADDONE(&hTx->stats.readyPbufPktEnq);
    /* Pass the packet to the translation layer */
    Lwip2Enet_sendTxPackets(pInterface, macPort);
    /* Packet has been successfully transmitted or enqueued to be sent when link comes up */
    return ERR_OK;
}
void pbufQ_enQ(pbufQ *pQ, struct pbuf *p)
{
    pbufNode* temp = NULL;

    uint32_t key = HwiP_disable();

    temp = mempQ_malloc();
    pbufQ_assert(NULL != temp);

    temp->hPbufPkt = p;
    temp->next = NULL;

    if(pQ->count == 0)
    {
        pQ->head = temp;
        pQ->tail = temp;
    }
    else if(pQ->count == 1)
    {
        pQ->head->next = temp;
        pQ->tail = temp;
    }
    else
    {
        pQ->tail->next = temp;
        pQ->tail = pQ->tail->next;
    }
    pQ->count++;

    HwiP_restore(key);

}
struct pbuf* pbufQ_deQ(pbufQ *pQ)
{
    struct pbuf *rtnPbuf = NULL;
    pbufNode *temp = NULL;
    uint32_t key = HwiP_disable();

    if(pQ->count != 0)
    {
        rtnPbuf = pQ->head->hPbufPkt;
        temp = pQ->head;
        pQ->head = pQ->head->next;
        if(pQ->count == 1)
        {
            pQ->tail = NULL;
        }
        pQ->count--;

        pbufQ_assert(rtnPbuf != NULL);
        mempQ_free(temp);
    }

    HwiP_restore(key);

    return rtnPbuf;
}

void Lwip2Enet_sendTxPackets(Lwip2Enet_netif_t* pInterface, const Enet_MacPort macPort)
{
    int packetsSubmitted = 0;
    /* If link is not up, simply return */
    if (pInterface->isLinkUp)
    {
        EnetDma_PktQ txSubmitQ;
        Lwip2Enet_TxHandle hTx = pInterface->hTx[0];
        EnetQueue_initQ(&txSubmitQ);

        if (pbufQ_count(&hTx->unusedPbufQ))
        {
            /* send any pending TX Q's */
            Lwip2Enet_pbufQ2PktInfoQ(hTx, &hTx->unusedPbufQ, &txSubmitQ, macPort);
        }

        /* Check if there is anything to transmit, else simply return */
        while (pbufQ_count(&hTx->readyPbufQ) != 0U)
        {
            /* Dequeue one free TX Eth packet */
            EnetDma_Pkt *pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&hTx->freePktInfoQ);

            if (pCurrDmaPacket == NULL)
            {
                /* If we run out of packet info Q, retrieve packets from HW
                * and try to dequeue free packet again */
                Lwip2Enet_retrieveTxPkts(hTx);

                pCurrDmaPacket = (EnetDma_Pkt *)EnetQueue_deq(&hTx->freePktInfoQ);
            }

            if (NULL != pCurrDmaPacket)
            {
                struct pbuf *hPbufPkt = pbufQ_deQ(&hTx->readyPbufQ);
                EnetDma_initPktInfo(pCurrDmaPacket);
								
                Lwip2Enet_setSGList(pCurrDmaPacket, hPbufPkt, false);
                pCurrDmaPacket->appPriv    = hPbufPkt;
                pCurrDmaPacket->txPortNum  = macPort;
                pCurrDmaPacket->node.next  = NULL;
                pCurrDmaPacket->chkSumInfo = LWIPIF_LWIP_getChkSumInfo(hPbufPkt);

                ENET_UTILS_COMPILETIME_ASSERT(offsetof(EnetDma_Pkt, node) == 0U);
                EnetQueue_enq(&txSubmitQ, &(pCurrDmaPacket->node));

                LWIP2ENETSTATS_ADDONE(&hTx->stats.freeAppPktDeq);
                LWIP2ENETSTATS_ADDONE(&hTx->stats.readyPbufPktDeq);
                
            }
            else
            {
                break;
            }
        }
        /* Submit the accumulated packets to the hardware for transmission */
        Lwip2Enet_submitTxPackets(hTx, &txSubmitQ);
    }
}

Hi,

Yes, This is possible. Since the Lwip2Enet_sendTxPackets function is not under any guards, it can be preempted which leads to this scenario. Since the packet queues are handled from netif perspective, but not local queues, it is hard to prevent this from happening. Let me consult our experts, and get back to you with some analysis. Please expect a response in 4-5 working days.

Thank you for your patience,
Teja.

Hello，

    Okay, I will wait for your reply.

    Regarding the LWIPIF_LWIP_send function, I have a question. When using this function, it is often paired with the pbuf_alloc and pbuf_free functions. For example, in our case, we first allocate memory from the memory pool before sending data, then assign values to the memory, send it via Ethernet, and finally release the allocated memory. However, during actual operation, if the Ethernet is not connected and our application executes the send operation without checking the network connection, it may result in the allocated memory not being released. This is because the LWIPIF_LWIP_send function internally performs a connection check. But if we add a network connection check in the application, it would mix the driver and application layers, which theoretically should not be the case. How should we handle such a situation more appropriately?

Hi,

wanglili 王 said:

it may result in the allocated memory not being released.

This is true. If the connection is down, and the stack tries to send more packets, then they will be dropped at the interface level due to unavailability of buffers. But the application need not perform connection check to send packets. Please explain the requirement for the application so that we can understand the usecase and suggest possible ways for your need.

Regards,
Teja.

Hello，

    Let me paste the code, and you should be able to see the issue. In one of our tasks, we periodically call this LWIPIF_LWIP_send function. Regardless of whether it succeeds or fails, the return value of this function is always ERR_OK. This means that when sending data without an active connection, the corresponding data won't be sent, but the allocated buffer is not released. The buffer currently has a fixed size of 18. Once more than 18 buffers are allocated, subsequent allocations will fail, triggering the assert function.

	    buf = pbuf_alloc(PBUF_RAW, ulSize, PBUF_RAM);
	    buf->payload = tptr;
	    buf->len = ulSize;
	
	    LWIPIF_LWIP_send(g_pNetif[1], buf);	//zhuangqianxin
	    goosesendcnt++;
	    //netif = g_pNetif[0];
	    //netif->linkoutput(netif, buf);

	    pbuf_free(buf);

Hi,

I will go through this code and provide feedback. Please give me 2 days time since I am engaged in other commitments now.

Thank you for your patience,
Teja.

Hi,

When the link goes down, and the application still ties to send packets, then we can enqueue packets in the buffer the same number of buffers declared in lwippools. After that, due to unavailability of payload pointers, pBuf allocation would fail, and pbuf_alloc would return NULL. So, before trying to send the packet using "LWIPIF_LWIP_send", it is suggested to have a check to ensure buf != NULL, ideally done right after pbuf_alloc().

Regards,
Teja.

Hello，

    I understand what you're saying, but there's still an issue. When the allocation is determined to be NULL, the memory that was successfully allocated earlier cannot be used for sending due to the network disconnection, and the corresponding memory is not released either. After this occurs, subsequent allocations will always return NULL, which means the Ethernet functionality essentially stops working. This does not meet the requirements of our use case.

Hi,

Once the connection is restored, the packets in the buffer will be sent out, and the earlier allocations will be released. This will let us send packets again like before. Can you give more details on your usecase so that I can understand it better? When the cable is disconnected, we can't send packets on to the wire, so the transmission stalls. Once the connection is restored, the transmission of packets resumes. Can you please what is the additional requirement for your system?

Regards,
Teja.

Hello，

    As mentioned earlier regarding our usage of the code, in our use case, when the network cable is disconnected, the system continues to allocate memory normally to send data packets. However, since Ethernet transmission is unavailable at this time, the corresponding pbuf_free(buf) function also fails. As a result, the allocated memory cannot be successfully released. After a period of time, even if the network cable is reconnected and the previously queued data can be transmitted, the pbuf_free(buf) function is no longer executed to release the allocated memory. Consequently, when attempting to allocate memory again, a NULL value is returned due to the lack of available buffer space.

    And regarding the previous issue of two tasks simultaneously calling the send function, is there a solution for handling this scenario now?

Hi,

As part of the LwIP interface layer, our driver contains a Tx task handler which polls onto the Tx queues, and once any unattended buffers are left, it will free them, and the memory will be available to be allocated. 

As part of our testing, we run iperf, disconnect the cable midway, and reconnect it, and the transmission would resume without issues. The driver is stable enough to handle loss of connection during transmission. 

Regarding the earlier call, I would need more time to analyse it as I am currently on a different commitment. Please expect a response on it by Tuesday.

Regards,
Teja.

Hello，

    And regarding the previous issue of two tasks simultaneously calling the send function, is there a solution for handling this scenario now?

Hi,

Due to other higher priority items, I was not able to allocate time for this. I will involve our development team and get some details from there. Please expect a response in 3 working days.

Thanks and regards, 
Teja

Hello，

    How about the results？Is anything news？

Please expect further delays as this week the India team is out of office and limited availability. 
Please expect responses later in the week or early next week

Regards

Mukul 

    How about the results？Is anything news？