TMS320F28388D: Low level debug analysis and workaround for: Ethernet / lwIP: RX stops under heavy load – ui32RXPbufAllocFailCount increases

Karan Thawani

Prodigy 10 points

Part Number: TMS320F28388D
Other Parts Discussed in Thread: C2000WARE

Hello TI Support Team,

We are using the C2000Ware v6.00.00.00 SDK on an F2838x device.

Note: The same issue exist in the latest SDK C2000Ware_26_00_00_00 also.

With reference to the issue reported earlier by another user:

https://e2e.ti.com/support/microcontrollers/c2000-microcontrollers-group/c2000/f/c2000-microcontrollers-forum/1572637/tmdscncd28388d-c2000ware-v6-00-00-00-ethernet-lwip-rx-stops-under-udp-load-ui32rxpbufallocfailcount-increases

We were getting the same issue(Ethernet Rx stops and it never recovers until a restart) in our system when we load our system with the heavy TCP load. (Consdering confidentiality of our ongoing development, we can not share our application/setup details for now).

There are mainly 2 reasons to post this here.

Very first one is to share our findings with TI's team so that they can provide a permanent fix in the next SDK.
Those who are still struggling in this issue and stuck in their developement, can use this workaround and continue their development.

So below are our observations:

The problem starts when the ethernet Rx is heavily loaded and RX OVERFLOW interrupt is fired. Now in C2000Ware v6.00.00.00: TI has implemented handling of Rx over flow interrupts in the

Ethernet_genericISRCustom() where the Rx channel is init again and Rx descriptor tail pointer is upadted which does not seems to be a proper handling of Rx over flow interrupts. So the main reasons for Lwip memory exhaustion seems to be this Rx overflow interrupt handling. (PBUF_POOL memory exhaust)

Now in C2000Ware_26_00_00_00: TI has completely removed handling of Rx over flow interrupts in the Ethernet_genericISRCustom(). So there is PBUF_POOL exhaustion in this case, but still the Ethernet Rx and Tx is not working.
We have done some low level debugging to know the Ethernet TX and RX DMA state, Tx and Rx interrupts counters after the Rx over flow interrupts. (with C2000Ware_26_00_00_00 ethernet driver).

C2000Ware_26_00_00_00: So after the RX OVERFLOW interrupt is fired, we have observed that the DMA Rx state machine goes into the suspend state and it never recovers from this state and Ethernet driver Tx stops after some time due to no available tx packet descriptors from the Lwip interface.

RCA for RX DMA goes into Suspend state: Rx interrupt is not fired from the ethernet module when the Rx overflow occurs. Due to this the ethernet Rx packets which are already processed by the DMA and handed over to the application are never seen to the application and never processed by the application and never given back to the DMA. (Because the whole Rx packet handling works based on the Rx interrupt)

RCA for Ethernet driver Tx Stops after some time: Tx interrupt is not fired from the ethernet module when the Rx overflow occurs. Due to this this, the driver sends the packet until the Tx packet FreeQ gets full (file: ports/c2000/netif/f2838xif.c: #define NUM_PACKET_DESC_TX_APPLICATION 20, TxPktFreeQ). After that the Tx stops as ethernet packet descriptors are not freed due to missing Tx interrupt and there are no Ethernet_Pkt_Desc available for Tx.

From the above analysis, it is clear that the main responsbile module for this whole problem is tx and rx interrupts which are not fired from the module when rx overflow occurs.

So for workaround, we have removed Tx and Rx packets handling based on the interrupts. We are continously polling Tx and Rx descriptors.

TI's reference implementation

if((uiISRsignal & RX_ISR_MASK) != 0)

{

//process Rx

Ethernet_removePacketsFromRxQueue(
(Ethernet_DescCh*)
&Ethernet_device_struct.dmaObj.rxDma[ETHERNET_DMA_CHANNEL_NUM_0],
ETHERNET_COMPLETION_NORMAL);

ETHERNET_DISABLE_INTERRUPTS();
CM_Eth_ISRsignal &= (~RX_ISR_MASK);
ETHERNET_ENABLE_INTERRUPTS();

}

if((uiISRsignal & TX_ISR_MASK) != 0)

{

//process tx

Ethernet_removePacketsFromTxQueue(
(Ethernet_DescCh*)
&Ethernet_device_struct.dmaObj.txDma[ETHERNET_DMA_CHANNEL_NUM_0],
ETHERNET_COMPLETION_NORMAL);

ETHERNET_DISABLE_INTERRUPTS();
CM_Eth_ISRsignal &= (~TX_ISR_MASK);
ETHERNET_ENABLE_INTERRUPTS();

}

Modification:

Just remove checks of uiISRsignal & RX_ISR_MASK and uiISRsignal & TX_ISR_MASK

if((1)

{

//process Rx

Ethernet_removePacketsFromRxQueue(
(Ethernet_DescCh*)
&Ethernet_device_struct.dmaObj.rxDma[ETHERNET_DMA_CHANNEL_NUM_0],
ETHERNET_COMPLETION_NORMAL);

}

if((1)

{

//process tx

Ethernet_removePacketsFromTxQueue(
(Ethernet_DescCh*)
&Ethernet_device_struct.dmaObj.txDma[ETHERNET_DMA_CHANNEL_NUM_0],
ETHERNET_COMPLETION_NORMAL);

}

Note: remove if(1), written here just for comparision with the reference implementation.

The workaround seems to be preety simple but we have spent many days in this low level debugging and finding the root cause.

10 days ago

0 Karan Thawani 10 days ago

Prodigy 10 points

typo error: Now in C2000Ware_26_00_00_00: TI has completely removed handling of Rx over flow interrupts in the Ethernet_genericISRCustom(). So there is NO PBUF_POOL exhaustion in this case, but still the Ethernet Rx and Tx is not working.

0 Gouri Siva M 9 days ago in reply to Karan Thawani

TI__Prodigy 750 points

Hi,
Thank you for your input regarding this issue.

Let us try to reproduce from our side and if found proper patch can be applied to next C2000ware release.

0 Karan Thawani 8 days ago in reply to Gouri Siva M

Prodigy 10 points

Hi, Thanks for your reply. the reproduction steps are well mentioned in the original issue which we have mentioned in the starting.

Otherwise you can simply add a intentional delay in any ethernet example (tcp/udp) before processing rx packets which will cause Rx overflow to hit.

reference code for adding delay in ethernet packet rx processing:

uint32_t uiISRsignal = g_uiISRsignal;

if((uiISRsignal & RX_ISR_MASK) != 0)
{

// add delay

if (1 == Cm_Main_DelayRxProcess.isDelay)
{
uint32_t delayTicks = Cm_Main_DelayRxProcess.DelayTicks;
while (delayTicks > 0)
{
delayTicks = delayTicks - 1;
}
}

Ethernet_removePacketsFromRxQueue(
(Ethernet_DescCh*)&Ethernet_device_struct.dmaObj.rxDma[ETHERNET_DMA_CHANNEL_NUM_0], ETHERNET_COMPLETION_NORMAL);

ETHERNET_DISABLE_INTERRUPTS();
g_uiISRsignal &= (~RX_ISR_MASK);
ETHERNET_ENABLE_INTERRUPTS();
}

Add around 100ms of delay by adjusting Cm_Main_DelayRxProcess.DelayTicks.

But it would be really helpful for us if you can confirm whether our analysis regarding this issue is in right direction or not? and the workaround which we have implemented is correct or not?

As we are going towards the production stage of our ongoing development, we do not want to take any risks. So it would be really good if you can confirm this at earliest.

Feel free to reach us if you need any help/more clarity.

0 Karan Thawani 2 days ago in reply to Gouri Siva M

Prodigy 10 points

Hi, Is there any update?

0 Gouri Siva M 1 day ago in reply to Karan Thawani

TI__Prodigy 750 points

Hi,

Sorry for the Delayed response.

I was able to reproduce the issue.

Reproduction steps followed -

- Added one second delay before calling Ethernet_removePacketsFromRxQueue API

- Normal ping - works

- Flood ping - drops more packets

- Again normal - crashes (Rx Process Stopped

Issue found to be that in Ethernet_genericISRCustom API unnecessarily clearing RX ISR was found, which causes not to Trigger Rx Process so the system stucks in RBU state.

Apply following patch in libraries/communications/Ethernet/third_party/lwip

diff --git a/driver/enet.c b/driver/enet.c
index 15cac49..5bd203e 100644
--- a/driver/enet.c
+++ b/driver/enet.c
@@ -124,17 +124,6 @@ interrupt void Ethernet_genericISRCustom(void)
                                      ETHERNET_DMA_CH0_STATUS_RBU));

     }
-    if(0U != (HWREG(Ethernet_device_struct.baseAddresses.enet_base +
-                                 ETHERNET_O_DMA_CH0_STATUS) &
-                           (uint32_t) ETHERNET_DMA_CH0_STATUS_RI))
-    {
-        ENET_DRIVER_STATS_INC(RIinterrupt);
-
-        Ethernet_clearDMAChannelInterrupt(
-                        Ethernet_device_struct.baseAddresses.enet_base,
-                        ETHERNET_DMA_CHANNEL_NUM_0,
-                        ETHERNET_DMA_CH0_STATUS_NIS | ETHERNET_DMA_CH0_STATUS_RI);
-    }

     for(i = 0U;i < Ethernet_device_struct.initConfig.numChannels;i++)
      {

Result on modiefied example with 1 sec delay

- Normal ping works

- Flood ping drops many packets

- Again normal ping works

C2000™︎ microcontrollers

C2000 microcontrollers forum

TMS320F28388D: Low level debug analysis and workaround for: Ethernet / lwIP: RX stops under heavy load – ui32RXPbufAllocFailCount increases