TMS320F28388D: Ethernet Issues - TX Missing Interrupt and Delayed TX Packet Transmission

Hello Jay,

The expert was on holiday, they should be able to get back to you soon.

Best regards,

Omer Amir

Thanks for the update!

Hi, 

Sorry but there are no pollling mode examples for Ethernet. 

Are you seeing any other messages in the WireShark before you see the dealyed transmit packet?

Best Regards

Siddharth

Nothing out of the ordinary. I am hitting it with ping traffic and some UDP (application-specific) packets that lead to the delay. It looks as if they are getting stuck/delayed in the driver/hw layer. I see a TX interrupt, but not the packet on the wire. What information would help troubleshoot it? Are there counter in the driver that would be useful?

If I was to use the driver in a simplistic polling mode, any thoughts on how to achieve it? As another user wrote, I put the interrupt routines in a polling loop and disabled interrupts. It _sort of_ works:

void pollEnet(void)

{

Ethernet_receiveISR();
Ethernet_transmitISR();

}

There is more information on the original post. Thanks.

Hi, 

There are counter variables in the driverlib that could help in narrowing down on the issue - Ethernet_txInterruptCount and Ethernet_numTxPktFreeCallback

For these the predefined symbol "ETHERNET_DEBUG" has to be added to the project.

As mentioned, haven't tried polling mode of the driver, so don't have any inputs on it.

Best Regards

Siddharth 

I was normally using the driver with the callbacks, but was trying other control methods. I see the issues with ISR/Callbacks as well.

The counters are what I have been looking at. The TX ISR is either firing but late, or not firing at all. 

I will collect some more data to show the issue. My previous post (link in the first post here) shows the data for the missing TX ISR/callback.

Thanks.

When the TX ISR is triggered, is this a good indication that the packet was sent? Is there a hardware buffer that could be holding TX messages beyond the interrupt time? 

Below is an example of the kind of thing I am seeing:

Packet 1246 is sent by host as part of ping.

1247 is actually a previous ARP response from three ARP requests not on the screenshot. (others are 1249 and 1252)

1248 is the first ping. Response eventually comes at 1254.

Next pings are 1250, 1251, and 1253, all with delayed responses.

The code runs on RX interrupt, processes packet, and sends TX response. There is no delay, especially of ~60s.

When TX interrupt hits, TX packet resource is freed.

Another example:

Packet 171/172 was normal (albeit slow response) ping.

Packet 183 was severely delayed. 

The last three ARP were finally received after some other random traffic was sent in the background. Not sure where these were buffered.

It looks as if some of the RX packets are sitting in a buffer and not either triggering the RX ISR, or not being fully emptied. How does the driver handle multiple packets in the queue? Does it rely on the ISR? What if I am responding to a RX ISR and another packet is received? Should the receive queue be manually checked before exiting RX ISR?

I get a burst of ARPs, which seems to cause the delay in RX -- like the input buffer is not being completely drained.

When I get this burst of ARPs, I only get 4 interrupts (instead of 8) and process only first 4 packets. I can see the data in the RX Buffer, but am not being notified (or even able to process when directly calling RX ISR).

The Ethernet_xxInterruptCount values are off since I am also polling on the ISRs, but the GetPacketBufferCallback is correct (initial 16 buffers + 4 processed).

The missing packets just appear to be unprocessed. The descriptors are allocated to the application, but were not handled in the interrupt.

The first 16 descriptors are TX, the second 16 are RX. All belong to DMA except those highlighted. The data buffers match the packets I expect.

How is the driver designed to handle this? The Technical Manual states to "poll the descriptors, reading the status of the descriptor owned by the Host (either transmit or receive)". In this case, I could check the next descriptor (as pointed to by Ethernet_device_struct.dmaObj) for the owned bit, and thereby determining that I need to process a packet. Suggestions?

Thanks.

Yes, the TX interrupt is a "Transmit Completion interrupt", so it indicates that the transmit packet was sent.

I am not sure why the interrupts are not triggered.  How frequently are you sending the ping request? 

Best Regards

Siddharth

My guess is that it is re-triggered during processing of the previous ISR, and cleared. The ping isn't doing this... its an ARP. The wireshark trace shows that the request packets are as low as 28us apart. The first four are processed, then the remaining four are received and sit in the descriptor queue. They only start to get processed when another packet arrives and triggers the RX interrupt again. This is where my RX ping delay is coming from... the ping packet sits in the descriptor queue until the previous ARP packets are processed.

So it appears that RX interrupts are not re-triggered/enabled when there are descriptors to be processed. How should this be handled? Polling per Technical Manual? There does not appear to be a method in the driver for this case. Is this a completely unexpected behavior? What should be different in the flow to prevent this?

I switched to a polling approach and found that the ARP burst was suspending the RX DMA. From the driver and manual, the process to unsuspend is:

* 1. Change ownership of current RX descriptor to DMA
*
* 2. Issue a receive poll demand command
*
* 3. Issue a write to the descriptor tail pointer register

What is a Poll Demand command? Both the manual and driver mention it, but the driver doesn't do anything:

        /* To un-suspend the DMA:
         *
         * 1. Change ownership of current RX descriptor to DMA
         *
         * 2. Issue a receive poll demand command
         *
         * 3. Issue a write to the descriptor tail pointer register
         */

        /* 1. Change current descriptor owernship back to DMA */
        descPtr = (Ethernet_HW_descriptor *)(HWREG(
                Ethernet_device_struct.baseAddresses.enet_base +
                (uint32_t)ETHERNET_O_DMA_CH0_CURRENT_APP_RXDESC));

        descPtr->des3 = ETHERNET_DESC_OWNER | ETHERNET_RX_DESC_IOC |
                        ETHERNET_RX_DESC_BUF1_VALID;

        /* 2. Issue a receive poll demand command */

        /* 3. Issue a write to the descriptor tail pointer register */
        tailPtr = (Ethernet_HW_descriptor *)(HWREG(
                Ethernet_device_struct.baseAddresses.enet_base +
                (uint32_t)ETHERNET_O_DMA_CH0_RXDESC_TAIL_POINTER));

        Ethernet_writeRxDescTailPointer(
            Ethernet_device_struct.baseAddresses.enet_base,
            ETHERNET_DMA_CHANNEL_NUM_0,
            tailPtr);

Also, I was trying to use ARP offloading to speed up processing. My understanding is that ARP offload will automatically process ARP requests without passing the descriptor to the firmware/application. Is this correct? I have enabled the ARPEN bit in the MAC Configuration register:

    Ethernet_setMACConfiguration(EMAC_BASE, ETHERNET_MAC_CONFIGURATION_ARPEN);

But it does not appear to work the way I thought. The description for this bit states "This bit is available only when the Enable IPv4 ARP Offload is
selected", which I thought was this very bit. Is this a typo? Are there other fields that need to be set (aside from IP address).

Thanks.

Hi, 

The Receive poll demand is nothing but update of the tail pointer register which the driver is already doing. 

Can you check the 'RBU' bit of DMA_CH0_Status Register to see if receiver buffer is unavailable? 

For the ARP offloading, have you configured the IP address in the MAC_ARP_Address Register register?   Is the ARPOFFSEL bit set in the MAC_HW_Feature0 Register?  This should indicate if the ARP Offloading is enabled or not.  Agree that the text 'Enable IPv4 ARP Offload' is confusing since it should refer to this bit itself.

Best Regards

Siddharth

Yes, I did check it and found that the ARP storm was depleting my pointers. That's why I was asking about the ARP Offload in order to help with the issue. So you are saying step 2 is the same as step 3? This matches my observed behavior of the DMA engine. It clears the buffer unavailable issue and resumes processing packets. In my polling loop, I just check the bit and run the clearing process if it is set.

For ARP Offload, I am calling both these driver APIs:

Ethernet_setMACConfiguration(EMAC_BASE, ETHERNET_MAC_CONFIGURATION_ARPEN);   

I am also setting the IP with a call I added to the driver.

MAC_HW_Feature0 (offset 0x11C) is = 0x0E1D73F5 with bit 9 ARPOFFSEL set. Interesting that by default it is set, per documentation.

which I assume are the two fields you are asking about. I still see packets being passed to the application. It a little unclear what exactly the behavior is supposed to be. Are _all_ ARP packets automatically processed? Or ARP packets that don't target our IP filtered, and matching ones are sent to the application?

What else am I missing? Thanks

Hi, 

I don't think all ARP packets are processed automatically .  Only the onee which match with the IP  Address  configured are handled by the hardware and not sent to the application/driver.  Also, it processes only one ARP request at a time. If it receives an ARP request when it is already processing an earlier ARP request, it forwards the new ARP request packet to application .

Best Regards

Siddharth

ok I will check if that is the case. In the example I posted above, it wouldn't have processed any of these packets and the forward to the app layer would be expected.

Can you confirm the byte ordering in the MAC_ARP_Address register? The documentation is a little vague. The desired IP in this case was 172.16.201.201 (0xAC10C9C9). 

I just confirmed that the ARP packets are still be forwarded to the app layer for the desired IP.

Thanks.

Hi, 

The IP address should be in host byte order format, 

Pls refer this post

https://e2e.ti.com/support/microcontrollers/c2000-microcontrollers-group/c2000/f/c2000-microcontrollers-forum/1082860/tms320f28388d-mac_arp_address-bytes-order

Best Regards

Siddharth

BTW, to sum up the issues named in the original ticket:

- TX interrupt was not being missed, but was a threading bug in a queue external to the module

- Polling RX resolved much of my processing issues and processing delays since I directly inspected the USER bit of descriptors and never missed an RX interrupt. One could use the RX interrupt, but should check USER bit regularly.