TMS320F28388D: C2000 TCP/IP Stack Issues, Support, and Roadmap plans

Mike , 

We do understand our concern. Will get more details and share with you. However as mentioned earlier TI NDK product is in long-term maintenance, and as such there is no plan to provide new updates .  The recommendation is to use LWIP and we do provide few examples (TCP/IP, UDP) in the current C2000Ware.  This is bare metal and do not use any RTOS.  Do you forsee any problems in using LWIP?

Best Regards

Siddharth

Hi Siddharth,

Thank you for the quick response.   Yes, it would be much appreciated if you could provide details on TI's F2838X network stack (etc) roadmap plans (what and when) so we can plan accordingly.

In the meantime, I am gathering more details for you on our current issues/implementation and our questions/concerns about moving to LWIP.   This will take me a couple of days as we have more than one product this is going into and a few FW engineers working on them.   I want to make sure I get all of their inputs and boil them down before I reach back out to you....probably Tuesday.

Again, we appreciate your support

Mike

Hi Mike,

Okay, you can provide your inputs. 

Have you already implemented your application using  TI-RTOS/NDK/NSP and is it in production? 

Best Regards

Siddharth

Hi Siddharth,

We were in beta during 2020 approaching a production release using the F28379.  In response to (essentially) a whole market shutdown, our new system 's launch got delayed.   This eventually resulted in significant feature->design updates where we moved over to the F2838XD. 

We are now in alpha approaching beta on one product  and in-design incorporating the F2838XD into another machine (multiple circuit boards in each).

Yes, the current implementation is using TI-RTOS/NDK.   A  huge amount of the embedded code is realtime controls and so there has been a lot of focus on that.  And, the ethernet connectivity/throughput has been adequate for the controls and status, but the  (inconsistencies of the) high speed streaming data's BW is now becoming an issue.   The streaming data is needed for backend algorithms running on GPU's.

Hope that helps.

Best regards

Mike

Hi Mike,

Thanks for the additional  info. Will get back to you.

Best Regards

Siddharth

Hi Siddharth,

We have spent some engineering time looking at your LWIP code base and examples:

• There are two examples that we could find in C2000ware: a webserver and a UDP example, but no TCP/IP examples
• We’ve attempted (for lack of an example) to implement a LWIP TCP/IP application fitted to our architecture
  • This initial attempt had achieved limited BW (it is slow) and its stability is lacking
  • We believe we are groping in the dark due to lack of a basic example that demonstrates how to
    • properly hand off to/from LWIP to achieve relatively high BW (duty cycle)  use of a 100Mb/s ethernet channel
    • while maintaining the CM in a stable state that allows other processes to continue to be serviced.
• For reference, our architecture (what we implemented using your RTOS/NDK)
  • We have a fairly classic asymmetric TCP-IP based client/server architecture.
  • The Delfino is the server with a listener.
  • The client PC (using WinSock and .NET socket library) opens a socket to the Delfino
    • Most PC->Delfino packets are low bandwidth command/response packets for Commands, Settings, Status, etc
  • Upon receiving a command to stream data, the Delfino will begin streaming ~64Kbyte data sets broken into roughly 64 packets each
    • This is a continuous streaming that lasts for 5minutes at a time and whose BW requirements are >> 24Mb/s
      • 1Kbyte or larger packet size
      • Average (not fixed) 32-64 packets per data set
      • 32 to 64 data sets per second. (16Mb/s to 32 Mb/s)
  • Can LWIP support this architecture
    • And if so, can you point us to (and/or provide us with) an example that we can build on.
    • And if NOT,  describe how we can use LWIP to achieve our requirements in an architecture  better suited to LWIP.
    • TI stated that LWIP is used in a lot of places but we only have two very basic examples that we can find in C2000ware. 
      • Are there other examples that TI has that are not in the C2000ware?
      • Or, does there exist any good non-TI examples of implementing raw data communication with LWIP, particularly high throughput bidirectional TCP/IP applications?
    • How much direct experience and expertise with LWIP does the TI engineering team have?  Can we expect technical support (for our implementation) from TI with questions and concerns specific to LWIP and its source code?

In regard to our RTOS/NDK implementation

• We are experiencing throughput rates that vary from session-to-session with a session being defined as a physically stable board/connection environment.  For example, we can apply power to the board and send data achieving 30Mbps across multiple data link connect and disconnect cycles. 
  • That is plug-it-up, turn-it-on, and look at that: it works and it works well……
  • This involves many socket connect/disconnect cycles.
  • But, physically unplug the cat-5, or power cycle the Delfino (leave the PC alone) and start again….and now we may only get 10Mbps (or less) throughput across multiple data connections/transfers.  
• We have found no culprit for this unknown drop in data transfer rate. 
  • Is this consistent with your experience in this configuration?
• We haven’t seen (in any documentation) and don’t know if our code is intended to manage the PHY registers.  Is there any configuration or monitoring of these registers that the user code is intended/required to perform?
  • We have dumped the PHY registers and do not see any configuration changes having occurred between an acceptable BW transfer vs one that is operating slowly?
• Below is a pseudo code block to illustrate how our system works. Some comments have been added to highlight that things coming and going from CPU1/CPU2 are independent of things coming and going from the CM.  We think this distinction really can’t be emphasized enough, because it’s what sets our application’s needs apart from the web server architecture that seems to be implied in the TI examples/comments of Ethernet connectivity.

Thank you for your help.   We have a large investment in delfino based HW and FW design at this point over many years and so we need to get to the bottom of this.

Mike

// connection handler that gets hooked up to the NDK stack
handleNetworkConnection(SOCKET s, Uint32 unused)
{
// the client has just connected--do some initial setup

  // once connected, we'll be sending data back and forth for quite a while
  while(true)
  {
    // NOTE: the clients are using industry-standard Berkeley socket implementations (Winsock and .NET)
    inDataLength = recv(s, inDataBuf, MAX_SIZE, 0); //configured with a short timeout if the connection is still active but not sending data

    if(connectionClosed)
      break;

    if(inDataLength != 0)
      processIncomingMessage(); // not tied to a particular response

    // CM's outbound traffic--may or may not be messages waiting to send
    outDataLength = getOutgoingMessagesCm();
    if(outDataLength != 0)
      send(s, outDataBuf, outDataLength, 0);

    // CPU1's outbound traffic--may or may not be messages waiting to send
    outDataLength = getOutgoingMessagesCpu1();
    if(outDataLength != 0)
      send(s, outDataBuf, outDataLength, 0);

    // CPU2's outbound traffic--may or may not be messages waiting to send
    outDataLength = getOutgoingMessagesCpu2();
    if(outDataLength != 0)
      send(s, outDataBuf, outDataLength, 0);
  }
  
  // the client has disconnected--do some final cleanup
}

Mike, 

Will take a look at your detailed post and will get back in few days. 

Best Regards

Siddharth 

Mike, 

Had few discussions over the last week regarding this topic and here is the summary. 

Regarding NDK, the product is no longer actively supported. Also, the team who was handling the NDK  earlier is dissolved.  Any anlysis of the throughput issue that you are facing will be difficult to support. 

TI is recommending to move to LWIP.  LWIP is a open source stack and TI has ported the stack for F2838x.  TI can support any porting related issues with LWIP,  however anything specific to LWIP, we are relying on the open source community forum to support.  Team within TI does not have expertise on internals of LWIP stack.

Regarding performace, we have done a test (running no OS/ LWIP Stack) and could achieve ~33Mbps using the Checksum Offload Engine feature in the hardware. CPU was only doing Etherenet operations during this test. 

On the roadmap, there is a plan to support LWIP with FreeRTOS but the schedule is not yet finalized. 

Best Regards

Siddharth

Siddharth,

If TI has an example test demonstrating 33Mbps using LWIP, will TI share this code?

Thanks

Mike

Mike, 

It's not an example, it was a performance test done using IPerf tool . Iperf tool was used to create data streams and measure the throughput.

Will try to get more details and share if possible

Best Regards

Siddharth

Hi Siddharth,

We believe sharing the code used in your performance test could be helpful.  Much appreciated if you can help to get this accomplished.

We had originally (~2017) used a F28379D with an OMAP and TI-RTOS  (for Ethernet connectivity) and so had patience (for a couple of years) in NOT moving to F2838XD until we saw the support by TI of  RTOS with examples for it (in 2020).  Given that TI support has now been eliminated, and that TI is (essentially) only pointing customers to external (non-TI forums) for potential support of ethernet applications/LWIP development  we have a few additional questions:

1. Is there an underlying F2838X HW issue and/or concern that has led to the elimination of internal support of ethernet connectivity applications?
2. Was the RTOS only pulled from the F2838X family or others and why?   Remember we still need that ethernet connectivity.
3. Is Texas Instruments discouraging new design-ins of the F2838X for Ethernet connected apps?
4. Should we be concerned that the delfino with a CM will have a short life span and be EOL'd in the near future?

Thanks again

Mike

Mike, 

Will try to get the code for performance test and share it.

Please find response to your other questions below

1) There is no underlying HW issue with F2838x, only the software support on Ethernet has been minimal

2)  TI-RTOS support has moved from a central TI-RTOS team to individual business units within TI , so it has affected other TI products as well.

3)  No, I don't think so that is the case. 

4) There is no such reason to be concened about, there is no such plan to make this device EOL in the near future. 

Best Regards

Siddharth

Michael Wohl said:

We have found no culprit for this unknown drop in data transfer rate.

I haven't attempted to use LWIP on a TMS320F28388D, but can you take a wireshark capture on the PC for both the "good" and "bad" data transfer rate?

Comparing the two cases may allow the issue to be identified.

E.g. to determine if maybe re-transmissions or if the TCP windows are closing due to running out of buffers.

Michael Wohl said:

• Average (not fixed) 32-64 packets per data set
• 32 to 64 data sets per second. (16Mb/s to 32 Mb/s)

What in your application causes the average data rate to change?

Not sure if the change in packet rate can trigger some interaction with buffering.

Hi Chester,

In regard to the NDK based TCP application/performance, I don't have wireshark data I can share at the moment as we have moved our immediate focus to LWIP (for obvious reasons).  I can say that my stated variance in the designed data rate is only from a testing perspective.    The actual use case is fixed.

In regard to LWIP, we have made progress using UDP and are encouraged.   We believe the root problem is not in the LWIP port, but in its use of the LWIP API by the TI UDP example.  Looking forward to see if TI can add additional insights by releasing the iperf-tested example from their side.

Thanks

Mike