• State Resolved
  • Locked Locked
  • Replies 7 replies
  • Subscribers 98 subscribers
  • Views 1853 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Multi TCP/IP connection crash with network latency

misteralex
Intellectual 760 points

Hello,

I am involved in a problem related to a network latency.

The system consist of 6 boards where one is a converter from standard RS232c serial to TCP/IP

port. The other 5 boards are slaves that answer to their proper requests.

This system is designed considering one task to send network request and five tasks for each

slave to process answer received from network. So all boards are connected by ethernet.

This system works fine. Anyway when there is a network latency of millisec (e.g. 10 msec), provided

for example from a firewall, all tasks related to network will crash after few seconds.

I tried to review the question from memory management side and also from SACK context.

No solution until now. I suspect that the TI TCP/IP stack has strange behavior while manage

delayed/lost packages. I mean the problem get start from repeated request on network while

there is delayed reply from network related to one socket. This means that I created the problem

only for one slave not for all of them. 

The system work fine until to 3 slaves including the delayed. it starts to work bad with a configuration

that include 4 slaves. This means that the master create 3 tasks related to each slave (included the delayed) 

and works fine but when add one more task (related to the new one slave without any network latency) all tasks go down after few seconds. Memory management problem? Heap? Stack? What to review there? I tried to extend th size of heap and stack assigned to each task without success.

The TI RTOS package used is the TI-RTOS v1.1.0.25 in ARM/Stellaris context.

Welcome any advice.

Thanks

  • 0
    TI__Mastermind 45025 points

    Hi AlexUnico,

    Which exact hardware platform(s) are you using in your setup?

    AlexUnico said:
    . This means that the master create 3 tasks related to each slave (included the delayed) 

    and works fine but when add one more task (related to the new one slave without any network latency) all tasks go down after few seconds. Memory management problem? Heap? Stack? What to review there? I tried to extend th size of heap and stack assigned to each task without success.

    It sounds like you may be running out of memory.

    Can you open the ROV tool at this point to see what's going on?  You should check out the HeapMem module (including free list), and Task module's detailed view, in particular.

    Steve

  • 0 in reply to Steven Connell
    Intellectual 760 points

    Hi Steve,

    the hardware platform consist of 6 board with ARM Stellaris LM3S9D96.

    All boards communicate through ethernet. The problem appears on the

    master board that convert packages between serial and ethernet port.

    Here in attachment a ROV screenshot about heap debug.

    It shows the message "Requested heap size is too small".

    It sounds like the board is running out of memory?

    What can I do to manage properly this architecture?

    Currently all tasks are created dynamically.

    I changed the stack size assigned to each task but without success.

    Here the heap and tcp stack memory configuration related:

    BIOS.heapSize = 40000;

    Global.pktSizeFrameBuf = 1024;
    Global.memRawPageSize = 2100;
    Tcp.transmitBufSize = 1024;
    Tcp.receiveBufSize = 1024;
    Tcp.receiveBufLimit = 1024;
    Ip.socketBufMinTxSize = 1024;

    Welcome any advice

    Thanks

    Alex



  • 0 in reply to misteralex
    TI__Mastermind 45025 points

    Alex,

    What kind of data are you transferring to each of the slave boards?  How big is the data being sent in a typical "send()" call?  And how often?

    Do you expect "heavy data transfer" to occur?  Or, are you just sending some small amount of communications data from the master board to each of the slaves?

    If your data being transferred is minimal, then perhaps you can lower the TCP socket buffer sizes.  This is handled by the following lines of your config:

    Tcp.transmitBufSize = 1024;
    Tcp.receiveBufSize = 1024;

    You see, with each call to "socket()" that you have to create a new socket (you probably have one per each of these tasks), it will dynamically allocate memory for these TCP send/receive buffers.

    Creating a Task dynamically will also take from the heap, mainly for the Task stack stack size.
      So for each of these Tasks that's created, you need memory for the Task stack and the socket send/recv buffers, which can add up quickly.

    You should make sure the Task stack sizes are minimized for your needs.  One technique for doing this is run through a "typical scenario" or better, a "worst case" scenario for each of your Tasks.  Then, halt the application and open the ROV tool.  Look at the "stack peak" for that task to see what the maximum stack space needed actually was.  If your Task stack size is, for example 2048, but you see a stack peak of 256, you obviously have too high of a stack size configured.

    Steve

  • 0 in reply to Steven Connell
    Intellectual 760 points

    Hi Steve,

    please read my answer in blue.

    Thanks

    What kind of data are you transferring to each of the slave boards? How big is the data being sent in a typical "send()" call? And how often?

    Do you expect "heavy data transfer" to occur? Or, are you just sending some small amount of communications data from the master board to each of the slaves?

    Typically I manage few bytes, lets say less of 10 bytes. Really a small amount of communications data.

     

    If your data being transferred is minimal, then perhaps you can lower the TCP socket buffer sizes. This is handled by the following lines of your config:

    Tcp.transmitBufSize = 1024;
    Tcp.receiveBufSize = 1024;

    Already done. I tried again to test the system considering a size of 100 bytes. Nothing. The issue is still there.

     

    You see, with each call to "socket()" that you have to create a new socket (you probably have one per each of these tasks), it will dynamically allocate memory for these TCP send/receive buffers.

    Creating a Task dynamically will also take from the heap, mainly for the Task stack stack size.
    So for each of these Tasks that's created, you need memory for the Task stack and the socket send/recv buffers, which can add up quickly.

    Yes. I create task dynamically and I suppose the problem is related to delayed network package management.

    The debug provide this message few times before system crash definitively: 

    TCP: Retransmit Timeout


    I suppose that the TCP stack get a lot of memory to manage properly this scenario. Anyway this network latency is well managed while there are only three tasks there.

     

    You should make sure the Task stack sizes are minimized for your needs. One technique for doing this is run through a "typical scenario" or better, a "worst case" scenario for each of your Tasks. Then, halt the application and open the ROV tool. Look at the "stack peak" for that task to see what the maximum stack space needed actually was. If your Task stack size is, for example 2048, but you see a stack peak of 256, you obviously have too high of a stack size configured.

    I already done this review. Anyway I tried to follow your advice about analysis of a single task. In attachment you will

    find the stack peak that for task "tcpWorker0" is just 696 and then I resized the stack size to 800. 

    Again: nothing new.

    Welcome any advice

    Thanks

  • 0 in reply to misteralex
    Intellectual 760 points

    Hello,

    from this experience, while I also tried to reduce the source code where possible, I can

    say that the TCP Stack provided from TI should be improved because currently is 

    hunger of memory. So, the powerful Stellaris processor LM3S9B95 che support smartly

    a network architecture of 1 master and 3 slaves within a very light network traffic.

    Currently I cannot improve the source code to avoid crash in stress network condition.

    Any advice to improve the memory management is welcome.

    Thanks

  • 0 in reply to misteralex
    TI__Mastermind 45025 points

    Hi Alex,

    Since you are transferring a very small amount of data, you can probably reduce the TCP send/receive buffer sizes even further.  Possibly down to 512K or even 256K.

    In case you have not seen these, I wanted to point out the following wikis that discuss the NDK's memory configurations:

    1. TI-RTOS Networking Stack Memory Usage

    2. NDK RAM Sizing: Concerto Example

    You can also find some example configurations shown in the NDK sizing benchmarks section of the NDK documentation.  This can be found in the NDK's release notes.  For example, the release notes are here on my machine's TIRTOS install location:

    C:/ti/TIRTOS/tirtos_tivac_2_00_02_36/products/ndk_2_23_01_01/ndk_2_23_01_01_release_notes.html

    Once you have opened the release notes, look for the section "Benchmarks" and then a link called "Timing and Size Benchmarks" and click on that link.  You'll see a page with more links, you want to click on the M3 link under sizing:

    This will bring you to a new page containing size/memory usage stats for some typical applications.  Notice that the application name is also a link that displays the configuration (*.cfg) file that was used to obtain the footprint info shown in the table.  I think the "tcpSocket Application" will be most useful to you.

    Note that when you click on a link to bring up the *.cfg file, there is a comment near the top of the file that describes some characteristics of that application:

    /*
 *  ======== tcpSocket.cfg ========
 *  Minimized configuration for the TI-RTOS tcpEcho example that contains a
 *  single TCP socket running in a single task.  The TCP send/receive buffers
 *  have been lowered to a value of 512 bytes. Uses DHCP to obtain IP address.

 *
 *  The program works with a modified version of the 'tcpSendReceive' tool
 *  which sends data packets of 512 bytes.
 */

    Steve

  • 0 in reply to Steven Connell
    Intellectual 760 points

    Hello,

    problem solved. 

    The TI TCP Stack now works very well also under stressful network.

    I had only to resize the memory and the most critical was parameter was:

    Global.memRawPageSize = 2100;

    I increased the page size to 3072.

     

    Here my full critical changes:

    Number of frames: from 16 to 8

    Frame buffer size: from 1024 to 256

    Page size: from 2100 to 3072

    See CCS > open your .cfg file > TI-RTOS > Products > NDK > Networking Stack Buffer Sizes And Placement

    All these info are well discussed on TI Wiki page "NDK Static Network Buffers" suggested from Steve.

    Thanks

    Alessandro