TM4C1294NCPDT: Help with malloc() Error from the NDK

Hi Doug,

  I'm not sure if you have a memory management issue. Can you please take a look at this thread on how to manage memory in NDK? Hopefully, it will provide some hints to your problem. https://e2e.ti.com/support/processors-group/processors/f/processors-forum/947313/faq-how-is-memory-managed-in-the-ndk

I had not seen those documents before so thanks for the links!  Just to add a bit more, the messages I send and receive over the TCP/IP sockets are small  fixed length messages (256 bytes) each and they are sent and received about 1 every 5 seconds.  It does usaully take a long time for the error to occur like if I run it overnight, often times in the morning there is a problem with the stack.  

I looked at the settings an my settings are the defaults with the exception of the "time settings" is TI-RTOS->Products->NDK->TransportLayer-TCP-Module settings.  I had the keep alive settings turned up to be more aggressive so the sockets time out faster.

When I looked again at the ROV tools to see stack sizer per the document NDK Dynamic Memory Manager on page 3 of 4, I do not see the ROV showing the task "tcpHandler" but I do see the other two tasks and can confirm that the ti_sysbios_knl_Idle_loop__E and ti_ndk_config_Global_stackThread both look OK for stack size. 

The question I have is the malloc() gets memory from a heap not a stack so I didn't see where to adjust the heap memory in any of the documents unless its implied by making the stacks smaller??? 

Doug

Sorry I hit the "this resolved my issue" button by accident, can you fix that so it's not marked solved?  

When I check those settings in my project, the System (Hwi and Swi) stack size was increased to 4096 and the heap size was 22528.  I see 22528 is 22*1024=22528 so  I increased the heap to 30*1024 = 30720 and will give that a try.  If it doesn't fail, I may need a couple days to be sure since it never failed right away.

Thanks,

Doug

I don't know how to undo it. Just reply back to the thread and it will automatically reopen and I will get the notification. 

I just wanted to give a update and say so far its working.  I'll give it a couple more days to see if really is fixed.  

Thanks,

Doug

Glad to hear that it is working so far. Let us know again if you have some update. 

For various reasons, I haven't been able to test for a while but last night I was able to run the system overnight and when I cam in this morning, the board was running on the probe and the heartbeat LED was flashing but the board was not pinging and I see the following errors in the console window.  

43555.300 NodeNew: OOM
43555.300 RtFind: Clone creation failed!
43559.378 NodeNew: OOM
43559.378 RtFind: Clone creation failed!
43559.378 NodeNew: OOM
43559.379 RtFind: Clone creation failed!
43559.379 NodeNew: OOM
43559.379 RtFind: Clone creation failed!
43559.379 NodeNew: OOM
43559.379 RtFind: Clone creation failed!
43560.351 NodeNew: OOM

A couple things to note:

* The error usually only happens when the system is running for a long time (overnight).

* I have a couple TCP/IP sockets and on one socket every 5 seconds the host requests a message from the Tiva and the Tiva has a simple reply so it is not using dynamic memory, it just sends a simple reply message.

* The ROV shows the stacks are all OK on the tasks.

* The sockets use keepalives enabled and I notice the lowest priority task blinking the LED briefly stops blinking when the system does a keepalive check.  The ROV shows the NDK stack has a priority 2 and the LED task is priority 0 so its expected the LED would stop blinking briefly but I wouldn't expect the keepalive to be costly enough to notice.

10954.500 TcpTimeoutKeep: Keep Timeout

Any more ideas?

Thanks,

Doug

I'm not sure what to do next to fix this.

Hi Doug,

  Is this after you increase the heap size? I thought a one point of time with the increase in heap you were able to continue running for much longer time? But it seems that you are having the same problem again with similar duration of time as in your original post. Or it is running longer than original but still fails eventually. 

 I find these two known bugs and wonder if it will make a difference to your problem if you use the suggested workaround. 

• TIRTOS-1488: "NDK can get into a state where unable to transmit packets". Please refer to this forum thread for the description of the problem and work-around: https://e2e.ti.com/support/microcontrollers/tiva_arm/f/908/t/674812

• TIRTOS-1551: "TM4C EMAC driver does not reset read/write indexes on restart of NDK". Please refer to this forum thread for the description of the problem and work-around: https://e2e.ti.com/support/microcontrollers/tiva_arm/f/908/t/710149

I was not able to leave the system on overnight in a while so I was not able to test till now but to your question yes this error happened after I made the increase in memory size. 

Do you think the errors I am seeing:  43559.379 RtFind: Clone creation failed! and  43559.379 NodeNew: OOM would be fixed by the changes in your links?    The errors I am seeing still looks like a memory issue to me.

Doug

Hi,

In addition to the two known issues that I think may impact you, I also find the below post with the same error as yours. It was suggested to increase Global.memRawPageCount to resolve the issue. What is your current Global.memRawPageCount? Can you increase a few more pages to see if it makes a difference?

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/568131/rtos-tm4c1294ncpdt-udp-sendto-32-severs-failed

This note talks about how to increase memRawPageCount and the ramifications of using it. 

/cfs-file/__key/communityserver-discussions-components-files/908/NDK-Static-Internal-Memory-Manager-_2D00_-Texas-Instruments-Wiki.pdf

Here are me settings, BTW, it ran all nigh last night.

Doug

Doug Baker said:

Here are me settings, BTW, it ran all nigh last night.

It ran all night with the increased heap size? How is it different compared to your earlier reporting where you said you were still getting the same error in the next day after you increased the heap size? Is it the same code and setting this time and the last time?

My last reply was to suggest increasing the memRawPageCount. You are showing 8 in your .cfg. Per the post I referenced, can you increase memRawPageCount from 8 to a higher value. Perhaps try 10 or 12 and will that make a difference. The post I referenced was showing the same errors as yours. 

To you question about whats different, my understanding is the problem does not always happen overnight so it's not easily reproducible.  We have two setups that reproduce this issue and I have one setup and a coworker is doing the Linux side.

The client on the other side is Linux and it should open a socket and do a periodic status message and it is doing that every 5 seconds.  What I found out is the host (Linux system) is rebooting every 1/2 hour and last nigh that was fixed.   I don't know if this is causing the NDK in the Tive a problem or not but I would assume the Tiva should be able to handle that since I'm using TCP/IP and keep-alives.  Last night, the host fixed the  reboot every 1/3 hour and it was running this morning.  I'll try changing the memRawPageCount from 8 to 12 and see what happens.  I have a feeling this is going to take a while to find the actual fix.  I suspect that Linux side rebooting could somehow be causing the problem I'm seeing but that's just a guess..

I have the 

Thanks,

Doug

Hi,

  Will you please take a look at these two posts that may help.

In this below post the poster uses SO_LINGER to handle situation when the connection is closed. In your case, perhaps this is the case due to your host reboot. 

https://e2e.ti.com/support/microcontrollers/arm-based-microcontrollers-group/arm-based-microcontrollers/f/arm-based-microcontrollers-forum/1019830/ek-tm4c1294xl-tcp-client-sockets-creation-fails-after-10-times-socket-calls/3770705#3770705

In the post, the poster increase the keepalive time. Compared to your keepalive time, this posters has a much larger time. Perhaps it may help to some extent. 

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/513818/so_keepalive-usage?tisearch=e2e-sitesearch&keymatch=SO_KEEPALIVE#

Those links look like they have good information, I'll try some of these ideas next week and let you know but it could be a while before I get to this and get the results.

Thanks,

Doug

Ok, let us know the results after trying the suggestions. 

I don't want to Jinx myself bu so fa so good.  I said I would give periodic updates so I wanted to let you know its been going good.

Hi Doug,

  Hope your issue is fully resolved pending your ongoing testing.