RTOS/AMIC110: Webpage crash

Stefan Haensel

Part Number: AMIC110
Other Parts Discussed in Thread: SYSBIOS

Tool/software: TI-RTOS

Hi,

I'm using the AMIC110 with the profinet example project to run a webpage, where I can see the status of our TMS320. The TMS is connected via SPI to the amic to send the current status.

The webpage is hosted by the amic and updates itself every 5 seconds to show the new status.

After leaving the webpage open for approximatly 30 minutes, the amic crashes with the notice: CortxA8: Unhandled ADP_Stopped exception 0x20023

So I started to debug everything. First I had a look on the heap and if there is a memory leak. Using the mmCheck I got the following results:

  47:ERROR4u ( 73%)    18:ERROR4u ( 56%)    63:ERROR4u ( 87%)    16:ERROR4u ( 66%)
   2:ERROR4u ( 33%)     1:ERROR4u ( 50%)     4:ERROR4u (100%)
(39936/49152 mmAlloc: 1070806/0/1070693, mmBulk: 6735/0/6734)

2 blocks alloced in 512 byte page
24 blocks alloced in 128 byte page
24 blocks alloced in 128 byte page
15 blocks alloced in 128 byte page
23 blocks alloced in 48 byte page
18 blocks alloced in 96 byte page
1 blocks alloced in 1536 byte page
6 blocks alloced in 256 byte page

Everything stays the same over this time, just the mmAlloc counts and free counts went up. But the difference between them (113) stayed the same. So it looks like, there is no problem with the memory, right?

Could it be, that there is some issue with the SNMP-Stack, which is limited to 1000 responses? Or is it independent to it?

Any Idea, what could be the problem?

Best regards,

Stefan

over 7 years ago

0 Biser Gatchev-XID over 7 years ago

TI__Guru**** 393215 points

The RTOS team have been notified. They will respond here.

0 Garrett Ding over 7 years ago in reply to Biser Gatchev-XID

TI__Mastermind 43296 points

Stefan,

Exception should not happen with SNMP stack limitation. Can you try the debugging techniques to see if it helps:

• training.ti.com/debugging-common-application-issues-ti-rtos
• processors.wiki.ti.com/.../Processor_SDK_RTOS:_TI_RTOS_Tips_And_Tricks

Regards,
Garrett

0 Stefan Haensel over 7 years ago in reply to Garrett Ding

Intellectual 680 points

Hi,

I just had a conversation with Mr. Walzer.

It looks like, that I have indeed a leak of memory. When the webpage gets recalled every 5 seconds, the totalFreeSize in HeapMem in ROV gets less and less. When I close the webpage in my browser, this totalFreeSize stays the same.

After ca. 30 minutes I get the following messages:

"CortxA8: Unhandled ADP_Stopped exception 0x20023"

And the lagestFreeSize is almost zero:

As I already told, is my webpage one big webpage, where measurement results are included via an iframe. This small webpages call an cgi function, which build a small webpage. Those webpages look like this:

static int measCurrent(SOCKET htmlSock, int ContentLength, char *pArgs ) {
    char htmlbuf[MAX_RESPONSE_SIZE];
    char numberBuf[20];

    httpSendStatusLine(htmlSock, HTTP_OK, CONTENT_TYPE_HTML);
    httpSendClientStr(htmlSock, CRLF );
    httpSendClientStr(htmlSock,"<html><head>\r\n");
    httpSendClientStr(htmlSock,"<meta content=\"text/html; charset=ISO-8859-1\" http-equiv=\"content-type\"><title>TI-RTOS HTTP Lab</title></head>\r\n");
    httpSendClientStr(htmlSock,"<meta http-equiv=\"refresh\" content=\"5\" >\r\n");

    ftoa(dcBrStatus.dcBrMeasSend.current, numberBuf, 3);
    System_sprintf( htmlbuf, "<body>%s A<br>\r\n", numberBuf);
    httpSendClientStr(htmlSock, htmlbuf );
    httpSendClientStr(htmlSock,"</body></html>\r\n");
    fdClose(htmlSock);

    return 1;
}

So as it can be seen, the refresh settings is to 5 seconds. So every 5 seconds this function will be called and updates this iframe.

I never call malloc or calloc in my code. So it looks like, it is not the fault of my code. Maybe a setting in the NDK? Or do I have to destroy the webpage after calling the cgi function? Any Idea?

Best Regards

0 Garrett Ding over 7 years ago in reply to Stefan Haensel

TI__Mastermind 43296 points

Hi Stefan,

Can you try to enable the HeapTrack from .cfg to see if it can catch the memory leak function easily? The details of the HeapTrack usage is in the slide - training.ti.com/.../TIRTOS_CCSDebugging.pdf
BIOS.heapTrackEnabled = true;

Also, there is a sample webpage.c file with some CGI function calls in PDK packages\ti\transport\ndk\nimu\example\client\src that you may refer to.

Regards,
Garrett

0 Stefan Haensel over 7 years ago in reply to Garrett Ding

Intellectual 680 points

Hi Garrett,

so it looks like if have a Orphan Task Problem.

I didn't run it for 30 minutes. But after 5 minutes I see already, that in the HeapTrack -> Orphan there are coming more and more blocks.

But I don't call or exit any task. So how is this possible?

0 Stefan Haensel over 7 years ago in reply to Stefan Haensel

Intellectual 680 points

0 Garrett Ding over 7 years ago in reply to Stefan Haensel

TI__Mastermind 43296 points

Can you check if this helps - e2e.ti.com/.../238997

I will ask Frank to share your project and try to reproduce in my end.

Regards,
Garrett

0 Stefan Haensel over 7 years ago in reply to Garrett Ding

Intellectual 680 points

This would be great. Thank you!

0 Garrett Ding over 7 years ago in reply to Stefan Haensel

TI__Mastermind 43296 points

I am not able to build your project - fatal error: webSource/webpage.h: No such file or directory.

Do you have these files in separate folder and not archived in your PRU-ICSS-Profinet_Slave_01.00.02.02.zipfile? From the project build repository and include directories, I don't see anything other than default Profinet ones though.

#include "webSource/webpage.h"
#include "tmsCom/spiStructs.h"
#include "CRC/myCrc.h"

Regards,
Garrett

0 Stefan Haensel over 7 years ago in reply to Garrett Ding

Intellectual 680 points

On this path should be everything included:

0 Garrett Ding over 7 years ago in reply to Stefan Haensel

TI__Mastermind 43296 points

OK, I directly imported your project and am able to build it now.

The profinet_slave_RT_MRP_AMIC11x_arm.txt in ccsproject_args\ is not updated, and I re-created the project with the projectCreate.bat script, so these folders in the profinet_slave_RT_MRP_AMIC11x_arm were lost.

You can ignore the 'TI Thinks Resolved' message. I wanted to click 'Reply' instead.

Thanks,
Garrett

0 Garrett Ding over 7 years ago in reply to Garrett Ding

TI__Mastermind 43296 points

I can reproduce the task orphan issue here. The requested heap size is always fixed 596 bytes and appears to be from daemon child (dchild) thread. There was some discussion in the thread regarding the memory release and dchild thread. It looks like adding Task_sleep( ) or adjusting task priority may able to resolve the issue.

Below is the log from live session and ROV which shows multiple dchild threads:

Regards,

Garrett

0 Stefan Haensel over 7 years ago in reply to Garrett Ding

Intellectual 680 points

I tried adding "Task.deleteTerminatedTasks = true;" into the cfg file and nothing changes.
And in all tasks there should be some task delays. So I cannot believe, that this should be the problem.

Can you try to solve the problem?

0 Garrett Ding over 7 years ago in reply to Stefan Haensel

TI__Mastermind 43296 points

Stefan,

The issue is root caused to GCC Newlib nano linked in SYS/BIOS release - Newlib nano does not close stdin/stdout/stderr file descriptors making app vulnerable to memory leaks.

SYS/BIOS migrated to using newlib-nano starting SYS/BIOS 6.51.00 release. Newlib-nano is a runtime support C library with a very small footprint that is suited for use with MCUs.

The issue can be fixed with the followings:

• Add the following into the .cfg file

var hooks = new Task.HookSet();
hooks.deleteFxn = '&cleanupGCC';
Task.addHookSet(hooks);

• Add the following into any source file in the application

#include <stdio.h>
#include <reent.h>
#include <xdc/std.h>
#include <ti/sysbios/knl/Task.h>
#include <ti/sysbios/rts/gnu/package/internal/ReentSupport.xdc.h>

void cleanupGCC(Task_Handle task)
{
struct _reent *pStoredContext;

pStoredContext =
(struct _reent *)Task_getHookContext(task, ReentSupport_module->taskHId);

if (pStoredContext != NULL) {
/*
* Newlib nano does not close the file descriptors allocated for
* stdin/stdout/stderr in _cleanup_r() called from _reclaim_reent().
* This can cause a memory leak as the stdin/stdout/stderr file
* objects are not marked as free when the Task is deleted. Therefore,
* the now unused file objects cannot be reused, causing Newlib to
* allocate more memory for new file objects.
*/
_fclose_r(pStoredContext, pStoredContext->_stdin);
_fclose_r(pStoredContext, pStoredContext->_stdout);
_fclose_r(pStoredContext, pStoredContext->_stderr);
}
}

The fix will be included in later SYS/BIOS releases (6.55/6.70).

Regards,
Garrett

Processors

Processors forum

RTOS/AMIC110: Webpage crash