RTOS/CC3220S-LAUNCHXL: Each MQTT connection takes 112 bytes from Heap

Team,

I have tried the same with Non secure MQTT connection and it does not seem to have this problem, i.e no memory leak at all.
I have also noticed that changing the certificate path in the secure mode to "" i.e char *Mqtt_Client_secure_files[CLIENT_NUM_SECURE_FILES] = {""} also reduced the memory leaked to 16, instead of 112. However I was unable to debug further.

Would really appreciate if someone can look into this at the earliest.

Thanks
Zac

Hi Zac,

I've replicated the memory leak that you've observed. As you pointed out, in the mqtt example every time you go through the loop in an attempt to connect to the target MQTT broker some memory from the heap is lost.

However, I have not been able to pin down the cause of the memory leak yet. I'll do some more investigation in the upcoming couple days and alert the developers of the MQTT library of the issue. I'll be sure to update you through this thread once I have more concrete findings.

Regards,
Michael

Hi Michael,

It seems to be related to the SSL layer as the issue disappears completely in non SSL mode.
Few of our devices at site is affected due to this issue. They run out of memory when the internet connection drops.
It would be really helpful if you can expedite the analysis and give a solution at the earliest. Please let me know if you need further details.
Appreciate your efforts.

Regards,
Zac

Hi Zac,

Thank you for providing the clear information that allowed me to reproduce this problem. I have since filed a bug report with the team that developed the MQTT library, and will report back when they have examined the issue and resolved it or find a workaround. I will also let you know if they want more details on your use case and how you're running into the bug.

Regards,
Michael

Thanks Michael.

Dear Michael,

Any update on this issue. Really sorry to push this, I am really stuck without a fix :(

Thanks
Zac

Team,

I would really appreciate if someone can help me to solve this issue.

Regards,
Zac

Dear Michael,
Any update on the issue ?

Thanks
Zac

Hi Zac,

As I mentioned last week, I have told the team responsible for the mqtt library that there is a memory leak in their library and how to reproduce the bug we observed. At this point, it's a matter of waiting for them to debug and provide a fix. I will let you know once the situation changes.

Regards,
Michael

Dear Michael,

We are stuck with this issue and couldn't roll out the products last week. Our products are installed at places where users don't have direct access to reset the system upon any failure, hence we are at risk if we deliver with this issue unresolved.
Is the MQTT team part of TI itself ? Is there any way by which you can request them to expedite ?
Appreciate all your efforts.

Regards,
Zac

Hi Zac,

While the MQTT memory leak is being fixed, there are a couple workarounds you can implement in your program to prevent your products from crashing due to you running out of heap memory.

The first would be to prevent your device from attempting to do an MQTT connect if there is no internet connectivity. You can do this by pinging a well-known host, and seeing if it responds. If there is no response, then you will know that you cannot connect to your MQTT server either. You would then ping every few seconds to a minute to see when internet connectivity has been restored, at which point you can safely connect.

The second option would be to implement the watchdog timer, which will automatically reset your device if it enters a state where it gets stuck to the point that it can no longer periodically give the watchdog timer a heartbeat. Even without this MQTT issue, having this watchdog reset might be a good idea if you cannot manually reset devices, since it gives you a way for devices to recover themselves if they get into a crash for whatever reason.

I hope that gives you a few ideas for possible ways to mitigate the issue while the library is being fixed.

Regards,
Michael

Hi Michael,

Thanks for your suggestions. The internet connection connection at our sites are inconsistent and the reconnection happens very often.

The first suggestion seems to help to a very good extend, however the only drawback is actual connection / reconnection take more time. During the positive flow, the system is expected to come online in 2 seconds.

Regarding the second option, we have already implemented a watchdog timer which helps to recover from a crash. However the problem with that approach is that the time synchronization will be lost and the schedules will not fire unless another NTP connection is successful.

Appreciate you help.

Regards,
Zac

Hi Michael,

Any progress on MQTT fix ?

Thanks,
Zac

Hi Michael,
It's been a long time since this issue was reported and it's very unfortunate that we haven't found a solution yet.
I would really appreciate if you / your team can look into this at priority, as this may have affected most of the devices using MQTT.

Regards,
Zac

Hi Zac,

I spoke with the software team responsible for the MQTT library the other day about this issue. It seems like any fix for the MQTT library will not be in the next Q2 SDK release due to time constraints. You will likely have to try debugging the library and figuring out where the memory leak occurs in order to get this fixed in your system in the short term.
That being said, there is also the MSP432E4 SimpleLink device which uses the exact same MQTT library. I'll go and see if this bug is present on that platform. If it does exhibit the same memory leak, then I can also get their SDK team involved in finding a fix. If not, it might provide some debug insight as to where the problem might be occurring.

Regards,
Michael

Hi Michael,
Any progress on this issue ? We have migrated from another micro-controller to Texas modules considering the brand and quality. However this is a real let down considering that this is an issue and should have affected other users as well. Needless to say that our product usage experience is also affected due to this issue. We would really appreciate if you could let the respective owners know about the impact and provide a quick solution.

Regards,
Zac

Hi Zac,

I tested the same mqttclient example on the MSP432E4 device and it exhibits the same problems, which makes sense since both devices are part of the SimpleLink platform and share the same MQTT library. I informed the MSP432 software team of the issue, and they have told me they will treat this as a higher priority given that it affects multiple devices. They hope to look into it this week and determine the root cause of the memory leak, and provide a fix committed in the Q2 SDK release.

Regards,
Michael

Thanks Micheal, appreciate if you can keep me posted.

Regards,
Zac

Hi Zac, Micheal:

We have identified a leak in SlNetSock_secAttribDelete() in C:/ti/simplelink_cc32xx_sdk_1_60_00_04/source/ti/net/slnetsock.c:

    /* Free all SecAttrib list nodes                                         */
    while (NULL != nextSecAttrib)
    {
        nextSecAttrib = tempSecAttrib->next;
        free((void *)tempSecAttrib);
        tempSecAttrib = nextSecAttrib;
    }

    free(secAttrib);
    return SLNETERR_RET_CODE_OK;
}

The line in red needs to be added. Could you try to add it, rebuild the SDK and your application, and see if this fixes the issue for you? On my end, I can set a breakpoint on the UART_PRINT call at the end of while loop in the mainThread() function in mqtt_client_app.c in the mqtt_client example and see that the amount of free space on the heap is constant at each breakpoint hit, after the fix. I have set the example to connect to a bogus URL, so that it keeps trying to reconnect.

Best regards,

Vincent

Hi Vincent,

Could you please help me with the step on rebuilding the SDK in CCS7 ? I am unable to check out C:/ti/simplelink_cc32xx_sdk_1_60_00_04/source/ti/net/ as a project in CCS.

Regards,
Zac

Hi Zac,

To rebuild the SlNetSock libraries:

1. Update your installation paths in C:/ti/simplelink_cc32xx_sdk_1_60_00_04/imports.mak as shown in section 6.2.1 in the Quick Start Guide (C:/ti/simplelink_cc32xx_sdk_1_60_00_04/docs/simplelink_mcu_sdk/Quick_Start_Guide.htm)

2. Open a Windows command prompt

3. Enter these commands at the prompt:

cd C:/ti/simplelink_cc32xx_sdk_1_60_00_04/source/ti/net

c:\ti\xdctools_3_50_04_43_core\gmake.exe

This should rebuild your libraries. You can verify the timestamps in the files located at C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net\lib\ccs\m4.

Best regards,

Vincent

Thanks a lot Vincent.

I got the below error while running gmake.exe, however I could see that some files in C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net\lib\ccs\m4 is updated.

C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net>c:\ti\xdctools_3_50_04_43_core\gmake.exe
building packages ...
making all: Wed Apr 25 00:09:32 IST 2018 ...
======== .interfaces [.] ========
making package.mak (because of package.bld) ...
iar.targets.arm.M4: warning: exec of 'c:/Program Files (x86)/IAR Systems/Embedded Workbench 8.0/arm/bin/iccarm -v' failed:java.io.IOException: Cannot run program "c:/Program" (in directory "."): CreateProcess error=2, The system cannot find the file specified; check tools installation
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
generating interfaces for package ti.net (because package/package.xdc.inc is older than ) ...
======== .interfaces [./sntp] ========
making package.mak (because of .xdcenv.mak) ...
iar.targets.arm.M4: warning: exec of 'c:/Program Files (x86)/IAR Systems/Embedded Workbench 8.0/arm/bin/iccarm -v' failed:java.io.IOException: Cannot run program "c:/Program" (in directory "."): CreateProcess error=2, The system cannot find the file specified; check tools installation
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
iar.targets.arm.M4: warning: version match failed on ''
generating interfaces for package ti.net.sntp (because package/package.xdc.inc is older than ) ...
======== .interfaces [./sntp2] ========
making package.mak (because of package.bld) ...
js: "C:/ti/xdctools_3_50_04_43_core/packages/xdc/bld/PackageContents.xs", line 546: xdc.services.global.XDCException: xdc.bld.INCORRECT_PACKAGE_NAME: Package name 'ti.net.sntp' does not match directory name C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net\sntp2
gmake[1]: *** Deleting file `package.mak'
making package.mak (because of package.bld) ...
js: "C:/ti/xdctools_3_50_04_43_core/packages/xdc/bld/PackageContents.xs", line 546: xdc.services.global.XDCException: xdc.bld.INCORRECT_PACKAGE_NAME: Package name 'ti.net.sntp' does not match directory name C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net\sntp2
gmake[1]: *** Deleting file `package.mak'
gmake[1]: *** No rule to make target `package.mak', needed by `.interfaces'.  Stop.
xdctools_3_50_04_43_core\gmake.exe: *** [sntp2,.interfaces] Error 2
gmake: *** [all] Error 2

Regards,

Zac

Hi Vincent,

Somehow my build is failing and I am unable to apply the patch. Appreciate your support.

Regards,
Zac

Hi Zac,

Are you sure you have IAR installed under c:/Program Files (x86)/IAR Systems/Embedded Workbench 8.0/arm/bin/iccarm? Alternatively, if you are using the TI compiler and don't care about the IAR compiler, you could just comment out this line in imports.mak:

#IAR_ARMCOMPILER ?= c:/Program Files (x86)/IAR Systems/Embedded Workbench 8.0/arm

Best regards,
Vincent

Also, if rebuilding the library is problematic, yet another alternative is to try including the modified slnetsock.c file directly into your application project. I think your project is likely to be set up with the right compiler include paths to rebuild it.

Best regard,
Vincent

Hi Vincent,

I have removed #IAR_ARMCOMPILER and rebuild, however there seems to be an error with SNTP package.

C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net>c:\ti\xdctools_3_50_04_43_core\gmake.exe
building packages ...
making all: Wed Apr 25 09:16:05 IST 2018 ...
======== .interfaces [.] ========
making package.mak (because of .xdcenv.mak) ...
generating interfaces for package ti.net (because package/package.xdc.inc is older than package/build.cfg) ...
======== .interfaces [./sntp] ========
making package.mak (because of .xdcenv.mak) ...
generating interfaces for package ti.net.sntp (because package/package.xdc.inc is older than package/build.cfg) ...
======== .interfaces [./sntp2] ========
making package.mak (because of package.bld) ...
js: "C:/ti/xdctools_3_50_04_43_core/packages/xdc/bld/PackageContents.xs", line 546: xdc.services.global.XDCException: xdc.bld.INCORRECT_PACKAGE_NAME: Package name 'ti.net.sntp' does not match directory name C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net\sntp2
gmake[1]: *** Deleting file `package.mak'
making package.mak (because of package.bld) ...
js: "C:/ti/xdctools_3_50_04_43_core/packages/xdc/bld/PackageContents.xs", line 546: xdc.services.global.XDCException: xdc.bld.INCORRECT_PACKAGE_NAME: Package name 'ti.net.sntp' does not match directory name C:\ti\simplelink_cc32xx_sdk_1_60_00_04\source\ti\net\sntp2
gmake[1]: *** Deleting file `package.mak'
gmake[1]: *** No rule to make target `package.mak', needed by `.interfaces'.  Stop.
xdctools_3_50_04_43_core\gmake.exe: *** [sntp2,.interfaces] Error 2
gmake: *** [all] Error 2

Regards,

Zac

Hi Vincent,

I temporarily removed sntp2 library from the source and the sdk rebuild seems to be successful and after that I have put the sntp2 back in place. However upon rebuilding my application the memory leak still exists.

The steps I have tried are,
1) First I will successfully connect to a valid MQTT server
2) Turn off just the internet, the WiFi connection is still present
3) Now my MQTT disconnects and retry connection
4) On every reconnection 112 bytes are lost.

Are you able to replicate the same and resolve it with the fix? Appreciate your support.

Regards,
Zac

After rebuilding the SDK, I am still experiencing a memory leak of 112 Bytes for every reconnection.

Regards,
Zac

Hi Zac,

It is strange you are having trouble with sntp after the rebuild. Could you reinstall and use a fresh copy of the SDK and just include the modified slnetsock.c into your application project as I mentioned in my previous post? I want to make sure you are indeed running the modified code.

Also, it'd help if you make the same change to the mqtt_client example bundled with the SDK and confirm if you see any memory leak when the example tries to reconnect when an invalid URL is set. You can set the URL the example connects to by changing this line in mqtt_client_app:

#define SERVER_ADDRESS           "m2m.eclipse.org"

The other thing I could think of is that I did my validation on the latest CC3220 SDK 2.10. I'll try to install the older version 1.60 and see if I can reproduce your issue. Maybe there are some fixes that were introduced between the two releases.

Best regards,

Vincent

I have tried the same experiment in the mqtt_client example on SDK 1.60, and see a leak of 88 bytes on every reconnection. On SDK 2.10, I only saw a leak of 16 bytes (prior to the change in SlNetSock_secAttribDelete()) - hence there were additional fixes that were introduced between 1.60 and 2.10. Would you be able to upgrade to SDK 2.10 for your project, just to ensure you benefit from all the bug fixes?

Best regards,

Vincent

Hi Vincent,

The SNTP issue while rebuild was due to a mistake I made. The build is now successful however the memory leak still exists. Please note that there is memory leak only in case of SSL MQTT connection and not for non secure connection. I hope you have tried SSL MQTT configuration in your experiments.

Myself and Michael got a memory leak of 112 bytes in our experiments (against the 88 bytes which you have mentioned).

Regards,
Zac

Hi Vincent,

I am still working on porting my application to SDK 2.10. It seems like too many thing in the MQTT library have changed from 1.60 to 2.10. I was also not able to run the example successfully. My build is successful now, however there seems to be some issues with the MQTT connection (connection fails with -2006 or it just hangs).

Meanwhile can Michael test this and confirm please ?

Regards,
Zac

Hi Vincent,

Finally I am able to run the MQTT example on SDK 2.10 and saw that the memory leak was 16 bytes (before the fix). And after the fix, there is no memory leak. So I think the issue is resolved. I am yet to complete our application porting to 2.10.
I really appreciate your timely support in resolving this. Thanks a lot.

Regards,
Zac