CC2652R: Bluetooth Stack does not work when memory allocations are made before ICall_createRemoteTasks

Hello Michal,

Thanks for reaching out. Please help me with the following you help you further:

1. Could you please specify what error are you getting from the radio or what functionality stops working?
2. Could you please use the Runtime Object Viewer to see if there is heap corruption problem?
3. What example are you using from the SDK?

BR,

David.

Thank you for your response. Here are answers to your additional questions:

1. To make myself more precise - when a memory allocation is done prior to initializing the stack (but after ICall_init - see code from the intro post) and this allocated memory is used (so it no longer has the value 0x00) and freed and therefore the same memory area is then used by the TI BLE5 Stack it causes Radio MCU to halt all operations as soon as Bluetooth connection is tried to be made (try to connect to the devkit using a phone)
2. Sure thing! Here is the ROV capture - let me know what you want to see. As you can see here we are in the HWI error handling - Radio MCU no longer works
   
   
   This does not seem like a memory corruption on my end - no access is done outside allocated memory. I am not sure If I can easily check memory corruption without making any further modifications to the SysCfg which could alter the testing environment
3. I am using the DMM sample named "DMM 15.4 Collector + BLE Remote Display" - dmm_154collector_remote_display_app_2_4g_CC26X2R1_LAUNCHXL_tirtos7_ticlang

Just to make sure my message gets across:

• If you memset memory area to any non-zero value then Radio MCU stops working completly and enters HWI handler
  
• If you memset memory area to zero value the Radio MCU works without any issues

This leads me to suspect that BLE5 Stack internally is allocating some memory for itself and does not initialize it to an expected value but assumes that whole memory area returned by ICall_malloc is initialized with 0x00 values (which may or may not be true). In my case Bluetooth Stack is returned a pointer to a memory area that has value 0xAB and not 0x00. The 6 lines of codes in the original post is the only modification I made to the sample to trigger that issue 

Hello Michal,

Thanks for the extra info. May I ask why do you need to allocate that much amount of memory? Could you please share with me what is the address of the data pointer you get returned? It might be useful to use this guide for Debugging Memory Problems. In addition, when using the auto heap size feature you have around 8839 bytes, so you might be overwriting some parameters used by the stack (why I ask the pointer address).

BR,

David.

The allocation of 8k of memory in this sample code is just to simulate my real project - a lot of memory allocations are made prior to initializing Bluetooth Stack. So in the code above this collectively simulates a RAM area that has been used before but now is free and can be used by (probably) Bluetooth Stack.

Address of the returned pointer is: 0x200072C8

I am very confused about the 8839 bytes limit you mentioned. Where is that limitation coming from? Out of curiosity working on the same sample I decided to try and allocate even more memory (10k bytes) and it had absolutely no issue with that - no NULL pointer was returned. Even more I decided to print heap stats and it looks like "a bit more" than 8k is available. In the screen below you can see a capture from debugging - both heap stats and allocated address are visible:

Additionally I had a look a the generated .map file and here are my heapStart and heapEnd symobls:

20013600 heapEnd
200071d8 heapStart

When calculating memory area for heap (following calculation method from the link you provided) I get the same value as returned by the get heap stats functionality: 50216

Hello Michal,

Could you please share the results of the ROV related to heap consumption? Debugging Common Heap Issues

Could you please try splitting the allocation into multiple mallocs?

BR,

David.

I am sorry but I feel like we are getting nowhere. To progress with this issue let me reiterate how to reproduce it – it is very simple to trigger it so I hope you can do it on your end and gather as much information and analytic data as you need.

How to cause the TI BLE5 stack to misbehave:

1. Import DMM enabled sample from TI SDK into workspace. Use this sample project: dmm_154collector_remote_display_app_2_4g_CC26X2R1_LAUNCHXL_tirtos7_ticlang
2. Open “main.c” file – located in “application” directory
3. Navigate to line 334 or find “ICall_init();” function call
4. Insert 5 lines of following code so that your sample main.c file looks like this:
   

   uint8_t *data = ICall_malloc(8192);
   if(NULL != data) {
       memset(data, 0xAB, 8192);
       ICall_free(data);
   }

5. Build and launch sample project on a supported development board
6. Using your favorite Bluetooth debugging app connect to the development board
7. Observe the board enter HWI error handler and stopping any operations

I’ve done my best to isolate the issue into your example project and find the exact moment it causes your TI BLE5 stack to misbehave. Please try to recreate it and see for yourself.

Memory corruption theory:

It is completely incorrect what you said about the auto heap size feature being limited to only 8839 bytes of heap. I have no idea where you got that information from (since you did not answer my question) but it might be misleading to anyone reading this thread. If that was true there would be no memory left after TI BLE5 Stack starts which is clearly not the case since TI15.4 Stack can be started after BLE Stack and gets all required memory without any issue and there is still plenty of memory left for a user application.

Once again - please provide me with the source of 8kB limit information

My theory about Bluetooth memory corruption:

However, your questions lead me to finding out why 8kB of memory allocation before starting ICall remote tasks causes TI BLE5 stack to malfunction. If one decides to analyse memory consumption before and after calling “ICall_createRemoteTasks” they would see that TI BLE5 Stack allocates around 8kB of heap.

Since I:

• allocated 8kB
• modified memory in that area
• freed that area

it means that TI BLE5 stack gets returned a memory area that is no longer filled with 0x00 values but any value that was previously set to that memory. If no allocation takes place prior to ICall remote tasks create, TI BLE5 stack gets memory area with 0x00 values as it is freshly after a reboot. This seems like a bug in TI BLE5 Stack implementation – it allocates a memory and assumes (incorrectly) that the returned area is filled with 0x00 values. It is not. Sure, in TI example projects there is no allocation prior to creating ICall remote tasks but nowhere in TI's documentation (at least no that I could find) is it stated that it is forbidden to allocate and free memory before creating ICall remote tasks.

If one tries to allocate an area smaller that 8kB and fills it with random data then no Bluetooth corruption occurs as some part of the area is still filled with 0x00 and this is what TI BLE5 Stack expects.

How to proceed:

• Please try to recreate that issue. I made sure that it is easy to reproduce and isolated to your example project.
• I can try to split my allocations into 8 chunks of 1kB and results will still be the same. If TI BLE5 Stack gets returned a dynamically allocated memory area that is NOT filled with 0x00 values, it will break.
• I am sure the same behavior could be triggered with a change to linker script that will initialize heap area in RAM with a random data instead of 0x00 values - I am not a linker expert I will not attempt it

Hello Michal,

Thanks for the detailed steps and information and sorry for the confusion, I got my values completely wrong and I will make sure to clarify this based on the example you are using. Thanks for pointing this out.

I am working on reproducing the issue now. So far I have not been able to reproduce it on a project that is only using BLE (simple_peripheral for instance) so this makes me believe there might be an issue with the DMM or another protocol involved. I will review this with the team as well in case I am missing something. I will report back today with my findings.

BR,

David.

Hello Michal,

After further inspection, we see that the issue comes from the fact that the memory space that has been previously allocated (for instance by applying your ICall_malloc(8196) and  memset(data, 0xAB, 8096) case) is not set back to 0 (which is expected by some of the stack functions to work correctly - for instance 0x00021504 (llReplaceRxBuffers + 0xC)  which is where the exception was occurring and a different value was being used (based on 0xABAB...)

As a work around I would suggest the following:

Inside "icall.c": set the data to 0x00 after allocating.                                                                                                         

void *ICall_malloc(uint_least16_t size)
{
  void* data = ICall_heapMalloc(size);
  memset(data, 0x00, size);
  return (data);
}

I will review this with the stack team to better understand what are the potential consequences of this modification.

BR,

David.

Perfect! Thank you for confirming my observations. This fix (although will work just fine) is not acceptable due to two reasons (possibly more):

• memsetting memory on every malloc is going to be very heavy on MCU
• this requires SDK files modifications which is a complete no-go - this will prevent me from easily upgrading to latest SDK in the future

I am looking forward for this issue to be fixed in the upcoming release of the SDK. This seems like a serious bug and I am more than happy to upgrade my project to the latest release

Hello Michal,

I understand. I think to be on the safe side here regarding the use of the ICall functions which should only be used inside ICall registered tasks, we should use ICall_malloc() after BIOS_start(), which starts the RTOS scheduler in TI-RTOS. I am reviewing this with the team to be 100% certain and give you a proper justification here, but from what I have tested, this works fine without having to memset the values to zero every time.

BR,

David.

There should be no issue with calling ICall_malloc before BIOS_start() because there is nothing in the implementation of ICall_malloc (internally HEAPMGR_MALLOC) that requires BIOS to be executing. This is a very simple implementation of the heap and it was easy to inspect – nothing in those functions requires BIOS running. Regarding calling ICall functions from ICall registered tasks – does that apply also to ICall_malloc? I don’t think so given the implementation of heap manager underneath and no abort functions being called. Not all tasks that wish to allocate some memory must be an ICall registered tasks.

The real issue is TI BLE5 Stack implementation assuming that ICall_malloc returns pointer to a memory area initialized to 0x00. As an embedded developer I would never assume that dynamically allocated memory is going to be initialized with any value. If I want my memory to have specific value, I set that value explicitly.

Thank you for all suggestions how to work around the issue but I will pass on them – I have a better solution for my enterprise project. As you have acknowledged there is an issue with TI BLE5 Stack implementation and  all I expect from TI is to make sure that this bug is fixed in the upcoming SDK release.

Hello Michal,

I have raised this to the R&D team and they will evaluate the best approach here. I will update you with the answer as soon as I have one.

Thank you for your patience.

BR,

David.

Additional information.

The same issue can be triggered by just enabling this during compilation: HEAPMGR_PROFILER.

It enables this piece of code in void HEAPMGR_INIT(void) function located in rtos_heaposal.h file:

#ifdef HEAPMGR_PROFILER
  (void)HEAPMGR_MEMSET(HEAPMGR_HEAP, HEAPMGR_INIT_X, (HEAPMGR_SIZE/HDRSZ)*HDRSZ);
#endif // HEAPMGR_PROFILER

which fills RAM with predefined data. This of course leads to the same issue.

Given there are two different ways the bug can be triggered it only makes sense to fix the root cause - TI BLE5 stack bug - rather than trying to avoid triggering it.

Hello Michal,

Thank you for the additional information. I have shared this with the team as well and I will report back as soon as possible.

BR,

David.

Hi David, any conclusion on this topic ?

Regards,
Andrzej

Hello Andrzej,

Apologies for the time this is taking. The team is still looking into the problem, I will ask for updates.

BR,

David.

When can I expect fix for that ?

Hello Andrzej,

The prioritization of the issue is being internally evaluated and we are doing our best to address it as soon as possible. We are communicating with your TI reference contact regarding time expectations. I apologies for the delay and inconvenience this may cause.

BR,

David.