RTOS/CC2640R2F: ICall abort during my application because of osal_snv_read() call when BIOS start

Hi Louis,
Is this reproducible with simple_central running on CC2640R2 Launchpad, or have you made any HW/SW modifications?

Hi J Lindh

We have only made some SW modifications in simple_central.c where we've deleted unused functions and added the following code to communicate with our extra task ( of priority 1, so bellow GAP priority of 5 ).

      System_printf("1\tDevice with addr : %s and frame %s\n",
                      Util_convertBdAddr2Str(pEvent->deviceInfo.addr),
                      Util_convertFrame2Str(pEvent->deviceInfo.pEvtData));
        System_printf("1\trssi: %d\tdataLen: %d\n", pEvent->deviceInfo.rssi, pEvent->deviceInfo.dataLen);
        System_flush();

        extern void             *event_collect_handle;
        extern Queue_Handle     frame_queue_handle;
        gapDeviceInfoEvent_t    *device_info = ICall_malloc(sizeof(gapDeviceInfoEvent_t));
        queueRec_t              *pRec = ICall_malloc(sizeof(queueRec_t));

        memcpy(device_info, &pEvent->deviceInfo, sizeof(gapDeviceInfoEvent_t));
        pRec->pData = (uint8_t *) device_info;
        Queue_put(frame_queue_handle, &pRec->_elem);
        Event_post((*((Event_Handle *) event_collect_handle)), NEW_FRAME_EVENT);
        break;

We have not changed central.c, icall.c or anything else.

We've tried to reproduce our problem with the original project.

On the first run, everything works fine.

Then when we run our modified project and encounter our problem, and switch back to the original project again : non of these two project work.

We must unplug our board and plug it back to only get the original version working again.

It's like if our modified project was breaking something that only an unplug can fix.

We have no clue where to look at : do you have an idea of what to look for ?

Regards, Louis

Hi Louis,

Please don't use ICall_waitMatch in you application, please create a new Clock object instead ( dev.ti.com/.../index.html )

Are you freeing the memory you're allocating anywhere?

Can you check the ICall Heap when this happens? ( dev.ti.com/.../debugging-index.html )

Hello,

We are not using ICall_waitMatch, this was just a quote from the original project to show where the problem is.

At the time the problem occures, we do not free or alloc any memory as our task have a lower priority , and thus, did not start yet.

Concerning the Heap, we have no allocation failure ( heapmgrMemFail = 0 )

Hi Louis,

This is strange.... Can you check the task stacks as well? ( dev.ti.com/.../debugging-index.html )

What happens if you just let the program run (no breakpoints etc)?

Hi Louis,

Did you figure out what's wrong?

I will close this thread due to inactivity.

Hi Marie H

I did not figure out what's wrong on my advanced project so I reproduce the problem on a simple one.

Part Number: CC2640R2F

SDK: 3.10.00.15

Base Project: simple_central

On this basic project I just modified the TASK PRIORITY of SimpleCentral_createTask() on 2 instead of 1 and I added my own Application Task of priority 1 and I added a binary semaphore in central.c to see when I pass the 3 osal_snv_read.

Here is my Application Task:

#define APP_TASK_STACK_SIZE 760
#define APP_TASK_PRIORITY   1

#include <ti/sysbios/knl/Task.h>
#include <ti/sysbios/knl/Semaphore.h>
#include <ti/sysbios/BIOS.h>
#include <xdc/runtime/System.h>

extern Semaphore_Handle start_app_sem;

Task_Struct appTask;
Char    appTaskStack[APP_TASK_STACK_SIZE];

static void Application_taskFxn(UArg a0, UArg a1) {
    Semaphore_pend(start_app_sem, BIOS_WAIT_FOREVER);
    while (1);
}

void Application_createTask(void) {
    Task_Params taskParams;

    Task_Params_init(&taskParams);
    taskParams.stack = appTaskStack;
    taskParams.stackSize = APP_TASK_STACK_SIZE;
    taskParams.priority = APP_TASK_PRIORITY;

    Task_construct(&appTask, Application_taskFxn, &taskParams, NULL);
}

Here is the end of my main:

  /* Initialize ICall module */
  ICall_init();

  /* Start tasks of external images - Priority 5 */
  ICall_createRemoteTasks();

  /* Kick off profile - Priority 3 */
  GAPCentralRole_createTask();

  /* Kick off application - Priority 2 */
  SimpleCentral_createTask();

  /* My application - Priority 1 */
  Application_createTask(); //ME

  /* enable interrupts and start SYS/BIOS */
  BIOS_start();

  return 0;

And here is the osal_snv_read part in central.c where I added a semaphore:

  //ME
    Semaphore_Params  start_app_sem_params;
    Error_Block       eb;

    Semaphore_Params_init(&start_app_sem_params);
    start_app_sem_params.mode = Semaphore_Mode_BINARY;
    start_app_sem = Semaphore_create(0, &start_app_sem_params, &eb);

    if (start_app_sem == NULL) {
        while (1);
    }
  // Restore items from NV
  VOID osal_snv_read(BLE_NVID_IRK, KEYLEN, gapCentralRoleIRK);
  VOID osal_snv_read(BLE_NVID_CSRK, KEYLEN, gapCentralRoleSRK);
  VOID osal_snv_read(BLE_NVID_SIGNCOUNTER, sizeof(uint32_t),
                     &gapCentralRoleSignCounter);

  Semaphore_post(start_app_sem);
}

When I comment my Application_createTask() call in main everything works fine, semaphore post start_app_sem.
But when I don't comment it I get the ICall_abort() because of a Timeout caused by the first osal_snv_read() call, my semaphore post does not happen.

Do you have any idea of what I'm doing wrong?

Regards, Louis

This did not resolve my issue

Please take a look, I posted a new reply just above.

Hi Louis,

I'm not surprised that you can make an error with the while(1) loop...

In the simple_central task, can you try using only events and add them to the same event queue as the application is already using?

You right, the while(1) loop was the problem..
But I'm still stuck on my advanced project. I am going to try to find again this bug by remaking the project.
It will take me some time. I will let you know.

Hello

I did not understand the while(1) problem above. How a while(1) which is never executed before the snv_read() thanks to my semaphore can make the snv_read timeout?

I think it's the same problem in my advanced project when I'm using Util_enqueueMsg() in my Application Task:

extern Queue_Handle     appMsgQueue;
extern ICall_SyncHandle syncEvent;

void    ble_start_scanning(void) {
    sbbEvt_t    *start_msg = ICall_malloc(sizeof(sbbEvt_t));
    appEvtHdr_t event_hdr;

    memset(start_msg, 0, sizeof(sbbEvt_t));
    event_hdr.event = START_SCANNING_EVENT;
    start_msg->hdr = event_hdr;
    Util_enqueueMsg(appMsgQueue, syncEvent, (uint8_t *) start_msg);
}

When I comment the Util_enqueueMsg() call everything works fine snv_read() is executed and my Application Task is running.

But when Util_enqueueMsg() is not commented I get a snv_read() timeout but Util_enqueueMsg() is never executed...

Even if I'm using Event instead of Semaphore the same problem occurs.

Please, can you tell me some explanations?

Regards, Louis

Hi Louis,

To debug this you should use the ROV (see dev.ti.com/.../debugging-index.html ). Pay close attention to what tasks are active when, if anything is blocking the other tasks etc.

Did you test if this is some kind of timing issue? E.g. did you test with different optimizations?

I found a post talking about the same problem than me:

e2e.ti.com/.../739415

But it's a "TI thinks resolved" post.

My Util_enqueueMsg() is used to call SimpleCentral_startGapDiscovery() and with this call I get the abort just like the post of Joshua Hess.

So what is the proper way to go about merging the functionality from the simple_central example into a custom application?

I am using BLE5

Hi Louis,

Did you get any information from ROV?

We have  chapter in the BLE5-Stack User's Guide about Developing a Custom Application (  ). 

In general I would advise either making a separate task for your operations. If you want to use the same task as simple central.c, use the same application event queue and add new events to it.

Hello Marie.

1 - We have no extra information from the ROV. I have extracted the Data from the ROV and attached it.

As you can see the ICALL aborted, due to a timeout on osal_snv_read in central.c

2- We have carefully read the documentation you have provided and did not see anything that we could have missed

3-  We followed your hint on the timing with no luck.

4 - This bug appears to happen "randomly" and when it does, even the original project does not work anymore. At some point, the bug simply disappears and everything work back again.

5- We have changed boards, and use the original Dev Board from Sable-X-R2 to make sure it wasn't a hardware problem from us ( with the correct 5X5 chip config wich has been proven to work on other project). No luck.

6 - We have uploaded a repo with the problem to make things perfectly clear (only one line of code have been added in simple_central.c (https://github.com/Ehlll/simple_central_project/blob/c9fa724ea5cab812574a3e5722b091c2c9bee1ce/simple_central_cc2640r2lp_app/Application/simple_central.c#L821)):

The full repo :
github.com/.../simple_central_project

7- We have been now working on this problem for over a week, 8 hours a day, and we are still at the exact same point.....

currentViews.rod.zip

up

Hi Louis,

1) Are you doing anything when it fails? (Pressing any buttons?)

2) Can you add scanningStarted = TRUE; before you call GAPCentralRole_StartDiscovery? (See line 1067 in simple_central.c.)

3) Did you update the board files for the 5x5 board or is the project still configured for a launchpad?

Hi Marie,

I'm doing anything when it fails.

I added scanningStarted = TRUE; before my call GAPCentralRole_StartDiscovery but the ICall_abort is still there. You can see it on my test branch on github:
https://github.com/Ehlll/simple_central_project/tree/test
On the line:
github.com/.../simple_central.c

I did not update the 5x5 board. I just replaced the predefined symbol CC2640R2_LAUNCHXL by CC2640R2DK_5XD.

Hi Louis,

I just tried but I am not able to reproduce. I tried 5-10 times both on a LP and on 5x5 em. Each time I waited until the device finished the first scan (GAP_DEVICE_DISCOVERY_EVENT).

1) Are you sure you made no other changes? E.g. change optimization, change preprocessor symbols etc.

2) How often does this happen?

Hi Marie,

We are testing 3 different hardware boards :

- The LaunchPad (LP)

- The Sable-X-R2 Dev Board  ( DB) which is a 5X5 version of the CC26

- Our custom board (CB) which runs on the Sable-X-R2, so 5X5 as well

I downloaded the ZIP test branch from the repositorie on Latest commit 141a808 "Add scanningStarted = TRUE".

I have replaced Predefined Symbols "CC2640R2DK_5XD" by "CC2640R2_LAUNCHXL" and picked in General Connection: "Texas Instruments XDS110 USB Debug Prob [Default]".

I Have cleaned the project to make sure to have a nice clean build.

I tried to launch Debug Mode using LP on CCS8 and we have ICall_abort.

Same procedure and problem encountered with the DB and CB on both Windows and Fedora.

( Note that for DB and CB we do change the predefined symbol back to CC2640R2DK_5XD and use a Segger j-Link debug probe )

Strangely, on our first attemp, we did get the LP to work 10 times in a raw as well (just like you), then when we tried with the DB it did not work, and when we switched back to LP it wasn't working anymore ( with project clean between each try). Eversince, the LP is not working just like the CB and DB.

It Feels like we are corrupting the stack on something. 

We have found a known issue that could be related ( https://e2e.ti.com/support/wireless-connectivity/bluetooth/f/538/t/683310 ) but the fixed proposed did not work for us.

Any Idea ?

Hi Louis,

We're getting into details here...

1) Can you try using CCS 9.0.1?
2) The BLE-Stack is provided as library files so you won't break it, but I guess there could be a different file which is corrupted? You can try re-installing the SDK or use dev.ti.com/tirex to use the unaltered files.
3) I don't know if you are supposed to be able to use TI SimpleLink SDK on the Sable-X device. Can you reach out to them and ask?