RTOS/CC2642R: Computationally expensive code corrupts app msg queue

Brett Miller

Part Number: CC2642R
Other Parts Discussed in Thread: LAUNCHXL-CC26X2R1

Tool/software: TI-RTOS

Hello,

I'v having trouble debugging my application. First a few specifics:

My application is expanded upon the Project_zero Bluetooth 5 project using the CC2642R1F Rev C. chip on a LAUNCHXL-CC26X2R1 Dev kit. (Rev B.) on the 2.30 SDK.

As a note i do have access to the Rev E. CC2642R1F chip and the latest SDK (3.10)

I am using CCS 9.0.1 on Windows 10.

The specific issues is that my App Msg queue in the Project_Zero Task appears to be corrupted after a long running computation. In detail, it appears that a message (of type pzMsg_t) had been de-allocated and is now pointing to memory outside of the memory map declarations.

Here is watch in the debugger:

Whats more interesting is that if i remove this for loop from the execution loop, my AppMsg queue is not corrupted:

void ToWavenumberCubic(uint16_t* pixelData, bool isCalibration, float* extended)
{
    for (int i = 0; i < 1024; i++)
    {
        extended[i] = pixelData[i];
    }
}
//Additional Code removed for brevity

Before this function there is much more computation, but this section seems to trigger the error. (If i remove the earlier computations and still run the above code, there is no error). I don't believe the above function uses any additional stack or heap. Casting may be an issue.

I have tried wrapping the long running code in an ICall_enterCriticalSection, so that HWI/SWI interrupts are not sent to the AppMsg Queue. I also turn off Clock interrupts.

Additional Thoughts:

I do not believe this is a Heap issue as i am watching the heapmgr expressions and do not see a failed allocation.

It appears the pMsg is allocated early on in the stack before the long-running code, as it appears early on in the stack.

Should i be using a different semaphore or perhaps a new task for computationally expensive code?

Perhaps either clear the msg queue or wait for it to be cleared before running computationally expensive code.

I'm just not sure how to debug from here, however i will continue to investigate stack or dealloc issues on that Msg.

Full project attached, please PM for password. The Calibrate_handler is the function where this behavior is seen (CommandHandler.c)

MiniRaman.zip

I greatly appreciate your time and expertise!

Thank you,

Brett

over 6 years ago

0 J Lindh over 6 years ago

TI__Guru 57865 points

Hi Brett,
I'm assigning an expert on the topic. stay tuned.

0 Aslak N. over 6 years ago in reply to J Lindh

TI__Mastermind 23440 points

Hi Brett,

It looks like the Queue is corrupted as you say. Or rather the queueRec_t that util.c uses, that contained the pointer to the to pMsg.

If you find on repeat runs that the addresses are always the same you could use a Hardware Watchpoint to break when the contents of queueRec->pData is overwritten, to see why/where this happens.

You could also use HeapTrack instead of the regular heap provider, it includes more checks and things for heap corruptions, and should also break on double frees. See the cfg file and/or the debug guide in the User's Guide.

I suppose you have checked that the task heaps are not near full? Near full (as seen by the watermark bytes) could still mean that another stack frame overran the stack but just didn't use the allocated watermark bytes for anything.

It's very strange that a simple 1024 count loop should be the hair that breaks the camel's back, except of course if the float *extended pointer is not long enough :p

Best regards,
Aslak

0 Brett Miller over 6 years ago in reply to Aslak N.

Intellectual 430 points

Hi Aslak,

Thank you for the debugging tips. That specific issues was that i was never allocating data for the message and always sending Characteristic updates with bad data!. See code below:

/*
 * Handles the Software Interrupt resulting from timeout of the clock object(s)
 * used to demonstrate notifications/indications.
 */
static void user_generic_clockSwiHandler(uint16_t svcUuid, uint16_t paramID)
{
    uint8_t notiData[20];
    uint16_t notiLen = sizeof notiData;
    uint8_t someCounter = 0;
    // Get loopback data if char is writable, otherwise fill with junk
    switch (svcUuid)
    {
    case NOTIFICATION_SERV_UUID:
        switch (paramID)
        {
        case N_ALERT_LEVEL_ID:
            // Characteristic is not writable, so send something, max 20 bytes (max default ATT MTU size minus header)
            notiLen = MIN(notiLen, N_ALERT_LEVEL_LEN);
            memset(notiData, someCounter++, notiLen);
            break;
        case N_BATTERY_LEVEL_ID:
            // Characteristic is not writable, so send something, max 20 bytes (max default ATT MTU size minus header)
            notiLen = MIN(notiLen, N_BATTERY_LEVEL_LEN);
            memset(notiData, (uint8_t) status.batteryLevel, notiLen);
            break;
        case N_INTERLOCKS_ID:
            // Characteristic is not writable, so send something, max 20 bytes (max default ATT MTU size minus header)
            notiLen = MIN(notiLen, N_INTERLOCKS_LEN);
            memcpy(notiData, &status.AllInterlocksStatus, 1);
            //memcpy(notiData + 2, &status.Interlock3, 1);
            //memcpy(notiData + 3, &status.safe, 1);
            //memcpy(notiData, &someVal, notiLen);
            break;
        case N_ISARMED_ID:
            // Characteristic is writable, so echo back contents.
            Notification_GetParameter(paramID, &notiLen, notiData);
            //memset(notiData, status.Armed , notiLen);
            break;
        }
        break;
    case SPECTRUM_DATA_SERV_UUID:
        switch (paramID)
        {
        case SD_TX_READ_ID:
            // Characteristic is writable, so echo back contents.
            SpectrumData_GetParameter(paramID, &notiLen, notiData);
            break;
        }
        break;

    case USER_100MS_PERIODIC_UUID:
        // Send message to application that it should update the value of the characteristic from Task context.
        ProjectZero_enqueueMsg(APP_MSG_100MS_UPDATE, notiData);
        break;

    case USER_1S_PERIODIC_UUID:
        // Send message to application that it should update the value of the characteristic from Task context.
        ProjectZero_enqueueMsg(APP_MSG_1S_UPDATE, notiData);
        break;
    }
    // Send message to application that it should update the value of the characteristic from Task context.
    pzCharacteristicData_t *pValChange = ICall_malloc(
            sizeof(pzCharacteristicData_t) + notiLen);
    if (pValChange != NULL)
    {
        pValChange->svcUUID = svcUuid;
        pValChange->paramID = paramID;
        memcpy(pValChange->data, notiData, notiLen);
        pValChange->dataLen = notiLen;

        if (ProjectZero_enqueueMsg(PZ_UPDATE_CHARVAL_EVT, pValChange) != SUCCESS)
        {
            ICall_freeMsg(pValChange);
        }
    }
}

I've since rewritten the clock handler:

/*
 * Handles the Software Interrupt resulting from timeout of the clock object(s)
 * used to demonstrate notifications/indications.
 */
static void user_generic_clockSwiHandler(uint16_t svcUuid, uint16_t paramID)
{
    if (!status.Busy && status.ClocksRunning)
    {
        uint8_t notiData[20];
        uint16_t notiLen = 20;
        uint8_t someCounter = 0;
        app_msg_types_t msgType = PZ_UPDATE_CHARVAL_EVT;
        // Get loopback data if char is writable, otherwise fill with junk
        switch (svcUuid)
        {
            case NOTIFICATION_SERV_UUID:
                switch (paramID)
                {
                    case N_ALERT_LEVEL_ID:
                        // Characteristic is not writable, so send something, max 20 bytes (max default ATT MTU size minus header)
                        notiLen = MIN(notiLen, N_ALERT_LEVEL_LEN);
                        memset(notiData, someCounter++, notiLen);
                        break;
                    case N_BATTERY_LEVEL_ID:
                        // Characteristic is not writable, so send something, max 20 bytes (max default ATT MTU size minus header)
                        notiLen = MIN(notiLen, N_BATTERY_LEVEL_LEN);
                        memset(notiData, (uint8_t) status.batteryLevel, notiLen);
                        break;
                    case N_INTERLOCKS_ID:
                        // Characteristic is not writable, so send something, max 20 bytes (max default ATT MTU size minus header)
                        notiLen = MIN(notiLen, N_INTERLOCKS_LEN);
                        memcpy(notiData, &status.AllInterlocksStatus, 1);
                        break;
                    case N_ISARMED_ID:
                        // Characteristic is writable, so echo back contents.
                        Notification_GetParameter(paramID, &notiLen, notiData);
                        break;
                }
                break;
            case SPECTRUM_DATA_SERV_UUID:
                switch (paramID)
                {
                    case SD_TX_READ_ID:
                        // Characteristic is writable, so echo back contents.
                        SpectrumData_GetParameter(paramID, &notiLen, notiData);
                        break;
                }
                break;

            case USER_100MS_PERIODIC_UUID:
                memset(notiData, someCounter++, notiLen);
                msgType = APP_MSG_100MS_UPDATE;
                break;

            case USER_1S_PERIODIC_UUID:
                memset(notiData, someCounter++, notiLen);
                msgType = APP_MSG_1S_UPDATE;
                break;
        }

        pzCharacteristicData_t *pValChange;
        pzEventData_t *eventData;
        switch (msgType)
        {
            case PZ_UPDATE_CHARVAL_EVT:
                // Send message to application that it should update the value of the characteristic from Task context.
                pValChange = ICall_malloc(sizeof(pzCharacteristicData_t) + notiLen);
                if (pValChange != NULL)
                {
                    pValChange->svcUUID = svcUuid;
                    pValChange->paramID = paramID;
                    memcpy(pValChange->data, notiData, notiLen);
                    pValChange->dataLen = notiLen;

                    if (ProjectZero_enqueueMsg(msgType, pValChange) != SUCCESS)
                    {
                        ICall_freeMsg(pValChange);
                    }
                }
                break;
            case APP_MSG_100MS_UPDATE:
            case APP_MSG_1S_UPDATE:
                eventData = ICall_malloc(sizeof(pzEventData_t) + notiLen);
                if (eventData != NULL)
                {
                    //eventData->pBuf = ICall_malloc(notiLen);
                    eventData->svcUUID = msgType;
                    memcpy(eventData->data, notiData, notiLen);
                    if (ProjectZero_enqueueMsg(msgType, eventData) != SUCCESS)
                    {
                        ICall_freeMsg(eventData);
                    }
                }
                //ProjectZero_enqueueMsg(msgType, notiData);
                break;
        }
    }
}

Event definitions:

//Generic Event. currently used for timers.
typedef struct
{
    uint16_t svcUUID;
    uint8_t data[];
} pzEventData_t;

// Struct for message about sending/requesting passcode from peer.
typedef struct
{
    uint16_t connHandle;
    uint8_t uiInputs;
    uint8_t uiOutputs;
    uint32_t numComparison;
} pzPasscodeReq_t;

This works great with the Busy and ClocksRunning flags. However when those flags are removed and events are allowed to process during my long-running/heap intensive operations i receive ICall_malloc exceptions probably due to the fragmentation of the heap. I am allocating large sections of the heap for arrays, ~8000 bytes. Heap fragmentation is understandable and its probably prudent to not address non priority events during intensive/large heap operations.

I think the bottom line here is that i am taxing my Heap intensively. Perhaps for smaller variables the stack is better to use?

Thank you for your time and explaining the additional tools.

Regards,

Brett

0 Aslak N. over 6 years ago in reply to Brett Miller

TI__Mastermind 23440 points

Hi Brett,

It's not impossible that you are taxing the heap and indeed getting fragmentation issues. If you switch to the HeapMem heap manager (see the cfg file) you can use the ROV to look at the FreeList and see if it's fragmented. With HeapMem you should also be able to make a secondary heap that you can use through the Memory_ APIs in the kernel.

See the User's Guide and the runtime APIs of the kernel: dev.ti.com/.../node

Best regards,
Aslak

Bluetooth®︎

Bluetooth forum

RTOS/CC2642R: Computationally expensive code corrupts app msg queue