This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2652R7: ZStack bug leading to race condition on commissioning with no networks

Part Number: CC2652R7
Other Parts Discussed in Thread: Z-STACK

Tool/software:

Hello,

We have encountered a race condition which results in the application task hanging during commissioning.

After startup and initialization:

zstack_bdbStartCommissioningReq_t zstack_bdbStartCommissioningReq;
zstack_bdbStartCommissioningReq.commissioning_mode = BDB_COMMISSIONING_MODE_IDDLE;
Zstackapi_bdbStartCommissioningReq(task_id, &zstack_bdbStartCommissioningReq);

The main APP loop then receives the following events:

1. zstackmsg_CmdIDs_BDB_NOTIFICATION
   - mode: BDB_COMMISSIONING_INITIALIZATION
   - status: NO_NETWORK
   - remainModes: 0

APP calls bdb_StartCommissioning(BDB_COMMISSIONING_MODE_NWK_STEERING);

2. zstackmsg_CmdIDs_BDB_NOTIFICATION
> - mode: NWK_STEERING
> - status: IN_PROGRESS
> - remainModes: 2

APP no action

3. zstackmsg_CmdIDs_DEV_STATE_CHANGE_IND
   - state: zstack_DevState_NWK_DISC

APP no action

4. zstackmsg_CmdIDs_BDB_FILTER_NWK_DESCRIPTOR_IND

APP calls Zstackapi_bdbFilterNwkDescComplete(app_task_id)

5. zstackmsg_CmdIDs_BDB_FILTER_NWK_DESCRIPTOR_IND

APP calls Zstackapi_bdbFilterNwkDescComplete(app_task_id)

*** App task hangs

Here's why:

The call to Zstackapi_bdbFilterNwkDescComplete in event 4 does the following:
[APP] Zstackapi_bdbFilterNwkDescComplete
[APP]   => sendReqDefaultRsp
[APP]      => OsalPort_msgSend
[APP]      => while(!gotRsp)
[APP]         {
[APP]             // Wait for the response message
[APP]             OsalPort_blockOnEvent(Task_self());
[APP]             pCmdStatus = (zstackmsg_genericReq_t*)OsalPort_msgFindDequeue(appServiceTaskId, pMsg->hdr.event);
[APP]             if(pCmdStatus)
[APP]             {
[APP]                 gotRsp = true;
[APP]             }
[APP]         }

The zstack task handles the incoming message like this:
processBdbFilterNwkDescCompleteReq
[ZSTACK] => bdb_tryNwkAssoc(); joinState = BDB_JOIN_STATE_NWK_DISC
[ZSTACK]    => bdb_SendMsg(task: 7, state: 3, status: fail)
[ZSTACK] => sent reponse to APP task

The BDB task takes this message and effectively does this:
bdb_event_loop
[BDB] => OsalPort_msgReceive [BDB_COMMISSIONING_STATE_JOINING]
[BDB]    => bdb_ProcessOSALMsg(msg)
[BDB] 	  => bdb_nwkDiscoveryAttempt(FALSE);
[BDB]          => If secondary channel not tried yet - try
[BDB]          => else
[BDB]             => bdb_reportCommissioningState(BDB_COMMISSIONING_STATE_JOINING, FALSE);
[BDB]                 => ...
[BDB]                 => bdb_setFN();
[BDB]                 => NLME_ResetRequest();
[BDB]                 => ZMacReset
[BDB]                     => MAP_MAC_MlmeResetReq
[BDB]                         => OsalPort_clearTaskQueues

Perhaps you can see where this is going. In response to APP event 4, the BDB task tries the secondary channel, which leads to APP event 5.

On APP event 5, the BDB task ultimately clears all task queues. Depending on the precise timing of the various tasks, logging dis/enabled etc. this can happen before the APP task has received the response to the message sent to the zstack task.

To summarize:

1. APP calls OsalPort_msgSend (zstackmsg_CmdIDs_BDB_FILTER_NWK_DESC_COMPLETE_REQ)
2. APP blocks on itself and waits for response from the ZSTACK task
   - OsalPort_blockOnEvent
   - OsalPort_msgFindDequeue
   - loop
3. ZSTACK task sends msg to BDB task
4. ZSTACK task puts response in APP message queue
5. BDB task clears all message queues
6. APP waits forever for a message that will never come

Please confirm that our understanding is correct and that we are not missing something.

Best regards

Environment:

  • CC2652R7
  • simplelink_cc13xx_cc26xx_sdk_8_30_01_01
  • TI Clang v4.0.2.LTD
  • Hello Kenny,

    Thank you for your in-depth report of this issue.  Another E2E user has reported the same behavior alongside TI's own internal observations.  There is a 15.4-Stack low-level issue which is being further investigated, meanwhile you can:

    • Use SimpleLink F2 SDK v7.41 which does not demonstrate the behavior
    • Configure the Watchdog or a clock timeout to reset the device
    • Try the workaround used by the relevant E2E post (unconfirmed by TI)

    Regards,
    Ryan

  • Hi Kenny,

    The following workaround should be used:

    void OsalPort_clearTaskQueues( void )
    {
    // uint8_t i;
    // uint8_t *pMsg;
    //
    // for(i = 0; i < taskCnt; i++)
    // {
    // while((pMsg = OsalPort_msgReceive(i)) != NULL)
    // {
    // OsalPort_msgDeallocate(pMsg);
    // }
    // }
    }

    TI R&D will implement a fix by the next SimpleLink F2 SDK release.

    Regards,
    Ryan

  • Hi Ryan,

    Thanks for the quick replies.

    Is there any issue with the queues not being cleared with that workaround?

    We fixed it by putting a binary semaphore around the app send->wait for response, and clear app queue code sections like so, but if there are no consequences to removing it completely, then we'll just do that

    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    // app_main.c
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    
    uint8_t app_task_id = 0xff;
    ...
    void app_initialize()
    {
    ...
        app_task_id = OsalPort_registerTask(Task_self(), h_app_loop_sem, &app_task_events);
    ...
    }
    
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    // osal_task_clear_fix.h
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    
    #ifndef APPLICATION_OSAL_TASK_CLEAR_FIX_H_
    #define APPLICATION_OSAL_TASK_CLEAR_FIX_H_
    
    #ifdef __cplusplus
    extern "C" {
    #endif
    
    #define TASK_CLEAR_FIX
    
    #ifdef TASK_CLEAR_FIX
    
    extern uint8_t app_task_id;
    
    uint8_t OsalPort_getTaskId(void* taskHndl);
    
    void osal_task_clear_fix_init();
    void osal_task_clear_fix_pend();
    void osal_task_clear_fix_post();
    
    #endif // #ifdef TASK_CLEAR_FIX
    
    #ifdef __cplusplus
    }
    #endif
    
    #endif /* APPLICATION_OSAL_TASK_CLEAR_FIX_H_ */
    
    
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    // osal_task_clear_fix.c
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    
    #include "osal_task_clear_fix.h"
    #include <ti/sysbios/knl/Semaphore.h>
    #include <ti/sysbios/BIOS.h>
    
    static Semaphore_Struct app_wait_for_rsp_sem;
    static Semaphore_Handle h_app_wait_for_rsp_sem = NULL;
    
    void osal_task_clear_fix_init()
    {
        Semaphore_Params params;
        Semaphore_Params_init(&params);
        params.mode = ti_sysbios_knl_Semaphore_Mode_BINARY;
        Semaphore_construct(&app_wait_for_rsp_sem, 1, &params);
        h_app_wait_for_rsp_sem = Semaphore_handle(&app_wait_for_rsp_sem);
    }
    
    
    void osal_task_clear_fix_pend()
    {
        if (h_app_wait_for_rsp_sem) {
            Semaphore_pend(h_app_wait_for_rsp_sem, BIOS_WAIT_FOREVER);
        }
    }
    
    void osal_task_clear_fix_post()
    {
        if (h_app_wait_for_rsp_sem) {
            Semaphore_post(h_app_wait_for_rsp_sem);
        }
    }
    ```
    
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    // [SDK FILE] osal_port.c
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    
    // ============================= EDIT BEGIN =============================
    #include "osal_task_clear_fix.h"
    // ============================= EDIT END =============================
    
    void OsalPort_clearTaskQueues( void )
    {
        uint8_t  i;
        uint8_t *pMsg;
    
        for(i = 0; i < taskCnt; i++)
        {
            // ============================= EDIT BEGIN =============================
    #ifdef TASK_CLEAR_FIX
            // app_task_id is statically init to 0xff and
            // is set after the semaphore is initalized.
            if (i == app_task_id) {
                osal_task_clear_fix_pend();
            }
    #endif
            // ============================= EDIT END =============================
    
          while((pMsg = OsalPort_msgReceive(i)) != NULL)
          {
              OsalPort_msgDeallocate(pMsg);
          }
          
          // ============================= EDIT BEGIN =============================
    #ifdef TASK_CLEAR_FIX
          // app_task_id is statically init to 0xff and
          // is set after the semaphore is initalized.
          if (i == app_task_id) {
              osal_task_clear_fix_post();
          }
    #endif
          // ============================= EDIT END =============================
    
        }
    }
    
    
    
    // ============================= EDIT BEGIN =============================
    uint8_t OsalPort_getTaskId(void* taskHndl)
    {
        for(uint8_t taskIdx = 0; taskIdx < taskCnt; taskIdx++)
        {
            if (taskTbl[taskIdx].taskHndl == taskHndl) {
                return taskIdx;
            }
        }
        return 0xff;
    }
    // ============================= EDIT END =============================
    
    
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    // [SDK FILE] zstackapi.c
    ///////////////////////////////////////////////////
    ///////////////////////////////////////////////////
    
    // ============================= EDIT BEGIN =============================
    #include "osal_task_clear_fix.h"
    // ============================= EDIT END =============================
    static zstack_ZStatusValues sendReqDefaultRsp(uint8_t appServiceTaskId,
                                                  zstack_CmdIDs cmdID, void *pReq,
                                                  int msgSize)
    {
        zstack_ZStatusValues status = zstack_ZStatusValues_ZMemError;
        uint8_t msgStatus;
    
        //Make sure allocate space enought for the msg, even
        //if the message does not have payload
        if(msgSize < sizeof(zstackmsg_genericReq_t))
        {
          msgSize = sizeof(zstackmsg_genericReq_t);
        }
    
        zstackmsg_genericReq_t *pMsg =
            (zstackmsg_genericReq_t *)OsalPort_msgAllocate(msgSize);
    
        // Make sure the allocation was successful
        if(pMsg != NULL)
        {
            // Fill in the message header
            pMsg->hdr.event = cmdID;
            pMsg->hdr.status = 0;
            pMsg->hdr.srcServiceTask = appServiceTaskId;
    
            // Update the messges's request field
            pMsg->pReq = pReq;
    
            // ============================= EDIT BEGIN =============================
    #ifdef TASK_CLEAR_FIX
            // Do any other tasks call this fn?
            bool is_app_task = OsalPort_getTaskId(Task_self()) == app_task_id;
            if (is_app_task) {
                osal_task_clear_fix_pend();
            }
    #endif
            // ============================= EDIT END =============================
    
            // Send the message
            msgStatus = OsalPort_msgSend( stackServiceTaskId, (uint8_t*) pMsg );
    
            // Was the message sent successfully
            if(msgStatus == OsalPort_SUCCESS)
            {
                bool gotRsp = false;
    
                // Return status
                zstackmsg_genericReq_t *pCmdStatus = NULL;
    
                while(!gotRsp)
                {
                    // Wait for the response message
                    OsalPort_blockOnEvent(Task_self());
    
                    pCmdStatus = (zstackmsg_genericReq_t*)OsalPort_msgFindDequeue(appServiceTaskId, pMsg->hdr.event);
    
                    if(pCmdStatus)
                    {
                      gotRsp = true;
                    }
                }
                // ============================= EDIT BEGIN =============================
    #ifdef TASK_CLEAR_FIX
                if (is_app_task) {
                    osal_task_clear_fix_post();
                }
    #endif
                // ============================= EDIT END =============================
    
                 // setup return of
                status = (zstack_ZStatusValues)pCmdStatus->hdr.status;
            }
    
            // pCmdStatus is the same as pMsg
            OsalPort_msgDeallocate( (uint8_t*)pMsg);
        }
    
        // function status
        return(status);
    }

  • There are no consequences with the solution proposed, the OsalPort_clearTaskQueues API should only be used for 15.4-Stack CoProcessor projects for which the Z-Stack solutions do not apply.

    Regards,
    Ryan