This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2652R7: Groups persist in NV flash across power cycles after removal.

Part Number: CC2652R7

Tool/software:

We are working on a typical ZCL on/off output switch (router) and are experiencing a problem where groups can't be removed from NV flash.

The problem occurs both when adding/removing groups over the network, or calling add/remove zstack functions from our own firmware on the device.

To confirm no issue without our firmware, we flashed the zr_light_LP_CC2652R7_tirtos7_ticlang light sample app from the SDK with no changes, but still see the same behavior.

For example, when sending ZCL commands from the coordinator:

SEND ZCL command groups_get_group_membership:
>
> { capacity: 16, groupcount: 0, grouplist: [] }

SEND ZCL command add_group { groupid: 0x01, groupname: "group a" }:
SEND ZCL command groups_get_group_membership:
>
> { capacity: 15, groupcount: 1, grouplist: [ 1 ] }

SEND ZCL command groups_remove_group { groupid: 0x01}:
SEND ZCL command groups_get_group_membership:
>
> { capacity: 16, groupcount: 0, grouplist: [] }

This is as expected. But after rebooting the device (CPU reset, board reset, or hard power cycle), the group returns:

SEND ZCL command groups_get_group_membership:
>
> { capacity: 15, groupcount: 1, grouplist: [ 1 ] }

The same behavior is seen when using Zstackapi_ApsAddGroupReq / Zstackapi_ApsRemoveGroupReq / Zstackapi_ApsFindGroupReq locally on the device. The group functionality works fine, but removed groups persist after reboot. Zstackapi_ApsRemoveAllGroupsReq has the same result.

The issue occurs both when the device is connected to the debugger and when running standalone.

It seems the deletion is happening within the groups table in memory (the group can even be re-added after removal), but the changes are not written to NV, and so are loaded into memory again on reboot.

The only function that does remove them is Zstackapi_bdbResetLocalActionReq, which does a full NV clear (via bdb_setFN, which calls zgWriteStartupOptions)

It's not possible to debug further, since aps_RemoveGroup source is not available.

Does the lib cache flash writes to reduce wear? Is there perhaps an API call we missed to "flush" the changes to NV flash?

Environment:

- Device: CC2652R7
- Board: RF-star RF-BM-2652B2
- SDK: simplelink_cc13xx_cc26xx_sdk_7_41_00_17
- CCS: v12.7.1.00001
- Compiler: ti_cgt_tiarmclang_3.2.0.LTS

EDIT:

I note that someone had a similar (same?) issue, which was determined to be a bug in the SDK. See here:

https://e2e.ti.com/support/wireless-connectivity/zigbee-thread-group/zigbee-and-thread/f/zigbee-thread-forum/1171018/cc2652r-last-group-added-using-aps_addgroup-can-never-be-deleted/4409476?tisearch=e2e-sitesearch&keymatch=Zstackapi_ApsRemoveGroupReq#4409476

That was two years ago, so I'm assuming it was fixed...?

  • Hello Kenny,

    You are correct, this is a bug with the SDK and the solution is as follows (modify aps_GroupWriteNV): 

    /*********************************************************************
     * @fn          aps_GroupsWriteNV
     *
     * @brief       Save the Binding Table in NV
     *
     * @param       none
     *
     * @return      none
     */
    void aps_GroupsWriteNV( void )
    {
      apsGroupItem_t *pLoop;
      apsGroupNVItem_t item;
      uint8_t x = 0;
    
      pLoop = apsGroupTable;
      if (pLoop == NULL)  //ADD THIS LINE
      {  //ADD THIS LINE
          aps_GroupsSetDefaultNV();  //ADD THIS LINE
      }  //ADD THIS LINE
      else  //ADD THIS LINE
      {  //ADD THIS LINE
          while (pLoop)
          {
            item.endpoint = pLoop->endpoint;
            OsalPort_memcpy( &(item.group), &(pLoop->group), sizeof ( aps_Group_t ) );
            // Save the record to NV
            osal_nv_write_ex( ZCD_NV_EX_GROUP_TABLE, x++, sizeof(apsGroupNVItem_t), &item );
            pLoop = pLoop->next;
          }
      }  //ADD THIS LINE
    }

    Thanks,
    Alex F

  • Hi Alex,

    Thanks for your confirmation that it's a bug.

    As far as I can tell, the source of aps_GroupsWriteNV is not available. It is part of the precompiled libraries in the SDK; specifically SDK_ROOT/source/ti/zstack/lib/ticlang/m4f/libZStack_nwk_zr.a <aps_groups.o>. As such, there is no way to modify the original.

    I can define my own version, but the linker will complain about the multiple definitions and I can't exclude the entire library.

    I am not very familiar with the TI linker, but I assume it supports the --wrap option, so that's a possibility, but I suspect that's not what you were suggesting.

    EDIT:

    I've also just noticed that the function is not called from anywhere in the ZStack/example source, so I assume it must be called from within the static libs [EDIT: it is called from apsAddGroup/apsRemoveGroup/apsRemoveAllGroup]. Since this is the case, the problem would need to fixed in the library. At the very least, a fixed version of aps_groups.o.

  • Hello Kenny,

    You are correct, the fix I mentioned unfortunately cannot be applied by developers in the SDK; however this fix can be provided by pre-built libraries, so I will inquire into getting this. 

    Thanks,
    Alex F 

  • Understood.

    As an aside, we can't really understand how this bug can still be present, when it (and the fix) have been known for more than two years. There have been many, many releases since then and this is a very simple fix. At the very least we would expect it to be listed as a known issue.

    Groups are core ZCL functionality and are a required cluster in just about every certified product. Although the ZCL doesn't specifically state that groups must be persistent after power cycling, we can't imagine deleted groups reappearing when the customer power cycles their light switch or plug-in module would be acceptable in a certified product; and TI claim ZCL 7 "integration". Of course the developer can implement their own NV flash routines, and perhaps TI don't consider this functionality (aps_*) part of the ZStack, but it is certainly presented as such.

    Are we missing something here? Is this a new issue and the old issue was resolved?

    EDIT: Actually persistent groups are required by the ZCL spec - "An APS group table entry and an APS binding are both persistent data across a restart.". So it seems ZStack is not compliant.

    We have temporarily resolved the issue in the following manner:

    Looking at the aps_groups.o code, aps_GroupsWriteNV is called by aps_AddGroup, aps_RemoveGroup, and aps_RemoveAllGroup after any work done by these functions. Since the existing aps_GroupsWriteNV is a NOOP when  apsGroupTable is null, then calling the below function immediately after any calls to aps_AddGroup, apsRemoveGroup, and apsRemoveAllGroup has the same effect as fixing the existing aps_GroupsWriteNV.

    void aps_GroupsWriteNVFix (void)
    {
      if (apsGroupTable == NULL)
      {
          aps_GroupsSetDefaultNV();
      } 
    }

    For example in zstacktask.c:

    static bool processApsRemoveGroup( uint8_t srcServiceTaskId, void *pMsg )
    {
      zstackmsg_apsRemoveGroup_t *pReq = (zstackmsg_apsRemoveGroup_t *)pMsg;
    
      pReq->hdr.status = zstack_ZStatusValues_ZInvalidParameter;
    
      if ( pReq->pReq )
      {
        // if ( aps_RemoveGroup( pReq->pReq->endpoint, pReq->pReq->groupID ) )        REMOVED
        uint8_t ret = aps_RemoveGroup( pReq->pReq->endpoint, pReq->pReq->groupID ) // ADDED
        aps_GroupsWriteNVFix();                                                    // ADDED
        if ( ret )                                                                 // ADDED
        {
          pReq->hdr.status = zstack_ZStatusValues_ZSuccess;
        }
      }
    
      return (TRUE);
    }

    We have made these changes and they appear to resolve the issue, but would appreciate your comments on whether you see any unintended consequences as, without access to the source code, we are basing this on what we see in the assembly.

    Thanks

  • Hello Kenny,

    We have reached out internally to see if this fix can be applied to the SDK in the near future, I will update you on the conclusion reached here; I am also looking into your code updates provided. 

    Thanks,
    Alex F

  • Hello Kenny,

    As an update, I received some files from RnD that will allow me to generate new files in SDK; I am currently trying to update these, and I will let you know when I get results. The intent here is to confirm the new files can generate in the SDK, then pass the pre-built library to you as a fix, and a more finalized solution I can notify the team to add the fix to a future SDK release. 

    Thanks,
    Alex F

  • Hello Kenny,

    The files that RnD passed to me are having some build errors at the moment, addressing that now. 

    Thanks,
    Alex F

  • Hello Kenny,

    RnD is still guiding me through addressing the build errors here to generate a proper files that I can pass to you (*new .a files). 

    Thanks,
    Alex F

  • Hello Kenny,

    We were able to get the fix applied to new .a file, attached.

    You will need to take the new .a file and replace the one in C:\ti\simplelink_cc13xx_cc26xx_sdk_7_41_00_17\source\ti\zstack\lib\ticlang\m4f. 

    (Note you will also have to delete your old project, and re-import it so the new setting applies) 

     libZStack_nwk_all.a

    Thanks,
    Alex F