• State TI Thinks Resolved
  • Locked Locked
  • Replies 14 replies
  • Answers 2 answers
  • Subscribers 56 subscribers
  • Views 1973 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2530: Zstack 3.0.1 network frame counter issue

Daniele Coati
Intellectual 270 points

Part Number: CC2530
Other Parts Discussed in Thread: Z-STACK,

Hi,
i am working on a project with a zigbee router with zstack 3.0.1; i have a problem regarding the network frame counter handled by the zstack.
Imagine this scenario:
- router and coordinator are in a zigbee network and are both working
- the router is turned off
- the router is powered back on
- after powering on the router, the coordinator ignores the messages received by the router, only after some time it begins to work again.

After analyzing this problem with a sniffer i found out that after powering up the router network frame counter was lower than the value that it had before turning it off.
The communication between router and coordinator begins to work again only after the network frame counter reaches the value that it had before.

I tried to look into the zstack code to understand when the frame counter is updated in nv memory but without success. Is it closed source? Does anyone have any suggestions regarding this behavior?

  • 0
    TI__Guru**** 222577 points

    Hi Daniele,

    Outgoing NWK frame counters are required to be persistent across all resets in Zigbee 3.0: http://www.ti.com/lit/an/swra615/swra615.pdf

    Please name the TI-supplied ZR project used and further modifications if this is not the case. Frame counters are maintained in the stack source and cannot be handled by users.   If possible, cross-check your observation with Z-Stack 3.0.2 and provide sniffer logs of the issue.

    Regards,
    Ryan

  • 0 in reply to Ryan Brown1
    Intellectual 270 points

    Hi Ryan,

    it is a project based on the sample light example with minimal changes to handle also a custom zigbee cluster. From my understanding zstack has to write to nv memory the frame counter once  MAX_NWK_FRAMECOUNTER_CHANGES changes happened; after every power up the counter is incremented by the same amount plus a delta to make sure the value is always higher. Do you confirm that this logic is closed source and cannot be modified? If so i don't understand why i am having this behavior.

    I will also test on zstack 3.0.2 as soon as possible and will provide to you the sniffer logs of the issue.

    Regards,

    Daniele

  • 0 in reply to Daniele Coati
    TI__Guru**** 222577 points

    Hi Daniele,

    Thank you for your patience, it is possible to change the outgoing frame counter of your local device as well as the incoming frame counters of neighboring devices (nwkSecMaterialDesc.FrameCounter) stored in NV (ZCD_NV_NWK_SEC_MATERIAL_TABLE_START).  In order to ensure that a remote device will not reject frames after a local reset, the last NV frame counter is restored from NV and incremented by 1000 + 250 (see the comments in zgUpgradeSecurityNVItems).  I have not observed the behavior you are describing, have you changed any stack parameters?

    Regards,
    Ryan

  • 0 in reply to Ryan Brown1
    Intellectual 270 points

    Hi Ryan,

    sorry for the long silence, but we are still trying to reliably reproduce the issue; i'll keep you posted when i have more information to share.

    I understand that in the function zgUpgradeSecurityNVItems the frame counter is restored from NV, but what about the point in the source code where the frame counter is written to NV memory periodically? I guess zstack must write the frame counter to NV every 1000 frames, right? But I couldn't seem to find it in the source code...

    Best Regards,

    Daniele

  • 0 in reply to Daniele Coati
    TI__Guru**** 222577 points

    Hi Daniele,

    ZDApp_SaveNwkKey is called every ZDO_FRAMECOUNTER_CHANGE event which is set by the source libraries when the frame counter is incremented up to MAX_NWK_FRAMECOUNTER_CHANGES, which determines the number of times the frame counter can change before saving to NV (default 1000 in ZDApp.h).

    Regards,
    Ryan

  • 0 in reply to Ryan Brown1
    Intellectual 270 points

    Hi Ryan,

    thanks for the feedback, now i understand. Attached you can find some sniffer logs of the issue (test done with zstack 3.0.1 for now); the device under test has network address 0xC9DB. Every capture file starts with a new power on of the device, notice how the first packet sent by the device in every file has the frame counter at 554548. Any ideas?

    sniffer_logs.zip

    Regards,

    Daniele

  • 0 in reply to Daniele Coati
    TI__Guru**** 222577 points

    Hi Daniele,

    Attached are my own test results with a CC2530 Z-Stack 3.0.2 SampleSwitch Router project in which the network frame counter is shown to increment correctly across two resets.  https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/158/NWK_5F00_Frame_5F00_Counter_5F00_Persistence.cubx

    Regards,
    Ryan

  • 0 in reply to Ryan Brown1
    Intellectual 270 points

    Hi Ryan,

    yes, my firmware too works like that normally, but i am trying to understand why sometimes i have this problem and what triggers the frame counter to remain stuck after a power up. I also noticed that when i send the device a network leave command and make it join the zigbee network again then the frame counter increases normally again.

    Regards,

    Daniele

  • 0 in reply to Daniele Coati
    TI__Guru**** 222577 points

    How are you able to consistently reproduce this behavior?

    Regards,
    Ryan

  • 0 in reply to Ryan Brown1
    Intellectual 270 points

    Right now we are still trying to understand how to reproduce consistently this issue, we just know that almost all our router zstack 3.0.1 devices showed at some point during testing this behavior. Current hypothesis are some problems related to writing values to nv memory or some sequence of events during the join procedure that makes the device not save the frame counter correctly.

    I'll keep you posted when i know more.

    Regards,

    Daniele

  • 0 in reply to Daniele Coati
    Intellectual 270 points

    Hi Ryan,

    still trying to investigate and understand this issue, i found out that if i leave the device on long enough to trigger the periodic store to nv memory after 1000 frames then after the power cycle the frame counter increases normally, but if i turn off the device before the first periodic store then the problem occurs.

    The only way to completely solve the issue is to erase and flash the device again, as this is the only way to completely reset the frame counter from nv memory.

    It is becoming more important to us to solve this problem as we have devices from our customer that show this behavior; it is not acceptable to flash these devices again, we normally update them over the air.

    Is there another level of support from TI beyond this forum?

    Regards,

    Daniele

  • 0 in reply to Daniele Coati
    TI__Guru**** 222577 points

    Hi Daniele,

    What is the value of your ZDAPP_UPDATE_NWK_NV_TIME?  This determines the time after which a device joins the network that the NV is updated http://processors.wiki.ti.com/index.php/Zigbee_Known_Issues_and_Proposed_Fixes#NWK.2FAPS_packet_retries.2Ffailure_when_ZED_polls_during_commissioning_.28Transport_Key.2C_Simple_Descriptor_Request.2C_etc.29

    There are also some Z-Stack profile OTA upgrade concerns which are addressed on this Wiki page.  This behavior has still not been reproduced or observed on TI's side, this could imply an issue within your application.  You can contact your local TI Sales or FAE Teams if you find the E2E support to be inadequate.

    Regards,
    Ryan

  • 0 in reply to Ryan Brown1
    Intellectual 270 points

    Hi Ryan,

    i guess i found the bug. In ZDApp.c function ZDApp_RestoreNwkSecMaterial the counter variable "i" is not handled correctly. If the "generic item" on the nv memory is read (not sure if it is normal that this item is used, any comment on this?) then the osal_nv_write at row 1943 will always fail because the id is wrong.

    Here is the fixed version:

    void ZDApp_RestoreNwkSecMaterial(void)
{
  uint8 Found = FALSE;
  uint8 i;
  nwkSecMaterialDesc_t nwkSecMaterialDesc;
  uint8 UpdateFrameCounter = FALSE;

  //Search if we do have security material for this network
  for( i = 0; i < gMAX_NWK_SEC_MATERIAL_TABLE_ENTRIES; i++)
  {
    osal_nv_read(ZCD_NV_NWK_SEC_MATERIAL_TABLE_START + i,0,sizeof(nwkSecMaterialDesc_t),&nwkSecMaterialDesc);
    {
      if(osal_memcmp(_NIB.extendedPANID,nwkSecMaterialDesc.extendedPanID,Z_EXTADDR_LEN))
      {
        UpdateFrameCounter = TRUE;
        Found = TRUE;
        break;
      }
    }
  }    
  //Check if we do have frame counter stored in the generic
  if(!Found)
  {
    //The last entry readed has the Generic item, thefore, no need to read it again
    if(nwkSecMaterialDesc.FrameCounter)
    {
      UpdateFrameCounter = TRUE;
      i--;                                                 <<<<<<--------- added this
    }
  }  

  if(UpdateFrameCounter && (!FrameCounterUpdated))
  {
    FrameCounterUpdated = TRUE;
    
    // Increment the frame counter stored in NV
    nwkSecMaterialDesc.FrameCounter += ( MAX_NWK_FRAMECOUNTER_CHANGES +
                              NWK_FRAMECOUNTER_CHANGES_RESTORE_DELTA );
    
    nwkFrameCounter = nwkSecMaterialDesc.FrameCounter;
    
    osal_nv_write(ZCD_NV_NWK_SEC_MATERIAL_TABLE_START + i,0,sizeof(nwkSecMaterialDesc_t),&nwkSecMaterialDesc);
    
    nwkFrameCounterChanges = 0;
  }
  return;
}

    Best regards,

    Daniele

  • 0 in reply to Daniele Coati
    TI__Guru**** 222577 points

    Hi Daniele,

    Thank you for the advanced debugging.  I see that in zd_sec_mgr.c when the generic nwk security material is initialized it does so with variable "i" decremented by one (lines 3358 to 3363) and it goes to follow that the same should occur in zd_app.c for ZDApp_RestoreNwkSecMaterial, as you've done.  As this bug is also prevalent in our SIMPLELINK-CC13X2-CC26X2-SDK I will pass this information along to our SW Team so that it can be fixed in future updates.

    Regards,
    Ryan