End device fails to go to sleep when response to rejoin request is 'PAN Access 'Denied'

I don't have anything further to add at this point.

My testing was done using a fresh install of Z-Stack Home 1.2.2a, 4 Smart RF06 + 4 CC2538EM, and using the SampleSwitch project as end devices and a SampleLight as the coordinator.

So it looks like this issue stems from a few different things:

- The power manager is never set to allow the device to go into sleep mode if it does not successfully connect to a network. i.e., the device won't go to sleep when it is in backoff unless the system is initialized with a call to osal_pwrmgr_device( PWRMGR_BATTERY ) instead of osal_pwrmgr_device( PWRMGR_ALWAYS_ON ) (the default), or unless the that function call is added in an appropriate place.

- Like you stated, the radio and polling must be turned off when the device goes into backoff as well.

- In ZDApp_ProcessNetworkJoin(), while the device attempts to connect to the network, it tries 4 different methods, ordered as follows: DEV_NWK_SEC_REJOIN_CURR_CHANNEL -> DEV_NWK_SEC_REJOIN_ALL_CHANNEL -> DEV_NWK_TC_REJOIN_CURR_CHANNEL -> DEV_NWK_TC_REJOIN_ALL_CHANNEL, but once reaching DEV_NWK_TC_REJOIN_ALL_CHANNEL there is no logic to prevent it from stopping and forcing it to go into backoff, e.g. after a certain number of failed attempts. However, the rejoin scan timer timeout will make it go into backout after a defined period of time.

- Your rejoin scan timeout value (-DREJOIN_SCAN) should be sufficiently large enough to account for the entire length of your scanning procedure, which in my case takes around 20 seconds real time. You generally wouldn't want ZDO_REJOIN_BACKOFF to activate in the middle of your scanning procedure.

- ZDApp_ProcessOSALMsg() was sometimes executed while devState == DEV_NWK_BACKOFF, and the devState changing logic was handling it incorrectly

I set my project up as you stated, but I used the following config values for testing purposes:

-DREJOIN_SCAN=60000

-DREJOIN_BACKOFF=30000

I've made the following changes in ZDApp.c:

Defines/Global variables:

#define MAX_TC_ALL_CH_REJOIN_FAILURES 3
uint8 rejoinFailureCount = 0;

in ZDApp_ProcessOSALMsg():

else
{
  if( prevDevState == DEV_NWK_SEC_REJOIN_ALL_CHANNEL )
  {
	ZDApp_ChangeState( DEV_NWK_TC_REJOIN_CURR_CHANNEL );
  }
  else if (prevDevState == DEV_NWK_TC_REJOIN_CURR_CHANNEL)
  {
	ZDApp_ChangeState( DEV_NWK_TC_REJOIN_ALL_CHANNEL );
  }
  // Here I increment how many times I have failed to connect to the network using
  // DEV_NWK_TC_REJOIN_ALL_CHANNEL mode. This is used by ZDApp_ProcessNetworkJoin
  else if (prevDevState == DEV_NWK_TC_REJOIN_ALL_CHANNEL){
	rejoinFailureCount++;
	ZDApp_ChangeState( DEV_NWK_TC_REJOIN_ALL_CHANNEL );
  }
  // if we are just coming out of backoff, we want to switch back to 
  // DEV_NWK_TC_REJOIN_ALL_CHANNEL mode
  else if (prevDevState == DEV_NWK_BACKOFF){
	ZDApp_ChangeState( DEV_NWK_TC_REJOIN_ALL_CHANNEL );
	prevDevState = DEV_NWK_TC_REJOIN_ALL_CHANNEL;
  }
}

in ZDApp_event_loop():

if( events & ZDO_REJOIN_BACKOFF )
  {
    // if we are here and devState == DEV_NWK_BACKOFF, we are coming out
    // of backoff after the timer elapsed, so start network rejoining process
    if( devState == DEV_NWK_BACKOFF )
    {
      ZDApp_ChangeState(DEV_NWK_DISC);
      prevDevState = DEV_NWK_BACKOFF;
      // Restart scan for rejoin
      ZDApp_StartJoiningCycle();
      osal_start_timerEx( ZDAppTaskID, ZDO_REJOIN_BACKOFF, zgDefaultRejoinScan );
    }
    
    // otherwise, this event was generated from the rejoin scan timer expiring
    // which means scanning was attempted and unsuccessful for
    // zgDefaultRejoinScan milliseconds, so we should go into backoff/sleep
    else
    {
      // Rejoin backoff, silent period
      ZDApp_ChangeState(DEV_NWK_BACKOFF);
      ZDApp_StopJoiningCycle();
      
      NLME_SetPollRate(0);
      uint8 rxOnIdle = false;
      ZMacSetReq( ZMacRxOnIdle, &rxOnIdle );
      
      #if defined ( POWER_SAVING )
        if(pwrmgr_attribute.pwrmgr_device != PWRMGR_BATTERY) {
          osal_pwrmgr_device( PWRMGR_BATTERY );
        }
      #endif
      
      osal_start_timerEx( ZDAppTaskID, ZDO_REJOIN_BACKOFF, zgDefaultRejoinBackoff );
    }

    return ( events ^ ZDO_REJOIN_BACKOFF);
  }

in ZDApp_ProcessNetworkJoin():

else 
{
 if ( devStartMode == MODE_RESUME )
      {	
		... // leave this code as is
	  }
else if(devStartMode == MODE_REJOIN)
      {
        if ( ZSTACK_END_DEVICE_BUILD )
        {
          devStartMode = MODE_REJOIN;
          _tmpRejoinState = true;
          _NIB.nwkState = NWK_INIT;

          if( prevDevState == DEV_NWK_SEC_REJOIN_CURR_CHANNEL )
          {
            runtimeChannel = MAX_CHANNELS_24GHZ;
            prevDevState = DEV_NWK_SEC_REJOIN_ALL_CHANNEL ;
          }
          else if ( prevDevState == DEV_NWK_SEC_REJOIN_ALL_CHANNEL)
          {
            // Set the flag that will ask the device to do trust center network layer rejoin.
            _NIB.nwkKeyLoaded = FALSE;
            ZDApp_ResetNwkKey(); // Clear up the old network key.
            runtimeChannel = (uint32) (1L << _NIB.nwkLogicalChannel);
            prevDevState = DEV_NWK_TC_REJOIN_CURR_CHANNEL ;
            
          }
          else if ( prevDevState == DEV_NWK_TC_REJOIN_CURR_CHANNEL )
          {
            runtimeChannel = MAX_CHANNELS_24GHZ;
            prevDevState= DEV_NWK_TC_REJOIN_ALL_CHANNEL ;
            
          }
          else if (prevDevState == DEV_NWK_TC_REJOIN_ALL_CHANNEL)
          {
            // rejoinFailureCount is incremented in ZDApp_ProcessOSALMsg
            // where devState is changed based on prevDevState
            
            // if we fail to connect MAX_TC_ALL_CH_REJOIN_FAILURES times,
            // go into backoff before trying again
            if (rejoinFailureCount >= MAX_TC_ALL_CH_REJOIN_FAILURES){
              rejoinFailureCount = 0;
              // ZDO_REJOIN_BACKOFF timer is currently running with 
              // zgDefaultRejoinScan timeout, stop it before continuing 
              osal_stop_timerEx(ZDAppTaskID, ZDO_REJOIN_BACKOFF);
              ZDApp_ChangeState(DEV_NWK_BACKOFF);
              ZDApp_StopJoiningCycle();

              NLME_SetPollRate(0);
              uint8 rxOnIdle = false;
              ZMacSetReq( ZMacRxOnIdle, &rxOnIdle );

              #if defined ( POWER_SAVING )
              if(pwrmgr_attribute.pwrmgr_device != PWRMGR_BATTERY) {
                osal_pwrmgr_device( PWRMGR_BATTERY );
              }
              #endif
              
              // start the backoff timer, go to sleep if power_saving is defined
              osal_start_timerEx( ZDAppTaskID, ZDO_REJOIN_BACKOFF, zgDefaultRejoinBackoff );
            }
          }
        }
      }

      // Clear the neighbor Table and network discovery tables.
      nwkNeighborInitTable();
      NLME_NwkDiscTerm();

      if (devState != DEV_NWK_BACKOFF) {
        // setup a retry for later...
        ZDApp_NetworkInit( (uint16)(NWK_START_DELAY
             + (osal_rand()& EXTENDED_JOINING_RANDOM_MASK)) );
      
      }
}

The event handling function for ZDO_REJOIN_BACKOFF may be somewhat unclear at first glance. With my code changes, it is entered under two conditions:
1. If a timer is started with zgDefaultRejoinScan as the timeout value and allowed to expire before stopping it, the ZDO_REJOIN_BACKOFF statement will be executed and the system will be put into backoff, following the else.
2. If a timer is started with zgDefaultRejoinBackoff as the timeout value and allowed to expire, the system will be coming out of backoff, hence the current state being DEV_NWK_BACKOFF, and the if part will be executed.

The system being put INTO backoff after a defined number of failed connection attempts is handled separately inside ZDApp_ProcessNetworkJoin()

Let me know if you have any questions and if you're able to integrate these changes.

Hi Leo Cahalan， thank you for your post ！

i am using the zstack 1.2.2 in CC2530. and i have a similar issue .

but my issue is that the device execute the REJOIN_BACKOFF case in ZDO_REJOIN_BACKOFF EVENT,

But  the device state is alway keep DEV_NWK_DIS.

do you find that?

SET UP:

when ZED join network , power off the ZC , so the ZED  running  rejoin scan === rejoin backoff  === rejoin scan

sometimes  my issue happens!!!

this issue happen both 1.2.2 and 1.2.2a

CHANGE ITEM TEST:

I have change the rejoin scan period  from 10s ,20s ,3 minutes  ,5minutes`~~~~~

rejoin backoff  30s, 30minutes  and so on.

it seems that long time period will easy to call this issue , ( just base on the result of test again and again,)

BR!

Hi  Yikai

         i  add  those code , could  i always add    "Start rejoin scan when key press" in  the application ?

if ( keys & HAL_KEY_SW_4 )//P04  Wakeup Press
{
 // make sure start rejoin scan only implementing one time
 if(Being_Scaning == FALSE)
 { 
   // only the device being DEV_NWK_BACKOFF
   // key press will make deivces turn to rejoin scan right now 
   if(zclEmerButton_NwkState == DEV_NWK_BACKOFF)
   {
    // make a flag ,don't let the devices going into two times ,
    //while it restart to rejoin scan
    Being_Scaning = TRUE;
    //stop the  zgDefaultRejoinBackoff  timer counting 
    osal_stop_timerEx( ZDAppTaskID, ZDO_REJOIN_BACKOFF );
    // set the device turn into rejoin scan right now 
    ZDApp_ChangeState(DEV_NWK_DISC);
    ZDApp_StartJoiningCycle();
    //make sure devices only scan (zgDefaultRejoinScan)  
    //then turn into rejoin backoff again  
    osal_start_timerEx( ZDAppTaskID, ZDO_REJOIN_BACKOFF, zgDefaultRejoinScan );
   }            

  }
}

  BR!

Hi ALL 

Unfortunately， this patch seems did not fix it totally。

SET UP：

rejoin backoff period ：1800000 // 30minutes

rejoin scan period ：180000// 3minutes

PollRate：45s

Shut down ZC ，when ZED success joinng the network， and then testing the rejoin logic。

After testing several rejoin backoff/Scan cycle， turn on the ZC ，checking whether ZED could success rejoin  

Today ，i found two strange case happen。

1. i update this patch in the IAS_ZONE devices，several testing cycle later ，i found that the ZED could not notice it is disconnect ZC。

   At this moment， ZC is being shut down ， i press the ZED key to send out the msg to ZC ， as normal case，when it does not receive the ACK it will   being orphan。 

    but  it doesn't ， and i saw the log ，there is not more ACK ，when it sending out the msg , And i do it again ，it always doesn't being orphan although there is not more ACK of its msg ！！！

    this case is rare ，i  test all the day and just happen one time。

2. the ZED could not follow the rejoin logic （rejoin scan 3minutes =====》rejoin backoff  30minutes  =====》oin scan 3minutes） sometimes。

    As the normal case ，it must follow the default rejoin time。

    but i found  one ZED rejoin scan 3minutes  and then sleep 3minutes  ， and then rejoin scan 3minutes again。  but it does not always keeping this rejoin cycle。

    after several cycle later ，it turn to the new one cycle ， the  default one  （rejoin scan 3minutes =====》rejoin backoff  30minutes  =====》oin scan 3minutes）

BR!

sorry JasonB , i  was so repent of mine carelessness。

****1

i forgot to save the log when “1 case（don't being Oprah）” happen.

****2

and i saw the “2 case ” in the record video，

《 i set a led blink in the ZDApp_NetworkInit(  )， when being rejoin scan i could know that is being scanning 》。 

Yesterday i found a new strange case  that the ZED stop rejoin scanning in sometimes，（but i don't know what state does it being at that moment）

SET UP:

rejoin  scan  period : 180000

rejoin backoff period： 1800000

pollrate ：45s

default channel：-DDEFAULT_CHANLIST=0x04888800  // 26 23 19 15 11

shut down ZC after ZED join the network.

Log : 5187.456.rar

it shows that After 15:08 there is not more any beacon request until 30minutes later， although rejoin backoff period is only 30minutes

i have some questions ：

1.  when “strange case”  happen ，what can i do  for you？

2. is that OK change  the default rejoin period as  what i want （just like 30minutes or 3minutes）？

3. is that matter set up 10 ZED to test the rejoin logic at the same time  same place？

BR! 

If you have not enabled POWER_SAVING, then the problem I encountered should not be an issue for you. However, I think that some of the suggested code changes may impact other error handling and so it may be safer to add it in your project.

Hi Jason,

The code change fixes the not going back to sleep problem. Thanks for sorting this out.

I now have spent more time examining the code looking for some explanation for an occasional problem we are experiencing in the filed where an end device leave the network for no apparent reason. The only area that I suspect may cause this is where the end device is attempting an insecure rejoin and when it does this, the key is deleted by a call to ZDApp_ResetNwkKey(). If the end device rejoins to a router and the coordinator is not reachable at the time, then the end device will not get a security key and will leave the network and reboot.

When I run a test with the setup described earlier where the end device is orphaned, the parent is the coordinator but the device entry is deleted in the coordinator table and the table is full, I only see the state machine using DEV_NWK_SEC_REJOIN_CURR_CHANNEL. ProcessNetworkJoin does not change devState as prevDevState is set to DEV_NWK_BACKOFF which is not one of the states that triggers a transition.

This is the state machine flow:
DEV_INIT
DEV_NWK_DISC
DEV_NWK_SEC_REJOIN_CURR_CHANNEL
ProcessNetworkJoin (devState 4, prevDevSte 12, nwkStatus 2)
ZDApp_NetworkInit 

If I reboot the end device, then it will cycle through the different join modes as expected. I am not sure though if the DEV_NWK_TC_ join modes are handled. I don't see anything specific to DEV_NWK_TC_REJOIN_CURR_CHANNEL when the end device attempts to rejoin.

In case I made a mistake pasting your code into  ZDApp.c, could you post your file?

One other worrying thing I have noticed while running this test is that if I leave the end device running for several hours where it is attempting to rejoin to the coordinator every 10 minutes, it will eventually stop seeing beacon requests and require a reboot to get out of this mode. Right now I am not sure if this may be some issue related to running the debugger. I will repeat this test in normal run mode and see if I can repeat it.

Thanks

Thanks for this patch. I have the following requirement on a portable end-device:

Transmit every 4-6 hours a status message

When out of range, use the 4-6 hours interval as backoff-time

On button press, leave the backoff state immediatelly and when the end-device joins, immediatelly send a package (personal alarm message).

Now in Switch_processZStackMsgs, I catch the zstackmsg_CmdIDs_DEV_STATE_CHANGE_IND. When changed to zstack_DevState_DEV_END_DEVICE, I immediatelly send the packet. This crashes the CC2630.

So you can see that after the rejoin some messges are exchanged on Endpoint 0x00. But my packet is not transferred (I'm using endpoing 0x05).

I cannot say if the CC2630 really crashes, but it does not respond on anything anymore.