CC2530: Coordinator and End Device interaction

Mark Galich said:

1.When LinkQuality between Coordinator and End Device is about zero I get ZDO_LEAVE_IND (0x45C9) in some time. Communication stops. Need to pair them again.
1.1.Who is initiator of this dispairing? Coordinator or End Device?
1.2.Why dispairing happens? Huge amount of events during low LinkQuality?
1.3.Is there a way to prevent dispairing?

Do you have sniffer log to elaborate such behavior?

Mark Galich said:

2.1.How to remove the Device knowing its short address only?

AFAIK, IEEE address is mandatory to send ZDO MGMT leave request.

Mark Galich said:

3.1.Does unpaired Device listen to tha radio?

No, device won't communicate with previous network after it leaves network.

Mark Galich said:

3.2.Is that possible to communicate to unpaired Device knowing its 64-bit IEEE address?

No.

Hi Mark,

1. The default END_DEV_TIMEOUT_VALUE is 8 meaning that the ZED will be aged out after not communicating for 256 minutes.  If the parent misses too many ZED data requests then the child may search for a better parent as well.  You could increase the TX power or reduce the distance between nodes to improve the link.

2. You could use UTIL_ADDRMGR_NWK_ADDR_LOOKUP to discover the extended address of devices in the association table.

Regards,
Ryan

Hello guys

>>> 1. The default END_DEV_TIMEOUT_VALUE is 8 meaning that the ZED will be aged out after not communicating for 256 minutes. If the parent misses too many ZED data requests then the child may search for a better parent as well.

In case of the End Device has been offline for days and then comes online, I get ZDO_END_DEVICE_ANNCE_IND (0x45C1) + ZDO_TC_DEV_IND (0x45CA). Communication continues. 256 minutes of offline do not lead to leave.
Leave happens much faster. When LinkQuality is high enough - End Device works without problem for days. But in case of some obstacle between End Device and Coordinator ZDO_LEAVE_IND may happen within dozen of seconds. Even if an obstacle was removed the connection with End Device was lost. The only way to restore communication is manual intervention: PermitJoin + push the button for pairing. End Device should never leave the network in any circumstances. The only way to leave should be manual intervention as well.

>>> You could increase the TX power or reduce the distance between nodes to improve the link.
We use LNA and external antennas for both Coordinator and End Device. But crowd of people or loader or metal door may randomly cross the way between Coordinator and End Device. Earlier or later the obstacle will away, communication should continue.

>>> Do you have sniffer log to elaborate such behavior?

We sniffed the radio. LinkQuality was reduced with the help of metal box.

--- Example 1:

Coordinator: 0x0000
ZDO: 0x3459
Packet #3492 - Leave from Coordinator to the End Device.
Packet #3494 - Leave from the End Device to broadcast.
Initiator is Coordinator.

--- Example 2

Coordinator: 0x0000
ZDO: 0x7F07
Packets #203-219 - a number of Leaves from Coordinator to the End Device.
Packets #220-223 - a number of Leaves from the End Device to broadcast.
Initiator is Coordinator as well.

We need to disable unintended leaving. May there be some #defines in source code to awoid such a behaviour of Coordinator?

Thanks for the sniffer log screenshots, feel free to provide the actual sniffer log files if capable.  It appears that the Leave is instigated by the end device's rejoin request.  Are you able to debug the Coordinator during a fail state, and if so then how does it process ZDSecMgrDeviceJoin and when does it send the NLME_LeaveReq (likely from ZDSecMgr.c)?  Is permit join on during this time and have you set zgAllowRejoins to TRUE (ZGlobals.c)?

// trustcenter allows rejoins using well known or default keys 
uint8 zgAllowRejoins = FALSE;   // FALSE by default

//if zgAllowRejoins is set to FALSE, the rejoining device will receive a rejoin rsp success and also a leave command.
//The leave command will have set the rejoin parameter = 'zgAllowRejoinsOptions' value.
uint8 zgAllowRejoinsOptions = FALSE;  //FALSE by default

Please also review the Z-Stack 3.0.2 Known Issues and Fixes E2E post.

Regards,
Ryan

Hi Rian

Both params in ZGlobals.c were FALSE by default
uint8 zgAllowRejoins = FALSE;
uint8 zgAllowRejoinsOptions = FALSE;

1.I changed them to TRUE as you suggested. The situation has not inproved drastically. These two params are used in the library only. What is their sense?

2.On low LinkQuality though there is no LEAVE, Coordinator just forgets the End Device and responses ZNwkNoRoute (0xCD) for any request to ZDO. UTIL_GET_DEVICE_INFO says there is no Device in the table at all.

--- Log example1:

Before 12:45:31 there were Rejoin Responses, just after - Route Requests. This is crucial moment, Coordinator forgot the Device. I suspect this moment coinsides with PermitJoin termination.

--- Log example2:

Similar situation. Before 12:53:49 there were Rejoin Responses, after - Route Request. Coordinator forgot the Device again. PermitJoin terminated as well.

Full log in attachment (time shift 12:41:00)

Log7_25_10_2022_Leave.zip

What is the reason of Device forgetting? Though the End Device instigates Coordinator to forget, how to prevent this situation? Is that possible to force Coordinator and End Device to recover connection infinitely?

Hi Mark,

Thank you for the sniffer log.  There appear to be several "Malformed Packets" from the ZC starting at packet 108, and the ZCL sequence number never appears to increment.  The ZC begins to miss ACKs quite early after commissioning, as soon as packet 136.  Do you observe this behavior when using a default CC2530 Z-Stack 3.0.2 ZNP interfaced with Z-Tool, and approximately how many devices are associated with the ZNP device?  Does similar behavior persist even with the nodes in close proximity to one another?  There could be memory corruption and I once again advise that you review the Z-Stack 3.0.2 Known Issues and Fixes E2E post as well as CC2530 RAM/flash limitations.  If possible, you should consider migrating your ZNP to a SimpleLink CC13X2 / CC26X2 device instead.

Regards,
Ryan

Hi Ryan

>>> There appear to be several "Malformed Packets" from the ZC starting at packet 108, and the ZCL sequence number never appears to increment.
These are my test requests. I generate them via API AF_DATA_REQUEST (0x2401) and send to Coordinator. There are always the same TransId and ZCL sequence number. Does it lead to a trouble?
When there is a good LinkQuality, there is no problem at all. Devices work correctly for days.

Do TransId and ZCL sequence number depend on each other? Like ZCL sequence number = TransId + 1?
Should TransId and ZCL sequence number be incremented in any message independently of response?
In case of overflow 0xFF->0, right?
Any details concerning these two fields?

>>> The ZC begins to miss ACKs quite early after commissioning, as soon as packet 136.
I guess this packet loss made manually to reproduce the situation. There is no problem when good LinkQuality. On real object it takes days to lose connection. On debugging we have to make poor conditions for communication to get the same trouble for minutes.

>>> Do you observe this behavior when using a default CC2530 Z-Stack 3.0.2 ZNP interfaced with Z-Tool
I have not tried it yet. It will not be easy to send a number of requests via Z-Tool manually, but will try...

>>> and approximately how many devices are associated with the ZNP device?
The only End Device.

>>> Does similar behavior persist even with the nodes in close proximity to one another?
Everything depends on LinkQuality. Good level - no problem. But we cannot guarantee 100% good level on the object.

>>> Z-Stack 3.0.2 Known Issues and Fixes E2E post
Most of these issues fixed. Tried to reduce NWK_MAX_DEVICE_LIST, MAX_NEIGHBOR_ENTRIES and increase NWK_MAX_DATABUFS. Seems to be more stable connection, but at last it broke.

This is screenshot:

As I can see 890 was the last ACKed successfull packet. After it my requests to Coorditator did not lead to any radio activity. Then Coordinator stopped responding for my API requests.
After reboot Coordinator did not forget the Device, but the End Device left the network.
Why? Looks like the End Device needs to be fixed as well.

Log8_26_10_2022_Leave.zip

(full log in attachment, time shift 09:37:30)

>>> If possible, you should consider migrating your ZNP to a SimpleLink CC13X2 / CC26X2 device instead.
Will migration to CC2652P solve all these problems?

Mark Galich said:

These are my test requests. I generate them via API AF_DATA_REQUEST (0x2401) and send to Coordinator. There are always the same TransId and ZCL sequence number. Does it lead to a trouble?

Thanks for clarifying, I wanted to make sure this was the intended effect.

Mark Galich said:

Do TransId and ZCL sequence number depend on each other? Like ZCL sequence number = TransId + 1?
Should TransId and ZCL sequence number be incremented in any message independently of response?
In case of overflow 0xFF->0, right?
Any details concerning these two fields?

Each Zigbee layer (IEEE, NWK, APS, ZCL) has its own count, and the ZCL can be configured by the user.  This too was a check to make sure the behavior was known and expected.

Mark Galich said:

After reboot Coordinator did not forget the Device, but the End Device left the network.
Why? Looks like the End Device needs to be fixed as well.

The ZED will enter an orphan state if the ZC becomes unresponsive, that way it can send beacons to possibly rejoin another router device.  This should occur within the same network unless the application dictates that it should forget the previous one and restart commissioning from the start.  This is why it would help to know the specific changes applied to each project.

Mark Galich said:

Will migration to CC2652P solve all these problems?

As stated earlier, the CC2530 has limited RAM and thus performs poorly as a ZC trust center when compared to the CC2538 or SimpleLink CC2652R.  Another consideration is that the last Z-Stack update for CC253X occurred four years ago whereas the SIMPLELINK-CC13XX-26XX-SDK still receives quarterly updates which include bug resolution and feature improvement.

The ZED and ZC appear to be attempting to maintain a connection but are constantly hindered by the poor link quality causing disconnection.  You could add routers in between devices to help improve stability, or increase the TX power of devices so that they can transmit further distances.  Ultimately there will be a limitation on what can be achieved given the circumstances.

Regards,
Ryan

Hi Ryan

We have made a new coordinator based on CC2652P and reproduced the same experiment with low LinkQuality. Coordinator (CC2652P), ZED (CC2652RB), request 0x2401 -> response 0x4481.
From time to time we got _ANNCE_IND, but finaly ZED sent LEAVE message.

Now coordinator does not forget ZED, but ZED still leaves the network.
Full log: Log8_16_11_2022_Leave.zip

We need to understand the reason of leaving. May be ZED considers its work from batteries and leaves the network to spare energy?
ZED based on examples\rtos\LP_CC2652RB\zstack\zed_temperaturesensor and has external power source
Our modifications:
1.CUI is still active
2.Added customized request/response (0z2401/0x4481)
3.Modified zd_config.c
void ZDConfig_UpdatePowerDescriptor()
{
ZDO_Config_Power_Descriptor.PowerMode = NODECURPWR_RCVR_ALWAYS_ON;
ZDO_Config_Power_Descriptor.AvailablePowerSources = NODEAVAILPWR_MAINS;
ZDO_Config_Power_Descriptor.CurrentPowerSource = NODEAVAILPWR_MAINS;
ZDO_Config_Power_Descriptor.CurrentPowerSourceLevel = NODEPOWER_LEVEL_100;
}

and zcl_sampletemperaturesensor_data.c
const uint8_t zclSampleTemperatureSensor_PowerSource = POWER_SOURCE_DC;

4.Built with preprocessor defines:
ZCL_READ
ZCL_DISCOVER
ZCL_WRITE
ZCL_BASIC
ZCL_IDENTIFY
ZCL_GROUPS
ZCL_TEMPERATURE_MEASUREMENT
ZCL_MS
BDB_REPORTING
TIMAC_ROM_PATCH
xCUI_DISABLE
MAX_STATUS_LINES=10
ZSTACK_SECURITY
BOARD_DISPLAY_USE_UART
FREQ_2_4G
OSAL_PORT2TIRTOS
OSAL_PORT2TIRTOS_OSALMAP
STACK_LIBRARY
ZDO_API_BASIC
TC_LINKKEY_JOIN
NV_RESTORE
NV_INIT
FEATURE_NON_BEACON_MODE
ZCL_STANDALONE
MAX_DEVICE_TABLE_ENTRIES=50
DEVICE_FAMILY=cc26x0
DeviceFamily_CC26X2
TIMAC_ROM_IMAGE_BUILD
NVOCMP_NVPAGES=2

Have we missed something? May there be heating up due to low LinkQuality and following leave? We need to prevent ZED from occasional leaving.

The ZED sending Leave commands should be not be related to your application and pre-define changes but you could test with a default ZED temperature sensor project to confirm.  If possible to actively debug the ZED, or since you still have the CUI enabled, then it would help to determine the call stack which leads to NLME_LeaveReq at either bdb_resetLocalAction or bdb_event_loop as these are the two feasible locations from which a leave would be instigated. 

The instance in bdb_event_loop depends on BDB_TC_LINK_KEY_EXCHANGE_FAIL which does not make too much sense for this context given that the ZED should already have the TCLK stored inside NV memory and does not appear to send any TCLK update requests over the air.  It could happen from a security perspective since the device had recently joined less than 15 seconds before the Leave command, and the ZC failing to ACK the Node Descriptor and Read Attribute Responses.

Both instances of Zstackapi_bdbResetLocalActionReq which cause the bdb_resetLocalAction are dependent on the CUI, either through UART selection or pressing BTN-2 during a reset.  These are even less likely than the previous instance.  You mentioned that the ZED is externally powered, typically through batteries.  What is the voltage level of the ZED during this behavior and what is its TX power level?  If not done so already, you should set NVOCMP_MIN_VDD_FLASH_MV=200 (Non-Volatile Memory Low Voltage Detection) so that the NV is not corrupted during low power states.

Regards,
Ryan

Hi Ryan

more experiments
Coordinator CC2652P SDK 6.30 based on ZNP_CC1352P_2_LAUNCHXL
Changes:
zgAllowRejoinsWithWellKnownKey = TRUE;
zgChildAgingEnable = FALSE;

EndDevice CC2652RB SDK 6.30 based on zed_temperaturesensor_LP_CC2652RB
Changes:
requestNewTrustCenterLinkKey = FALSE;

ZED under debugger with breakpoints on every NLME_LeaveReq function call.
During communication we reduce LinkQuality to about zero by means antenna wringing, polarization changing, device shielding. The purpose is to make reconnections ZDO_TC_DEV_IND and ZDO_END_DEVICE_ANNCE_IND.

None of breakpoins was cought. Will continue exploring.
It seems to be there are two problems:
1.ZC sends leave to ZED

Full log 09_18_11_2022.zip the trouble is below 1072.

2.ZC forgets ZED without leave

Full log 11_18_11_2022.zip the trouble is below 1608.

Full log 12_18_11_2022.zip the trouble is below 2148.

Is there any idea how to prevent unintential leaving?

1st log: As with the ZED, you can debug the ZC to determine why it enters ZDSecMgrDeviceRemove to execute a NLME_LeaveReq.  It could be due to missing Device Announce messages.  You may have noticed that the Rejoin bit is true and that the ZED rejoins successfully a few seconds afterwards.

2nd log: I don't believe the ZC received the ZED Device Announce message and thus believes another routing device is the parent.  It is acknowledging the data requests sent directly to it, then attempting to find the correct parent device to receive the queued messages.

Same symptoms for the third log.  You will need to decrease the distance between nodes or increase the TX power of each to improve behavior.

Regards,
Ryan

Dear friend Ryan

>>> you can debug the ZC to determine why it enters ZDSecMgrDeviceRemove to execute a NLME_LeaveReq.

Sure, but we have the only debugger, so either ZC or ZED can be debugged at once. Those logs were collected with ZED under debug. Now will debug ZC.

>>> I don't believe the ZC received the ZED Device Announce message and thus believes another routing device is the parent. It is acknowledging the data requests sent directly to it, then attempting to find the correct parent device to receive the queued messages.

There is no problem if device turned off for long time (hours, days, weeks), when turned on it continue working as usual. There is no problem if good LinkQuality between ZC and ZED, they work for days and weeks (I believe they can work for months and years as well). The problem happens when the LinkQuality is about zero. As I said earlier we cannon assure 100% good LinkQuality. The only reason for LinkQuality reducing during our experiments is to reproduce the situation on real object.

>>> You will need to decrease the distance between nodes or increase the TX power of each to improve behavior

ZC and ZED have no power limit and supplied from AC/DC converter. ZC based on CC2652P with amplification +20dB and external wifi antenna. ZED based on CC2652RB with amplification +5dB and external pcb antenna. The distace is vary about 15-20m but obstacles may happen. We cannot predict the time, duration and amount of obstacles. We admit loosing of the connection for minutes or even hours. But when the obstacles are beeing out the connection MUST recover. Manual repeat of the device pairing after connection unexpectedly lost is NOT we really need.

All we need is to force the devices to recover connection in any circumstances without manual intervention.
We suspect such a behaviour is linked with some counter/buffer/table overload or even with policy of security. If the last one, lets talk of security simplification, we can cypher our traffic ourselves.
And we need your support.

Hey Mark,

Thank you for the detailed description of your environment and project requirements.  Concerning relaxed security, you will want to review zd_sec_mgr.c for the sections where AddrMgrEntryRelease is used to remove device associations.  This is either during APSME_TCLinkKeyLoad (not likely since the TCLK should be maintained since the device is allowed to rejoin the network, however I'm not too sure given the changed zgAllowRejoinsWithWellKnownKey bit) or ZDSecMgrAddrClear.  You could further debug to determine if ZDSecMgrAddrClear is used at any point, then review the call stack to further determine the correct action to take inside of zd_sec_mgr.c for your application.

Do you have a recommended method of replicating this behavior so that I can further test from my end?

Regards,
Ryan

Dear friend Ryan

We continue experimenting with our Zigbee devices. Eventually we discovered what was going on with ZED.
1.We noticed ZED lives longer without CUI. So we excluded it with the keyword CUI_DISABLED. Unfortunately leds and buttons have gone as well.

2.Added #define RESET_ASSERT

3.File zd_app.c line 516 was commented
//zgWriteStartupOptions( ZG_STARTUP_SET, ZCD_STARTOPT_DEFAULT_NETWORK_STATE | ZCD_STARTOPT_DEFAULT_CONFIG_STATE);
This line forces ZED to forget the connection. Without it we got the desired result.
ZED does not forget the network inspite of low LinkQuality.

We have a number of questions we did not find response to:

1.We are not sure commenting line 516 was the best decision. Though we got the result, probably we lost something else. Can there be some side effect?
2.What is ZBA_FALLBACK_NWKKEY for? Can it help us in maintaining the connection?
3.DEV_END_DEVICE_UNAUTH // Joined but not yet authenticated by trust center
We expected this event happen every join, but actually not. What conditions it can happen in? Some timeout?

4.There is SysCtrlSystemReset() function to reset microcontroller. Does ZED make reset in absense of connection? In unpaired state?

5.In case of good LinkQuality ZC is used to send me AF_DATA_CONFIRM with status 0x00 = Ok. In case of low LinkQuality I used to get 0xF0 = No response, but sometimes 0xCD = No route. So there is some reason ZC to remove ZED from its table. If ZED continue requesting data from ZC, ZC continue talking to ZED. Before line 516 in ZED was commented we used to loose connection. Probably we should make similar changes in ZC to prevent route loosing.

6.zgChildAgingEnable=TRUE; What effects will happen if zgChildAgingEnable=FALSE? Table overload?

7.We need an example how to add our own cluster and command. Now we use ZCL_CMD_READ handler. Just added the line
if(pInMsg->msg->clusterId==0x8888) {OurHandler(pInMsg); return TRUE;}
to zclProcessInReadCmd( zclIncoming_t *pInMsg ) function.
It works, but we suspect there must be more correct way to add our own handler.
It is preferable to add our own command and to use our own data type. Could you please provide with some example?

Thanks

1. The ZDO_DEVICE_RESET event is intended for instances where the device needs to leave the network for any given reason (requested by another device or user interaction), thus removing the startup options will break this feature.
2. ZBA_FALLBACK_NWKKEY will resort to an existing fallback nwk key if device authentication fails.  This may be an option since the ZED cannot currently communicate with the ZC.
3. DEV_END_DEVICE_UNAUTH is the state in between DEV_NWK_JOINING and the final DEV_END_DEVICE, it will not occur every rejoin since the device should already be authenticated.
4. If ZDApp_RestoreNwkKey returns false, or the state is DEV_NWK_TC_REJOIN_CURR_CHANNEL or DEV_NWK_TC_REJOIN_ALL_CHANNEL, without proper authentication from the TC then the device will leave the network.  Please consult the following code excerpts:

//bdb.c

  // Transition state machine to correct rejoin state based on nwk key
  if ( (bdbSecureRejoinAttempts < zgBdbMaxSecureRejoinAttempts) && (ZDApp_RestoreNwkKey( FALSE ) == TRUE) )
  {
    ZDApp_ChangeState( DEV_NWK_SEC_REJOIN_CURR_CHANNEL );
  }
  else
  {
    ZDApp_ChangeState( DEV_NWK_TC_REJOIN_CURR_CHANNEL );
  }

//zd_app.c

      // Verify NWK key is available before sending Device_annce
      // Device performing TC rejoin shall wait until (new) NWK key received
      if ( ZDApp_RestoreNwkKey( TRUE ) == false ||
           devState == DEV_NWK_TC_REJOIN_CURR_CHANNEL ||
           devState == DEV_NWK_TC_REJOIN_ALL_CHANNEL )
      {
        if ( ZSTACK_END_DEVICE_BUILD )
        {
          nwk_SetCurrentPollRateType(POLL_RATE_TYPE_JOIN_REJOIN,TRUE);
        }
        // wait for auth from trust center
        ZDApp_ChangeState( DEV_END_DEVICE_UNAUTH );

        // Start the reset timer for MAX UNAUTH time
        ZDApp_ResetTimerStart( MAX_DEVICE_UNAUTH_TIMEOUT );
      }

You could choose for the device to continue attempting DEV_NWK_SEC_REJOIN_CURR_CHANNEL and not resort to DEV_NWK_TC_REJOIN_CURR_CHANNEL, or not use ZDApp_ResetTimerStart on failed authentication.
5. If zgChildAgingEnable = TRUE then the ZED association will be removed from the ZC after END_DEV_TIMEOUT_VALUE expires.
6. zgChildAgingEnable allows the ZC to purge ZED associations which are no longer communicating, thus the table could be filled if this is neglected.
7. Please refer to this relevant E2E thread: https://e2e.ti.com/f/1/t/1124463 

Regards,
Ryan

Dear friend Ryan

Thanks for your response. We got a lot of helpful info and reached certain results.
But we still have some questions.
Delay between request to ZED and response from ZED is 2-4 seconds in average.
On looking through the code there seems to be two ways to reduce the delay:

1.To decrease POLL_RATE from 3000 to let us say 500. It will speed up the responses, right? But if there are many of ZEDs, e.g. 50. And all of them request data from ZC every 500 milliseconds. Will it be a problem?

2.To switch to RxAlwaysOn mode.
Will ZC send requests to ZED in RxAlwaysOn mode immediately without queueing?

We tried these two defines in ZED:
RFD_RX_ALWAYS_ON_CAPABLE=TRUE
RFD_RX_ALWAYS_ON=TRUE
But with no success. ZED could not pair to ZC. There is ZDO_TC_DEV_IND (0x45CA), but no ZDO_END_DEVICE_ANNCE_IND(0x45C1).
Probably there is another way to turn ZED to RxAlwaysOn mode.

What is better from ZC point of view: 50 ZEDs in RxAlwaysOn mode or 50 ZEDs in Sleepy mode with POLL_RATE = 500?

I'm glad to hear that you are making progress.

1. Decreasing POLL_RATE will reduce data latency while also increasing network traffic.

2. The non-sleepy status will be broadcast during the device announce, thus the parent node will be aware that it can immediately send messages to a non-sleepy ZED without pending on the data request.

Please try to set RFD_RCVC_ALWAYS_ON=TRUE in f8wConfig.cfg, also is there any reason for continuing to use ZED nodes instead of ZRs?

Regards,
Ryan

Dear friend Ryan

Thanks.

Dut, can't find f8wConfig.cfg file.
I find it in zStack 1.2 and zStack 3.0.

But my ZED is built on SDK 6.30. Is there no such file?

My apologies as I though you were referring to Legacy CC253X changes.  For the SIMPLELINK-CC13XX-CC26XX-SDK devices you will refer to the SysConfig file under the Z-Stack-> Power Management module and change "Power Mode" to "Always On + Sleepy" (RFD_RX_ALWAYS_ON_CAPABLE = TRUE) and "Power Mode of Operation" to "Always On" (RFD_RX_ALWAYS_ON = TRUE) from the generated z_stack_config.h file.  I've just confirmed successful operation of this mode using the zc_light and zed_sw projects from SIMPLELINK-CC13XX-CC26XX-SDK v6.30

Regards,
Ryan