CC2652P: ZDO messages not able to be sent out from ZNP in large network

Hi Shuyang,

I notice several MAC Sequence numbers missing from 0x3049 but this could be due to the sniffer log filter.  How far is the sniffer from the node in question?  The most likely reason is that there is too much channel noise and hence CSMA collision is preventing the ZNP from transmitting.  If MT_AF_CB_FUNC is defined then you will be able to monitor the zstack_ZStatusValues of MT_AF_DATA_CONFIRM.

Regards,
Ryan

Hi Ryan,

During our test, we set the CCA threshold to -63 and the broadcast table to 30 in ZNP, and all devices are placed on the same table (of course, the node density will be higher).

We will then monitor the zstack_ZStatusValues of AF_DATA_CONFIRM as you suggest.
In addition, when i defined MT_MAC_FUNC and MT_MAC_CB_FUNC, ZNP compilation will report an error. Can you please confirm it?

I uploaded a sniffer file when the problem occurred, and the comments are as shown in the screenshot below:

node req fail_panid0x55B3_0917.zip

Thanks,

Howjie

Hi Howjie,

The transmit failures you are encountering are likely due to the change in CCA threshold value.  I was able to build the ZNP with MT_MAC_FUNC and MT_MAC_CB_FUNC defined after changing MAC_CHAN_END to 27 in mt_mac.c  and replacing DMMGR_SAveMacCbReg(0xFFFF); with _macCallbackSub = 0xFFFF; in mt_util.c, however please understand that this layer has not been maintained or tested by TI for several years and it is thus recommended that you take caution when attempting to use it for your application purposes.

Regards,
Ryan

Hi Ryan,

When we reproduced the problem, ZNP received the node_desc_req message and sent the node_desc_rsp message, but the AF_DATA_CONFIRM status of the message was ZNwkNoRoute.
This situation can be resolved by triggering route discovery in advance.

We have a new situation during the test:
The MAC layer of ZNP received the node_desc_req message, but the ZDP_IncomingData function did not receive the message. How to determine the cause of this?

Thanks,

Howjie

There is no correlation since ZDP_IncomingData is a callback from the AF layer whereas Node_Desc_req are processed by the ZDO layer in ZDO_ProcessNodeDescReq.

Regards,
Ryan

Hi Ryan,

From the perspective of code execution, the Node_Desc_req message will be processed by calling ZDO_ProcessNodeDescReq in the ZDP_IncomingData function.

void ZDP_IncomingData( afIncomingMSGPacket_t *pData )
{

    ...
    
    
  while ( zdpMsgProcs[x].clusterID != 0xFFFF )
  {
    if ( zdpMsgProcs[x].clusterID == inMsg.clusterID )
    {
      zdpMsgProcs[x].pFn( &inMsg ); //Processed here++++++++++++++++++++++
      return;
    }
    x++;
  }



}

Thanks,

Howjie

Thanks for clarifying.  So ZDO_ProcessNodeDescReq is never executed?  Can you provide a sniffer log of the Node Descriptor Request and does the ZNP give a response to it?

Regards,
Ryan

Hi Ryan,

Only when the problem recurs, ZDO_ProcessNodeDescReq will not be executed.

The location when the problem reappears is shown in the screenshot:

nodeDescReq0924.zip

Thanks,

Howjie

Hi Howjie,

I appreciate the sniffer log as it shows that this behavior is an exception to normal operation.  The Node Descriptor Request is formatted properly and MAC ACKed by the ZC, and previously the ZC has returned a Node Descriptor Response to other devices.  I notice that 0x0000 and 0x7C0E are not listed in each other's Link Status messages which could explain why 0x7C0E initiates a Route Request (no route available) and 0x0000 rejects the packet (node not listed in its association/neighbor tables).

The issue may also be due to network congestion, filled Z-Stack buffers or insufficient heap.  Can you please consider the following to improve stack behavior?

• Increase MAC_CFG_* in znp_conf.opts
• Increase NWK_MAX_DATABUFS_* in nwk_globals.c
• Increase HEAPMGR_SIZE in app.cfg (if evaluating SDK v5.20)

Regards,
Ryan

Hi Ryan,

As shown in the screenshot above：
0x0000 received the Dev Annce message broadcast by 0x7C0E, and 0x0000 also returned the corresponding Route Reply message, which indicates that 0x7C0E has been stored in a table (at least the routing table) of 0x0000.

packet number 57132: 0x7C0E directly sent node_desc_req to 0x0000 and did not initiate route req;
It seems normal here, but the coordinator did not send the node_desc_rsp message.

packet number 57680 & 57695: ZNP's serial port printing shows that these two data packets have been received, but it appears to be retransmitted at the network layer of 0x7C0E. If 0x7C0E never receives the 0x0000 MAC layer ACK, it finally initiates route discovery to 0x0000.
We will modify the relevant configuration according to your suggestions and test it. If there are new results, I will update you.

Thanks,

Howjie

Hi Ryan,

we doubled the parameters you mentioned and tested it. It seems that there is some improvement, but the phenomenon still exists.

The phenomenon looks similar to the previous one.

In the sniffer, the MAC layer ACK during data transmission can be seen, but the receiver does not seem to receive it and process it.

Thanks,

Howjie

The Route Reply from the ZC definitely appears to be an error since the TCLK exchange is tunneled between intermediary routers.  Can you please refresh me on the current ZC network settings and SDK version evaluated?  Can the issue only occur through a large network setup and is it possible for any joining router device to experience this problem?  Is the router capable of joining properly on subsequent attempts?

Regards,
Ryan

Hi Ryan,

1." Can the issue only occur through a large network setup and is it possible for any joining router device to experience this problem?  "

ZC: ZNP (SDK5.20)
We configure ZNP mainly according to swra650b.pdf.

Additional modifications:
NV: 5Page, Region Base-0x4C000, Region Size-0xA000
CCA threshold: -63
MAX Route Req Entry: 20
MAX_RTG_ENTRIES: 100

If the coordinator receives the DevAnnce of the new device, the coordinator ZNP will trigger an AODV Route Req.

ZR: onOffLight(SDK4.40)
The network parameters have not been greatly modified, and they are basically in the default state.

2." Is the router capable of joining properly on subsequent attempts?"

The routing device can finally join successfully, but it will take many attempts.

3."The Route Reply from the ZC definitely appears to be an error since the TCLK exchange is tunneled between intermediary routers.  "

Yes, 0x0000 and 0x7C0E are not valid neighbor devices, and their direct communication behavior will be very strange.

However, the subsequent 0x7C0E can send Node_Desc_req directly, indicating that 0x7C0E processes the RouteReply message "normally" and uses this path.

The MAC layer of 0x0000 received the Node_Desc_req and did not pass it to the upper layer. Maybe the nwk layer discarded it because it could not find a conter that matches the device when checking the secrity frame counter? If this is the reason, why can 0x7C0E process the Route Reply message?

At present, a similar problem may occurs in a network with 1 ZC（ZNP） and 20 routes, but the message with the problem is not the Node_Desc_req.
We saw that after the coordinator received a ZCL message from an invalid neighbor device, the coordinator did not send the data to the Host through the serial port.

Thanks,

Howjie

Thanks for the details Howjie, I am continuing to look into this.

Regards,
Ryan

Hello Howjie,

I apologize for the delay in response, thank you for your patience.  Upon further review, there are some fundamental differences between your network topology and SWRA650 which should be reflected in the configuration settings.  The setup of SWRA650 consists of one ZC, 12 ZR, and 62 ZED.  Since your network is practically all ZR, there are several more routing options that must be considered and routing variables should be increased as such, as shown in the Routing Settings Quick Reference section of the Z-Stack User's Guide.  You may want to consider increasing MAX_RTG_SRC_ENTRIES or using CONCENTRATOR_ROUTE_CACHE.  Broadcast storms certainly appear to exist on your network.  I see that the concentrator is sending a MTO request every MTO_RREQ_LIMIT_TIME instead of CONCENTRATOR_DISCOVERY_TIME.  This shows multiple broken routes existing which could be verified by further monitoring ZDO_ManytoOneFailureIndicationCB.

Concerning the sniffer log, as the ZC does not appear to have stored a route path to the 0x7C0E node it is providing a direct Route Reply as a last effort to complete the Route Request.  If this error can be detected on the ZC then it could choose to carry out a Zstackapi_DevNwkRouteReq to determine the best path to the device in question.  You can also try staggering device joins so that the ZNP does not have to process multiple joins around the same time.  

Regards,
Ryan

Hi  Ryan,

Thanks for you reply.

In the sniffer file, I saw that you said that you frequently send MTO route requests. Is the MTO route request here triggered automatically after the concentrator receives the Router Error?

"Concerning the sniffer log, as the ZC does not appear to have stored a route path to the 0x7C0E node it is providing a direct Route Reply as a last effort to complete the Route Request."
According to the specification, once the device responds or receives Route Reply, The path will be established and stored. If my understanding is wrong, you can correct it at any time.

Is there a way to control whether the router sends beacons to avoid excessive Beacon in the network? We tried to handle it in MAC_CbackEvent, but it doesn't seem feasible.

Regards,

Howjie

You could choose different logic inside ZDO_ManytoOneFailureIndicationCB other than RTG_MTORouteReq (or disable as a test), increase MTO_RREQ_LIMIT_TIME, and/or define CONCENTRATOR_ROUTE_CACHE.

Regards,
Ryan

Hi Ryan,

When the device received the Route Error message, we processed the abnormal path, and it seemed that there was some improvement.

At present, we want to try to selectively control whether the router sends BEACON messages to reduce the traffic on the current channel.
Where can I receive the Beacon Request message in the SDK and can I intervene to respond to it?

Thanks,

Howjie

I'm glad to hear that you've been able to improve on your current system implementation.  Unfortunately Z-Stack does not provide the ability for users to intercept and/or modify the response to Beacon Requests.

Regards,
Ryan

ZDO messages can't support MT_AF_DATA_CONFIRM.

ZNwkNoRoute? The ZNP might be routing-busy. If ZNP is sending ZDP-commands to multiple target, ZNP is in routing-busy status.  

Hi Lord,

We obtain and process the data confirm of the ZDO message in afDataConfirm().

We check and confirm that the current state of the device does not have a corresponding communication path in the relevant table.

Thanks,

Howjie

Will you consider adding this feature interface to users in the future, so that users can make their products more competitive?

Thanks,

Howjie

I've passed this feedback along to the Software Development Team before, but without any further action being taken.

Regards,
Ryan

It is not meaning that Data-Confirm of ZDO message needs to be processed, but it needs to be waited. ZNP wait Data-Confirm before next ZDO Message sent.

you can try my fixed SDK, the Data Confirm message of ZDO can be set as a callback function.

https://gitee.com/zigbee_luo/simplelink_zstack_sdk.git

Thanks，We will look at your modified project later.

If there are new news about this function, can you inform us? the function is very needed for us.

Thanks,

Howjie

I was reminded from a previous E2E thread that Appendix D.9.1.5 of the Zigbee 2017 PRO Specification that "All coordinators/routers SHALL respond to Beacon Requests with a Beacon."  To not do so would be violating the Zigbee Specification.

Regards,
Ryan

If it is only used as an optional feature, ZStack is still compliant with the specification by default.

Is this function easy to implement in ZStack?

If it can be achieved, will it be beneficial to the solution of the problem if the business is involved? Because this feature allows our products to have a better user experience.

Thanks,

Howjie

I've sent you a Friendship Request.

Regards,
Howjie