RTOS/CC2538: Z-Stack 3.0.2 fatal bug report whenever join the coordinator network(only Association request and Response success,join network not successful)

behold li

Part Number: CC2538
Other Parts Discussed in Thread: Z-STACK

Tool/software: TI-RTOS

@Ryan Brown1
@JasonB

We are in urgent need of help.

All CC2538 device use the newest Z-Stack 3.0.2(include coordinator and router,end-device),capture all OTA packet with newest Ubiqua.

1. When there are more than 20 routers around the coordinator, there is no signal saturation at the right distance. End-device has been tried many times. It has been beacon request, Association Request, Association Response: Success, but there is nothing behind the key transports. It has been repeated all the time. It is impossible to joined the network successful, cut off all router power supply and test again, except the coordinator. and end-device, access to joined the network immediately succeeded, in other words, can not joined the network in the "key-tunnel" way,I don't know why coordinator not issue key transport whenever it send Association Response Success?

2.The above steps will lead to internal abnormalities in the coordinator protocol stack (as the coordinator no longer issue control instructions at all, test the coordinator issue permit join, also can not come out, but each time the coordinator receives data from other routers, it has the Acknowledgement response, indicating that it did receive truely, but the routing is not feasible, all router issue Route Failure occurs, and eventually the upper gateway of the coordinator has to be restarted (the coordinator's issue MT command to ZNP executes the reset command) ,the network of coordinator will resume.

I can confirm that this is the bug. It has been repeated at least a dozen times.

over 5 years ago

0 Toby Pan over 5 years ago

TI__Mastermind 28440 points

Hi,

Are you running the examples without any local edits? Which example(s) are you using?
When both ZC and ZED started from FN, does the ZC transport key to ZED?
By default, the examples use APS key transport for the TC link key exchange.

Please send the logs for further analysis.

Regards,
Toby

0 Jesse Huang over 5 years ago

Intellectual 780 points

I am also concerned about this issue.

0 behold li over 5 years ago in reply to Jesse Huang

Expert 2745 points

I think this is a fatal bug internal z-stack even newest 3.0.2,please JasonB confirm it

0 YiKai Chen over 5 years ago in reply to behold li

Guru 735455 points

Can you attach your sniffer log?

0 Ryan Brown1 over 5 years ago in reply to behold li

TI__Guru**** 202256 points

behold,

Please provide the information requested in order to receive further support for your issue. I also highly recommend that you try increasing the nwk_globals.h/c macros affecting the maximum number of devices and data buffers. processors.wiki.ti.com/.../Zigbee_Known_Issues_and_Proposed_Fixes

Regards,
Ryan

0 behold li over 5 years ago in reply to Ryan Brown1

Expert 2745 points

ubiqua capture is only have Association Request and Response when join the network failure,no more packet be capture over the air(include Transport Key ,Request Key and Verify Key etc.),I think even upload sniffer log not be helpful.

0 YiKai Chen over 5 years ago in reply to behold li

Guru 735455 points

Oops....

0 Toby Pan over 5 years ago in reply to behold li

TI__Mastermind 28440 points

Can you confirm whether Ryan's suggestion works for you?
It would also be helpful to know what edits you've made to the macros (specifically the macros in nwk_globals.h).

0 behold li over 5 years ago in reply to Ryan Brown1

Expert 2745 points

What is difference NWK_MAX_DATABUFS_WAITING and MAC_CFG_TX_DATA_MAX in *.cfg file?

0 Toby Pan over 5 years ago in reply to behold li

TI__Mastermind 28440 points

They define the size of queues/buffers used by the corresponding layer.

NWK_MAX_DATABUFS_WAITING is the number of entries in the buffer that's used for sending frames from the NWK layer to the MAC layer.

MAC_CFG_TX_DATA_MAX is the number of entries in the transmit queue that's used by the MAC layer for holding frames that are transmitted to sleepy end devices.

0 behold li over 5 years ago in reply to Ryan Brown1

Expert 2745 points

hi Ryan:

Attachment is ubiqua capture log not join network successful:

Customer TCLK: 5A:69:67:42:65:65:41:6C:6C:69:61:6E:63:65:30:3A

Customer network key: 01:02:03:08:09:32:78:86:01:02:03:08:09:32:78:86

not join network successful.zip

0 Ryan Brown1 over 5 years ago in reply to behold li

TI__Guru**** 202256 points

Hi behold,

The number of routers directly connected to the ZC is very near the default value of MAX_NEIGHBOR_ENTRIES, please increase this as well as NWK_MAX_ROUTERS, MAX_RTG_SRC_ENTRIES, and the max buffer values previously mentioned then report any observed changes.

Regards,
Ryan

0 behold li over 5 years ago in reply to Ryan Brown1

Expert 2745 points

Hi Ryan:

I have been try the following steps and diagnostic this bug continually:

1. modify nwk_globals.h

#if !defined( NWK_MAX_ROUTERS )
#define NWK_MAX_ROUTERS           6
#endif

2. modify nwk_globals.c

/*********************************************************************
 * CONSTANTS
 */
// Maximums for the data buffer queue
#if !defined( NWK_MAX_DATABUFS_WAITING )
#define NWK_MAX_DATABUFS_WAITING    8     // Waiting to be sent to MAC
#endif
   
#if !defined( NWK_MAX_DATABUFS_SCHEDULED )
#define NWK_MAX_DATABUFS_SCHEDULED  5     // Timed messages to be sent
#endif
   
#if !defined( NWK_MAX_DATABUFS_CONFIRMED )
#define NWK_MAX_DATABUFS_CONFIRMED  5     // Held after MAC confirms
#endif
   
#if !defined( NWK_MAX_DATABUFS_TOTAL )
#define NWK_MAX_DATABUFS_TOTAL      12    // Total number of buffers
#endif

3. based CC2538ZNP project,modify znp.cfg as following

/* purpose of many to one */
-DINT_HEAP_LEN=3280
-DNWK_MAX_DEVICE_LIST=50
-DCONCENTRATOR_ENABLE=TRUE
-DCONCENTRATOR_DISCOVERY_TIME=120
-DMAX_RTG_SRC_ENTRIES=430
-DSRC_RTG_EXPIRY_TIME=2
-DCONCENTRATOR_RADIUS=15
-DCONCENTRATOR_ROUTE_CACHE=TRUE
-DMTO_RREQ_LIMIT_TIME=5000
-DLINK_DOWN_TRIGGER=12
-DNWK_ROUTE_AGE_LIMIT=30
-DBCAST_DELIVERY_TIME=100
-DDEF_NWK_RADIUS=15
-DDEFAULT_ROUTE_REQUEST_RADIUS=8
-DROUTE_DISCOVERY_TIME=13
-DZDNWKMGR_MIN_TRANSMISSIONS=0
-DNWK_LINK_STATUS_PERIOD=30

/* purpose of facilitate join network successful */
-DMAX_NEIGHBOR_ENTRIES=430
-DNWK_MAX_ROUTERS=430
-DNWK_MAX_DATABUFS_WAITING=16
-DNWK_MAX_DATABUFS_SCHEDULED=10
-DNWK_MAX_DATABUFS_CONFIRMED=10
-DNWK_MAX_DATABUFS_TOTAL=24

0 YiKai Chen over 5 years ago

Guru 735455 points

Try to increase the number of network layer data buffers in nwk_globals.c to test again.

// Maximums for the data buffer queue
#define NWK_MAX_DATABUFS_WAITING 16 //8 // Waiting to be sent to MAC
#define NWK_MAX_DATABUFS_SCHEDULED 10 //5 // Timed messages to be sent
#define NWK_MAX_DATABUFS_CONFIRMED 10 //5 // Held after MAC confirms
#define NWK_MAX_DATABUFS_TOTAL 24 //12 // Total number of buffers

0 behold li over 5 years ago in reply to Ryan Brown1

Expert 2745 points

Hi Ryan:

The newest test again,attachment ubiqua capture 736 packet(recall scene) caused coordinator(CC2538ZNP) network layer crashed,coordinator can't send out any control command even permit join request etc.

caused coordinator network layer crash.zip

0 Ryan Brown1 over 5 years ago in reply to behold li

TI__Guru**** 202256 points

Hi behold,

You could be having memory issues due to some macro value options. For example, the last we see from the ZC is #656 after adding an end device and yet in the settings above NWK_MAX_DEVICES < NWK_MAX_ROUTERS and therefore gNWK_MAX_SLEEPING_END_DEVICES is less than zero. Maybe try:

-DINT_HEAP_LEN=6144
-DNWK_MAX_DEVICE_LIST=60
-DSRC_RTG_EXPIRY_TIME=10
-DZDNWKMGR_MIN_TRANSMISSIONS=10
-DMAX_NEIGHBOR_ENTRIES=50
-DNWK_MAX_ROUTERS=50

Regards,
Ryan

0 behold li over 5 years ago in reply to Ryan Brown1

Expert 2745 points

Hi Ryan:

I will test continue.

Zigbee & Thread

Zigbee & Thread forum

RTOS/CC2538: Z-Stack 3.0.2 fatal bug report whenever join the coordinator network(only Association request and Response success,join network not successful)