CC2530: Rejoin failure capture analysis

Hi Peter,

Can you please provide the NWK key?  Otherwise the packets in the sniffer log are encrypted and thus difficult to decipher.  Which short address is the ZED node in question?  I recommend that you review the Known Issues and Fixes E2E post since there are some frame counter issues which could affect your device's rejoin behavior.  Otherwise, please further debug your device within your IDE to gather more clues about MCU behavior during the problem.

Regards,
Ryan

Thanks Ryan,

sorry. Here are the keys I have configured  in Wireshark:

5A:69:67:42:65:65:41:6C:6C:69:61:6E:63:65:30:39

6F:8B:C6:95:74:85:3F:39:B4:9B:DF:AC:CF:7A:9D:7E

Regarding the known issues.. yes I have seen the list. That's why I was curious if it can be seen from the capture..

It is quite complicated to debug since it happens after several days, randomly..

But I have some diagnostic print on display and device is in BDB_COMMISSIONING_PARENT_LOST with status BDB_COMMISSIONING_NO_NETWORK

And device state is DEV_NWK_ORPHAN which seem to sometimes change to DEV_NWK_DISC..

Any hint would be highly appreciated.

Thank you.

..and yes.. the device in question is 0x0B7F. One strange thing is that it reports manually once in an hour by 

zclStatusPanel_DstAddr.addrMode = (afAddrMode_t)Addr16Bit;
zclStatusPanel_DstAddr.addr.shortAddr = 0;
zclStatusPanel_DstAddr.endPoint=1;

zcl_SendReportCmd(....)

and somehow.. even the ZED is not on the network obviously and cannot receive WRITE commands.. the coordinator is able to receive such reports..

Probably the device has a orphan state but NWK material in NV memory is intact allowing to send reports to hardcoded address..

I'm really kind of confused...

Thanks, the first key is the global TCLK and the second was the NWK key I needed.

The ZC continuously sends back Rejoin Responses with status of Successful, thus ZDApp should receive a ZDO_JoinConfirmCB.  Are you able to rejoin successfully before the issue occurs?  Then can also debug bdb_rejoinNwk of a successful rejoin and compare against a failing instance.

How did you migrate the application from Z-Stack HA 1.2 to 3.0?  Are you experiencing this same issue when using a default TI example?  Are you able to read/monitor your NV entries, and have they changed any between working and broken states?

Regards,
Ryan

Hi Ryan,

thanks for the tips. I will try to address them one by one.. I haven't done it recently but in the past I noticed that in suuccessful rejoins coordinator replied with rejoin response in much shorter interval.. Usually when it fails, the response takes som time for some reason.. I was thinking if there is theoretically chance that ZED already turned off receiver after some timeout and didn't catch it? Or is the following MAC ACK sent by ZED as a response? I can see they have the same sequence number, so this is probably not the case.. So as it seems coordinator considers the device as joined but ZED refuses to accept rejoin response..

I need to experiment with forced rejoins to see the healthy procedure, which is not that easy in my network..

Thanks a lot for analysis an tips.. 

I will come back with my findings if I have any..

Hmmm.. very good tip Ryan. I have a bad feeling that the problem goes that direction.. I will investigate further.. I cannot thank you enough for your support. 

When I analyzed longer sniffer capture.. I can see that ZED sends Data Request usually around 2500ms after Rejoin Request.. it is kind of a pattern.. This is pretty strange, because my settings in f8wconfig are:

-DRESPONSE_POLL_RATE=100

-DREJOIN_POLL_RATE=440

-DPOLL_RATE=10000

So .. none of it appears to be any near the used value...

After Data Request, the coordinator replies promptly, within few ms ..

With such behavior, the theory of NWK_REJOIN_TIMEOUT_EVT is getting more and more probable..

Now I need to find from where the damn strange poll rate is set..

Do you add those patches in the Known Issues and Fixes E2E post ?

Hi YiKai,

not yet.. I wanted to understand what's going on, before I possibly introduce additional unexpected problems with the fixes.. But if I don't find anything meaningful by my analysis I will try to implement fixes and go the "trial and error" method..

thank you.

One thing that is worth mentioning, that this behavior can be observed even after device restart.. so it should not be related to some memory leaks, resources exhaustion or something like that..

Further debug your application to determine the poll rate set to evaluate the DEV_NWK_DISC devState which will call bdb_rejoinNwk if bdb_isDeviceNonFactoryNew returns true.  You could try forcing NLME_SetPollRate(zgRejoinPollRate) before NLME_ReJoinRequest to observe whether this changes behavior.

Regards,
Ryan

Hi guys, I have some first results from debugging. It seems that the ZED is able to move among coordinator and routers and rejoin the network without any problems while on debugger for several hours. Rejoin poll rate is indeed 440ms and it can be observed also in sniffer and everything works fine. So I'm wondering how it can go wrong after several days of usage.. I will let the ZED running on debugger and see if I can reproduce the problem. In the past I was able to reproduce it always after around 3 days.. So now I'm wondering.. is it possible that something overwrites the memory? Some overflow which rewrites zgRejoinPollRate or NVRAM gets somehow corrupted? So far, it is quite mysterious problem..

It may be worth enabling OSALMEM_METRICS to make sure that there is no memory leakage leading up to the issue, even though the behavior is observable immediately after resets.  You could also try to monitor ZCD_NV_REJOIN_POLL_RATE and reading NV memory before/after the issue occurs.  Are you able to replicate this problem with a default TI example and what is the power supply (battery) and voltage level of your ZED?  Based on the description it is possible that NV memory corruption is involved.  I'm suspicious of low voltages from the battery causing problematic NV memory writes, or flash writes performed by the application.

Regards,
Ryan

Thank you Ryan, I will keep investigating with all your tips. I will publish here as soon as I come to any resolution. It would be such a nice open source project if it succeed.. It displays temperature from indoor and outdoor sensor which is updated by Write cmd once in 10 minutes and it also displays immediate warning icons about opened windows, smoke alarm, flood, etc.. All that on 2.9 eink display. Something like a status display, you wanna check when you leave home.. all of that in nice SLA printed custom enclosure.. wouldn't it be a shame if it fails on such problem...:(

Very strange. I let the ZED running with slight changes in logging outputs. I implemented low battery callback and in case of orphan state printing of zgRejoinPollRate on the display. What a surprise.. again after approx 3 days, device is in error state, not able to rejoin. Sending Data request after Rejoin request again after 2500ms. But.. NO low battery warning and display prints 440ms as a value of zgRejoinPollRate... I don't get it :( It has to be probably some problem with nvram values, since the problem persist after restart and only reflashing helps..

btw.. could it be coincidence that the delay is the same as TOUCHLINK_INITIATOR_REJOIN_TIMEOUT..?

I agree that this appears to be an issue with NV flash memory.  Would it be possible to print other NV values to determine whether there are any inconsistencies outside of the expected value?  Have you attempted any of the known issue fixes recommended by YK or myself?  Are there any flash writes performed by the application?  I expect TOUCHLINK_INITIATOR_REJOIN_TIMEOUT is a coincidence given that this code should not be compiled if BDB_TL_INITIATOR is not defined.

Regards,
Ryan

Thanks Ryan for additional suggestions. My application does not directly read or write to NV.. I have basically only stripped version of genericApp.. only handling of e-ink is added. I examined the recommended fixes, but I didn't find any, which would seem to fix the problem I'm experiencing. For sure I will implement them later, but I didn't want to introduce additional bugs or strange behavior. Currently I'm experimenting with comparison of NV values before and after the problem happens.. so hopefully I will find some differences... I'm really quite curious ... Thanks for your support.

You can also keep track of the heap with OSALMEM_METRICS for more debugging assistance (unfortunately most likely not the issue at hand given that a reset does not fix the issue).  Since the issue appears to concern NV writes perhaps it would be possible to greatly reduce the value of MAX_APS_FRAMECOUNTER_CHANGES for debug purposes to cause the issue to exist much quicker than it is currently (i.e. hours instead of days).

Regards,
Ryan

Hi Ryan, thanks for the tip with MAX_APS_FRAMECOUNTER_CHANGES.. I will have a look at it.. But just now I experienced the very new effect. I implemented some additional printouts to display regarding NV values.. But what was shocking for me, is that now the issue appeared just a few minutes after I inserted batteries. Now the delay between RejoinRequest and DataRequest is approx. 2800ms.. In the past it was usually 2500ms.. So far the NV related values printed on the display does not seem altered.. So.. It does not make much sense.. I would say memory, because I extended printing and I needed to import additional font.. but restart does not help.. Anyway I will keep investigating.. Maybe I will try at least the fix with "XDATA stack" or MAXMEMHEAP.. Maybe with the e-ink support, the device is so limited in resources that under some circumstances it starts such strange behavior..

Hi guys, it seems I'm able to reproduce the behavior. It seems that defining more data in the code, e.g. fonts in byte arrays.. leads to such strange problem..I use only single 300 byte array as an operational buffer for graphics, but icons and fonts are defined as constant arrays.. I would expect they occupy data part of memory..I'm kind of lost in this situation. What limitations I should take into account? Should I adapt any settings regarding xdata? What would you recommend for my scenario? Do you think that such structures as larger arrays could cause such instability.. I don't understand why restart does not help.. but maybe the device state requires more resources which triggers the problem..

Hi Peter,

I a agree that constants should occupy flash memory, however the 300 byte buffer could greatly affect RAM as XDATA stack and INT_HEAP_LEN sizes are marginal by default.  Please consider increasing them to observe whether this changes behavior, as well as removing any unnecessary code which could affect RAM usage (review SWRA635 in this regard).  Any overflow could result in unexpected device behavior, and perhaps the unique combinations of factors results in a RAM violation while attempting to modify the NV flash memory region.

Regards,
Ryan

Hi Ryan,

It seems I need to study a lot.. I found a lot of different discussions on similar topics, regarding memory models, optimizations, etc.. e.g. https://e2e.ti.com/support/wireless-connectivity/zigbee-thread-group/zigbee-and-thread/f/zigbee-thread-forum/93583/help-the-input-param-of-one-function-was-changed-when-running-timac-cc2530-1-3-0

But to be honest.. My knowledge in this area is rather limited, so I need to catch on.. From what I've seen so far, also the large constants could cause issues and require attention.. and in my case, it seems that adding more const arrays e.g. with font related stuff worsen the situation, even though, that RAM buffer remains the same and I don't allocate more memory directly. So it seems that the constants themselves are problematic.. Currently I have no idea what are my limits and what changes should I make in the project settings to optimize that.. so I need to get some documentation and understand the things much more in depth.

Hi guys, could you please confirm my understanding of constants mapping between Flash and XDATA? In f8w2530.xcl I found that the constants from Flash are mapped to XDATA instead of copying them.. I'm not sure I understand the ranges correctly, but it seems to me that 4000B of XDATA (RAM) is reserved for such constants? If this is true, I counted, that only fonts and icons in my scenario which fails after few minutes take around 3700 B and I'm not counting other constants in the program.. So is my assumption correct, that my constants could overflow somehow in RAM? Or am I totally wrong? Can I consider 4000B as a limit for my constants? Or if not, what is the limit? Thank you!

Can you please provide your f8w2530.xcl file or tell me where it is different from the default one I've provided below?

/**************************************************************************************************
  Filename:       f8w2530.xcl
  Revised:        $Date: 2014-05-17 12:12:11 -0700 (Sat, 17 May 2014) $
  Revision:       $Revision: 38578 $

  Description:    This is a linker command line file for the IAR XLINK tool for the
                  CC2530 SoC and Z-Stack sample applications where the General Options
                  for location for constants and strings is "ROM mapped as data".


  Copyright 2009-2014 Texas Instruments Incorporated. All rights reserved.

  IMPORTANT: Your use of this Software is limited to those specific rights
  granted under the terms of a software license agreement between the user
  who downloaded the software, his/her employer (which must be your employer)
  and Texas Instruments Incorporated (the "License").  You may not use this
  Software unless you agree to abide by the terms of the License. The License
  limits your use, and you acknowledge, that the Software may not be modified,
  copied or distributed unless embedded on a Texas Instruments microcontroller
  or used solely and exclusively in conjunction with a Texas Instruments radio
  frequency transceiver, which is integrated into your product.  Other than for
  the foregoing purpose, you may not use, reproduce, copy, prepare derivative
  works of, modify, distribute, perform, display or sell this Software and/or
  its documentation for any purpose.

  YOU FURTHER ACKNOWLEDGE AND AGREE THAT THE SOFTWARE AND DOCUMENTATION ARE
  PROVIDED “AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,
  INCLUDING WITHOUT LIMITATION, ANY WARRANTY OF MERCHANTABILITY, TITLE,
  NON-INFRINGEMENT AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT SHALL
  TEXAS INSTRUMENTS OR ITS LICENSORS BE LIABLE OR OBLIGATED UNDER CONTRACT,
  NEGLIGENCE, STRICT LIABILITY, CONTRIBUTION, BREACH OF WARRANTY, OR OTHER
  LEGAL EQUITABLE THEORY ANY DIRECT OR INDIRECT DAMAGES OR EXPENSES
  INCLUDING BUT NOT LIMITED TO ANY INCIDENTAL, SPECIAL, INDIRECT, PUNITIVE
  OR CONSEQUENTIAL DAMAGES, LOST PROFITS OR LOST DATA, COST OF PROCUREMENT
  OF SUBSTITUTE GOODS, TECHNOLOGY, SERVICES, OR ANY CLAIMS BY THIRD PARTIES
  (INCLUDING BUT NOT LIMITED TO ANY DEFENSE THEREOF), OR OTHER SIMILAR COSTS.

  Should you have any questions regarding your right to use this Software,
  contact Texas Instruments Incorporated at www.TI.com.

**************************************************************************************************/

////////////////////////////////////////////////////////////////////////////////
//
//
// Segment limits
// --------------
//
//
// XDATA available to the program.
//
// Reserving address 0x0 for NULL.
-D_XDATA_START=0x0001
-D_XDATA_END=0x1EFF
//
//
// The 8052 IDATA is overlayed on the SoC XDATA space from 0x1F00-0x1FFF.
//
-D_IDATA_END=0xFF              // Last address of Idata memory.
//
//
//    CODE
//
-D_CODE_START=0x0000
-D_CODE_END=0x7FFF             // Last address for ROOT bank.
//
-D_FIRST_BANK_ADDR=0x10000
//
//
//
// Special SFRs
// ------------
//
//    Register bank setup
//
-D?REGISTER_BANK=0             // Default register bank (0,1,2,3).
-D_REGISTER_BANK_START=0       // Start address for default register bank (00,08,10,18).
//
//
//    PDATA page setup
//
-D?PBANK_NUMBER=00             // High byte of 16-bit address to the PDATA area.
//
//
//    Virtual register setup
//    ----------------------
//
-D_BREG_START=0x00             // The bit address where the BREG segments starts.
                               // Must be placed on: _BREG_START%8=0 where _BREG_START <= 0x78.
-D?VB=0x20                     // ?VB is used when referencing BREG as whole byte.
                               // Must be placed on: ?VB=0x20+_BREG_START/8.
//
////////////////////////////////////////////////////////////////////////////////



////////////////////////////////////////////////////////////////////////////////
//
// IDATA memory
//

// Setup "bit" segments (only for '__no_init bool' variables).
-Z(BIT)BREG=_BREG_START
-Z(BIT)BIT_N=0-7F

-Z(DATA)REGISTERS+8=_REGISTER_BANK_START
-Z(DATA)BDATA_Z,BDATA_N,BDATA_I=20-2F
-Z(DATA)VREG=08-7F
-Z(DATA)PSP,XSP=08-7F
-Z(DATA)DOVERLAY=08-7F
-Z(DATA)DATA_I,DATA_Z,DATA_N=08-7F

-U(IDATA)0-7F=(DATA)0-7F
-Z(IDATA)IDATA_I,IDATA_Z,IDATA_N=08-_IDATA_END
-Z(IDATA)ISTACK+_IDATA_STACK_SIZE#08-_IDATA_END
-Z(IDATA)IOVERLAY=08-FF

////////////////////////////////////////////////////////////////////////////////
//
// ROM memory
//
//
// The following segments *must* be placed in the root bank. The order of
// placement also matters for these segments, which is why we use the -Z
// placement directive.
//
-Z(CODE)INTVEC=_CODE_START
-Z(CODE)BIT_ID,BDATA_ID,DATA_ID,IDATA_ID,IXDATA_ID,PDATA_ID,XDATA_ID=_CODE_START-_CODE_END
//
// Sleep PCON instruction must be 4-byte aligned.
//
-D_SLEEP_CODE_SPACE_START=(_CODE_END-7)
-D_SLEEP_CODE_SPACE_END=(_CODE_END)
-Z(CODE)SLEEP_CODE=_SLEEP_CODE_SPACE_START-_SLEEP_CODE_SPACE_END
//
// The following segments *must* be placed in the root bank, but the order
// of placement within the root bank is not important, which is why we use the
// -P directive here.
//
-P(CODE)CSTART,BANK_RELAYS,RCODE,DIFUNCT,NEAR_CODE=_CODE_START-_CODE_END
//
// Setup for constants located in code memory:
//
-P(CODE)CODE_C=_CODE_START-_CODE_END
//
// Define segments for const data in flash.
// First the segment with addresses as used by the program (flash mapped as XDATA)
-P(CONST)XDATA_ROM_C=0x8000-0xFFFF
//
// Then the segment with addresses as put in the hex file (flash bank 1)
-P(CODE)XDATA_ROM_C_FLASH=0x18000-0x1FFFF
//
// Finally link these segments (XDATA_ROM_C_FLASH is the initializer segment for XDATA_ROM_C,
// we map the flash in the XDATA address range instead of copying the data to RAM)
-QXDATA_ROM_C=XDATA_ROM_C_FLASH
//
// The directive below ensures that the remaining space in the root bank gets
// filled, then starts filling the banks.
//
-P(CODE)BANKED_CODE=_CODE_START-_CODE_END,0x18000-0x1FFFF,0x28000-0x2FFFF,0x38000-0x3FFFF,\
0x48000-0x4FFFF,0x58000-0x5FFFF,0x68000-0x6FFFF,0x78000-0x7F7FF

////////////////////////////////////////////////////////////////////////////////
//
// XDATA memory
//

-Z(XDATA)XSTACK+_XDATA_STACK_SIZE=_XDATA_START-_XDATA_END
-Z(XDATA)XDATA_Z,XDATA_I=_XDATA_START-_XDATA_END
-P(XDATA)XDATA_N=_XDATA_START-_XDATA_END

-cx51

////////////////////////////////////////////////////////////////////////////////
//
// Texas Instruments device specific
// =================================
//
//
// Layout of CODE banks
// -------------------
//
//-D_BANK0_START=0x08000
//-D_BANK0_END=0x0FFFF
//
//-D_BANK1_START=0x18000
//-D_BANK1_END=0x1FFFF
//
//-D_BANK2_START=0x28000
//-D_BANK2_END=0x2FFFF
//
//-D_BANK3_START=0x38000
//-D_BANK3_END=0x3FFFF
//
//-D_BANK4_START=0x48000
//-D_BANK4_END=0x4FFFF
//
//-D_BANK5_START=0x58000
//-D_BANK5_END=0x5FFFF
//
//-D_BANK6_START=0x68000
//-D_BANK6_END=0x6FFFF
//
//-D_BANK7_START=0x78000
//-D_BANK7_END=0x7FFFF
//
//
// Include these two lines when generating a .hex file for banked code model:
-M(CODE)[(_CODEBANK_START+_FIRST_BANK_ADDR)-(_CODEBANK_END+_FIRST_BANK_ADDR)]*\
_NR_OF_BANKS+_FIRST_BANK_ADDR=0x8000

// When -M is used to build debug output, XLINK gives warning [w69] which is safely ignored.
-ww69=i

//
//
// Internal flash used for NV address space: reserving 6 pages.
// NV memory segment size must coincide with page declarations in "hal_board_cfg.h" file.
//
-D_ZIGNV_ADDRESS_SPACE_START=(((_NR_OF_BANKS+1)*_FIRST_BANK_ADDR)-0x3800)
-D_ZIGNV_ADDRESS_SPACE_END=(_ZIGNV_ADDRESS_SPACE_START+0x2FFF)
-Z(CODE)ZIGNV_ADDRESS_SPACE=_ZIGNV_ADDRESS_SPACE_START-_ZIGNV_ADDRESS_SPACE_END
//
//
//
// The last available page of flash is reserved for special use as follows
// (addressing from the end of the page down):
//   16 bytes  Lock bits
//    8 bytes  IEEE address space (EUI-64)
//   22 bytes  Device Private Key (21 bytes + 1 byte pad to NV word size)
//   22 bytes  CA Public Key (22 bytes)
//   48 bytes  Implicit Certificate (48 bytes)
// 1932 bytes  Reserved for future Z-Stack use (1932 bytes)
//
-D_LOCK_BITS_ADDRESS_SPACE_START=(((_NR_OF_BANKS+1)*_FIRST_BANK_ADDR)-0x10)
-D_LOCK_BITS_ADDRESS_SPACE_END=(_LOCK_BITS_ADDRESS_SPACE_START+0x0F)
-Z(CODE)LOCK_BITS_ADDRESS_SPACE=_LOCK_BITS_ADDRESS_SPACE_START-_LOCK_BITS_ADDRESS_SPACE_END
//
-D_IEEE_ADDRESS_SPACE_START=(_LOCK_BITS_ADDRESS_SPACE_START-0x08)
-D_IEEE_ADDRESS_SPACE_END=(_IEEE_ADDRESS_SPACE_START+0x07)
-Z(CODE)IEEE_ADDRESS_SPACE=_IEEE_ADDRESS_SPACE_START-_IEEE_ADDRESS_SPACE_END
//
-D_DEV_PRIVATE_KEY_ADDRESS_SPACE_START=(_IEEE_ADDRESS_SPACE_START-0x16)
-D_DEV_PRIVATE_KEY_ADDRESS_SPACE_END=(_DEV_PRIVATE_KEY_ADDRESS_SPACE_START+0x15)
-Z(CODE)DEV_PRIVATE_KEY_ADDRESS_SPACE=_DEV_PRIVATE_KEY_ADDRESS_SPACE_START-_DEV_PRIVATE_KEY_ADDRESS_SPACE_END
//
-D_CA_PUBLIC_KEY_ADDRESS_SPACE_START=(_DEV_PRIVATE_KEY_ADDRESS_SPACE_START-0x16)
-D_CA_PUBLIC_KEY_ADDRESS_SPACE_END=(_CA_PUBLIC_KEY_ADDRESS_SPACE_START+0x15)
-Z(CODE)CA_PUBLIC_KEY_ADDRESS_SPACE=_CA_PUBLIC_KEY_ADDRESS_SPACE_START-_CA_PUBLIC_KEY_ADDRESS_SPACE_END
//
-D_IMPLICIT_CERTIFICATE_ADDRESS_SPACE_START=(_CA_PUBLIC_KEY_ADDRESS_SPACE_START-0x30)
-D_IMPLICIT_CERTIFICATE_ADDRESS_SPACE_END=(_IMPLICIT_CERTIFICATE_ADDRESS_SPACE_START+0x2F)
-Z(CODE)IMPLICIT_CERTIFICATE_ADDRESS_SPACE=_IMPLICIT_CERTIFICATE_ADDRESS_SPACE_START-_IMPLICIT_CERTIFICATE_ADDRESS_SPACE_END
//
-D_RESERVED_ADDRESS_SPACE_START=(_IMPLICIT_CERTIFICATE_ADDRESS_SPACE_START-0x78C)
-D_RESERVED_ADDRESS_SPACE_END=(_RESERVED_ADDRESS_SPACE_START+0x78B)
-Z(CODE)RESERVED_ADDRESS_SPACE=_RESERVED_ADDRESS_SPACE_START-_RESERVED_ADDRESS_SPACE_END
//
////////////////////////////////////////////////////////////////////////////////

Where do you see the 4 KB (0x0FFF) limitation?  There is an area from 0x8000 to 0xFFFF defined for const data in flash, it is further described in the TRM:

The upper 32 KB of XDATA is a read-only area called XBANK (see Figure 2-1). Any of the available 32 KB flash banks can be mapped in here. This gives software access to the whole flash memory. This area is typically used to store additional constant data.

I do not have any further documentation to provide beyond what I am confident you are already aware of.

Regards,
Ryan

Hi Ryan, 

I'm sorry. That was obviously only my misunderstanding.. Ok then.. It seems that this problem is little bit over my knowledge and abilities.. I don't understand what's going on.. Maybe it is really heap issue.. Still OSALMEM_METRICS is left for me to try an experimenting with graphics buffer.. Thanks Ryan for such a great support!

Hi Ryan, it seems that the strange behavior might have been really result of RAM related problems. I completely removed the graphic buffer and now graphics is directly copied to eink module RAM and so far I haven't seen the issue..

BUT..

Now I've got new phenomenon..and I'm not sure who to blame..ED, Router or Coordinator..

Sometime I get into situation when ZED polls directly coordinator, it happens probably after switching parent from router to coordinator..

But coordinator is still trying to route WriteCommand via router, instead directly to ZED. Coordinator still thinks that ZED is behind the router, but ZED polls directly coordinator so it obviously changed parent.. as a result WriteCommand is never delivered, because it is trapped at router and since router is never polled, it never releases the message.. It seems that no participant is willing to do anything about that.. ZED happily polls coordinator and coordinator still delivers commands to router.. I have a susspicion that probably coordinator is guilty.. It is zigbee2mqtt wit zzh stick.. running 3.x.0 zstack firmware.. I'm not sure which participant shoul start some kind of repair in such situation..

Thank you.

Hi Peter,

Can you please provide a sniffer log which demonstrates this new behavior?  A ZED which sends a Rejoin Request to the ZC should trigger the association and routing tables to automatically update so that messages can be directly sent to the ZED as its child device.  Are you also aware of the SimpleLink SDK version on the CC2652R ZNP?

Regards,
Ryan