CC2530: Device Stops sending Write attributes

SDB@23

Intellectual 340 points

Part Number: CC2530
Other Parts Discussed in Thread: Z-STACK

Hi,

Revisited a problem found in Z-stack 2.6.1 (Existing deployments)

Setup : 9 routers, 1 Master router writes a attribute value to 8 other routers periodically

Approach 1: Multicast was used

Problem : Master would stop Writing attributes after 'X' times and eventually crash

Approach 2: Unicast was used

Problem : Master would stop Writing attributes after 'Y' times and eventually crash

Sequence of crash:

i) Link status stopped

ii) Write attribute stopped

iii) Report attribute stopped

iv) All communication stopped

As per rough calculation: (with 9 routers)

ZCL_CMD_WRITE was used

disabledefaultrsp = FALSE (0)

After 70 Multicast the device stopped writing attributes

After 530 Unicast the device stopped writing attributes

The only thing that was common with both these cases were the number of write responses received by the Master

i.e 70 (multicast) x 8 = 560

```````````````````````````530 (unicast)

Instead if we have only 4 routers instead of 8, then the number of successful write attributes increased to approx '2X'

Based on this observation another test was conducted with three routers, out of which 1 router (Master) writes a attribute to 2 other routers with and without write responses

The write attribute was done every 10 seconds

TEST 1: Unicast without write attribute response

i.e ZCL_CMD_WRITE_NO_RSP was used

disabledefaultrsp = TRUE

The problem was not noticed in this setup, for 2006 unicast

Then we stopped the test

TEST 2: Unicast with the write attribute response

i.e ZCL_CMD_WRITE was used

disabledefaultrsp = FALSE

The firmware crashed in this case

The following are the details

1) After 450** unicast by master, i.e 450 write attribute responses, it stopped sending link status (** appoximate)

2) After 480** unicast by master, i.e 480 write attribute responses, it stopped sending write attributes ( ** appoximate)

3) After 2720 periodic report attribute (not write attributes) reports to coordinator the node

We doubt that the problem is caused due to the Write attribute responses received by Master,

1) zcl_SendWriteRequest returns 0x10 (ZMemError) on failure, which indicates memory leak

How can we resolve this issue in Z Stack 2.6.1 since we know the program flow in this case ?

2) How can we make sure that in Z Stack 3.0.2 this problem does not occur before migrating ?

Regards,

SDB@23

over 5 years ago

0 YiKai Chen over 5 years ago

Guru 735695 points

zcl_SendWriteRequest returns 0x10 (ZMemError) is usually caused by application not managing memory well or keep sending too many requests in short period. I would suggest you to check your application first.

0 Ryan Brown1 over 5 years ago

TI__Guru**** 220087 points

Hi SDB@23,

I agree that this appears to be a memory leak. Can you please compare the contents of zcl.c between Z-Stack 2.6.1 and Z-Stack 3.0.2, particularly zcl_SendWriteRequest and zcl_SendCommand? Z-Stack 2.6.1 has been deprecated for several years and this bug has most likely been resolved alongside others for the Z-Stack 3.0.2 release, at least it has not been observed recently or recorded on the Known Issues E2E page.

Regards,
Ryan

0 SDB@23 over 5 years ago in reply to YiKai Chen

Intellectual 340 points

We dont do so many transmissions at any given point in our application

The problem was being caused only when Write Attribute responses were enabled, Write attribute responses are handled in zcl_HandlerExternal function

In the cmd table, when we used NULL instead of the function zcl_HandleExternal function,

ZCL_CMD_WRITE and disabledefaultrsp = 0

The node is working properly

We suspect some issue in the zcl_HandleExternal function

0 SDB@23 over 5 years ago in reply to Ryan Brown1

Intellectual 340 points

In zcl_SendCommand

APS counter is used instead of zcl_TransID in 3.0.2

There is no change in zcl_SendWriteRequest function from 2.6.1 in 3.0.2

zcl_HandleExternal function

2.6.1

The message is sent to zcl_RegisteredTaskID, it would go to zll_target_event_loop

In 3.0.2

Task ID is obtained from zcl_getExternalFoundationHandler and the message is sent to the zcl_samplelight_event loop

I see there is a difference in how the response is treated in z stack 2.6.1 and z stack 3.0.2 function zcl_ProcessIncomingMsg is introduced

Will try to implement the same in z stack 3.0.2 soon

Is there any problem in after the message is formed in zcl_HandlerExternal and sent to zclRegisteredTaskID in 2.6.1

0 Ryan Brown1 over 5 years ago in reply to SDB@23

TI__Guru**** 220087 points

This is great debugging progress, it appears that an allocated message is not being de-allocated as expected for a memory leak issue. I cannot provide further comments on the capabilities of the deprecated stack. Please be sure to update this thread with any further information that is uncovered.

Regards,
Ryan

0 SDB@23 over 5 years ago in reply to Ryan Brown1

Intellectual 340 points

The reason for the memory leak was spotted and the problem was fixed :

1) zll_target.c

zllTarget_Init function

zcl_registerForMsg( zllTarget_TaskID );

Here the zcl messages are registered to zllTarget_TaskID

2) zcl.c

zcl_HandleExternal function

/* send message through task message */
osal_msg_send( zcl_RegisteredMsgTaskID, (uint8 *)pCmd );

Here the message is formed and sent to the zcl_RegisteredMsgTaskID which is zlltarget_TaskID

3) zll_target.c

zllTarget_event_loop

Here any messages from ZCL_INCOMING_MSG is not handled, thus by default the message is getting deallocated in the function

4) The attrCmd which was allocated memory in zcl_ProcesMessageMsg(), was supposed to be cleared, but was not cleared in the zll_target_event

leading to a memory leak

5) By freeing the attrCmd in the zll_target_event for ZCL_INCOMING_MSG, the leak can be stopped

osal_mem_free( pInMsg->attrCmd );

Thanks Ryan and YK for the support

Regards,

SDB@23

Zigbee & Thread

Zigbee & Thread forum

CC2530: Device Stops sending Write attributes