This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2650: zstack linux gateway: gateway sever goes into permanent busy state for attribute requests

Part Number: CC2650
Other Parts Discussed in Thread: Z-STACK, CC2530,

This is a question related to "CC2530: Problem with modifying Z-Stack Linux Gateway code, Gateway server doesn't answer after a while" because we also run into the problem that the gateway server stops working after minutes to hours. It then goes into a state where si_send_packet() in demo/framework/socket_interface.c returns early because waiting_for_confirmation is and remains true. The system doesn't recover.

We decided to use the »zstack linux gateway« because this was recommended by a TI employee here in the forum a while ago and because it supports HA. I've meanwhile read that the gateway is no longer supported, but replacements don't support HA and we are kind of stuck now.

We haven't changed much in the gateway code, we only fixed:

  • several cases of "use of uninitialized values", some of which are definitely bugs, while others might be false positives. (This was complained about by cppcheck.)
  • a few cases of unaligned memory access, which, depending on the configuration of the kernel, can be a problem on ARM hardware
  • a few other bugs, like dereferencing of null-pointers or leaks
  • code that caused compiler warnings

Depending on the frequency of attribute requests, the gateway server group fails sooner or later. Restarting the gateway_server often helps, sometimes restarting the network_manager does. Even when attribute requests fail with error message "BUSY - please wait for previous operation to complete", zigbee commands do still work for a while, until they start to fail as well.

Is there anything I could try or tune to avoid the zstack linux gateway failing completely? Has the gateway ever been tested with any serious real-world software, or has this never left the demo stage with simple examples?

  • Hi,

    We've had a few gateway releases, some of which are now superseded by the latest offering:  https://www.ti.com/tool/download/ZIGBEE-LINUX-SENSOR-TO-CLOUD 
    This should still allow HA devices.

    We do still support the above tool.

    Can you provide Gateway server logs? You can enable this by adding "-v 0xFFFFFFFF" to the end of the command which starts the Gateway server (this will enable full logs from the Gateway server).

    Can you also provide sniffer logs during the process? This is in order to map over-the-air activity to the Gateway logs.

    Regards,
    Toby

  • Hey,

    thanks for the reply! I'll look into ZIGBEE-LINUX-SENSOR-TO-CLOUD and see how much time a switch over would take. We have already spent a *lot* of time with the linux gateway, though, and the time is pressing, so we'd prefer a fix to a new begin. Debugging the linux gateway is a PITA, though, because of its multiple-server architecture. It's hard to see, why the whole communication breaks down, even though the servers keep running and still exchange TCP messages. That's why I had hoped to see some progress in that other thread. I only realized later, that that thread had been closed and locked without resolution.

    I'll send the requested logs later today or tomorrow.

    BTW: On the end devices we use Z-Stack_Home_1.2.2a, which we also had to modify, because it often ended up with a hard fault. The reason was that the heap is a little small and so malloc() sometimes returned NULL, which the code apparently didn't expect. There was no check for the return value. Now there is, but it doesn't signal the failure to the caller at the moment. It just drops the pending message. Unfortunately, zstack 3 is too large for the cc2650, so we are stuck here as well.

    diff --git a/Projects/zstack/ZNP/CC26xx/StackTask/stack_task.c b/Projects/zstack/ZNP/CC26xx/StackTask/stack_task.c

    index 8c8eb12..9be9408 100644
    --- a/Projects/zstack/ZNP/CC26xx/StackTask/stack_task.c
    +++ b/Projects/zstack/ZNP/CC26xx/StackTask/stack_task.c
    @@ -187,15 +194,21 @@ static void StackTask_handleNPIReq( ICall_EntityID srcEntityID, void *pMsg )
    
         /* Allocate memory for the MT message */
         pOsalMsg = (mtOSALSerialData_t *)osal_msg_allocate( sizeof ( mtOSALSerialData_t ) );
    
         if (pOsalMsg)
         {
             /* Fill up what we can */
             pOsalMsg->hdr.event = CMD_SERIAL_MSG;
    
             pOsalMsg->msg = osal_mem_alloc ( MT_RPC_FRAME_HDR_SZ + pReq[MT_RPC_POS_LEN] );
    
    +       if (!pOsalMsg->msg) {
    +               osal_msg_deallocate((void *)pOsalMsg);
    +               return;
    +       }
    +
             memcpy(pOsalMsg->msg, pReq, (MT_RPC_FRAME_HDR_SZ + pReq[MT_RPC_POS_LEN]) );
    
    

    PS: The line information in this patch doesn't add up, because I removed some code that keeps track of the number of prevented hard faults.

  • Hi,

    Toby Pan said:

    We've had a few gateway releases, some of which are now superseded by the latest offering:  https://www.ti.com/tool/download/ZIGBEE-LINUX-SENSOR-TO-CLOUD 

    This should still allow HA devices.

    So this should still be compatible with Z-Stack_Home_1.2.2a, as far as you can tell?

  • Hopefully the debug logs will shed some light on the issue.

    Yes, the ZIGBEE-LINUX-SENSOR-TO-CLOUD should support 1.2.2a devices joining/operating in its network.

  • Toby Pan said:

    We've had a few gateway releases, some of which are now superseded by the latest offering:  https://www.ti.com/tool/download/ZIGBEE-LINUX-SENSOR-TO-CLOUD 

    Could you please tell the project team to let a static analyzer run over that code? cppcheck would be a good start. I only checked three of the reported 41 errors, and two of them were actual bugs, while the third one was a possible bug and certainly bad coding.

  • Toby Pan said:

    Hopefully the debug logs will shed some light on the issue.

    I haven't gotten around to creating them yet, but I will post them by the end of the week. I was busy with some other topic in that same project. Sorry for the delay!

  • Here are the logs.

    case 1:

    • the coordinator requesting attributes from an enddevice worked for a while
    • then the enddevice sent custom commands (which were received correctly and without errors)
    • the corodinator kept requesting attributes, and the sniffer log confirms that they were sent back by the enddevice, it also looks like the gateway server received them, but it only delivered an error message to attr_process_read_attribute_response()  [status != success]
    • as soon as the enddevice stopped sending custom commands, attributes were handled correctly again

    13:52:03.760 >>> ZigBee_EndDevice: requesting attributes
    [MUTEX] SRSP Lock status: 0
    [GATEWAY/HNDL] INFO: Waiting for synchronous response...
    [GATEWAY/HNDL] INFO: [MUTEX] Wait for SRSP Cond signal...
    [GATEWAY/READ] INFO: Received 4 bytes, subSys 0x71, cmdId 0x22, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x22
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: [MUTEX] SRSP Cond signal set
    [GATEWAY/READ] INFO: Client Read: (len 0):
    [GATEWAY/LSTN] INFO: 0x08
    [GATEWAY/LSTN] INFO: 0x22
    [GATEWAY/LSTN] INFO: 0x10
    [GATEWAY/LSTN] INFO: 0x00
    [GATEWAY/LSTN] INFO: [MUTEX] Unlock SRSP Mutex
    [GATEWAY/LSTN] DEBUG: Checked transaction table size: 50
    [GATEWAY/LSTN] DEBUG: Checked transaction table number of entries: 12
    [GATEWAY/LSTN] INFO: (GW posted) cmdId: 204, zclTransId 48
    [GATEWAY/LSTN] : Sending ZigBee Generic Confirmation
    [GATEWAY/LSTN] INFO: ...sent 10 bytes to Client
    [GATEWAY/LSTN] INFO: !Done
    [GATEWAY/LSTN] INFO: Receive message...
    [GATEWAY/LSTN] : Processing Read Device Attribute Request
    [GATEWAY/LSTN] : Sending Read Device Attribute Request
    [GATEWAY/LSTN] INFO: preparing to send 11 bytes, subSys 0x19, cmdId 0x02, pData:
    [GATEWAY/LSTN] INFO:  0x08
    [GATEWAY/LSTN] INFO:  0x02
    [GATEWAY/LSTN] INFO:  0x11
    [GATEWAY/LSTN] INFO:  0x83
    [GATEWAY/LSTN] INFO:  0x23
    [GATEWAY/LSTN] INFO:  0x8A
    [GATEWAY/LSTN] INFO:  0x0C
    [GATEWAY/LSTN] INFO:  0x00
    [GATEWAY/LSTN] INFO:  0x4B
    [GATEWAY/LSTN] INFO:  0x12
    [GATEWAY/LSTN] INFO:  0x00
    [GATEWAY/LSTN] INFO: [MUTEX] Lock SRSP Mutex[GATEWAY/LSTN] INFO:
    [MUTEX] SRSP Lock status: 0
    [GATEWAY/READ] INFO: Received 8 bytes, subSys 0x79, cmdId 0x03, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0xD2
    [GATEWAY/READ] INFO:  0x8E
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO: [MUTEX] SRSP Cond signal set
    [GATEWAY/READ] INFO: Client Read: (len 0):
    [GATEWAY/LSTN] INFO: Waiting for synchronous response...
    [GATEWAY/LSTN] INFO: [MUTEX] Wait for SRSP Cond signal...
    [GATEWAY/LSTN] INFO: 0x08
    [GATEWAY/LSTN] INFO: 0x03
    [GATEWAY/LSTN] INFO: 0x10
    [GATEWAY/LSTN] INFO: 0x00
    [GATEWAY/LSTN] INFO: 0x18
    [GATEWAY/LSTN] INFO: 0xD2
    [GATEWAY/LSTN] INFO: 0x8E
    [GATEWAY/LSTN] INFO: 0x03
    [GATEWAY/LSTN] INFO: [MUTEX] Unlock SRSP Mutex
    [GATEWAY/LSTN] DEBUG: AF DstAddr Info:
    [GATEWAY/LSTN] DEBUG: AddrMode: 2
    [GATEWAY/LSTN] DEBUG: ShortAddr: C752
    [GATEWAY/LSTN] DEBUG: EndpointId: 13
    ../../../../Components/stack/zcl/zcl.c +945: zcl_SendCommand
    [GATEWAY/LSTN] DEBUG: Sending AF Data Request - dstAddr:c752, srcEp:2, dstEp:13, cID:a000, bufLen:15
    zcl_port.c +349: AF_DataRequest
    [GATEWAY/LSTN] INFO: preparing to send 45 bytes, subSys 0x11, cmdId 0x22, pData:
    [GATEWAY/LSTN] INFO:  0x08
    [GATEWAY/LSTN] INFO:  0x22
    [GATEWAY/LSTN] INFO:  0x12
    [GATEWAY/LSTN] INFO:  0x0C
    [GATEWAY/LSTN] INFO:  0x08
    [GATEWAY/LSTN] INFO:  0x02
    [GATEWAY/LSTN] INFO:  0x10
    [GATEWAY/LSTN] INFO:  0xD2
    [GATEWAY/LSTN] INFO:  0x8E
    [GATEWAY/LSTN] INFO:  0x03
    [GATEWAY/LSTN] INFO:  0x20
    [GATEWAY/LSTN] INFO:  0x0D
    [GATEWAY/LSTN] INFO:  0x28
    [GATEWAY/LSTN] INFO:  0xF3
    [GATEWAY/LSTN] INFO:  0xED
    [GATEWAY/LSTN] INFO:  0x02
    [GATEWAY/LSTN] INFO:  0x20
    [GATEWAY/LSTN] INFO:  0x02
    [GATEWAY/LSTN] INFO:  0x28
    [GATEWAY/LSTN] INFO:  0x80
    [GATEWAY/LSTN] INFO:  0xC0
    [GATEWAY/LSTN] INFO:  0x02
    [GATEWAY/LSTN] INFO:  0x30
    [GATEWAY/LSTN] INFO:  0x31
    [GATEWAY/LSTN] INFO:  0x3A
    [GATEWAY/LSTN] INFO:  0x00
    Calling confirmation callback
    attr_read_confirm_handler: Received ZIGBEE_GENERIC_CNF
    attr_read_confirm_handler: Status SUCCESS.
    [GATEWAY/LSTN] INFO:  0x40
    [GATEWAY/LSTN] INFO:  0x1E
    [GATEWAY/LSTN] INFO:  0x4A
    [GATEWAY/LSTN] INFO:  0x0F
    [GATEWAY/LSTN] INFO:  0x00
    [GATEWAY/LSTN] INFO:  0x31
    [GATEWAY/LSTN] INFO:  0x00
    [GATEWAY/LSTN] INFO:  0x00
    [GATEWAY/LSTN] INFO:  0x00
    [GATEWAY/LSTN] INFO:  0x01
    [GATEWAY/LSTN] INFO:  0x80
    [GATEWAY/LSTN] INFO:  0x02
    [GATEWAY/LSTN] INFO:  0x80
    [GATEWAY/LSTN] INFO:  0x03
    [GATEWAY/LSTN] INFO:  0x80
    [GATEWAY/LSTN] INFO:  0x04
    [GATEWAY/LSTN] INFO:  0x80
    [GATEWAY/LSTN] INFO:  0x05
    [GATEWAY/LSTN] INFO:  0x80
    [GATEWAY/LSTN] INFO: [MUTEX] Lock SRSP Mutex
    [GATEWAY/LSTN] INFO:
    [MUTEX] SRSP Lock status: 0
    [GATEWAY/LSTN] INFO: Waiting for synchronous response...
    [GATEWAY/LSTN] INFO: [MUTEX] Wait for SRSP Cond signal...
    [GATEWAY/READ] INFO: Received 9 bytes, subSys 0x51, cmdId 0x91, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x91
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x2F
    [GATEWAY/READ] INFO: RPC_CMD_AREQ cmdId: 0x91
    [GATEWAY/READ] INFO:
    [DBG] Allocated @ 0x00000000B6C007C0 (received  104 messages)...
    [GATEWAY/READ] INFO: Filling new message (@ 0x00000000B6C007C0)...
    [GATEWAY/READ] INFO: Received 4 bytes, subSys 0x71, cmdId 0x22, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x22
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: [MUTEX] SRSP Cond signal set
    [GATEWAY/READ] INFO: Client Read: (len 0):
    [GATEWAY/READ] INFO: Received 63 bytes, subSys 0x51, cmdId 0x92, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x92
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0xA
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0xD2
    [GATEWAY/READ] INFO:  0x8E
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x38
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x40
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x48
    [GATEWAY/READ] INFO:  0x31
    [GATEWAY/READ] INFO:  0x50
    [GATEWAY/READ] INFO:  0x4
    [GATEWAY/READ] INFO:  0x58
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x60
    [GATEWAY/READ] INFO:  0xC6
    [GATEWAY/READ] INFO:  0xD3
    [GATEWAY/READ] INFO:  0x13
    [GATEWAY/READ] INFO:  0x68
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x70
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x78
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x82
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x5
    [GATEWAY/HNDL] INFO: 0x08
    [GATEWAY/HNDL] INFO: 0x22
    [GATEWAY/HNDL] INFO: 0x10
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: [MUTEX] Unlock SRSP Mutex
    [GATEWAY/HNDL] INFO: [MUTEX] AREQ (Handle) (message @ 0x00000000B6C00938)...
    [GATEWAY/HNDL] INFO: [DBG] Clearing @ 0x00000000B6C00938 (processed 79 messages)...
    [GATEWAY/HNDL] INFO: [MUTEX] Wait for AREQ semaphore
    [GATEWAY/HNDL] INFO: [MUTEX] Mutex for AREQ unlocked
    [GATEWAY/HNDL] INFO:
    
    [DBG] Processing @ 0x00000000B6C00988
    [GATEWAY/HNDL] INFO: [MUTEX] AREQ Calling asynchMsgCback (Handle)...
    [GATEWAY/HNDL] INFO:
    
    [DBG] asyncCB: subSys:0x0000000000000011, cmdId:0x0000000000000092, len:0x000000000000003B, pData:0x00000000B6C00990
    [GATEWAY/HNDL] : Processing Af Incoming Message Indication
    [GATEWAY/HNDL] INFO: preparing to send 6 bytes, subSys 0x19, cmdId 0x00, pData:
    [GATEWAY/HNDL] INFO:  0x08
    [GATEWAY/HNDL] INFO:  0x00
    [GATEWAY/HNDL] INFO:  0x10
    [GATEWAY/HNDL] INFO:  0xD2
    [GATEWAY/HNDL] INFO:  0x8E
    [GATEWAY/HNDL] INFO:  0x03
    [GATEWAY/HNDL] INFO: [MUTEX] Lock SRSP Mutex
    [GATEWAY/HNDL] INFO:
    [MUTEX] SRSP Lock status: 0
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x2F
    [GATEWAY/READ] INFO:  0xB
    [GATEWAY/READ] INFO:  0xCC
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: RPC_CMD_AREQ cmdId: 0x92
    [GATEWAY/READ] INFO:
    [DBG] Allocated @ 0x00000000B6C006F0 (received  105 messages)...
    [GATEWAY/READ] INFO: Filling new message (@ 0x00000000B6C006F0)...
    [GATEWAY/READ] INFO: Received 54 bytes, subSys 0x51, cmdId 0x92, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x92
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0xA
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0xD2
    [GATEWAY/READ] INFO:  0x8E
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x38
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x40
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x48
    [GATEWAY/READ] INFO:  0x31
    [GATEWAY/READ] INFO:  0x50
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x58
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x60
    [GATEWAY/READ] INFO:  0x9B
    [GATEWAY/READ] INFO:  0xD6
    [GATEWAY/READ] INFO:  0x13
    [GATEWAY/READ] INFO:  0x68
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x70
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x78
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x82
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x6
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x4E
    [GATEWAY/READ] INFO:  0x6E
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: RPC_CMD_AREQ cmdId: 0x92
    [GATEWAY/READ] INFO:
    [DBG] Allocated @ 0x00000000B6C00928 (received  106 messages)...
    [GATEWAY/READ] INFO: Filling new message (@ 0x00000000B6C00928)...
    [GATEWAY/READ] INFO: Received 54 bytes, subSys 0x51, cmdId 0x92, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x92
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0xB
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0xD2
    [GATEWAY/READ] INFO:  0x8E
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0xFF
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x1A
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x38
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x40
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x48
    [GATEWAY/READ] INFO:  0x31
    [GATEWAY/READ] INFO:  0x50
    [GATEWAY/READ] INFO:  0xA0
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x58
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/HNDL] INFO: Waiting for synchronous response...
    [GATEWAY/HNDL] INFO: [MUTEX] Wait for SRSP Cond signal...
    [GATEWAY/READ] INFO: Received 13 bytes, subSys 0x79, cmdId 0x01, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x19
    [GATEWAY/READ] INFO:  0x83
    [GATEWAY/READ] INFO:  0x23
    [GATEWAY/READ] INFO:  0x8A
    [GATEWAY/READ] INFO:  0xC
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x4B
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: [MUTEX] SRSP Cond signal set
    [GATEWAY/READ] INFO: Client Read: (len 0): [GATEWAY/MAIN] INFO: (GW timer) deleted zclTransId: 63
    [GATEWAY/READ] INFO:  0x60
    [GATEWAY/READ] INFO:  0xA4
    [GATEWAY/READ] INFO:  0xD6
    [GATEWAY/READ] INFO:  0x13
    [GATEWAY/READ] INFO:  0x68
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x70
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x78
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x82
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x4
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x4F
    [GATEWAY/READ] INFO:  0x66
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO: RPC_CMD_AREQ cmdId: 0x92
    [GATEWAY/READ] INFO:
    [DBG] Allocated @ 0x00000000B6C00F60 (received  107 messages)...
    [GATEWAY/READ] INFO: Filling new message (@ 0x00000000B6C00F60)...
    [GATEWAY/READ] INFO: Received 85 bytes, subSys 0x51, cmdId 0x92, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x92
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0xC
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0xD2
    [GATEWAY/READ] INFO:  0x8E
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xE4
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x33
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x38
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x40
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x48
    [GATEWAY/READ] INFO:  0x31
    [GATEWAY/READ] INFO:  0x50
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x58
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x60
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0xDA
    [GATEWAY/READ] INFO:  0x13
    [GATEWAY/READ] INFO:  0x68
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x70
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x78
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x82
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x23
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x50
    [GATEWAY/READ] INFO:  0x78
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/HNDL] INFO: 0x08
    [GATEWAY/HNDL] INFO: 0x01
    [GATEWAY/HNDL] INFO: 0x10
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x19
    [GATEWAY/HNDL] INFO: 0x83
    [GATEWAY/HNDL] INFO: 0x23
    [GATEWAY/HNDL] INFO: 0x8A
    [GATEWAY/HNDL] INFO: 0x0C
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x4B
    [GATEWAY/HNDL] INFO: 0x12
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: [MUTEX] Unlock SRSP Mutex
    [GATEWAY/HNDL] : Processing ZCL Frame Receive Indication
    [GATEWAY/HNDL] INFO: preparing to send 11 bytes, subSys 0x19, cmdId 0x04, pData:
    [GATEWAY/HNDL] INFO:  0x08
    [GATEWAY/HNDL] INFO:  0x04
    [GATEWAY/HNDL] INFO:  0x11
    [GATEWAY/HNDL] INFO:  0x83
    [GATEWAY/HNDL] INFO:  0x23
    [GATEWAY/HNDL] INFO:  0x8A
    [GATEWAY/HNDL] INFO:  0x0C
    [GATEWAY/HNDL] INFO:  0x00
    [GATEWAY/HNDL] INFO:  0x4B
    [GATEWAY/HNDL] INFO:  0x12
    [GATEWAY/HNDL] INFO:  0x00
    [GATEWAY/HNDL] INFO: [MUTEX] Lock SRSP Mutex[GATEWAY/HNDL] INFO:
    [MUTEX] SRSP Lock status: 0
    [GATEWAY/READ] INFO:  0xCC
    [GATEWAY/READ] INFO:  0x32
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x11
    [GATEWAY/READ] INFO:  0x5C
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: RPC_CMD_AREQ cmdId: 0x92
    [GATEWAY/READ] INFO:
    [DBG] Allocated @ 0x00000000B6C00FA8 (received  108 messages)...
    [GATEWAY/READ] INFO: Filling new message (@ 0x00000000B6C00FA8)...
    [GATEWAY/READ] INFO: Received 9 bytes, subSys 0x51, cmdId 0x91, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x91
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO: RPC_CMD_AREQ cmdId: 0x91
    [GATEWAY/READ] INFO:
    [DBG] Allocated @ 0x00000000B6C007F0 (received  109 messages)...
    [GATEWAY/READ] INFO: Filling new message (@ 0x00000000B6C007F0)...
    [GATEWAY/READ] INFO: Received 4 bytes, subSys 0x71, cmdId 0x22, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x22
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: [MUTEX] SRSP Cond signal set
    [GATEWAY/READ] INFO: Client Read: (len 0):
    [GATEWAY/READ] INFO: Received 60 bytes, subSys 0x79, cmdId 0x05, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x5
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x1A
    [GATEWAY/READ] INFO:  0x36
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0xD2
    [GATEWAY/READ] INFO:  0x8E
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x11
    [GATEWAY/READ] INFO:  0x83
    [GATEWAY/READ] INFO:  0x23
    [GATEWAY/READ] INFO:  0x8A
    [GATEWAY/READ] INFO:  0xC
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x4B
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x19
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x9
    [GATEWAY/READ] INFO:  0x92
    [GATEWAY/READ] INFO:  0xC
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x4B
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x2A
    [GATEWAY/READ] INFO:  0x1A
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x84
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x81
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: [MUTEX] SRSP Cond signal set
    [GATEWAY/READ] INFO: Client Read: (len 0): [GATEWAY/HNDL] INFO: Waiting for synchronous response...
    [GATEWAY/HNDL] INFO: [MUTEX] Wait for SRSP Cond signal...
    [GATEWAY/LSTN] INFO: 0x08
    [GATEWAY/LSTN] INFO: 0x22
    [GATEWAY/LSTN] INFO: 0x10
    [GATEWAY/LSTN] INFO: 0x00
    [GATEWAY/LSTN] INFO: [MUTEX] Unlock SRSP Mutex
    [GATEWAY/LSTN] DEBUG: Checked transaction table size: 50
    [GATEWAY/LSTN] DEBUG: Checked transaction table number of entries: 12
    [GATEWAY/LSTN] INFO: (GW posted) cmdId: 16, zclTransId 49
    [GATEWAY/LSTN] : Sending ZigBee Generic Confirmation
    [GATEWAY/LSTN] INFO: ...sent 10 bytes to Client
    [GATEWAY/LSTN] INFO: !Done
    [GATEWAY/READ] INFO: Received 9 bytes, subSys 0x51, cmdId 0x91, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x91
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/HNDL] INFO: 0x08
    [GATEWAY/HNDL] INFO: 0x05
    [GATEWAY/HNDL] INFO: 0x10
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x1A
    [GATEWAY/HNDL] INFO: 0x36
    [GATEWAY/HNDL] INFO: 0x08
    [GATEWAY/HNDL] INFO: 0xD2
    [GATEWAY/HNDL] INFO: 0x8E
    [GATEWAY/HNDL] INFO: 0x03
    [GATEWAY/HNDL] INFO: 0x11
    [GATEWAY/HNDL] INFO: 0x83
    [GATEWAY/HNDL] INFO: 0x23
    [GATEWAY/HNDL] INFO: 0x8A
    [GATEWAY/HNDL] INFO: 0x0C
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x4B
    [GATEWAY/HNDL] INFO: 0x12
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x19
    [GATEWAY/HNDL] INFO: 0x01
    [GATEWAY/HNDL] INFO: 0x09
    [GATEWAY/HNDL] INFO: 0x92
    [GATEWAY/HNDL] INFO: 0x0C
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x4B
    [GATEWAY/HNDL] INFO: 0x12
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x20
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x2A
    [GATEWAY/HNDL] INFO: 0x1A
    [GATEWAY/HNDL] INFO: 0x08
    [GATEWAY/HNDL] INFO: 0x0D
    [GATEWAY/HNDL] INFO: 0x10
    [GATEWAY/HNDL] INFO: 0x84
    [GATEWAY/HNDL] INFO: 0x02
    [GATEWAY/HNDL] INFO: 0x18
    [GATEWAY/HNDL] INFO: 0x81
    [GATEWAY/HNDL] INFO: 0x02
    [GATEWAY/HNDL] INFO: 0x20
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x28
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x28
    [GATEWAY/HNDL] INFO: 0x03
    [GATEWAY/HNDL] INFO: 0x28
    [GATEWAY/HNDL] INFO: 0x80
    [GATEWAY/HNDL] INFO: 0xC0
    [GATEWAY/HNDL] INFO: 0x02
    [GATEWAY/HNDL] INFO: 0x30
    [GATEWAY/HNDL] INFO: 0x00
    [GATEWAY/HNDL] INFO: 0x30
    [GATEWAY/HNDL] INFO: 0x03
    [GATEWAY/HNDL] INFO: 0x30
    Calling confirmation callback
    attr_read_confirm_handler: Received ZIGBEE_GENERIC_CNF
    attr_read_confirm_handler: Status SUCCESS.
    13:52:05.955 >>> ZigBee_EndDevice: response timeout MLB
    

    case 2:

    • After a while with enddevice and coordinator both sending commands, the gateway stops sending attribute requests altogther. All the coordinator sends are "link status" broadcasts. Maybe one of the zstack_linux_gateway's servers went into a thread lock, where it cannot get out any more. None of the servers crashed, and there's plenty of free RAM.
    • Killing the gateway_server makes the system work again, unless there are custom commands, then attributes fail (see case 1).

    08:36:08.216 >>> ZigBee_EndDevice: request MLB
    [GATEWAY/READ] INFO: Received 84 bytes,  subSys 0x51, cmdId 0x92, pData:
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x92
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x12
    [GATEWAY/READ] INFO:  0xA
    [GATEWAY/READ] INFO:  0x8
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x10
    [GATEWAY/READ] INFO:  0xD2
    [GATEWAY/READ] INFO:  0x8E
    [GATEWAY/READ] INFO:  0x3
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x18
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x20
    [GATEWAY/READ] INFO:  0x80
    [GATEWAY/READ] INFO:  0xC0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x28
    [GATEWAY/READ] INFO:  0xB8
    [GATEWAY/READ] INFO:  0x9
    [GATEWAY/READ] INFO:  0x30
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x38
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x40
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x48
    [GATEWAY/READ] INFO:  0xF
    [GATEWAY/READ] INFO:  0x50
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x58
    [GATEWAY/READ] INFO:  0x13
    [GATEWAY/READ] INFO:  0x60
    [GATEWAY/READ] INFO:  0x98
    [GATEWAY/READ] INFO:  0xD6
    [GATEWAY/READ] INFO:  0x13
    [GATEWAY/READ] INFO:  0x68
    [GATEWAY/READ] INFO:  0xD
    [GATEWAY/READ] INFO:  0x70
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x78
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x82
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x23
    [GATEWAY/READ] INFO:  0x1
    [GATEWAY/READ] INFO:  0x4C
    [GATEWAY/READ] INFO:  0x78
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x2
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x35
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x9A
    [GATEWAY/READ] INFO:  0x5B
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO:  0x0
    [GATEWAY/READ] INFO: RPC_CMD_AREQ cmdId: 0x92
    [GATEWAY/READ] INFO:
    [DBG] Allocated         @ 0x00000000B6B4DC68 (received  27666 messages)...
    [GATEWAY/READ] INFO: Filling new message (@ 0x00000000B6B4DC68)...
    confirmation_timeout_handler: Operation timed out
    08:36:10.219 >>> ZigBee_EndDevice: response timeout MLB
    

    I'll now try to find out, why and where the gateway_server gets locked. Custom commands are probably one cause of this failure, though I can't see why that would be. Or is it a problem that commands are coming in between attribute request and attribute response?

  • Thanks for providing the logs.

    For case 1, one thing I notice is "(GW timer) deleted zclTransId: 63". This means a transaction, in which the gateway expects a response to, has timed out.

    Can you also provide the sniffer log(s) (and NWK key)? It will be helpful to map ZCL sequence number to what the gateway logs indicate.

    I agree, using custom ZCL commands should not break things. The issue (probably caused by timeout) may be related to a ZCL command sent by the end device in quick succession to the gateway sending the Read Attribute request.

    On the gateway application side, if you don't send the Read Attribute requests, does sending custom ZCL commands from the end device cause unexpected behavior on the gateway servers?

  • I've meanwhile discovered, that there was another bug in the gateway server – a pthread_mutex deadlock. This, however, is fixed in https://www.ti.com/tool/download/ZIGBEE-LINUX-SENSOR-TO-CLOUD (api_server.c) and I backported the fix.

    There is still the problem that after a while attribute responses are still sent correctly (according to the sniffer log) and received by the gateway, but only an error message is sent to the application. I'll look into this now and will either post more information in the next days, so that you can look into this, or I'll close the report as resolved otherwise.

  • diff --git a/Z-Stack_Linux_Gateway-1.0.1-src/Source/Projects/zstack/linux/srvwrapper/api_server.c b/Z-Stack_Linux_Gateway-1.0.1-src/Source/Projects/zstack/linux/srvwrapper/api_server.c
    index 27a769f..1f23f38 100644
    --- a/Z-Stack_Linux_Gateway-1.0.1-src/Source/Projects/zstack/linux/srvwrapper/api_server.c
    +++ b/Z-Stack_Linux_Gateway-1.0.1-src/Source/Projects/zstack/linux/srvwrapper/api_server.c
    @@ -306,18 +306,13 @@ static int reserveActiveConnection( int connection )
     {
       ConnectionInfo_t *entry;
     
    -  if ( pthread_mutex_lock( &aclMutex ) != 0 )
    -  {
    -    perror( "pthread_mutex_lock" );
    -    exit( 1 );
    -  }
       entry = activeConnectionList;
       while ( entry )
       {
         if ( entry->sock == connection )
         {
           aclNoDeleteCount++;
    -      pthread_mutex_unlock( &aclMutex );
    +
           if ( pthread_mutex_lock( &entry->mutex ) != 0 )
           {
             perror( "pthread_mutex_lock" );
    @@ -327,7 +322,6 @@ static int reserveActiveConnection( int connection )
         }
         entry = entry->next;
       }
    -  pthread_mutex_unlock( &aclMutex );
       return -1;
     }
     
    @@ -345,11 +339,6 @@ static void unreserveActiveConnection( int connection )
     {
       ConnectionInfo_t *entry;
     
    -  if ( pthread_mutex_lock( &aclMutex ) != 0 )
    -  {
    -    perror( "pthread_mutex_lock" );
    -    exit( 1 );
    -  }
       entry = activeConnectionList;
       while ( entry )
       {
    @@ -364,7 +353,6 @@ static void unreserveActiveConnection( int connection )
         }
         entry = entry->next;
       }
    -  pthread_mutex_unlock( &aclMutex );
     }
    
     /*********************************************************************
    @@ -380,12 +368,6 @@ static void unreserveActiveConnection( int connection )
     static void dropActiveConnection( int connection )
     {
       ConnectionInfo_t *entry;
    -
    -  if ( pthread_mutex_lock( &aclMutex ) != 0 )
    -  {
    -    perror( "pthread_mutex_lock" );
    -    exit( 1 );
    -  }
       entry = activeConnectionList;
       while ( entry )
       {
    @@ -396,7 +378,6 @@ static void dropActiveConnection( int connection )
         }
         entry = entry->next;
       }
    -  pthread_mutex_unlock( &aclMutex );
     }
    
     /*********************************************************************
    @@ -480,18 +461,18 @@ static int apisSendData( uint16 len, uint8 *pData, int connection )
       int bytesSent = 0;
       int ret = APIS_LNX_SUCCESS;
    
    +  if ( pthread_mutex_lock( &aclMutex ) != 0 )
    +  {
    +    perror( "pthread_mutex_lock" );
    +    exit( 1 );
    +  }
    +
       // Send to all connections?
       if ( connection < 0 )
       {
         // retain the list
         ConnectionInfo_t *entry;
    
    -    if ( pthread_mutex_lock( &aclMutex ) != 0 )
    -    {
    -      perror( "pthread_mutex_lock" );
    -      exit( 1 );
    -    }
    -
         entry = activeConnectionList;
    
         // Send data to all connections, except listener
    @@ -541,7 +522,6 @@ static int apisSendData( uint16 len, uint8 *pData, int connection )
           }
           entry = entry->next;
         }
    -    pthread_mutex_unlock( &aclMutex );
       }
       else
       {
    @@ -583,6 +563,7 @@ static int apisSendData( uint16 len, uint8 *pData, int connection )
         }
       }
    
    +  pthread_mutex_unlock( &aclMutex );
       return (ret);
     }
    

  • That other problem was a my very own creation. So I'll now close this issue as resolved. Thanks, Toby, for your support!

  • Understood, thanks for the update!