CC2538: Why "SRSP Cond Wait timed out! " condition occur in Zigbee linux gateway 3.0 application.

Yes,

It occurs eventually,
But still it is showing SRSP Time out still  " SRSP Cond Wait timed out! " Error.
even Increasing SRSP timer out interval not resolved this problem & slows down the gateway application as we increases the timer value.

As I worked on increasing the baud rate,
Now I am able to know that this condition occurs for each Sync Response type of commands send to ZNP device on Serial BUS (i.e. UART),
when ZNP Device doesn't response to Gateway Server. It will throw a error SRSP Cond Wait timed out!

But I am not 100% sure about this,
Because I have few questions regarding this,

I have enabled the Handshake by setting variable flowcontrol=1 ; in NPI_Gateway.cfg. file. 

CTS & RTS pins are grounded on ZNP device.

Because I am unable to found CTS & RTS PINs on Beaglebone. there is no definition for them in Gateway application.

1. Is it happening because I am not connected CTS & RTS pins for the handshaking between ZNP & Gateway Server?.
           If -> Yes
           Would you please let me know which PINS are to configured as a CTS & RTS on beagle bone side?

2. I think it is a critical problem In gateway application, how we can handle this?
           So that even it arries we take a immediate action to recover it.

Regards,

Shiv Patil.

Hello Toby Pan,

I am able to configure CTS & RTS pins for Hardware Flow control on ZNP device ( CC2538 ) & Beaglebone successfully.

I have tested the Zigbee Gateway Application by sending commands to different devices over the network & its working fine.

But,

I am still facing that problem ERROR :SRSP Cond Wait timer out! 

Does the gateway recover after this error shows up?

Ans : It will be only recover when we RESTART the Gateway Application OR Gateway Application it self causes RESTART for certain condition.

            ( i.e. when NWK / GATEWAY / OTA any one of network dies, application get a call of restart by it self. )

Can you provide the complete log for more context?

Ans : Find the attachment given below will help you to understand the problem.  

Note : Now the gateway application has somehow better performance than previous one. ( Baudrate = 921600 & enabled flowconrtol using CTS & RTS )

           If you look at the logs & do search for the "  ERROR : SRSP Cond Wait timed out!  ",  You will be able to know that more than 100s of time this

           Error will be generated.

           Even though system is in working state we have no control over it. It will be continually prompted on console.

           Major drawback is it will be slows down gateway application responses as per the values of 

Hello Toby Pan,

I have tried both of the solutions as you mentioned above.

1. Enabling compilation flag -D__BIG_DEBUG__ will not affect logs very much. though I am able to get more informations using this,
    but it is not showing any additional information related to the SRSP Cond Wait timed out!

2. making separate logs.out for all types of server components on gateway server will also not help me to get information related to this
problem.

And I found something different that,

When we are sending multiple commands to a device rapidly. it goes in to offline mode after some time.

Is this problem that device doesn't response or packet are dropped on ZNP or Gateway server due to it's offline status?.

If device goes offline how we get back it in online mode?..

Hello Toby Pan,

I would like to answer your question 

Q. What do you mean "it goes in offline mode" and "device goes offline" ?

Ans : 

if you look at the Zigbee Gateway Application code. you will be able to found that there is a way to get the Device Network Status. 

In file nwkmgr / nwkmgr.pb-c.h you will able to find its definition. also same reference is mentioned in serverpb / server.pb-c.h

typedef enum _NwkDeviceStatusT {
  /*
   * Device is off-line (non-responsive to service discovery)             // Here TI clearly mentioned that the device is in non-responsive mode.
   */
  NWK_DEVICE_STATUS_T__DEVICE_OFF_LINE = 0,
  /*
   * Status good
   */
  NWK_DEVICE_STATUS_T__DEVICE_ON_LINE = 1,
  /*
   * Device has been removed
   */
  NWK_DEVICE_STATUS_T__DEVICE_REMOVED = 2,
  /*
   * Not Applicable (this value is returned when gateway device)
   */
  NWK_DEVICE_STATUS_T__DEVICE_NA = 255
    PROTOBUF_C__FORCE_ENUM_TO_BE_INT_SIZE(NWK_DEVICE_STATUS_T)
} NwkDeviceStatusT;

& this Device State is maintained in the file type.h  > device_info_t  struct.

/* Display-related information for a Device in the network */

typedef struct
{
  endpoint_info_t     ep_list[MAX_ENDPOINTS];
  uint64_t            ieee_addr;
  uint16_t            nwk_addr;
  uint16_t            manufacturer_id;
  bool                  valid;
  bool                  selected;
  bool                  selected_as_bind_destination;
  uint8_t             device_status;
  uint8_t             num_endpoints;
  uint8_t             selected_endpoint_index;
} device_info_t;

STRING_ADD(device_string, DARK_WHITE " |%s%s" , ds_device_table[i].selected_as_bind_destination ? GREEN :   \
 ((ds_device_table[i].device_status == NWK_DEVICE_STATUS_T__DEVICE_OFF_LINE) ? BOLD BLACK : RED), ds_device_table[i].selected ? BOLD : "");

STRING_ADD(device_string, " ");

Yes it tries up the MAX_DEVICE_FAILED_ATTEMPTS if device doesn't responds to the requests it goes in to offline mode.

& When Device goes offline we are not able to see the commands on sniffer. // shows something like Route Request.

Find the attachment give below will help you to understand the problem

My question is, how we get device back to online state?...

Do one thing,

Run the ./zigbeeHAgw bbb & ./start_application script.

Add any device in network supports ON/OFF OR anye Actuation Command related to it.

Continuously send ON &  OFF commands by pressing keys 'n' & 'f'.

Try this as ferster as you can.

After some time you will able to know that device is going to be offline now.

The device details on start application will be display in gray color as given below.

>00:12:4B:00:09:E2:91:21 B5F0 0F 0100(HA) | F2 0061(A1E0)   

Try this for few more time you will able to see msg on start application logs.

TIMEOUT waiting for confirmation      

This means on ./zigbeeHAgw produces the error.

ERROR : SRSP Cond Wait timed out!

I am not sure but, this problem is may related to the NIP Linux server.

Regards,

Shiv Patil.

Hello Toby Pan,

I have further tested the code in more details, now able to understand the problem,

See the logs given below. will help you to understand it.

ERR:: Interrupted system call
[ERR] npi_ipc_errno 0x02030100
Exit socket while loop
cat: /proc/7128/cmdline: No such file or directory
pid 7128 is not there
count is 0, not 4
kill -SIGUSR2 6985
caught SIGUSR2, a server other than NWKMGR died!
[12:50:12.281,096] [NWK_MGR/LSTN] CONNECT: Disconnected from client GATEWAY (connection # 7)
waiting for GATEWAY SERVER to exit
[12:50:12.624,424] [Z_STACK/LSTN] ERROR  : SRSP Cond Wait timed out!
[12:50:12.624,610] [Z_STACK/LSTN] ERROR  : apicSendSynchData() failed getting response
[12:50:12.625,185] [Z_STACK/LSTN] CONNECT: Disconnected from client GATEWAY (connection # 6)
tracker exiting
[12:50:14.625,804] [Z_STACK/LSTN] ERROR  : SRSP Cond Wait timed out!
[12:50:14.626,238] [Z_STACK/LSTN] ERROR  : apicSendSynchData() failed getting response
[12:50:17.398,019] [NWK_MGR/LSTN] CONNECT: Disconnected from client OTASRVR (connection # 8)
waiting for OTA SERVER to exit
ZLSZNP_arm: no process found
waiting for Zstack linux to exit
NPI_lnx_arm_server: no process found
waiting for NPI to exit
NETWORK MANAGER exited with code 140 on Fri May 17 12:50:17 IST 2019
./zigbeeHAgw: line 583: kill: (7173) - No such process
making sure there are no lingering servers...

In above logs, I you look at the line  npi_ipc_errno 0x02030100 & trace this error in npi_lnx_error.h file

you will able to see that this error mapped to,

How this causes the SRSP time out Condition?

Also How to solve this error OR handle it so that our zigbee servers does not crashed or get the delayed responses.

Regard's 

Shiv Patil.

Yes, it seems this happens in the NPI server.

The non-success return value of npi_write causes the npi_lnx_ipc.c loop to break. This is the loop which runs the NPI server. When one server dies, zigbeeHAgw will kill the other servers.
You can try handling it based on the actual error value returned by npi_write (which is a wrapper for the write function).

// this is in npi_sendframe in npi_lnx_uart.c
   if (npi_write(pBuf, frlen) < 0) {
    perror("ERR:");
    free(pBuf);
    npi_ipc_errno = NPI_LNX_ERROR_UART_SEND_FRAME_FAILED_TO_WRITE;
    return NPI_LNX_FAILURE;
}

// this is in main() of npi_lnx_ipc.c
else
{
    //                          debug_
    printf("[ERR] npi_ipc_errno 0x%.8X\n", npi_ipc_errno);
    // Everything about the error can be found in the message, and in npi_ipc_errno:
    childThread = ((npiMsgData_t *) npi_ipc_buf[0])->cmdId;
    sprintf(toNpiLnxLog, "Child thread with ID %d in module %d reported error:\t%s",
            NPI_LNX_ERROR_THREAD(childThread),
            NPI_LNX_ERROR_MODULE(childThread),
            (char *)(((npiMsgData_t *) npi_ipc_buf[0])->pData));
    //                          printf("%s\n", toNpiLnxLog);
    writeToNpiLnxLog(toNpiLnxLog);
}
break;

Can I enable SPI communication between the CC2538 & Beagle Bone?..

In Z-Stack 3.0.x, ZNP over SPI is deprecated so you cannot use it.

You can reference the SPI module on the HA 1.2.2a ZNP from Z-STACK-ARCHIVE and port the functionality to your Z-Stack 3.0 ZNP accordingly.

Regards,
Ryan

Thank You Ryan, 

I will work on it.

I would also like to know that,

Is Zigbee Linux Gateway has SPI Support?  (i.e. Can we use it as it is just by enabling the flags & GPIO's)

OR

we need to do modifications in code?...

SPI functionality is built into the HA 1.2.2a ZNP but modifications are required for the Z-Stack 3.0.  We already had this conversation: https://e2e.ti.com/support/wireless-connectivity/zigbee-and-thread/f/158/t/784775/

Regards,
Ryan

Yes, I remember it.

I am Asking about the Zigbee Linux gateway Application.

Shall we need to do modifications on Zigbee Linux Gateway Application along with the Z-stack 3.0 Application code?..

Or we can use it by just making changes in NPI_Gateway.cfg.

Ah, thank you for clarifying.  I believe SPI support has remained on the the Zigbee 3.0 Linux gateway application from the Z-Stack 1.2.2a implementation, however as this is unsupported you will have to change NPI_Gateway.cfg and test this for yourself.

Regards,
Ryan