This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2538: Packet loss in zigbee communication.

Guru 14805 points
Part Number: CC2538
Other Parts Discussed in Thread: Z-STACK, CC2592, , CC2590, CC2650

Hello ,

 I have 3 router and 1 gateway based on cc2538-cc2592emk and z-stack 3.0

Setup is operating on channel 15

Routers are 7 meter,10 meter, 15 meter away from gateway, the setup is deployed in a normal office work space environment.And routers are sending report command to gateway every 5 minute.

I have observed that during week days(Monday to Friday) routers to gateway there is significant packet loss of 10%  to 40 %.

What can be the possible reasons??

  • Do you use sniffer to check what happens?
  • Unfortunately i don't have sniffer log, that's a client location so not able to run the sniffer.
  • Without sniffer log, it is difficult to judge anything.
  • Hello Dhanraj,

    Link quality could still be an issue depending on the network setup. Have they defined HAL_PA_LNA_CC2592 inside of hal_board_cfg.h? Is the customer able to provide any LQI/RSSI reports per router? Can they alter the router location and firmware for debug purposes?

    Regards,
    Ryan
  • Hello Ryan,

    1) HAL_PA_LNA_CC2592 is defined in compile flag , not in hal_board_cfg.h
    2) How to generate LQI/RSSI report?
    3) yes, location can be altered and what changes you are suggesting in firmware?

    Regards
    DC
  • You can send ZDP_MgmtLqiReq from ZC to all ZR to collect LQI data.
  • Is there any command available to send from gateway application ??
  • Does your GW uses ZAP-ZNP architecture?
  • Yes , RPi as ZAP , and cc2538-cc2592emk as znp
  • You can use ZNP/MT command ZDO_MGMT_LQI_REQ to do LQI request.
  • I have checked in Z-stack Linux Gateway API document v1.1 , but there is no API available to trigger ZDO_MGMT_LQI_REQ from gateway
  • OK, you actually use Z-stack Linux Gateway. If there is no API available in Z-stack Linux Gateway, you might need to implement this by yourself.
  • From znp/coordinator if i send ZDO_MGMT_LQI_REQ, then routers and end devices both will respond or only router.
  • If router/device supports LQI command, both of them would respond.
  • If I dont want to send LQI request command from znp/coordinator , instead of that is there any API in z-stack(3.0) router which can return LQI value . Anyway my router is is reporting other data to gateway periodically , so i can accommodate that LQI value also into the same reporting packet.
  • There’s no such capability in current Z-Stack.
  • Hi Dhanraj,

    Do you have any updates to provide on this issue?  Please continue the investigation or state your solution and mark the thread as resolved.

    Regards,
    Ryan

  • Hello Ryan,

    I had continued the investigation.

    I am not able to observe any consistent pattern.

    Please have a look at attached report about packet loss. Each router should report maximum 288 times in a day, when 0% packet loss.

    You can see the par day packet received for each router out of 288.

    packet loss report.xlsx

  • Dhanraj:

         I don't think 15 meter LOS(Line-of-sight) is a problem for CC2538+CC2592 RF front ,22 dbm RF power this enough extension range to 100 meter LOS,even that is smaller than high performance WIFI router(30dbm).

         If you deploy on Z-Stack 3.0.x stack ZNP with CC2592,you should check the CC2538ZNP\hal_board_cfg.h carefully, as far as I am concerned,there have a bug on hal_board_cfg.h,don't case macro 

    HAL_PA_LNA_CC2592

    /* ----------- RF-frontend Connection Initialization ---------- */
    #if defined HAL_PA_LNA || defined HAL_PA_LNA_CC2590
    extern void MAC_RfFrontendSetup(void);
    #define HAL_BOARD_RF_FRONTEND_SETUP() MAC_RfFrontendSetup()
    #else
    #define HAL_BOARD_RF_FRONTEND_SETUP()
    #endif

  • Z-Stack 3.0.1 does have the macro included as far as I can tell:

    /* ----------- RF-frontend Connection Initialization ---------- */
    #if defined HAL_PA_LNA || defined HAL_PA_LNA_CC2590 || defined HAL_PA_LNA_CC2592
    extern void MAC_RfFrontendSetup(void);
    #define HAL_BOARD_RF_FRONTEND_SETUP() MAC_RfFrontendSetup()
    #else
    #define HAL_BOARD_RF_FRONTEND_SETUP()
    #endif

    But this does bring up a good point that all hardware power table settings should be checked carefully as incorrect values could affect packet reception.

    Regards, Ryan

  • I didn't get you -what do you mean by -don't case macro ?


    I have already modified hal_board_cfg.h as shown below-

    /* ----------- RF-frontend Connection Initialization ---------- */
    //#if defined HAL_PA_LNA || defined HAL_PA_LNA_CC2590 changes made to enable cc2592 by Dhanraj
    #if defined HAL_PA_LNA || defined HAL_PA_LNA_CC2590 || defined HAL_PA_LNA_CC2592
    extern void MAC_RfFrontendSetup(void);
    #define HAL_BOARD_RF_FRONTEND_SETUP() MAC_RfFrontendSetup()
    #else
    #define HAL_BOARD_RF_FRONTEND_SETUP()
    #endif

    // #ifdef HAL_PA_LNA changes made to enable cc2592 by Dhanraj
    #if defined HAL_PA_LNA || defined HAL_PA_LNA_CC2592
    #define HAL_BOARD_PA_LNA_INIT() st(GPIOPinTypeGPIOOutput(HGM_BASE, HGM_PIN); )
    #else
    #define HAL_BOARD_PA_LNA_INIT()
    #endif

  • And you define HAL_PA_LNA_CC2592 to start?

    Ryan
  • yes I define HAL_PA_LNA_CC2592 in compiler> preprocessor flag.
  • You mention that the design is based on the cc2538-cc2592emk but does the system actually use these boards or is it a custom EVM? Can you revert the application back to a stable point (similar to an example provided) or does this still appear to be hardware-related?

    Regards,
    Ryan
  • I am using readymade  cc2538-cc2592 evm.

    I don't think its hardware related bug.

    I will explain you about the test scenario- 

    1) so i have started this setup in a 3 floor building  ground floor, first floor, second floor.

    Gateway is located in first floor,

    in the ground floor i have 2 router and 28 zed

    in the first floor 3 routers and 31 zed

    in the second floor 3 routers and 29 zed 

    this was my initial setup -

    in this setup - packet loss in the ground floor and first floor devices routers and ZED  was very less (<10%).

    But in the second floor there was huge packet loss (>30%) ,even zed were not running more than 2 days they stopped sending data.

    so then i decided to isolate the network and first identify the packet loss issue in the routers

    so for the second floor i have deployed the separate network with three routers only. And then i have observed that still there is a significant packet loss as i shown in excel sheet in previous reply.

      

    I have collected ubiqua sniffer log for 9 hours , but a am not able to find anything from that.

    Please find attached sniffer log.

    Packet loss log 21032018.rar

    short address and extended address of the router for second floor which are suffering from packet loss

    1) 0X870B , 0X00124B00060d90ED

    2) 0X9000,  0X00124B00060BEE57

    3) 0XAB02, 0X00124B00060D91D1

  • Hi Ryan,

        Do you confirm you test on Z-Stack 3.0.1? I don't think this hal_board_cfg.h source code is correct when Z-Stack 3.0.1.exe default installation.

  • behold,

    We too have made changes like this to enable the CC2592 in the ZNP target. It seems to have been omitted by TI.
  • Hi behold, Mathew,

    You are correct that the CC2538ZNP hal_board_cfg.h is missing the CC2592 definition, I have too many threads running in tangent and was as such accidentally looking at the CC2538 general resources.

    Dhanraj,

    Your link status for these nodes seems fairly consistent, all 0x01 cost (best) with the exception of 0xAB02 outgoing to 0x9000 and 0x870B outgoing to 0x0000 which are occasionally 0x00 (disconnect). With 68 different PANs there appears to be a lot of noise on channel 15. Is this expected? I'm thinking back to when you expressed that the issue is worse on weekdays. Have you tried testing on a different channel?  And why consistently broadcast permit join requests every one to two minutes?  It is apparent that some of your attribute reporting is taking multiple retries, have you increased the number of attempts in your f8wConfig.cfg?

    We can back up on the length of future sniffer logs, 9 hours is a tad excessive.

    Regards,
    Ryan

  • Hey Ryan,

    A good candidate for comment in  I think. 

    I'm glad you confirmed the issue

  • Hi Ryan,

    1)With 68 different PANs there appears to be a lot of noise on channel 15.

     > actually there are only two pan, i don't know why this ubiqua is showing so much pan , and also it is showing end device as routers. similar problem i have already reported earlier .

    Currently there is one network have 5 routers and around 65 zed, And the second one which we are observing for packet loss is contains only 3 routers.

    https://e2e.ti.com/support/wireless_connectivity/zigbee_6lowpan_802-15-4_mac/f/158/t/663751?tisearch=e2e-quicksearch&keymatch=end%20devices%20showing%20as%20router

    2) Is this expected?

    > No this much of noise is not expected , its a ideal working office environment.also WiFi Router is also there but ,WiFi will be there in most of the office environment.  

    3) I'm thinking back to when you expressed that the issue is worse on weekdays. Have you tried testing on a different channel?

    > currently network is using channel 15 ,should I go towards channel 1 or towards channel 25 ?

     4) And why consistently broadcast permit join requests every one to two minutes? 

     > each router enable permit join every 5 minute, i have implemented this because end devices were frequently leaving the network in the same environment, so they can join the network again , that is  why i have enabled this.  if it bad practice/ not recommended then i will disable this.   

    5)  It is apparent that some of your attribute reporting is taking multiple retries, have you increased the number of attempts in your f8wConfig.cfg?

    > No I have not increased the number of attempts .

  • Channels 1 through 10 are 868/915 MHz PHY so stay in the 2.4 GHz PHY range, channels 11 through 26. Channels 15, 20, 25, 26 are always free from WiFi interference and 11, 14, 15, 19, 20, 24, or 25 are preferred for HA. Try a different channel with more retries, and consider using a hardware interaction (GPIO ISR) to enable permit join and only on one coordinator/router since the others will follow suit (hence all enable permit-join about every minute, I doubt it ever turns off on all nodes). Network rejoins do not require permit-joining enabled if the NWK information is stored in NV memory.

    Regards,
    Ryan
  • I have already enabled channel 15,20,25 during configuration. whenever network starts its selects channel 15.
    Anyway i will enable single channel at a time and test for channel 20,25,26.
    fyi-our application is not for Home automation, our application for large network in BIG office infrastructure/environment .

    1) How much range (communication distance , end device cc2650 to router cc2538-cc2592emk)will be affected when we use channel 26.
  • 26 has the same range as all other 2.4 GHz Zigbee channels.

    Regards,
    Ryan
  • Any updates on this, Dhanraj?

    Ryan
  • To check interference with wifi, I have conducted wifi signal test, and I found that in my test environment two wifi networks are operating on channel 2 and channel 4 with good signal strength.And wifi channel 2 and 4 overlap with ZigBee channel 15.

    Please find attached screenshot of wifi traffic in the test environment.

  • So then have you tried testing your ZigBee network on another channel?

    Regards,
    Ryan