CC2538: How to modify cc2538-cc2592emk znp MAC/RF parameter using z-tool

Do you use the same HW and SW when you test under environment 1 and 2. If so, I would say environment 1 might have interference and you can try to change channel to test again.

in both cases, HW and software are the same.

but if it is interference problem in the environment 1 then how the router is working fine there??

does router and znp both have different initialization process ??

Also my gateway in environment is start ..zigbee server and nodejs server working fine ..but Radio wise I can't see anything on sniffer.

In the environment 1, how far between coordinator and router and how far is it between sniffer and router/coordinator?

all three , router ,coordinator and sniffer are in an area of 1-meter radius.

Does your gateway as coordinator contain other 2.4G RF such as WiFi or Bluetooth?

the placement and distance  is as shown below 

My cordinator/gateway is based on Raspberry pi 3B+ and that has onboard wifi and Bluetooth, wifi is not disabled but it is not connected to any access point in both the environment.

I still suspect there is interference in your test environment 1. Do you try to change to other Zigbee channel and see if the problem is still there?

I am seeing the problem on multiple channel - 11, 21,24

in one floor i have total 5 gateway out of that 2 gateways showing problem one is on channel 11 and another is on channel 21

For example, if you change the one with problem on channel 11 to channel 15, do you still see the issue?

I am going to test that today.

any other suggestion to conduct any particular test

Also modifying below-mentioned jammer related parameter in the config.ini file of the gateway will help??

; Jammer Detection parameter for the time between noise level readings. 
; This value is in milliseconds. A value of 100 means that the jammer detecting algorithm will take a 
; noise level readying every 100 milliseconds. 
JAMMER_DETECT_PERIOD_TIME = 100 

; Jammer Detection parameter for the number of continuous events needed to detect a Jam. 
; A value of 150 means that the jammer detecting algorithm needs 150 consecutive readings that 
; are above the JAMMER_HIGH_NOISE_LEVEL to detect a "Jam". A single reading below the noise level 
; will restart the consecutive counting. 
JAMMER_CONTINUOUS_EVENTS = 150 

; Jammer Detection parameter for the High Noise Level comparison. This value will be multipled by -1. 
; A value of 60 means that the noise level reading must be greater than -60 to count as High Noise. 
JAMMER_HIGH_NOISE_LEVEL = 60

I don't think your issue is related to jamming detection defines in config.ini.

Make sure HAL_PA_LNA_CC2592 is defined for the ZNP, verify radio range with SMARTRFTM-STUDIO, test additional CC2538-CC2592EMKs, and perform A-B-A testing (switch the router and ZNP software between hardware) to further debug this problem.

Regards,
Ryan

no mto no link status from znp.rar

PFA attached sniffer log, This is captured in environment 1, routers(3199 and FB75) are able to send the data like link status.and reports command.

But there is no data from ZNP/coordinator like link status, mto route request.

Can you provide the network key for your sniffer log so I can decrypt it?

Hello YK,

I am sorry, Since the sniffer log is captured from client place,we don't have the network key.

I cannot do anything if there is no network key to decrypt sniffer log.

Hello YK,

Please refer below key.

1D2CC9FD96616F08AE46FDD2811DA029

The sniffer log only shows the results that there is no signal from coordinator. Do you have full log that coordinator was sending link status and turned to silence?

This issue m seeing consistently that, the coordinator/znp does not send any packet since it starts.But at the same time znp on usb port is getting detected on Raspberry pi gateway, and necessary zigbee servers nodejs server and application starting properly.

 I am trying to capture the scenario where the coordinator was sending link status and turned to silence?

Do you mean your coordinator doesn't send link status since you start it to form a Zigbee network?

not like that ...

step-1 

All the cordinator/gateway are tested once that is network formation is done, devices(routers are commissioned) and verified that data is sent by routers and received by the gateway.

step -2

After that gateway are packed and sent to the client location.

step -3

when we have setup/installed the gateways ate client location we have used 5v and 1.2 Amp(But recommended is 5v 2.4Amp) adaptor to power the gateways.

step-4

After this installation, we have started observing the issue where out of 10 gateways 3 were not sending any data since the time it is powered On and the remaining 7 they were sending data when started but eventually stopped sending and receiving data within a day time.

step -5

when we have seen this issue we removed one of the znp from gateway and connected to laptop and started testing with Ztool.But we have observed that with Ztool also when we start the znp by sending "ZB_START_REQUEST" under simpleAPI section,  we dont see any packet like link status , mto request etc on the sniffer.

step-6

after this we have used 5v and 2.4amp adaptor to power 7 gateways out of 7 gateways, 4 are able to send and receive data but remaining 3 are showing the same issue where they are not sending and receiving data.

So now we are confused where is the problem actually. 

Please let me know if you need further info.

Do you use TI CC2538-CC2592EMK on your 10 gateway or your own custom CC2538-CC2592 HW?

TI cc2538-cc2592emk as it is without any change ..directly procured from TI store.

Oops... run out of idea. Ryan Brown1 any suggestion or opinion?

Hello YK,

PFA attached sniffer logs ...

in that PAN - C081 and C082 are my network. out of those C081 is acting wired ...c081 is sending packets inconsistently.

nwk key for c081 -   98 02 AA A1 7D 78 A2 E2 BB E9 F2 8E 47 04 73 C3

Now i have removed znp of c081 pan from gateway and connected to z-tool still m not able to any packet from c081 znp ....

let me know if want me do any experiment using z-tool

c081 no packet after packet 31 32.rar

c081 no packet after packet 31 32 and inconsistent after 54.rarc081 inconsistency.rar

Dhanraj,

Do these devices return to normal operation if reprogrammed or factory reset?  If not then the field setup could be damaging components.  TER will continue to help you debug hardware issues: https://e2e.ti.com/support/wireless-connectivity/zigbee-and-thread/f/158/t/845980

Regards,
Ryan

i took the non working znp , erased memory and connected to smart rf studio ..to  send radio packets ...and througfh smart rf studio it is sending the packets ...

What happens when you perform a factory reset?  Are you using the default ZNP or have any changes been made?  Can you read out the flash contents and check for memory corruption?

Regards,
Ryan

Please find my answer in line below-

What happens when you perform a factory reset?

I have got one of the gateways(C081 PAN) from the field to the development environment and then done mass erase and flashed again with same firmware and it started working normally except few(not all) missing link status.

Are you using the default ZNP or have any changes been made?

I am using a modified ZNP firmware, but with the same firmware I have done multiple deployments and I am observing the issue only with this 1 deployment.

Can you read out the flash contents and check for memory corruption?

I have done the mass erase, how can I check for memory corruption.

Can you elaborate what you modify to ZNP source code?

Do you know of any hardware differences with the deployment in question?  You would use Uniflash or SMARTRF-PROGRAMMER to check your flash contents, ideally you are comparing a production image against the corrupted device's memory contents.

Regards,
Ryan

Hello YK,

please refer the attached file to see the changes in my znp

//Added  > in static void npInit(void)  > in znp_app.c
//--------------------------//
#ifdef HAL_PA_LNA_CC2592
   ZMacSetTransmitPower(TX_PWR_PLUS_22);//edited by Prem default is TX_PWR_PLUS_19
#else
   ZMacSetTransmitPower(TX_PWR_PLUS_4);
#endif 
//------------------------//


//Added at the end of  > typedef enum{ TX_PWR_MINUS_22 = -22,.....,TX_PWR_PLUS_19 > In  ZMAC.h
  
//----------------//
 TX_PWR_PLUS_20,//edited by Prem
  TX_PWR_PLUS_21,//edited by Prem
  TX_PWR_PLUS_22//edited by Prem
//------------------------//

//modified in > void ZDSecMgrDeviceRemove( ZDSecMgrDevice_t* device ) > in ZDSecMgr.c
//------------------//
leaveReq.rejoin         = TRUE; //edited by Prem default is FALSE
//-------------------//

//Modified  >  in ZDSecMgr.h
//---------------------//
#if !defined ( ZDSECMGR_TC_DEVICE_MAX )
  #if (ZG_BUILD_COORDINATOR_TYPE) 
    #define ZDSECMGR_TC_DEVICE_MAX 210//edited by Prem default is 40
  #else
    #define ZDSECMGR_TC_DEVICE_MAX 3
  #endif
#endif
//---------------------//

//Modified  >  in ZDNwkMgr.h
//--------------------//
#if !defined ( ZDNWKMGR_MIN_TRANSMISSIONS )
  #define ZDNWKMGR_MIN_TRANSMISSIONS      0//edited by Prem default is 20
#endif
//------------------------//

//modified in > ZDApp.c
//----------------//
uint16 ZDApp_CoordStartPANIDConflictCB( uint16 panid )
{
  return ( panid /*+ 1*/ ); //edited by Prem
}
//------------------//

//modified in > ZDApp.c
//----------------//
#if ZDO_NV_SAVE_RFDs
#define ZDAPP_UPDATE_NWK_NV_TIME 5000 //edited by Prem default is 700
//------------------//

//modified in > ZGlobals.h
//----------------//
#if !defined ( CONCENTRATOR_ROUTE_CACHE )
  #define CONCENTRATOR_ROUTE_CACHE     true//edited by Prem default is false // true if concentrator has route cache
#endif
//----------------//

//modified in > ZGlobals.h
//----------------//
#if !defined ( CONCENTRATOR_DISCOVERY_TIME )
  #define CONCENTRATOR_DISCOVERY_TIME  120//edited by Prem default is 0
#endif
//----------------//

//modified in > ZGlobals.h
//----------------//
#if !defined ( CONCENTRATOR_ENABLE )
  #define CONCENTRATOR_ENABLE          true//edited by Prem default is false // true if concentrator is enabled
#endif 
//----------------//

//modified in > ZGlobals.h
//----------------//
#if !defined ( ROUTE_DISCOVERY_TIME )
  #define ROUTE_DISCOVERY_TIME     13//edited by Prem default is 5
#endif
//----------------//

//modified in > ZGlobals.h
//----------------//
#if !defined ( BCAST_DELIVERY_TIME )
  #define BCAST_DELIVERY_TIME      100//edited by Prem default is 30
#endif
//----------------//


//modified in > nwk_globals.h              
//----------------//
#define DEFAULT_TC_LINK_KEY             { 0x5a, 0x69, 0x67, 0x42, 0x65, 0x65, 0x41, 0x6c,\
                                          0x6c, 0x69, 0x61, 0x6e, 0x63, 0x65, 0x30, 0x38 }  //edited by Prem   private TC key 
//----------------//


//modified in > nwk_globals.h              
//----------------//
#if !defined ( DEF_NWK_RADIUS )
  // the default network radius set twice the value of <nwkMaxDepth>
  #define DEF_NWK_RADIUS           15//edited by Prem default is ( 2 * BEACON_MAX_DEPTH )
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if !defined ( DEFAULT_ROUTE_REQUEST_RADIUS )
  #define DEFAULT_ROUTE_REQUEST_RADIUS 8//edited by Prem default is DEF_NWK_RADIUS
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if !defined ( LINK_DOWN_TRIGGER )
  #define LINK_DOWN_TRIGGER          12//edited by Prem default is 3   // Link is down if txCounter exceeds this
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if !defined ( MTO_RREQ_LIMIT_TIME )
  // in milliseconds. The time limited to one MTO RReq (Concentrator Announce)
  #define MTO_RREQ_LIMIT_TIME 5000//edited by Prem default is 1000
#endif
//----------------//


//modified in > nwk_globals.h              
//----------------//
#if !defined ( SRC_RTG_EXPIRY_TIME )
  #define SRC_RTG_EXPIRY_TIME 0//edited by Prem default is 10   // seconds before the source route entry expires
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if !defined ( MAX_RTG_SRC_ENTRIES )
  #define MAX_RTG_SRC_ENTRIES 210 //edited by Prem default is 12
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if !defined ( NWK_ROUTE_AGE_LIMIT )
  #define NWK_ROUTE_AGE_LIMIT 15//edited by Prem default is 3       // 3 missed link satus frames
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if !defined ( NWK_LINK_STATUS_PERIOD )
  #define NWK_LINK_STATUS_PERIOD 30       // 15 seconds  // modified by prem 
#endif
//----------------//


//modified in > nwk_globals.h              
//----------------//
#if !defined ( MAX_NEIGHBOR_ENTRIES )
  #if ( ZG_BUILD_RTR_TYPE )
    #define MAX_NEIGHBOR_ENTRIES    16
  #else
    #define MAX_NEIGHBOR_ENTRIES    16//edited by Prem default is 4
  #endif
#endif
//----------------//


//modified in > nwk_globals.h              
//----------------//
#if !defined ( NWK_MIN_ENDDEVICE_CHILDREN )
  #define NWK_MIN_ENDDEVICE_CHILDREN       10 // edited by Prem default is 0
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if !defined ( NWK_MIN_ROUTER_CHILDREN )
  #define NWK_MIN_ROUTER_CHILDREN          20 // edited by Prem default is 0
#endif
//----------------//


//modified in > nwk_globals.h              
//----------------//
#if !defined( NWK_MAX_DEVICE_LIST )
#define NWK_MAX_DEVICE_LIST     30//edited by Prem default is 20  // Maximum number of devices in the
                                    // Assoc/Device list.
#endif
//----------------//

//modified in > nwk_globals.h              
//----------------//
#if defined ( ZIGBEEPRO )
  #if !defined ( NWK_LINK_STATUS_PERIOD )
    #define NWK_LINK_STATUS_PERIOD 30//edited by Prem default is 15       // 15 seconds
  #endif
//----------------//


//modified in > nwk_globals.c             
//----------------//
#define NWK_MAX_DATABUFS_WAITING    16//edited by Prem default is 8     // Waiting to be sent to MAC
#define NWK_MAX_DATABUFS_SCHEDULED  10//edited by Prem default is 5     // Timed messages to be sent
#define NWK_MAX_DATABUFS_CONFIRMED  10//edited by Prem default is 5     // Held after MAC confirms
#define NWK_MAX_DATABUFS_TOTAL      24//edited by Prem default is 12    // Total number of buffers
//----------------//


//modified in > osal_nv.c             
//----------------//
#ifndef OSAL_NV_PHY_PER_PG
  #define OSAL_NV_PHY_PER_PG    2//edited by Prem default is 1
#endif
//----------------//


//modified in > hal_board.cfg.h             
//----------------//
#define HAL_NV_PAGE_CNT           12//edited by Prem default is 6
//----------------//


// Preprocessor symbol list
//----------------//
HAL_UART_USB
USB_SETUP_MAX_NUMBER_OF_INTERFACES=5
xHAL_SPI=TRUE
HAL_UART=TRUE
BDB_FINDING_BINDING_CAPABILITY_ENABLED=0
DISABLE_GREENPOWER_BASIC_PROXY
TC_LINKKEY_JOIN
ewarm
CC2538_USE_ALTERNATE_INTERRUPT_MAP=1
CC2538ZNP
ZNP_ALT
xPOWER_SAVING
FEATURE_SYSTEM_STATS
FEATURE_RESET_MACRO
ZDNWKMGR_MIN_TRANSMISSIONS=0
MT_UART_DEFAULT_OVERFLOW=FALSE
ASSERT_RESET
MAKE_CRC_SHDW
xSBL_CLIENT
ZCL_READ
ZCL_DISCOVER
ZCL_WRITE
ZCL_BASIC
HAL_PA_LNA_CC2592
CONCENTRATOR_ENABLE

//----------------//

//modifed in CC2538.icf
//----------------//
do not initialize { section .noinit };
place at end of SRAM { section .noinit }; // ++++++++++ ADD THIS LINE ++++++++++    // added by prem
//----------------//


//modifed in CC2538.icf
//----------------//
define region SRAM = mem:[from 0x20000000 to 0x20007FFF];//edited by Prem default is from 0x20004000 to 0x20007FFF
//----------------//


//modifed in CC2538.icf
//----------------//
define region NV_MEM = mem:[from 0x00279800 to 0x0027F7FF];//edited by Prem default is from 0x0027C800 to 0x0027F7FF
//----------------//

//modifed in CC2538.icf
//----------------//
define region FLASH = mem:[from 0x00200000 to 0x002797FF];//edited by Prem default is from 0x00200000 to 0x0027C7FF
//---------------

@ Ryan Brown 

There are no hardware differences with the deployment in question, except - the location and power source.

for deployment in question -

power supply adaptor used is 5v & 1.2A  Initially , after that changed and used  5v & 2.4 A   ... and input to adaptor is 220V & 60 Hz Raw power(without UPS)

for working deployment -

power supply adaptor used is 5v & 2.4 A  ... and input to adaptor is 230V & 50 Hz UPS power 

According to znp changes .txt, you revise a lot of defines in ZNP. I would suggest you to test default ZNP project without modification except enabling CC2592 to see if you still see the same issue.

Dhanraj,

Does the deployment in question have more nodes connected into the network or heavier channel traffic than other deployments?  Do any software resets or power cycles occur before observing the issue?  Can you replicate the issue by connecting a few devices and performing multiple resets to the fixed device inside your development environment?

Regards,
Ryan

Please find my answers in line below-

Does the deployment in question have more nodes connected into the network

> Deployment in question have very limited nodes -max 10 routers in a network and minimum 1 router in a network.

heavier channel traffic than other deployments

> channel traffic also very less..you can see in picture below 

Do any software resets or power cycles occur before observing the issue?

> NO software resets or power cycles occur before observing the issue, But if we do intentionally even after that also we see the same issue.

Can you replicate the issue by connecting a few devices and performing multiple resets to the fixed device inside your development environment?

I will test this and update you.

Hello Ryan Brown and YK,

I was doing testing and trying to recreate the issue in my development environment with one of the gateways which I got back from the deployment in question.

So during one of the experiments(as mentioned below), I am able to see somewhat similar symptoms consistently(i.e. > No link status and No MTO route request from ZNP)

>  whenever I keep the gateway on the CPU of my desktop, it stops sending link status and MTO route request but at the same time, znp sends mac acknowledgment to the report command sent by the router. (Basically, in this case, I don't see any network layer or application layer packet from ZNP )

> if I remove the gateway from the top of my CPU and keep it 10 -15 cm or more away from the CPU, it starts sending link status and MTO route request.

So far I have tested this issue with only 1 gateway and the above-mentioned behavior is consistent. 

Further, I will do this testing with multiple gateways. 

you can see in below picture how I have placed the gateway on my CPU during testing.

It sounds your GW are easily interfered by environmental changes. Can you take pictures of your PCB to show me your CC2538-CC2592EMK connection?

Please refer below picture to see the placement of znp and connection.

If you move CC2538-CC2592EMK outside of your case and still put them above your PC, do you still see the same issue?

If I move CC2538-CC2592EMK outside of the case and still put them above the PC,I still see the same issue.

You can see the picture of inside component of my pc.

Have you tested original ZNP only with CC2592 enable to see if you still see the same issue?

CASE 1 - 

I have used "CC2538_GW_ZNP_EM_StandAlone_USB.hex"  with my CC2538-CC2592EMK  znp.

in this case znp does not get affected by EMI of my PC. But the range is very less, I have to keep sniffer and znp both close to each other (in 25 cm range )

CASE 2 - 

I have built cc2538 znp firmware from original z-stack 3.0.2, in that, I have not defined "HAL_PA_LNA_CC2592"

in this case znp does not get affected by EMI of my PC. But the range is very less, I have to keep sniffer and znp both close to each other (in 25 cm range )

CASE  - 3

I have built cc2538 znp firmware from original z-stack 3.0.2, in that, I have defined "HAL_PA_LNA_CC2592"

in this case znp gets affected by EMI of my PC. (I am able to see the issue)

CASE  - 4

I have built cc2538 znp firmware from original z-stack 3.0.2, in that, I have defined "HAL_PA_LNA_CC2592" and TX power I have set to "TX_PWR_PLUS_8"

in this case znp gets affected by EMI of my PC. (I am able to see the issue )

Cases 1 & 2 are expected due to the CC2592 being connected but not used by the firmware.  Cases 3 & 4 insinuate that perhaps the LNA causes receiver sensitivity to believe that the channel is too noisy to send Zigbee packets due to CSMA/CA.

Regards,
Ryan

So far I am not able to observe the issues after the changes ...znp is working fine as expected.

I will continue the testing.

But why the znp(cc2538-cc2592emk) works normally when there is less noise(normal environment) ..even if cc2592 rf front end is not initialized. 

Oops, this is a known issue since Z-Stack Home 1.2.2a which you ca refer to . Not aware it's not fixed even in Z-Stack 3.0.2.