CC1310: what are the possible causes of a sensor node disconnects from a collector in TI 15.4 network with frequency hopping enabled?

Hi Zhiyong, 

Which version of the SDK are you seeing this behavior? 

Regards,

Sid

Both sensor and collector nodes are customized from and built with CC13x0 SDK 4.20.1.03.

Upon further analysis, I found that if more than one data packet are pushed to sensor node by an external MCU, within up to 5 seconds, sometimes the later data packets, sometime all of them, will not be sent to collector.

Am I doing something wrong or there is a limit on how many data packets a sensor node can handle within a period of time?

Thanks.

Hi Zhiyong,

The reporting interval itself on the sensor can be less than 5 seconds. The default we provide it self is 3 seconds. So that shouldn't be an issue. But when you say data being pushed to the sensor with an external MCU, do you mean there is another serial interface?

Regards,
Sid

Yes, we implemented an UART interface so we only use CC1310 as radio, all other functions are handled by a MSP430. We also didn't find problems with small number of sensor nodes with less than 5 seconds interval of data reporting. But the problem became obvious as the number of sensor nodes in a network increases. The network I am seeing constant lose of data packets have 9 sensor nodes.

We also enabled frequency hopping, without FH, the limit on TX power is a lot lower. Will FH make the network more delicate too?

Hi Sid,

Thanks for the idea to setup and use packet sniffer. I will try that next.

For now, I suspect saturation might be the cause of at least some of the problems. I reduced the test network to 3 radios, 1 collector and 2 sensor nodes, placed them10s of feet apart, and everything started to work just fine. Even with burst TX at once per second. While in previous test setups, there were sometimes 10+ radios, and radios were sometimes only several feet away from each other. Also, the TX power of those radios are left at default 26dB.

Are there any guidelines on how to manage/avoid saturation? We designed the radio to cover large areas, but sometimes we need to install a lot of them into small spaces as well. I assume reducing TX power whenever possible is one way to avoid saturation.

Best,

ZL

Hi Zhiyong, 

Reducing Tx power seems to be the correct way to test if saturation is the cause, also please can you measure the RSSI with Smart RF studio on the Receiver. With the same Phy settings on the RX? 

Regards,

Sid

We track RSSI on both sensor node and collector radios by using the function built-in TI 15.4 stack. In the test setup I am working on, RSSI on both ends range between -26 and -42, average ~38.

Hi Zhiyong,

It seems that the received power is within reasonable limits. Do you think it is an issue with the traffic over the air? I think the sniffer log would provide more insights. Or a reliable failure pattern would also help root cause the issue at this point. 

Regards,

Sid 

The actual cause of this network instability turns out to be some sort of stack overflow on collector radio. We tried to send a data packet to each sensor node every 5 minutes. On the surface, this doesn't seem much. But in our network sensor nodes only polls for data once every minute, and there are some background tracking data packets. I guess this is too much to handle for CC1310. Once this feature is turned off, all sorts of weird behavior just went away.

Hi Zhiyong, 
Thanks for getting back. I want to clarify the behavior a little. What is the feature you are turning off for the issue to go away? 

Please can you define the actual network traffic, maybe some change to the MAC_CFG_TX_DATA_MAX may help. But it would be great to understand what you are limited by. 

Regards,
Sid

Thanks for the update!

Sid said:

maybe some change to the MAC_CFG_TX_DATA_MAX may help.

/* maximum number of data frames in transmit queue */
// Assuming: 50 sensor nodes, host state packet 1 per 5 minutes, plus 1 time sync packet per hour
// Total packets: 650 packets in one hour
// Worst case: all 50 host state packet queued in 1 polling interval
#ifndef MAC_CFG_TX_DATA_MAX
#define MAC_CFG_TX_DATA_MAX         50
#endif

We already changed that to 50, that doesn't seem prevent the problem from happening.

Have you tried a smaller number here? 50 maybe a bit too much. But a number greater than default(2?). Maybe 10. Sorry that I do not have a better calculation for this number, but it is worth testing once. 

Hi Sid,

A related question, is there a way to query pending TX data packets on MAC layer? We implemented a way to manually track data packets pushed in from external MCU. Unfortunately, that doesn't work well. It doesn't track background tx packets by stack itself. And I just found out that calling that function frequently also contributes to network instability.

Error_t Cllc_getTotalPendingDataReq(uint16_t *totalPendingDataReq)
{
    uint16_t pendingDataReqCount = 0;
    for (size_t i = 0; i < CONFIG_MAX_DEVICES; i++)
    {
        pendingDataReqCount += Cllc_associatedDevList[i].pendingDataReq; // Added during customization
    }

    *totalPendingDataReq = pendingDataReqCount;

    return ERROR_OK;
}

Hi Zhiyong, 

If you just want to check if there is an overflow, checking the status of the ApiMac_mcpsDataReq is the easiest way. It should return a transactionOveflow status. 

But to keep track of the actual number of pending messages, this function might be useful: MAC_CbackCheckPending() 

Regards,

Sid

Hi Sid,

I tried 10 and with the following settings, MAC_CbackCheckPending() always returns 0 even when ApiMac_mcpsDataReq() returns overflow error.

Is there a better way to avoid overflow?

Thanks,

MAC_CbackCheckPending()

// mac_cfg.c
/* determine whether MAC_MLME_POLL_IND will be sent to the application */
#ifndef MAC_CFG_APP_PENDING_QUEUE
#define MAC_CFG_APP_PENDING_QUEUE   TRUE
#endif

Hi Zhiyong, 

Avoiding overflow itself I think comes from scheduling the commands at a larger interval. But I will look into how to track these messages better and will get back to you. 

Regards,

Sid

Hi Zhiyong, 

In macTask.c you see there is a macData instance being used. That structure has the count of indirect and direct messages left in the queue. 

That could be used to track the pending messages. 

Regards<
Sid