CC1312R: 15.4 stack loosing device link connectivity during high traffic situations

Hi Dirk,

this sounds like the sensor devices time out during the transmission of an OAD. The reason could be that the collector misses the poling requests of the other sensors thus they transition to orphan state.


Could you share the configuration of your network (polling, reporting, broadcast, timeout)?

How are the requests made during and OAD - do you use the process of the TI 15.4 Stack Linux SDK or a custom implementation?

Kind regards,
Theo

Hi Theo

I am working with Dirk on this issue. 
We are using 6s polling interval, 60s reporting interval and 60s braoadcast interval
We are still using the TI original OAD protocol described in below link.
https://software-dl.ti.com/simplelink/esd/ti15.4stack_linux_x64/2.03.00.06/exports/docs/ti154stack-users-guide/ti154stack/native-oad.html#oad-concept-overview
During the OTA we don't see problem on receive and handle the polling request from other devices.
Thanks

Daniel

Hi Daniel,

thank you for the details.

Daniel Tsang said:

During the OTA we don't see problem on receive and handle the polling request from other devices.

Are you tracking on the collector the received polling requests (dataIndCB) and do you check the packet sent callback (dataCnfCB -> ApiMac_status_success) to ensure it sent a reply?

I think to get more insight into which packets are lost it would be great if you could set both channel maps to only one channel and then you can observer the network traffic with the packet sniffer: https://www.ti.com/tool/de-de/download/PACKET-SNIFFER-2/1.11.0 

Could you share a sniff log with me that shows the szenario where a sensor is lost?

Kind regards,
Theo

Hi Theo

Thanks for your reply

I didn't use the snipper before, so may need some time to setup
btw, I dig into the packet flow as your advice, addding some more log messages on the api-mac especially the data-cnf. Maybe something is related to this
I find we are sometime get long delay or even missing data-cnf, and mishandling on this may cause failure on following sending messages
May I know, on missing of data-cnf, what should be the proper handling ? is it the purge request with correct msduhandle can always help to recover ? or any other action need to be done before sending message again ?

Thanks again

Daniel

Hi Daniel,

please let me know if you need additional guidance to set up the sniffer.

When you say "log messages" are you using already the TI logger? - https://dev.ti.com/tirex/explore/node?node=A__AUtBCHw9KYI99xItuPjn4w__com.ti.SIMPLELINK_ACADEMY_CC13XX_CC26XX_SDK__AfkT0vQ__LATEST 

The stack implements re-sending if no confirmation is received. On the sensor side if no confirmation is received it transitions to orphan mode and will try to re-join the network with a backoff timer.

Can you see the mac success on the collector for sending the message to the sensor?

I think we need the sniffer log to confirm which messages are actually sent and missed.

Kind regards,
Theo

Hi Theo

I've installed the sniffer for our setup, and attached is the catpure during the issue of devices disconnected during ota process. Please help check if this log can show the cause. 
 ota.zip

Thanks a lot

Hi Daniel,

please help me to fully understand the log that you shared.

I can see that all sensors try to join the network by sending out the PAS (Pan Advertisement Solicit) message but I also see that the collector sometimes does not respond with the PA (Pan Advertisement) message as it would be expected.

Dirk Beiner said:

during high data transmission to 1 individual RF client the stack starts loosing connections to random other clients (MAC_WS_ASYNC_IND)

scenario is that we send data packets (150 bytes) with a delay of 200ms between packets, about 1k packets in cue (effectively trying to do a firmware update)

I can not see this high data transmission to one of the sensors in the log.

Which APIs and communication pattern do you use to run it?
Is it running while you recorded the log?

Kind regards,
Theo

Hi Theo

Is this sniffer config for US band 915MHz correct ? seems I only captured the association messages but not the data exchange !

Hi Daniel, 

the settings look correct.

Could you please share with me how you implemented the "high data transmission" ?

Kind regards,
Theo

Hi Theo

We implemented this high data tranmission when we do the OTA for the sensors, sending 128bytes of firmware blocks per 200ms from collector to the sensors. Totally will send from 900 to 2000 block,  depends on different kinds of sensors.

Hi Daniel,

What processor are you using as your gateway, and what UART interface are you using to communicate with the coprocessor?

Regards,

Arthur

Hi Theo

We are running Linux SDK on Pi CM4, communication with CC1312 coproccess module with mt-msg layer and uart interface.

Hi Daniel,

thank you for the details. I think there should be no issues on the UART communication.

The base problem here is that when you start the OAD the sensor will keep requesting data until it has received the image.

Since there is no reserved communication slots for the other sensors there is a high chance of collision with the OAD messages.

If multiple collisions in a row happen the sensors will transition from resending to orphaned.

Now, to improve this situation you could transitions the sensors directly to re-join when orphaned which would allow for a faster re join after the collector finished the OAD.

Kind regards,
Theo

Hi Theo

I've checked again that our devices have already implemented the re-join algorithm, and the devices can paired again to the collector after lost link in the OAD process. But we are stil want to get some more understand on it. 
While one devices is doing OAD, with requesting firmware packets per 200ms, and collector keep replying the 128 bytes firmwarre packets
Then another deivce is sendiing a sequence of 5 readings to the collector, with each reading up to 150bytes.
Then the collision will happened can cause the OAD devices go into orphan mode ?
Is this can be captured through the sniffer software ? 
We just want to understand more in order to see how can we adjust our algorithm to improve it.
Thansk 

Daniel

Hi Daniel,

1. I'm trying to replicate the issue but it doesn't happen yet. Can you share the F2 SimpleLink SDK version you are using?

2. When using the sniffer are you mapping only one channel for the frequency hopping?

Best,

Daniel

Collector is using CC1312 6.10.002 Linux co-processor
Sensors are using CC1312 6.10.002 and CC1310 1.50.0.08
When using sniffer, we are configured using all channels

Thanks Daniel,

I've tried two sensor nodes and a coprocessor node running SDK 6.10.01.01 and I still cannot replicate the issue, maybe I'm missing a configuration. Are you running the default example codes or have they been modified?

Unfortunately the Sniffer can only work at one frequency at a time, so for a Frequency Hopping mode, you would have to use only channel 0 (Set on SysConfig for the sensor, and collector.cfg for the collector) to get everything recorded on the sniffer.

Have you enabled all debug flags on collector.cfg? Can you see the sensor requests arriving to the coprocessor?

Best,

Daniel

Hi Daniel

We are started from example code but shold have already so much modifications.
I will try sniffer again with set to channel 0 on sensors and collector.

Regards
Daniel

Alright,

Let me know how it goes.