CCS/LAUNCHXL-CC1310: Frequency hopping: Network testing in small and medium size

Hey Slev1n,

I am looking into your questions and follow up soon.

Quick Update: I've also tested with standard US PHY (50kbps) and I come to the same result.

1. The sensors easily connect/associate with the network.
2. Most but not all sensors have polling/config issues, where the collector does not receive an acknowledge for the config message send to the sensors (one sensor just was able to receive a config request after 16hrs, when polling is each hour? In non beacon mode the update occurs just after the association during the first polling)
3. Besides, sensor messages polling/data are not always acknowledged by the collector which causes a disconnect and rejoin (quite often) of the sensor
4. Some of the data messages from the sensors show a very high RSSI value -90dBm or worse though they are lying directly next to the collector. Sometimes the same sensors have RSSI values of -16 to -30dBm. Clearly an indication for a transmission on a close by but wrong channel.

My assumption, incorrect transmission channel on both sides.

Hey Ammar, thanks for responding. Some info. The code we are using bases on the out of the box example. We have added some stuff to using non-beacon mode. This system works as we want it. Join, rejoin, disassociation, polling and data messages or retry have been changed. Now, we want to achieve the same in FH mode. I have made some minor changes to the join/rejoin section and the disassociation section. These two parts seem to work fine except for the PAN_ID issue.

Now to the questions.

Ammar N said:

1. Hm, you should see the same behavior regardless of panID. Are you changing the panID dynamically after startup, or just changing the configuration before building your binary file? Can you try resetting the NV and see if this behavior persists? I'm wondering if the collector actually changed the panId or if it's operating on the old panId. I guess another way we can test is to set the channel mask to 1 channel on each device to see this in a sniffer.

I can change it dynamically using: ApiMac_mlmeSetReqUint16(ApiMac_attribute_panId, netInfo.devInfo.panID); By changing the PAN_ID all following messages from the sensor will be ignored until they disconnect and look for a new network and join the collector again. In non beacon mode, this works without issues.

Ammar N said:

2. I'm not sure what you mean, I see that the polling clock is stopped in all modes in the disassoCnfCb. Can you show me what you're referring to?

Oh, you are right I misread !CONFIG_RX_ON_IDLE condition...I had CONIFG_FH_ENABLE in mind.

    if(!CONFIG_RX_ON_IDLE)
    {
        /* stop polling */
        Ssf_setPollClock(0);
    }

Ammar N said:

4. Are you seeing this behavior with modifications, or on the out of box example?

See the beginning of my post. After association, I did not change much in the code. Besides, the same code works for non-beacon mode. I think something with the acknowledging goes wrong.. In dataCnfCB(ApiMac_mcpsDataCnf_t *pDataCnf) the status pDataCnf->status is 0xF0 meaning: ApiMac_status_transactionExpired = 0xF0

The sensors do disconnect after two consecutive messages are not acknowledged and according to many async messages being send in the network and the number of successful associations is supporting my theory that there is something wrong with acknowledgments.

Ammar N said:

For your most recent reply, if you separate the boards, does the RSSI change accordingly?

Yes it does. And I know from sniffer in non-beacon mode, that these sensors perform better than -90dBm in this distance...

It is really strange. In non beacon mode everything works fine. Enabling FH data exchange seems to be very bad. I went through the collector logfile. One sensor transmits a message and the collector indicates an RSSI of -107dBm. I see, that the sensor does not receive an ack. Next time, the message from the same sensor has an RSSI of -15dBm, to me this is clear, that there is some sync error and transmission is performed on a wrong channel.Another sensor has -113dBm, -96dBm and then -15dBm. I also have one sensor in the network which performs 100% perfect...

What do you think about that?

best wishes

Slev1n

I have some updates:

I changed the channel mask to enable only 1 channel and tested again decreasing the interval from 1 hour to 30s. The network works fine. I did then enable the 64 channels again keeping the same low message interval time of 30s. Again, everything works fine. But if I increase the interval again (polling to 20min and data to 1h) again a lot of messages are not reaching the collector causing the devices to disconnect and rejoin.

Interestingly, the launchpad is the only sensor seeming not to have any issues, only the TIDA 00489 boards...I am quite sure, the problem are timing/sync issues. Any idea what else I could try? Such low intervals are too power consuming.

EDIT: I set the channel mask to 1 channel again and kept the 1hour reporting interval and 20min polling interval and everything works fine and stable. Again, this points to a sync error between hopping sequence of collector and transmission channel of sensors.

Best wishes

Slev1n

This is useful information, I wasn't aware you were using the TIDA boards as a base. I see the TIDA board's antenna is tuned for 915 Mhz. The antenna on the launchpad also looks larger, so it makes sense that the launchpad wouldn't miss any packets. If the same software works well on a launchpad, there may not be any timing/sync issues.. Do you have a second launchpad to confirm it functions as expected?

Can you summarize which FH parameter's you've modified and to what?

Hey Ammar,

Ammar N said:

If the same software works well on a launchpad, there may not be any timing/sync issues.

I do have 10 Tida boards and 3 launchpads (2x CC1310 for sniffer and sensor software, CC1352R1 for collector). It is the launchpad with sensor software that seems to be working even with higher intervals, though I cannot imagine that there should be an performance issue between tida and cc1310 launchpad regarding sync. However, I wanted to inform you.

Ammar N said:

Can you summarize which FH parameter's you've modified and to what?

Sensor changes:

- I made changes during join phase. I changed the async send behavior and when to set trickle timer or go to sleep.

- I made changes during disassociation to allow disconnecting sensors to search for new collectors again (e.g. Ssf_clearNetworkInfo();)

- I changed back_off_exp to 5 and 8 (CMSA is still 4)

- My FH channel masks (the same for collector):

#define CONFIG_FH_CHANNEL_MASK        { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, \
                                        0x00, 0x00, 0xF8, 0xFF, 0xFF, 0xFF, \
                                        0xFF, 0xFF, 0xFF, 0x3F, 0x00 }

#define FH_ASYNC_CHANNEL_MASK         { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, \
                                        0x00, 0x00, 0xF8, 0xFF, 0xFF, 0xFF, \
                                        0xFF, 0xFF, 0xFF, 0x3F, 0x00 }

- non sleepy sensor neighbours 2 and non sleepy with fixed channel is set to 20.... Maybe this is something to change? because I actually dont have any non sleeping neighbours except for the collector.

- a lot of debug messages using System_printf()

- I've already asked before, but my reporting interval is only an approximation, the sensor may deviate from the exact interval timing. But you told me it does not matter what the sensor reporting interval says, since the stack will always take care of the correct channel no matter when I transmit, correct? The 1 hour interval is rather an average value.

Collector changes:

- backoff exponents to 5 and 8 (instead of 3 and 5)

- channel neighbours are all set to 0...should I do the same for sensor?

- broadcast dwell time set to 0

- new NetName (same on sensor and collector)

- a lot of debug messages using System_printf()

Hope you will find something interesting.

Best wishes

Slev1n

Any update?

Hey Slev1n,

Sorry for the delay, I was OOO. Taking a step back, have you cleared the NV on each sensor before running the test? I just want to make sure old network data is not being used.

I'm curious to see if changed the backoff exponents back to defaults changes the performance.

When you refer to "channel neighbours are all set to 0", do you mean FH_NUM_NON_SLEEPY_HOPPING_NEIGHBORS/FH_NUM_NON_SLEEPY_FIXED_CHANNEL_NEIGHBORS? On the collector, this should be OK if you aren't connecting any non-sleepy sensors. I am also curious to see what the network performance is like if you enable the sensors in a non-sleepy config. (If you do run this test, be sure to set the variable above accordingly to allocate enough memory. It sounds like the issue may be on the collector side, and this should rule out any issue with the sensors receiving anything from the collector.

Hey Ammar,

Ammar N said:

Taking a step back, have you cleared the NV on each sensor before running the test? I just want to make sure old network data is not being used.

The sensors have been flashed and the memory was erased before loading the program. I assume this should have cleaned the NV.

Ammar N said:

When you refer to "channel neighbours are all set to 0", do you mean FH_NUM_NON_SLEEPY_HOPPING_NEIGHBORS/FH_NUM_NON_SLEEPY_FIXED_CHANNEL_NEIGHBORS?

Yes, I meant those variables. In collector both were set to 0 and in sensor both were set to 2. However, for the test you asked me to perform I changed those two variables for sensor and collector to 10 each.

Ammar N said:

I am also curious to see what the network performance is like if you enable the sensors in a non-sleepy config.

I changed the backoff exponents back to their initial value for sensor and collector and I set the sensors to CONFIG_RX_ON_IDLE = true, hence, they were non-sleepy.

I ran the test with 6 TIDA boards and the CC1352 as collector. The sensors connect and get configured immediately (initial polling on sensor side was 10s). After one hour the sensors send their message, 5 of 6 arrive without issues having an RSSI value of 15-17 dBm, one sensor enters handleMaxFail, reconnects and transmits successfully. Again, one hour later only 3 out of 6 arrive, whereas two of the messages have an RSSI of -101 and -69, again, a clear indicator for loss of synchronization. 2 sensors enter handleMaxFail -> disconnect -> reconnect and transmit with an RSSI value of -15dbm..

I have no idea why this happens and what could be the reason, hope you can help me out once more. Could the application somehow be the reason for the loss of the synchronization or is the stack somehow immutable regarding the hopping sequence and timings?

best wishes

Slev1n

I ran another test, with two CC1310 launchpads and 4 TIDA boards, again only the launchpads were able to keep track.

I dont know if its the sensors or the collectors problem, but it seems to be a fact, that only the tida boards have issues. Thus, could you get an expert from the TIDA side asking if he has an idea?

Besides, if it is a timing issue where the devices get out of track if messages have a too long interval, how could I proof this? I mean I am quite sure since decreasing the intervalls to 30s makes the system run perfectly, though I'd like to compare tick numbers or something like that. Any idea how to achieve that?

best wishes

Slev1n

Hey Slev1n,

Thanks for running those tests. It looks like the issue may be with the TIDA boards in particular, and am trying to think of a valid workaround at this moment. 

We have another TIDA using the same board, but geared to sensor to cloud. In Section 2.4.5 Polling Intervals and Frequency Hopping, it states that the max amount of time data/acks should be sent is 20 minutes. Source: http://www.tij.co.jp/jp/lit/ug/tiduda8/tiduda8.pdf

Again, I'm looking into possible workarounds. Have we tried to reduce the dwell time on the collector? This should allow the collector to hop between channels faster and increase the likelihood of receiving packets on a given channel.

Ammar N said:

We have another TIDA using the same board, but geared to sensor to cloud. In Section 2.4.5 Polling Intervals and Frequency Hopping, it states that the max amount of time data/acks should be sent is 20 minutes. Source: http://www.tij.co.jp/jp/lit/ug/tiduda8/tiduda8.pdf

Thank you for that hint, at least we do now know where to look for an answer.

Ammar N said:

Again, I'm looking into possible workarounds. Have we tried to reduce the dwell time on the collector? This should allow the collector to hop between channels faster and increase the likelihood of receiving packets on a given channel.

I havent tried that, I will test with 100ms and report. But could you elaborate what causes the problem of early de-sync? Maybe I can also think of a solution/work around.

best wishes

Slev1n

The frequencies of the oscillator will drift over time on each device and could cause a de-sync scenario, especially in a timing critical application.

I will reach out to some colleagues to see if they have any ideas. 

Ammar N said:

The frequencies of the oscillator will drift over time on each device and could cause a de-sync scenario, especially in a timing critical application.

Ok I understand. Are there different oscillating quartz used on the boards?

Ammar N said:

I will reach out to some colleagues to see if they have any ideas. 

This would be very helpful. We are close to certification and would like to know if hardware changes are necessary or if we can start certification.

best wishes

Slev1n

Hi,

32.768KHz Crystal is used for RTC.

You can tune the Crystal loading Caps for better frequency accuracy.

Please refer to SWAR640 on how to measure the frequency. The following is the link to SWRA640.

Thanks,

PM

Hey PM,

I read through chapter 6 and still have few questions.

1. The schematics of the TIDA board and the CC1310 launchpad seem to be identical. Both have a 32kHz oscillator and both condensators are 12pF. Why are they performing differently in the first place? Are the oscillators different and would replacing the TIDA's oscillators with the launchpad's oscillator in the future (I assume no re-certification would be necessary) be a solution?
2. The whitepaper states, that the clock signal should be output on a pin and then measured by means of an oscilloscope. IOCPortConfigureSet(IOIDn, IOC_PORT_AON_CLK32K, IOC_STD_OUTPUT); The only argument which has to set by myself should be IOID, right?

EDIT: Question 2 has been solved: The TIDA boards have a frequency of approx. 32.775Hz (I tested 4 of them) and the launchpad CC1310 has a frequency of about 32.768 Hz.

3. How would I have to alter the capacity to achieve a better performance?

4. Even with 10min reporting time and 20min polling time, the system is a little bit more stable but still looses messages and hence, loose of sync after two or 3 successful transmissions.

5. Can I somehow compensate the drift with software changes?

Best wishes

Slev1n

PM said:

Q1: Oscillator circuit is same but due to Manufacturer tolerances of Crystal and parasitic on the PCB makes a Frequency little different to each other. That's why we need to trim the Loading Caps of the Crystal for Frequency Accuracy.

The package and footprint of the 32kHz oscillator on the launchpad looks different to the one on the TIDA board.Are they really the same (cant get enough info from the BOM)? However, if manufacturer tolerances are an issue, you would still suggest a manual trimming of the loading caps? Or do you mean if we find the correct loading cap once, we can always be sure that this loading cap value will do the trick?

PM said:

Q4: Did you check the Frequency Accuracy of the Main Crystal (24MHz)? If not please check and adjust the Loading Caps / Cap Array value for better freq accuracy. Alternatively, you can widen the Rx filter Bandwidth on the Receiver and test again.

I havent checked the 24MHz crystal so far, but since in non-beacon mode I never had any issues transmitting on a certain channel (I tested on many channels in 868MHz and 916MHz band) and when using frequency hopping, a reporting interval of 30s does the trick, I am quite sure that this is not where to look for a solution.

PM said:

Q5: Yes, you can do it. But you need to characterize your board for a full operating freq range and test the drift of Crystal frequencies and then use compensation (adjusting the Frequency) in the code. Instead of this you can widen the Rx Filter Bandwidth. This helps if there is a drift in the RF Frequency.

Assuming only the 32kHz crystal is the problem. Identifying the deviation from 32.768Hz once, should be sufficient (the same approach like adjusting the caps once in Q1). If let's say the deviation is +7Hz (like I measured), which equals 213ppm, than when the µC thinks that 1s = 1,000,000µs has passed, actually only 999,786.4µs have passed so far, because the frequency is higher. So, since 1tick = 10µs, the counter has to be adjusted with -22 ticks after 1s. There would be still a drift error of 0.6 ticks = 6µs per second, which means after 3600s = 1h, a drift of 21ms occurs. Still not acceptable, but I guess you guys know how to compensate for the drift more precisely?

Q5.1: How could I implement a compensation scheme?

Q5.2 I have the same worries like for Q1, can I be sure, that this deviation is the same on all boards? Obviously, it is impossible to adjust drift by hardware or a software factor manually in mass production.

I am really thankful for your support, since we are close to certification, we really have to get this issue solved.

Best wishes

Slev1n

Q1: What is TIDA board?

When you mentioned Oscillator, I thought you ware asking the Oscillator circuit inside IC. If the 32.768KHz are different from different manufacturers then they will have different tolerances and different loading Caps unless we ask them the same specs.

Yes, we have to tune the loading Caps manually to obtain better freq accuracy. This should be done on 1st prototype boards. Then you can implement the same cap value for the subsequent production boards ( if you use same Crystal). If you use different Crystal then you have to check again.

Just I am curious, do you have any RF experience?

Q4: Please check the RF Freq Accuracy. Please check it in TX CW-Unmodulated Mode and put the Spectrum Analyzer to 10KHz or less span for measurement.

Q5: First you adjust the Freq offset by using the Loading Caps. We recommend to use the Crystal  (20ppm) from Epson or equivalent one. Attached is the data sheet. If you use this Crystal and adjust the accuracy then you may not need any compensation mechanism. We have no issue on our Launchpad design.

FC-135.pdf

Thanks,

PM

PM said:

Q1: What is TIDA board?

When you mentioned Oscillator, I thought you ware asking the Oscillator circuit inside IC. If the 32.768KHz are different from different manufacturers then they will have different tolerances and different loading Caps unless we ask them the same specs.

I am sorry for the confusion. We have the TIDA 00489 board. It looks like the external 32kHz crystal has an offset of 7Hz compared to the launchpad (CC1310 or CC1352R1 which are used as collectors). The reason why we think that is because we get a de-sync in frequency hopping quite fast. 30s transmitting interval is fine, but even at 5min, we start getting de-sync.

PM said:

Just I am curious, do you have any RF experience?

Before I started this project, I didnt have any experience about RF and hardware design.

PM said:

Q4: Please check the RF Freq Accuracy. Please check it in TX CW-Unmodulated Mode and put the Spectrum Analyzer to 10KHz or less span for measurement.

You mean the 24MHz crystal? I assume TX-unmodulated cw can be achieved with SmartRF Studio. We only have an oscilloscope but no spectrum analyzer.. Could you elaborate why you think we have an issue with the 24MHz crystal?

PM said:

Q5: First you adjust the Freq offset by using the Loading Caps. We recommend to use the Crystal  (20ppm) from Epson or equivalent one. Attached is the data sheet. If you use this Crystal and adjust the accuracy then you may not need any compensation mechanism. We have no issue on our Launchpad design.

We will keep that in mind. We used the original BOM from TIDA 00489, but if this the way to go, though we do not have the resources for much hardware experiments, since we can only manufacture externally (of course we can exchange condensator by ourselves)

To sum it up, I think we have issues with the 32kHz crystal not the 24MHz one, since the RTC seems to be the core issue. We do not have much testing equipment and not the resources for much redesigning and tests regarding hardware. Thus, I think the best way for us is either a software compensation or adaption of the crystal caps. Either way we will have to stick with the quartz or at least with the same footprint. Any comment on that is welcome.

best wishes

Selv1n

Hey PM,

measuring the crystals frequency was done by a oscillocope using 250MS/s and averaging over 512 values. The maximum resolution achieved was 7Hz. The launchpad showed a jittering value between 32765Hz and 32772Hz leading to the assumption of an average frequency of about 32768Hz at RT. The TIDA 00489 boards showed a deviation of about 7Hz. I replaced C41 with 39pF to get the same results as with the launchpad. Running some tests over night, again at RT showed that the two TIDA boards were able to send messages with 20min and 1h reporting/polling interval.

We will run a test in a climate chamber to verify functionality at higher and lower temperatures.

Q: Do you also think, that we can keep our current design and only change C41 to 39pF if the rest stays "identical"?

Best wishes

Slev1n

Glad to hear.

Thanks PM for your support here!