CC2640R2F: After a period of time, the CC2640R2 stops reporting beacons.

Hello,

Thanks for posting. I've assigned your post to an expert to comment. In the meantime, can you tell me what SDK version and project the devices are based on? Do you see similar behavior with simple_observer? How often does this occur? Is it reproducible 100% of the time, or intermittent? Can you provide more context as to when the devices fail (are they running for days, hours, etc before they fail)?

Hi,

Ammar's questions are very pertinent to better describe this issue. In addition, you mentioned you are testing these "additional features" and seeing challenges but which ones? Did you move to Bluetooth LE 5 or is still in 4?

During scan, do you see anything strange shown on a bluetooth sniffer capture? Perhaps some condition was indeed sent over the air that might clarify the trigger that is collapsing the CC2640R2.

Regards,

Rafael

The project is based upon our cayman project.  Not sure what the simple_observer is.  This seems to happen after hours of operation.  We cannot get this to fail quickly and if we have a console hooked up to the unit, we have not seen what the microcontroller is doing when this occurs.  Also, once this state occurs, it will not clear until we do a power reset to the entire board.  Since we do not know when this happens, we have not tried a reset just to the BLE chip but our experience says this will probably work.  But we cannot be rebooting the BLE chip whenever we get a scan with no beacons found.  This will get reported as a potential error condition and could upset customers.  One sw engineer thinks this state is induced when we do a reboot.  Reboots might happen if the micro-controller loose WIFI connectivity for some period of time or maybe the unit has its lease expires and reboots to get a new IP address.  If we could capture a console log when this happens we would have more input.  Can I add our BLE firmware engineer's email to this thread and he can answer the question regarding the TI SDK he is using.  It is interesting to us that the unit appears to be scanning and responding with a completion code within the window of our watchdog timer, just don't transfer any beacons.  I think our firmware engineer could also tell you what flags are set during the scan that determine what beacons are passed to the MCU and what are tossed.  I think we have the best RSSI per Mac address set which will give us 1 instance of each beacon heard during the scan.  Not sure if there are other parameters set.

I do have wireshark running on the WIFI so I can see packets from our hub which show when beacons events stop happening.  But this is not what you are looking for.  I also have a BLE sniffer which interfaces to wireshark and I could do a capture with that.  But seeing that I have 6 units all working at the same time in a screen room, 3 are now not reporting any beacons while the other 3 have been reporting the same beacon set for over 4 days, I don't think there is a BLE RF event happening that three units would respond to at various times while the other 3 remain operational.  But RF is a tricky business.  I wondering if there is a command we can sent to the unit that might reverse this operation without doing a BLE reset. 

For CC2640R2, we are going to upgrade to the TI SDK 5.10, simplelink_cc2640r2_sdk_5_10_00_02.  We have asset tags that also are using the CC2640R2F and this same SDK.  Asset tags only transmit and we do not see anything problems with the TX operation.  We also have a product which beacons always and periodically listens which uses the CC2640R2.  we have not had a problem with these as well.  I am going to try to see if the 5.1 SDK fixes this issue and will respond with results.

Hi,

Please apologize for the delay. I agree the debugging of such system is quite difficult, as the faults happen over days and not seconds.

The move to a newer SDK indeed makes sense. I will think about possible ways to debug this but it seems you covered most of your bases. I will wait for your findings as well.

Regards,

Rafael

We have 12 units now running the updated code, see how the newer SDK behaves.  I think the previous version was pretty old for the CC2640R2. Check status on Monday.  Before I upgraded units, I did have another hang which ran more than 5 days before slipping into this mode.

We have update our CC2640R2 code to the 5.1 SDK and the problem still remains.  We have caught on our wireshark monitoring that the problem occurs when the unit reboots for some reason.  Our reboot code indicates the reboot was from a CPU driven event.  The CPU may reboot the unit due to WIFI connectivity, network communication with a server, IP renewal or failure, etc.  The error code we see is general so we specifically don't know exactly but if the reboot was a BLE issue, there would be a specific BLE code.  We also see units that have a greater number of beacons to report seem to be more likely to show this behavior but we have units in both environments (lots of beacons vs very few) that are running fine.  We have had only one unit that has stop hearing BLE beacons in a low beacon environment.  We have extended the RESET timing on the BLE chip on bootup to 1000 msec but this did not improve the problem.  I have 5 units hung after running 5 days.

Hi,

Thank you for sending additional details. I am sorry to hear the update did not work to circumvent the issue. Given the rebooting of the unit seems to be caused to an intrinsic aspect of the unit (the CPU Halt/reset), if possible, perhaps keep one of the failing units tied to a standalone JTAG/CCS running the code and with exception traps enabled, so at least you could get additional visibility about the status of the CPU when it locks/fails.

Details on how to enable the Exception trap in CCS is shown at the e2e post below:

https://e2e.ti.com/support/wireless-connectivity/bluetooth-group/bluetooth/f/bluetooth-forum/973731/cc2642r-debugging-memory-issues/3598030#3598030

Apart from this, I really am out of ideas on how to move forward in this case. Sorry.

Regards,

Rafael

We are testing a fix we think is promising.  Our sw engineer reported that one of the first things he does after a system reboot is that he reboots the CC2640R2 and then attempts to get the version information out of the device as a test to see if the device is responsive.  He indicated that he tries 4 times and then proceeds with the regular BLE init process if the chip doesn't respond.  After a brief discussion, we decided that we should not proceed and instead issue a new RESET to the CC2640R2 and try again without doing an entire system power reset.  I have 11 units that have ran over 5 days without the hang issue appearing.  Prior to this change, we were seeing at least 1 unit hanging in the first 24 hours.  We have been using the CC2640 for over 4 years and never seen this behavior (which probably means within the timing of the 4 requests one always gets a response, not so with the CC2640R2).  I will let you know if we decide that this fix is golden.

Hi,

Thank you for the additional details provided. This might prove helpful for other developers in the future.

Regards,

Rafael