CC2640: Device stops working at random times.

Hello Iain,

Thanks for reaching out.

There are several elements coming into the picture here. Please help me with the following questions to better understand the case:

1. How are the devices communicating between each other? Are yo using a network processor interface for example (assuming yes from the simple_np)?
2. How many of the products are failing (%) and how frequent is this reset happening (% reproducible)? This is for CC2640 devices right? or is it that the ones that work are only CC2650 and CC2640 are not working?
3. Do all the samples share exactly the same hardware?
4. Could you please provide some debug logs showing this "communication with the CC2640 getting stuck - no response back to the MSP on the UART and the watchdog then causes the reset".
5. In the second use case, is the application initiating a connection after discovering? What are the scanning parameters?
6. Considering the case where there is no unwanted reset and you are not holding the device in reset before discovering, could you please provide some debug logs showing this "random times the data from any discovered devices stops getting sent to the MSP., so looks like we have no BLE detected.". Have you validated that the device is actually scanning at those moments?
   

You can look into the "Debugging Output" section at the User Guide.

Please also specify the SDK version you are using.

Best Regards,

David.

Thank you for the response.

See below for answers to your questions:

1. How are the devices communicating between each other? Are yo using a network processor interface for example (assuming yes from the simple_np)?

ANS. - Yes, using NPI over UART with MRDY and SRDY providing control.

1. How many of the products are failing (%) and how frequent is this reset happening (% reproducible)? This is for CC2640 devices right? or is it that the ones that work are only CC2650 and CC2640 are not working?

ANS. - It appears to be about 20% exhibit the problem the others work as expected. Each reset event in a product is at a random time from the last one, between 0s to a few hours. It is only CC2640 devices, the latest production build only has CC2640 devices. We have not seen the issue previously with CC2650.  

1. Do all the samples share exactly the same hardware?

ANS. - Yes, all the samples are the same hardware build and the same firmware and configured to operate the same.

1. Could you please provide some debug logs showing this "communication with the CC2640 getting stuck - no response back to the MSP on the UART and the watchdog then causes the reset".

ANS. - We don't have any debug logs but here are some screenshots of the logic analyser trace showing the point where it stops and reset and one zoomed in a bit. What we have seen is that the MRDY and SRDY lines stay high. The second screenshot  "CC2640 last comms before device reset.jpg" show response to power init, we then send advertising, scan response data and then send scanning data and the start scanning command , we get the correct responses and 1 block of data which is incorrect length and that seems to cause the issue.

1. In the second use case, is the application initiating a connection after discovering? What are the scanning parameters

ANS. - No, we do not connect we just look for devices and filter on specific data in their advertising message.  The scanning duration is 829ms, interval 97ms and window 65ms and that auto restarts for the 4s detection period.

1. Considering the case where there is no unwanted reset and you are not holding the device in reset before discovering, could you please provide some debug logs showing this "random times the data from any discovered devices stops getting sent to the MSP., so looks like we have no BLE detected.". Have you validated that the device is actually scanning at those moments?

ANS. - No we don't have any logs for this either, we are trying to get some signals etc captured on logic analyser. In this case we know the MSP432 still tries to stop and start scanning as it normally does but it never receives any data back, as none is available from the MSP432. The only way to recover this is to do a reset of CC2640 and again the random times of the event are that each time we run it will be a different time before it re-occurs.

Please also specify the SDK version you are using.

ANS. - We based the CC2640 firmware on simple_np in CC2650BP from the BLE SDK. TI BLE SDK 2.02.04.06 and TI-RTOS CC13xx/CC26xx V2.21.01.08.

/resized-image/__size/320x240/__key/communityserver-discussions-components-files/538/CC2640-comms-with-device-reset.jpg

/resized-image/__size/320x240/__key/communityserver-discussions-components-files/538/CC2640-last-comms-before-device-reset.jpg

Hello Iain,

Thanks for the information provided, sadly I cannot see the logic analyzer images, they are very small. I would suggest just copy pasting an screenshot of this traces into the post directly.

Iain Maclean said:

1 block of data which is incorrect length and that seems to cause the issue.

Do you know where is this coming from?

BR,

David.

had some issues getting the images pasted into this. The block of data is from a device detected as to why it says length is 40 bytes but there is only 38 I do not know. this is the point at which it fails.

This is a zoomed out capture showing the last good data, the failure, reset and the re-start.

Hello Iain,

Thanks for the information.

Could you please specify (just by text is fine) what logic line refers to what exactly?

Also, I think digging deeper into the reason of the reset can be helpful. The AON_SYSCTL:RESETCTL.RESET_SRC register should give us more insight about this. Could you please read it and share back the result? You can read more abut this in chapter 6.7 in the Technical Reference Manual

BR

David.

Looking at the screenshots and reading down in order the signals are all relative to the CC2640

UART TX

UART RX

Bootloader input

Reset input

MRDY input

SRDY output

I will try and see if I can get the AON_SYSCTL:RESETCTL.RESET_SRC register value.

Hello Iain,

Understood.

What is exactly the logic behind the reset input signal?

Let me know if you need help with AON_SYSCTL:RESETCTL.RESET_SRC.

BR,

David.

The logic behind it, is to use it as a type of shutdown of the device. We run the BLE to look for specific beacons that are around and then report them at a specified interval, we do the bulk of processing and look at BLE for 4 seconds and turn BLE off for the rest of the interval and use it as a way of reducing the power used to preserve battery life.

I do not have a debugger that I can connect to the sample units, so any help with reading the AON_SYSCTL:RESETCTL.RESET_SRC would be appreciated.

 Hello Iain,

The AON_SYSCTL:RESETCTL.RESET_SRCS register located in memory 0x40090000 shows the source of the last system reset that we can access after the device boots up again. Do you have a way of displaying information? What interfaces do you have? 

BR,

David.

Hi David

The main controller, MSP432, has a debug UART that we use. The only connection we have between the MSP432 and the CC2640 is the UART used to send the NPI commands and data between them or update new firmware when put in bootloader mode. 

Hello Iain,

Can you use the MSP432 debug UART to display the RESET_SRCS register information?

Have you consider our CC13xx/CC26xx Hardware Configuration and PCB Design Considerations (Rev. G) during the design?

BR,

David.

Hi

Sorry for the delay in responding I have been investigating and looking at ways to debug the CC2640 in a better way that we currently have.

I think there is some confusion also, the CC2640 is only reset under control of the MSP so it will be known why it has done a reset. 

We have created some firmware to stop holding the cc2640 device in reset after the 4s period we want to gather scan data and just stop and start scanning and have found that after a period of a few hours the CC2640 stops responding to the start scanning command in that when we set the MRDY line and wait for the SRDY to be set to allow the command data to be sent the SRDY line stays low.

We are going to create some firmware to handle this and if the line stays low then do a reset of CC2640.

Is it possible that this is the same reason why the MSP was getting stuck trying to read data and its watchdog timed out causing the reset originally reported. 

Hello Iain,

This firmware modification will be applied to only the devices that are not working fine (reseting constantly)? Considering the fact that only 20% of the devices are having this issue, I am inclined to think this may be related to HW differences.

How can I further help you with what you are describing?

Best Regards,

David.

Hi

We would prefer to update the firmware on devices not working and also have it as a new release for all future devices we manufacture.

We have done further testing on a larger number of devices, we checked 500 and 5% of those are exhibiting the issue.

You mention HW differences, do you mean differences that are maybe due to incorrect tolerance of components fitted to our PCB or differences in the CC2640 devices.

Hello Iain,

I mean differences on a custom PCB level. If you consider this is worth double checking, then I can forward this request to a HW specialist.

BR,

David.

Hi David

Thank you for the offer but during this investigation our hardware designer has reviewed our circuit design and compared against reference designs and we are also going to send some failed products back to our PCB suppliers to investigate and inspect them for any manufacturing issues.

I do have one last question that maybe you can answer. We are using code pretty much as is from TI BLE SDK 2.02.04.06 and TI-RTOS CC13xx/CC26xx V2.21.01.08, with minimal additions from ourselves in configuration values and to filter for specific BLE devices detected, before sending data to MSP432, is it possible that there is something in the operation of that firmware and RTOS that is causing the issue we are seeing.