• State
  • Locked Locked
  • Replies 25 replies
  • Subscribers 52 subscribers
  • Views 3907 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2640 BLE Stack 2.1.0 Memory Corruption

Shaba Abhiram
Intellectual 540 points

Other Parts Discussed in Thread: CC2640

We are trying to deploy a large number of peripherals with a few central nodes to monitor and update them. In our network, we have been noticing that some of our peripherals will eventually stop sending connectable adverts until a CPU reset is performed. I am almost sure that this is a result of a BLE connection attempt, but am not 100% sure on what goes wrong in the system yet. However, in our environment - there are more than one central that might want to connect to a given peripheral's broadcast so perhaps that makes it worse. The other interesting item to note is that, when the peripheral stops sending connectable ads, there are certain events which still trigger non-connectable broadcasts from the peripheral. This suggests that the system is not completely compromised, but rather some part of the program is corrupted.

While trying to figure out why this might be happening, we stumbled across this release note for the BLE 2.1.1 stack:

"Fixed an issue that may result in memory corruption if the connection is terminated on the GATT Server during an ATT Prepare Write Request procedure"

I am thinking that this is very possibly what we are seeing, and wanted to confirm the side effect of this bug and what was done to address it. I will be trying to review the code changes between the two stacks to see if I can figure out how to prove that this is indeed our issue. I would appreciate it if anyone at Ti who is familiar with this change could shed some light on what its side effects could be, and what the fix actually is. 

Here are our specific configurations:

XDC Tools: xdctools_3_31_01_33_core

Compiler: ti-cgt-arm_5.2.5

RTOS: tirtos_simplelink_2_13_00_06

BLE Stack: ble_cc26xx_2_01_00_44423

Thank you for the time and advise, looking forward to what your experts have to say!

  • 0
    TI__Guru** 105180 points
    Hello Shaba,

    To encounter the issue described in the release notes, you must be 1) in a connection, 2) the Client is performing a ATT Prepare Write Request, and 3) the connection is terminated prior to the Execute Write request. The issue was address in the stack library code, so to try it out, you can follow the porting instructions on the BLE wiki.

    Are you doing this operation (GATT Long Writes) prior to seeing the unexpected stop in advertising? The actual failure may present after the connection is terminated, such as on the next connection.

    Can you take a sniffer of the connection request attempt that is causing the failure? That would be a good start based on your description.

    Best wishes
  • 0 in reply to JXS
    Intellectual 540 points
    Thanks for the quick reply JXS. We are not doing any GATT Long Writes that I am aware of (however, some of the connection attempts might arrive from central devices that are quite far and have bad links, our intention is that these would fail gracefully and allow another closer central to pick up the slack).

    Our flow is the following:

    1. Central sees peripheral's connectable advert
    2. Central decides it needs to connect to this peripheral and issues a GAPCentralRole_EstablishLink() in the ble task main(). At the same time the central also kicks off a clock to close unresponsive peripheral connections.
    3. Once I get a successful GAP_LINK_ESTABLISHED_EVENT - I kick off a start discovery event (again handled in the main thread) and exchange MTUs
    4. This will trigger a pairing request to the central, and once the central gets a successful GAPBOND_PAIRING_STATE_COMPLETE - it subscribes for notifications from the peripheral using GATT_WriteCharValue() to the correct handle for the notification attribute on the peripheral. If this fails, it signals the ble task to close this connection.
    5. Once the write is complete for the notification request, the central starts writing data to another characteristic handle using the GATT_WriteCharValue() method. This happens in a loop 16 bytes at a time until the central has sent the data over to the peripheral.
    6. Although the peripheral can respond using the notification (if it needs to send data back to the central), in the cases where we see the device get stuck - I do not believe any responses are expected to be sent back.

    I am trying to reproduce this with a single device at my desk - could you recommend any ways to reproduce this issue by changing the central code. If I can prove that the peripheral gets into this state when this happens, I can update the stack and verify that the issue is resolved.

    Thanks again for all the help!

    -Shaba
  • 0 in reply to Shaba Abhiram
    Intellectual 540 points
    I have kicked off an experiment to have a couple of centrals constantly connect to this single peripheral I have. I have the peripheral quitting the link as soon as the pairing is successful (before the central tries to write to it to enable notifications etc).

    Would this be the best place to try and trigger this to verify the issue?
  • 0 in reply to Shaba Abhiram
    Intellectual 540 points
    One other clue - very rarely when I get multiple connections trying to hit the peripheral - my device will claim to be in "connected advertising" mode which is strange given that this peripheral should never be allowed to be in that mode.
  • 0 in reply to Shaba Abhiram
    Intellectual 540 points
    I am directly not using GATT Long Write - but I am subscribing for notifications and using the auto param update. I would like to know if this error that we are seeing is plausibly fixed by the changes made in the stack code. However it worries me that we do not use the Long write which I am guessing is the only place where Prepare write is called from?
  • 0 in reply to Shaba Abhiram
    TI__Guru** 105180 points
    Since you are not doing any GATT Long Writes you won't encounter the specific issue described in the BLE 2.1.1 release notes.

    However, I still think you should try to reproduce your actual problem, along with a debugger attached to analyze the system once the failure occurs. This can include checking for a CPU abort or other conditions described in the SW Dev Guide (SWRU393) Ch 9. Perhaps reducing the TX power on your Central will help.

    If the connect request has a bad CRC, it will not be accepted by the PHY, so the device will not respond to it.

    For 'connected advertising', do you enable the PLUS_BROADCASTER predefined option in your App? Do you observe the peripheral advertising once connected? If you are using the Stack config from SimpleBLEPeripheral, then only one connection can be accepted.

    Best wishes
  • 0 in reply to JXS
    Intellectual 540 points
    The central device uses a range extender, do you think this could have a negative impact on the peripheral?

    Aside from that, we never enable PLUS_BROADCASTER, but I do recall the device going into connected advertising one time when I was casually observing a target device. I do not recall seeing this often. The difficult part is, this issue is very hard to reproduce in the lab setting and only seems to occur on the field. We only expect one connection to be valid at a time as well.

    I will check the frozen boards to see if they perhaps can tell us anything useful. Please let me know if you can think of any other issues which might have been resolved in the latest stack update. Also, could it be an issue if our central is running the 2.1.1 and our peripheral is running 2.1.0?
  • 0 in reply to Shaba Abhiram
    Intellectual 540 points
    I also had to drop the amount of heap size to 2048 from 3072. I know that dropping this further can have bad consequences. Could this be a potential culprit as well? The number of PDUs was dropped from 8 to 6. I see that the newer stack 2.1.1 sets this to 10. Any advise?
  • 0 in reply to JXS
    Intellectual 540 points

    How could the Tx power of the central affect the peripheral? Could you clarify that for me please?

  • 0 in reply to Shaba Abhiram
    TI__Guru** 105180 points
    Hello - were you able to reproduce on your bench?

    The suggestion to lower the TX power on the Central was in response to an earlier comment about using a range extender, although the power should not, by design, cause the stated failure.

    Best wishes
  • 0 in reply to JXS
    Intellectual 540 points

    Nope, we are not able to repro this in a lab setting. It only seems like this occurs when there are more than 10-20 central nodes all which might attempt to simultaneously connect to a single peripheral which might send a connectable advertisement from time to time. 

    Note that we have about 200 peripherals and 15 centrals running for the last 3 weeks in a test deployment, and we have been noticing this issue in about 1-2 devices every day. Each peripheral will send a message to all centrals asking for a connection once every 8 hours or so.

    We even enabled a watchdog to catch the issue in progress, but the way these fail - even the WDT does not reset the peripheral board. Have you guys tested > 10 centrals connecting to a single peripheral at the same time? Could there be a race condition / overflow in any of the Ti BLE stack code?

  • 0 in reply to Shaba Abhiram
    TI__Guru** 105180 points
    Hello,

    Regarding "Have you guys tested > 10 centrals connecting to a single peripheral at the same time? ", can you elaborate on the test condition? The radio can only receive/decode one valid connection request at a time, and once the connection is accept/established, it will no longer listen for connection requests unless instructed by the application (i.e, in a v4.1 multi slave configuration).

    Are you able to attach a debugger to a board in this state while it's in the field? This would be the most direct way to understand what is going on and determine if it's a problem in the stack or application.

    Also, have you installed an abort handler to catch any CPU exceptions and reset the board?

    Best wishes
  • 0 in reply to JXS
    Intellectual 540 points

    Hey JXS, unfortunately we are not able to debug these field devices. I will try and bring one back to our lab after they have failed for more inspection.

    Could you please point me at an example of an abort handler so I can figure out how to include this correctly from the config?

    Are you referring to these?

    System.abortFxn = System.abortSpin;
    System.exitFxn = System.exitSpin;


    Thanks!

  • 0 in reply to Shaba Abhiram
    TI__Guru** 105180 points
    Hi Shaba,

    Please see "9.8 Deciphering CPU Exceptions" in the CC2640 BLE SW Dev Guide (SWRU393). It has instructions on how to install an exception handler.

    Best wishes
  • 0 in reply to JXS
    Intellectual 540 points
    One more data point. I was able to get a couple of "dead" devices from the field. When I probe their crystals - both the 32Khz and the 24Mhz seem to be off. Any idea what might put them into that state? Almost seems like the device is in shutdown mode?
  • 0 in reply to Shaba Abhiram
    TI__Guru** 105180 points
    Hi Shaba,

    When the crystals are off, is the board / CC2640 drawing current? Please note that you should not directly probe the 24MHz while the device is powered or you may internally damage the CC2640. Can you also review the crystal load impedance & tuning guidance in the articles under the HW column "Design Resources" on the TI BLE Wiki. Are you using crystals listed on the wiki or crystals with different properties?

    Best wishes
  • 0 in reply to JXS
    Intellectual 540 points

    I will check that as soon as I am back in the lab. 

    The only other thing I can think of is that we have an i2c sensor which is being sampled using the Sensor Controller. It samples about 16 Hz and wakes up the M3 once a second. I recall seeing another thread talking about sensor controller + BLE lockups, so wanted to mention our use case if it points us at any clues.

  • 0 in reply to Shaba Abhiram
    TI__Guru 56590 points
    Hello Shaba,
    If you disable the sensor controller task will the issue still occur?
  • 0 in reply to Eirik V
    Intellectual 540 points
    Erik, These errors only happen out on the field after anywhere from 2 days to 5 weeks of run time. We have about 200 peripherals deployed in a test site and they fail fairly randomly. For what its worth, I do not think what we are seeing is related to the sensor controller. What types of issues do you anticipate as a result of the sensor controller?
  • 0 in reply to Shaba Abhiram
    Intellectual 540 points

    Additionally - we are using the 2.21.01.15600 version of the driverlib. Interestingly I noticed this little snippet in the release notes for the cc26xxware product(s) for the 2.21.01.15602 release:

    This is a patch release for 2.21.01.15600.

    Bugfixes and new features:

    • [DRV_SRC] DDI: Added missing semaphore wrapper around function call HapiHFSourceSafeSwitch(). 
      This addresses a bus lock-up issue triggered when System CPU calls OSCHfSourceSwitch() at the same time as Sensor Controller calls tdcSetCntSource().
    • [DRV_SRC] SysCtrl: Added temperature compensation to the VDDR sleep trim.

    Known issues (at release):

    • None

    The first point above seems rather interesting - however I cannot for the life of me find the patch needed to update this. If I remember correctly, to use the BLE stack (2.1.1) - I will need to keep my RTOS at the simple link 2_13 release correct. Furthermore, For the release 2.21.03.15980, I see this note:

     

    Bug fixes:

    • [DRV_SRC] OSC: XOSC startup time calculation done by function OSCHF_GetStartupTime() returned maximum startup time too early.
  • 0 in reply to Shaba Abhiram
    TI__Guru 56590 points
    Hello Shaba,
    The only issue I know of is related to use of the TDC with TI RTOS (2.13.00.06 ). It only applies if your sensor controller task is changing the TDC clock source. Refer to the thread below for some more details:
    e2e.ti.com/.../1704076

    This specific issue is fixed in version tirtos_cc13xx_cc26xx_2_15_00_17. Look at the porting guide for using this release with BLE stack v2.1:
    github.com/.../simple_ble_peripheral_tirtos_2p15
  • 0 in reply to Eirik V
    Intellectual 540 points
    I also came across this document: www.ti.com/.../swra495c.pdf which refers to two mystical sounding functions:

    OSCHF_DebugGetCrystalAmplitude and OSCHF_DebugGetExpectedAvarageCrystalAmplitude

    We are suspecting that long running boards end up maybe going out of sync due to crystal problems. We would love to verify if we are w/in range using some S/W solution as the above document recommends. Like I stated, we are using the 2_13 RTOS, but I installed the 2_15 and 2_16 RTOS and neither of their respective osc.h / osc.c files in the driverlib define or declare these functions.
  • 0 in reply to Shaba Abhiram
    TI__Guru 56590 points

    Hello Shaba,

    Based on your two statements below I suggest you profile your stack/heap usage as described in this thread: e2e.ti.com/.../1696288

    "other interesting item to note is that, when the peripheral stops sending connectable ads, there are certain events which still trigger non-connectable broadcasts from the peripheral. This suggests that the system is not completely compromised, but rather some part of the program is corrupted."

    And

    "I also had to drop the amount of heap size to 2048 from 3072."

  • 0 in reply to Eirik V
    Intellectual 540 points

    I have actually been able to free up considerable space for the stack. The heap now runs much more than 3072 bytes. I am more interested in a possible 32Khz crystal issue hence why I asked about those debug functions above. Where do they exist? Why do you guys have a software document which is not backed by the code it asks you to use?

  • 0 in reply to Shaba Abhiram
    TI__Guru 56590 points
    Hello Shaba,
    These function is planned to be released in a later version of CC26xxWare. They are related to the high speed crystal (24 MHz) only. You observed that the device still send non-connectable broadcasts which indicates that the crystal is ok. Because the device till operate, but in a undefined/unpredictable manner that is why I suspected there might have been issues with stack or heap overflow which can be the cause. Profile both he stack size in addition to the heap to find the safe level to use.