• State Resolved
  • Locked Locked
  • Replies 39 replies
  • Subscribers 54 subscribers
  • Views 857 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC2642R: Power Saving with enabled Standby Policy: After 1 minute working we observe stopped data collection form the Sensor Controller and stopped timer(clock) - related activities

Ludmil Kalaydjiyski
Prodigy 200 points
Part Number: CC2642R
Other Parts Discussed in Thread: SYSBIOS, CC2640, , ENERGYTRACE, CC2640R2F, SYSCONFIG

Tool/software:

We use SimpleLink SDK CC13xx CC26xx version 6.10.0.29, and the TI tool CCS12:

We use POWER_SAVING in the Compiler predefined symbols, and we enabled Standby policy in the same way as it is enabled in the TI FW app simple_peripheral in the same SDK 6.10.0.29: parameter comparison attached in a screenshot:

When we start our Firmware in this Power saving mode, it works well for approx. 1 minute, after that the data collection from sensor controller stops, and some timer(clock) related activities stop working as well.

Details:

1) We use Sensor Controller RTC-driven Alert interrupts to the main processor in the MCU. These interrupts/events invoke ADC read operations.

2) For LED on/off scheduling we are using timer (clock) APIs from file utils.c.

Util_constructClock: it calls ti_sysbios_knl_Clock_construct

Util_startClock: it calls Clock_start(handle) that calls ti_sysbios_knl_Clock_start

Question 1 for TI: Is it possible that the failures in these 2 above areas 1) and 2) are caused by a problem in the communication between drivers and the Power manager?

According to: SimpleLink SDK Power Management: MSP432, MSP432E4, CC13xx/CC26xx, and CC32xx    User’s Guide (2019)

“1.3 Power Manager API

Driver development: A device driver communicates with the Power Manager to enable/disable access to its peripherals. “

The above spec is a bit old, Question 2 for TI: What is the best Power management TI spec to follow for our SDK CC13xx CC26xx version 6.10.0.29? We also used a TI spec (link below), but it is for a higher SDK version 6.40:

software-dl.ti.com/.../power.html

According to our battery usage measurements: our previous TI MCU CC2640 saved battery 2 times better compared to our new current prototype using 6.10.0.29: We also had POWER_SAVING there as a predefined symbol. We did not have .syscfg file with power policy options. We had the standby policy defined in a .cfg file, no longer used in the new SDK.

  • 0
    Prodigy 200 points

    For our new MCU cc2642 I found a good sources of info:
    file:///C:/ti/simplelink_cc13xx_cc26xx_sdk_6_10_00_29/docs/drivers/doxygen/html/_power_8h.html#ti_drivers_Power_Examples_enable
    and file PowerCC26X2.h: they show several settings and the following steps to enable Power Policy:

    // Import Power Driver definitions
    #include <ti/drivers/Power.h>
    // One-time initialization of Power manager
    Power_init();
    // Enable power policy
    Power_enablePolicy();

    We have in our code the first two: #include for the drivers above, and the Power_init(). Looking at our FW app .cfg file: when I select the box: "Enable Policy", Power_enablePolicy() will be called, correct? I could not find other setup suggested in the TI docs.

    I tested and measured the initial period when all works OK on our device in Power saving mode: This period is: the first 3 min 35 secs during an established Bluetooth 5 connection. After this period the connection works OK for at least 1 hour and longer, and our I2C bus chip in the device works OK communicating with a mobile app, but our device RGB LED and events from the sensor controller get lost, i.e stop working at the end of this period.

    The CC2642R datasheet says:
    "SCLK_MF is an internal 2 MHz clock that is used by the Sensor Controller in low-power mode and also for internal power management circuitry. The SCLK_MF clock is always driven by the internal 2 MHz RC Oscillator (RCOSC_MF)."

    Will it help if I enable this symbol: USE_RCOSC  ? We inherited this code from TI examples:
    #ifdef USE_RCOSC
    // Set device's Sleep Clock Accuracy
    #if ( HOST_CONFIG & ( CENTRAL_CFG | PERIPHERAL_CFG ) )
    HCI_EXT_SetSCACmd(500);
    #endif // (CENTRAL_CFG | PERIPHERAL_CFG)
    RCOSC_enableCalibration();
    #endif // USE_RCOSC

    I tried but had RCOSC_enableCalibration rejected as unresolved symbol. It does not exist in the TI src code as part of the 6.10 SDK, that we are using.
    I should probably use the TI doc on link:
    Measuring CC13xx and CC26xx current consumption (Rev. D)
  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    Thanks for reaching out. Please help me with the following questions to assist you further:

    1. From what I understand, you have been using the CC2640 device and now migrating your code to CC2642 correct? or is this a new SW development for CC2642 based on an out of the box example? Have you checked this migration guide: https://software-dl.ti.com/simplelink/esd/simplelink_cc13xx_cc26xx_sdk/7.40.00.77/exports/docs/ble5stack/ble_user_guide/html/ble-stack-5.x-guide/porting-guides/cc2640r2_to_cc26x2.html
    2. Could you confirm if the entire sensor controller functionality also stops working after > 1 hour or the LEDs are the only ones failing here (previously I understood the ADC and comms (?) were also not working correctly)?
    3. If the standby policy is not enabled, will the issue reproduce?
    4. Regarding the RCOSC: The RCOSC build configuration no longer exists. To use RCOSC, add the following define in the project predefined symbols: USE_RCOSC.
    5. Do you know where is the extra power consumption (x2 more than 2640) coming from? Measuring CC13xx and CC26xx current consumption (Rev. D) can me help for for this. You can also leverage Energy Trace, especially EnergyTrace++ Instructions.

    BR,

    David

  • 0 in reply to David
    Prodigy 200 points

    Hi David, Thanks very much for your reply.
    RE 1: Yes: we have an old CC2640 device, and we migrated most of our code to CC2642R. We used the TI migration guide and re-used bits from Simple Peripheral TI app.

    RE 2:     The cc2642R datasheet says on p49: "In Standby mode, only the always-on (AON) domain is active. An external wake-up event, RTC event, or Sensor Controller event is required to bring the device back to active mode. …. All GPIOs are latched in standby mode."

    When the periodic RTC event comes, our SC code task runs and enables power to a voltage sensor and all LEDs by driving a digital output:

    gpioClearOutput(AUXIO_O_SG_POWER);

    After this our SC code reads the ADC to provide voltage sensor data to the MCU, and schedules its own execution:
    // Schedule the next execution <in ticks>  (1 sample period on the SC: default 100 ms)

    fwScheduleTask(1);

    All this works well in the following modes:
    - Standard “No Power policy enabled” mode,
    - In WFI Power Management mode,
    - In the first 1 to 3 mins in standby mode.

    After this variable period 1 to 3 mins in Power standby policy mode, we observe the following:
    1) My debug shows all LEDs turned off (red, blue, green), so the above SC gpio call has not been made by our SC code, or it has not enabled the power for the LEDs.
    2) No voltage sensor values are reported by the MCU over Bluetooth. I added debug code to the alert callback function: scTaskAlertCallback() to confirm that the alert interrupts from the SC to our MCU Firmware app have stopped. Also: If interrupts were available, our MCU code would read and report some ADC values: random or 0, but we see last value frozen (no more values present) in the mobile app still connected OK to our device.
    3) The Bluetooth connection is still in place and a separate IMU chip on our PCBA controlled only by the MCU (not the SC), is sending correct data to our mobile app.
    As a next debugging step, I can modify our SC task to remove the suspected dig output operation: gpioClearOutput(AUXIO_O_SG_POWER);


    RE your 3:     If the standby policy is not enabled, will the issue reproduce?  Reply : No.

    RE your 4: Thanks for the info. As I indicated in my previous message: I tried the predefined this symbol: USE_RCOSC, but had RCOSC_enableCalibration rejected as unresolved symbol. It does not exist in the TI src code as part of the 6.10 SDK, that we are using:
    #ifdef USE_RCOSC
    // Set device's Sleep Clock Accuracy
    #if ( HOST_CONFIG & ( CENTRAL_CFG | PERIPHERAL_CFG ) )
    HCI_EXT_SetSCACmd(500);
    #endif // (CENTRAL_CFG | PERIPHERAL_CFG)
    RCOSC_enableCalibration();
    #endif // USE_RCOSC

    RE your 5: "Do you know where is the extra power consumption (x2 more than 2640) coming from? " Reply: No, not measured yet, we will try. I looked at the doc you mentioned. We do not have such Power measurement equipment. Will check if it is possible to hire.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi David, I ran another test on the Sensor Controller: No digital outputs driven by our SC code, no ADC read, empty SC task code apart from only periodic RTC events (alert interrupts) generated from SC to the main MCU FW application core + our FW rescheduling itself. This setup generated successfully periodic alert interrupts for over 1 hour in MCU standby mode. This shows that these events can survive the standby mode alone, but they probably get stopped when all our SC code runs, because of a crash or a deadlock related to gpio calls to power up or down sensors, or an ADC-related FW call. 

    Further debugging showed that the suspected Digital output operations:
    Enable then disable sensor power supply:
    gpioClearOutput(AUXIO_O_SG_POWER);
    gpioSetOutput(AUXIO_O_SG_POWER);

    stop the SC alert interrupts to the MCU, when added to the beginning of our empty (reduced) SC FW app that runs every sample period in Power standby mode (using external crystals, indicated via X in the .syscfg file).
    Our empty (reduced) FW app contains only: generate a periodic alert interrupt + reschedule our SC FW task by calling fwScheduleTask(1).


    I suspected a race condition between our gpio calls and the TI power manager operation: "enable digital io" during wakeup, so I added 2ms delay before our first dig out operation => the RTC alert interrupts were recovered and received by our MCU FW app.

    Next step: I kept this 2ms delay and restored all our FW code to run in Power policy mode => after 2 mins the alert interrupts disappeared similar to our first configuration in Standby mode.
    The .syscfg file setup used, is shown below:



    SC power setup shown below:

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    Thanks for the extra information.

    Would it be possible to ask you to try to reproduce this issue using one of our out of the box examples considering the modifications you have done (utilizing drivers and sensor controller)? Also, are you using a custom board or an evaluation board from TI?

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi Dave, My experiments use configuration that is close to the standard TI environment:
    I use a custom board, but as I said in my first message: We use POWER_SAVING in the Compiler predefined symbols, and we enabled Standby policy in the same way as it is enabled in the TI FW app simple_peripheral in the same SDK 6.10.0.29: parameter comparison attached in a screenshot:

    During my experiments when the SC alert interrupts stop, I cannot reduce our SC FW app any further, I removed the main code. As I said above it only has:
    - 2x digio API calls
    - generate a periodic alert interrupt
    - reschedule our SC FW task by calling fwScheduleTask(1).

    Looking at the results, I think the issue is related to the 2x digio API calls in our SC code: they somehow cause the alert interrupt to stop in Standby mode.
    I may need to check the effect of the logic: We power up sensors and L    EDs by setting the dig. output to 0: gpioClearOutput(AUXIO_O_SG_POWER);

    These APIs work reliably when Power Saving is not enabled, and when it is enabled in WFI mode. I will think again, but at present I cannot see how changing our main FW and custom board will resolve the problem. 

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    I see, please allow me until tomorrow to go through these findings and be able to take a more detailed look at it while also requesting help from the team experts in sensor controller because it seems it will be difficult to reproduce the issue on our side at the moment.

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi David, thanks a lot. I checked the usage of XOSC boxes in the syscfg power settings in the simple peripheral example TI app and our FW app:
    The settings are the same: We use and tick the same XOSC LF and HF options/boxes: i.e. external crystal oscillators are used on our PCBA, and on the simple peripheral Hardware:



     

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    Thanks. I have discuss this further with the team and it does not seem like a known issue. Please help me with the following:

    1. If you use the simple_peripheral out of the box example with the minimal SC functionalities and same power saving setups, does this issue reproduces in your board?
    2. Are you running this test in debug mode?
    3. Have you had your custom board review by TI?
    4. Can you dump register data after the devices crashes to analyze some of the flags by connecting to the target in debug mode?
      1. AON_RTC register (SEC / SUBSEC / CH2CMP) - Page 1385 of TRM. This is done with the objective of checking if the RTC has been modified (if CH2CMP value is more than SEC and SUBSEC then that will evidence this scenario that will potentially cause issues.
      2. AUX_SCE --> CTL Register to see if CLK_EN is enabled - Page 1811 of TRM.
      3. AUX_SCE --> FETCHSTAT - Page 1812 of TRM.
      4. AUX_SCE --> CPUSTAT  - Page 1813 of TRM.
    5. Could you please try setting Default Power Mode to "Low Power"?                                                 

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi David, thanks a lot for the advice. RE your 5. Could you please try setting Default Power Mode to "Low Power"?    I will try today. I wanted to do this, but noticed in the SC Power help table that Low-power mode is not available for ADC (screenshot below), is this correct? We use ADC enable and select API calls in our SC FW code.

    RE your 1. If you use the simple_peripheral out of the box example with the minimal SC functionalities:
    I wanted to do this but I only have a simple peripheral project file that has no SC files and does not use the sensor controller:
    simple_peripheral_CC26X2R1_LAUNCHXL_tirtos_ccs

    If you know a link for a simple peripheral variant using SC code and provides the SC project, it will be useful.
    I will have to search for such project to see the SC source code.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi Daved, RE your 5. Try Default power mode set to "Low Power" for SC: I tried 2 variants with "Low Power" for SC: With this config shown below, I cannot connect over Bluetooth to our mobile app, or rarely connect, but the LEDs and main voltage sensor are not working:

    With this second possible variant I cannot connect over Bluetooth:

    RE: Your questions:
    2. Are you running this test in debug mode? LK: I am running in Release mode. WFI Power mode runs OK in Release mode and saves 32% more power compared to Release mode with no Power policy.

    3. Have you had your custom board review by TI? LK: The updated board with CC2642R has not been reviewed by TI. I cannot say this now for the original board with cc2640: it was designed 8 years ago. 

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi David, RE your 4.: dump register data:
    I found some TI code that reads these registers, so I think I can read them in our code after my detection of the missing alert interrupt from SC:

    // Read the current RTC value
    key = scifOsalEnterCriticalSection();
    sec = HWREG(AON_RTC_BASE + AON_RTC_O_SEC);
    subsec = HWREG(AON_RTC_BASE + AON_RTC_O_SUBSEC);

    ch2Cmp= HWREG(AON_RTC_BASE + AON_RTC_O_CH2CMP);

    RE your 2.: Are you running this test in debug mode?
    As I said I was running in Release mode, I switched to Debug mode now, I am trying to reproduce the issue and use the serial port to dump the RTC registers.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi David, please find attached an RTC log showing the loss of the SC alert interrupt event on the main MCU application code. The RTC registers are dumped every sample instant around the loss event, marked with "ERROR" in the log (sample period 100 ms). Time to the loss event is different on different runs: range 1-6 minutes, one run of 11 mins did not have the problem. 

    Our FW app runs in debug mode, I am using the serial port for the debug messages. The app is in standby Power management mode. SC is in active mode, 'cos the low power mode did not work.

    I did a quick sanity check and could not find an apparent issue in the AON, but a specialist is needed to check.
    In case RTC is OK, I think one scenario to explain the events, could be: RTC ticks are OK. Our SC task crashes probably on digio 0 output attempt (this stops voltage from SC and PCBA LEDs, we see and know the dependency). Our SC task cannot re-schedule itself on the next RTC interrupt, and never runs again, so it never generates again the SC Alert interrupt to the main core. Our main FW finds no Alert. This matches the attached log.

    AON_RTC_O_SEC log 4 ERROR.txt
    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    44 335675392 2894531
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    44 765460480 2901084
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    44 1195507712 2907637
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    44 1625292800 2914190
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    44 2055340032 2920743
    voltage: 3587
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    44 2485125120 2927296
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    syscfg and SC Power settings used when collecting RTC logs:
     

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Another log attached: includes SC registers as well (decimal numbers are shown):
    Point of Error:

    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    23 2295857152 1543048

    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 1281

    ERROR: LOST SC Alert AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    23 2729050112 1549601

    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 774
    ------------------------------------------------------------------

    AON_RTC_O_SEC+SCreg-s log 7 ERROR 23 seconds from start.txt
    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    17 982515712 1130209
    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 1281
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    17 1412300800 1136762
    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 1281
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    17 1842085888 1143315
    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 1281
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    17 2272002048 1149868
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Looks like the moment of Lost SC alert is random: can be even 23x100ms = 2.3 seconds after the Bluetooth connection established with our Mobile app. The app just displays data on the screen.

  • 0 in reply to Ludmil Kalaydjiyski
    Guru 86485 points

    Hi,

    I read this post and what you are experiencing is a code hang or an implementation issue. Also, I do not think the power management is causing your issue. By default if you use simple peripheral is set to low power, it will go to standby mode if there is no event.

    I did this project using CC2640R2F integrated sensor controller to simple peripheral. If you do the integration of sensor controller to the simple peripheral correctly, then there will be no problem.

    Soil Moisture Sensor Controller Project.

    -kel

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    H David,
    RE: your a: Here is my decoding of the RTC registers': From log 7 above: AON_RTC Seconds numbers seem to be  always correct, including and after the error. A new log entry comes every 100 ms.  Around the error we have all 10 RTC second indications: 23, then number 24 repeated 10 times correctly as well. I see no loss of RTC ticks. We have a battery measurement task 

    RE: your b: Decoding of the log for     AUX_SCE --> CTL Register to see if CLK_EN is enabled - Page 1811 of TRM.
    CTL is 1 in all cases before and after the error: hence CLK_EN bit is set, clock always enabled.

    RE: your d: Here is my decoding of the SC CPUSTAT register, at the transition point:" all good" => "SC Alert to main core lost":
    Value 1281 decimal in SC CPUSTAT in all good cases "SC Alert OK" before the error:
    0101 0000 0001 => 2 bits set: HALTED and SLEEP

    Value 774 decimal in SC CPUSTAT in all bad cases: ERROR: ALERT to main LOST
    0011 0000 0110 => 2 bits set: HALTED and WEV

    RE: WEV: Is this some kind of "Write Event" to a register that gets frozen - did not complete? My log indicates that 774 remained in all subsequent samples, is this an SC crash for all tasks? I found that 3 secs after the ERROR: Alert interrupt missing, our other SC battery monitor task starts reporting 0 for battery voltage, previous value: almost full battery.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    H David,
    RE: your a: Here is my decoding of the RTC registers': From log 7 above: AON_RTC Seconds numbers seem to be  always correct, including and after the error. A new log entry comes every 100 ms.  Around the error we have all 10 RTC second indications: 23, then number 24 repeated 10 times correctly as well. I see no loss of RTC ticks. 

    RE: your b: Decoding of the log for     AUX_SCE --> CTL Register to see if CLK_EN is enabled - Page 1811 of TRM.
    CTL is 1 in all cases before and after the error: hence CLK_EN bit is set, clock always enabled.

    RE: your d: Here is my decoding of the SC CPUSTAT register, at the transition point:" all good" => "SC Alert to main core lost":
    Value 1281 decimal in SC CPUSTAT in all good cases "SC Alert OK" before the error:
    0101 0000 0001 => 2 bits set: HALTED and SLEEP

    Value 774 decimal in SC CPUSTAT in all bad cases: ERROR: ALERT to main LOST
    0011 0000 0110 => 2 bits set: HALTED and WEV

    I removed different parts of our SC task source code to debug and try to remove the failure, but in all experiments, I reproduced the loss of SC Alert interrupts at random moments in time after startup (ie. clicking play button in CCS debugger). The transitions to the error had the same CPUSTAT change: 1281 => 774.

    RE: WEV CPUSTAT bit: Is this some kind of "Write Event" to an SC register that gets frozen when the error occurs ? Write did not complete?
    My log indicates that 774 remained in all subsequent sample periods after the error occurred. Is this an SC crash for all tasks? I found that 3 secs after the ERROR: Alert interrupt missing, our other SC battery monitor task started reporting 0 for battery voltage, all previous values: almost full battery.

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    Thank you for providing this information! I will review these findings today with the expert and come back at you as soon as possible.

    BR,

    David

  • 0 in reply to David
    TI__Genius 14005 points

    Hello Ludmil,

    The WEV refers to the "Wait for EVent" instructions. These instructions are not used by the run-time framework. Could you please confirm if you are using any of procedure that waits for some event, which might not occur at all or has occurred in the past? I would suggest to take a look at the documentation of Sensor Controller Studio.

    Do you have a Texas Instruments point of contact that could help you share the sensor controller studio project file with us to take a quick look at?

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi David, Thanks a lot for the info. In the SC task code that fails: we have a 5ms timer call to delay our SC task code: fwDelayUs(5000);. I removed this timer call as an experiment yesterday, but I still had WEV bit set: log attached. We used to have a TI contact years ago, but in 2022 were asked to use the forum. We are based in San Diego, CA.

    AON_RTC_O_SEC+log12a -5msDelay+MoreDebug ERR 500 secs from start.txt
    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 1281
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    41 405405696 2699484
    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 1281
    SC Alert OK AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    41 837156864 2706037
    AUX_SCE_O_CTL AUX_SCE_O_CPUSTAT:
    1 1281
    . . ..
    ERROR: LOST SC Alert AON_RTC_O_SEC AON_RTC_O_SUBSEC AON_RTC_O_CH2CMP:
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • 0 in reply to Markel Robregado
    Prodigy 200 points

    Hi Markel, thanks for the information. Our MCU is CC2642R. We ran successfully the same SC code for many months with no Power policy selected in the .syscfg file. The problem occurred when we selected Power Policy Standby in the .syscfg file. Can you provide a link to the simple peripheral source code including SC project and task that runs in standby mode?

    RE low-power mode:
    SC Power help table indicates that Low-power mode is not available for ADC (screenshot below), is this correct? We use ADC enable and select API calls in our SC FW code.


    As I indicated above in this post: 

    I tried 2 variants with "Low Power" for SC: With this config shown below, I cannot connect over Bluetooth to our mobile app, or rarely connect, but the LEDs and main voltage sensor are not working:

    With this second possible variant I cannot connect over Bluetooth:



    Regards

    Ludmil

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    I replaced the fwDelay() with a hard delay in the code and still observed the same error as before: frozen SC.

    //fwDelayUs(5000); //LK: removed

    for (U16 n = 0; n < 5000; n++) {

    sampleCounter = 0;

    }

    I found in a TI SC doc that this delay API is independent of power modes:

    fwDelayUs

    Prototype: fwDelayUs(delay)

    Delays execution of the next line by the specified number of microseconds.

    To avoid busy waiting, the delay is implemented using the microsecond delay timer.

    The delay is independent of power modes (active and low-power) and switching between power modes.

    Parameter value(s)

    • delay : Delay value in microseconds (1-65534)


    I tried custom / dynamically switched on/off low-power mode on SC: same bad result.
    I tried no Power policy in CCS, .syscfg file: all good: Alerts coming well in Release FW mode on the main core.
    Is it possible that WEV is set because SC is waiting for an ACK from the main core, and this ACK does not come in standby mode ? I made an experiment that shows this:
    I left only Alert interrupt and self-reschedule in our SC code, tried longer and had the loss of the Alert on the main core with this very reduced SC code, which makes me think that the problem is no ACK from the main MCU core. SC waits on this event, main waits on Alert, so we have a deadlock. Our main core FW calls the API for Ack to SC, it works well in no power Manag. mode and in WFI Power Manag. mode. Maybe I can switch to custom Power mode on the main core and use this option.

    Regards

    Ludmil

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    It would seem you still have a procedure that waits for an event (WEV). Could you please help me with the following:

    1. Get the value after the devices crashes from AUX_SCE --> FETCHSTAT - Page 1812 of TRM.
    2. Look for that value (program address) inside the "sce.lst" listing file, which should be able to tell you which instruction is causing the WEV.

    3. Could you please confirm you are using the ADC only from the sensor controller and not MCU?
    4. Could you please share how you are reading the ADC buffer (code-snipped) would be helpful. There could be a risk of incorrect reading that generates the program to get stuck waiting for data that does not exists.

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi David, thanks a lot, I will work on 1. and 2. today.

    RE: 3 and 4:
    3. Could you please confirm you are using the ADC only from the sensor controller and not MCU?
    After the alert interrupt showing "data ready: on the main MCU core we only call: AUXADCAdjustValueForGainAndOffset(. . .) to read  .output and adjust value. 

    4. Could you please share how you are reading the ADC buffer (code-snipped) would be helpful
    LK: We use standard APIs that exist in TI manual:
    ADC — Sensor Controller Studio v2.9.0 CC13x2 CC26x2 documentation

    From the manual: 
    // Select ADC input
adcSelectGpioInput(AUXIO_A_SENSOR_OUTPUT);

// Enable the ADC (fixed reference, 2.7 us sample time, manual trigger)
adcEnableSync(ADC_REF_FIXED, ADC_SAMPLE_TIME_2P7_US, ADC_TRIGGER_MANUAL);

// Sample the sensor and store the ADC value
adcGenManualTrigger();
adcReadFifo(output.adcValue);

// Disable the ADC
adcDisable();

    --------------------------------------------------------------------------

    Our code sequence used successfully on the previous MCU CC2640 for more than 5 years:
    adcEnableSync(. . . )

    for (U16 n = 0; n < ADC_INPUT_COUNT; n++) {
    // Sample the sensor
    adcSelectGpioInput(Input[n]);

    . . . .

    . . . . 
    U16 sample;
    adcGenManualTrigger();
    adcReadFifo(sample);

    . . . .;
    }

    . . . . 
    // Disable the ADC
    adcDisable();

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    I added more info in the following message

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi David RE your 1. and 2:

    1. Get the value after the devices crashes from AUX_SCE --> FETCHSTAT - Page 1812 of TRM.
    2. Look for that value (program address) inside the "sce.lst" listing file, which should be able to tell you which instruction is causing the WEV

    I ran a variant of our full SC code with dynamic power control (start/stop low-power mode). I decoded FETCHSTAT and found: The blocked place in the code is an ADC read at address 00e0. Instruction code 9990: input/read ADC. Looks like it is actually blocked by the preceding wev1 instruction. It shows the Program Counter at the next ADC input instruction:
                       ;? adcReadFifo(sample);
    00df ---- edb1 wev1         #WEVSEL_ADC_FIFO_NOT_EMPTY
    00e0 ---- 9990 in R1,       [#IOP_ANAIF_ADCFIFO]

    See attached below for all info:


    One possible root cause explanation: we are waiting for ADC input that does not happen because the previous dig output instruction did not supply power to our voltage sensor and also affected an OP-AMP circuit (low voltage impact confirmed by our HW engineer):
    Dig output not succeeded:
    ;? // Enable power supply
    ;? gpioClearOutput(AUXIO_O_SG_POWER);
    00d0 ---- 54bc iobclr #(20 & 0x7), [#(IOP_AIODIO0_GPIODOUT + (20 >> 3))]

    Another impact from the above dig output: The same power supply route supplies our LEDs, which are down every time when the error occurs. 
    We can try to also confirm with an oscilloscope tomorrow.

    We defined this digio power pin only in our SC configuration, not in main MCU .syscfg file. The error however only occurs when Standpy policy is selected in the main FW syscfg file. All works OK in standard non-Power saving mode on the main MCU, and in WFI mode on the main MCU. CC2642 datasheet: I found this text on page 49: All GPIOs are latched in standby mode.

    This dig output is declared in our SC code section, for which I explicitly request active mode according to TI SC docs:
    // Enter active power mode
    pwrRequestAndWaitForActiveMode();

    The error also occurs if I remove this dynamic power control config in SC and the above API.

    Configs screenshots attached as well: .syscfg and SC: Power and Clock Settings.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    I started using adcGetFifoStatus() and adcPopFifo() as a workaround. Will check a HW test point on the PCBA to confirm the above results.

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    Is this workaround avoiding the issue of the SC stopping? Could you confirm you are not using the DAC module from sysconfig?

    Is there a way for you to share with us the scp file and the relevant code for the application?

    Best Regards,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi David,
    The workaround worked fine for 65 mins yesterday in Debug mode: no SC crashes. I am testing it in Release mode today and will let you know. I am using the attached .syscfg and SC power setups:

      

    I confirm that I am not using the DAC module from sysconfig: screenshot attached below:

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    Understood. Let us know how it goes.

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi Dave, We hooked an oscilloscope and confirmed that my findings are correct for the dig output on SC: it stops working at the random time moment of freeze/deadlock of the SC:
    gpioClearOutput(AUXIO_O_SG_POWER);

    To remind you: Thiis output stops power to the voltage sensor and the LEDs. I reported at the time: We deadlock at WEV when we read the ADC, so I tried first: a more robust FIFO read: replaced adcReadFifo() with FIFO status read and check for status, retry the above Dig output...etc, all this helped a bit but the dig output problem was still there. 

    Further changes and 2nd SC freeze:
    The TI SC help table is saying that ADC is not supported in low power mode, so I started calling in our SC code: 
    pwrRequestAndWaitForActiveMode();   
    then apply the dig output and do all operations, after which I switch to Low-power mode. On the main MCU code I also disable standby mode before any SC operations start, and re-enable it after all SC, BLE and other operations complete. I resolved my freeze on the dig output, but I got another 2nd type of freeze, see summary below in screenshots:



    Further change and 3rd freeze: very similar to the 2nd type:
    I replaced the above API pwrRequestAndWaitForActiveMode(); with the following menu selection that seems equivalent:

    This is a change in settings: Set "wake up active" option in the SC power settings. I also removed SCOSC calibration from .syscfg: This probably has no effect 'cos we use XOSC. I observed an SC crash after 33 mins:
    ERROR: LOST SC Alert  SEC  SUBSEC CH2CMP:
    2040  2230190080  0

    CTL  CPUSTAT  FETCHSTAT: 
    1 1281 1879769235 

    FETCHSTAT 1879769235 => Hex: 700B 0093
    lst file shows:

    PwrRequestAndWaitForActiveMode:
    ; Update Reference DAC clock divider before entering active mode
    0093 ---- 700b Id R7, #((ACTIVE_MODE_SCE_CLK_FREQ_MHZ / 2) - 1)
    0094 ---- fb96 out R7, [#IOP_ANAIF_DACSMPLCFG0]

    I think in freeze case 2 and 3 we have an output operation related to DAC (similar to my case1: dig output to control power).
    I think I resolved my old case 1, but I hit other "output" cases 2 and 3 in a different switching TI code area: how can I work in Active mode on SC 100% of the time without switching Active <=> Low-power mode ?  
    I tried 4th case: no switch to low-power mode in our SC code, no dynamic Power control in SC settings: but had the same freeze as case 3, because I still have "wake up active" option in the SC power settings and it causes a switch in TI code in the same above PC=0093, op code 0093. 5th case using "init only" option in SC power settings also ends up freezing SC in this place:

    FETCHSTAT: 700B 0092

    PwrRequestAndWaitForActiveMode:
    ; Update Reference DAC clock divider before entering active mode
    0092 ---- 700b ld R7, #((ACTIVE_MODE_SCE_CLK_FREQ_MHZ / 2) - 1)
    0093 ---- fb96 out R7, [#IOP_ANAIF_DACSMPLCFG0]

    I tried power settings on SC: "Active" set for each task code block, with either option for default power mode: at every wake up / on init only: same result: frozen SC
    at the same point in the .lst file, shown below:
     
    PwrRequestAndWaitForActiveMode:
    ; Update Reference DAC clock divider before entering active mode
    0093 ---- 700b ld R7, #((ACTIVE_MODE_SCE_CLK_FREQ_MHZ / 2) - 1)
    0094 ---- fb96 out R7, [#IOP_ANAIF_DACSMPLCFG0]

    I tried setting the frequencies for low-power and active mode the same at 2MHz (SC setting), but had the same problem in the TI code: frozen SC at the same address, opCode slightly different:
    PwrRequestAndWaitForActiveMode:
    ; Update Reference DAC clock divider before entering active mode
    0093 ---- 7000 ld R7, #((ACTIVE_MODE_SCE_CLK_FREQ_MHZ / 2) - 1)
    0094 ---- fb96 out R7, [#IOP_ANAIF_DACSMPLCFG0]

    Maybe I need custom power mode in .syscfg? Or use constraints?

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    To further be able to help you with this, would it be possible for you to try the following?

    1. Could you please confirm the issue occurs with default simple_peripheral and/or SCS ADC examples (either separately or together)?
    2. Do you have a TI CC2652XR1 LaunchPad you can use to reproduce the issue?

    3. Try with the latest F2 SDK (7.10.00.98 which has last update for SCS) and SCS (that has all patches applied).

    All this so that we can try to reproduce the issue on a TI environment that we can use a base reference.

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi David, Thanks for the advice and the doc link, I can try the SCS ADC examples with SDK 7.10, but this will take some time to port code / migrate from our current SDK 6.10. I checked the SC examples. I see that they are using RTC. I switched to Util_restartClock() before some recent code updates, so it will be faster to go back to RTC in my code first, test, and then look at SDK 7.10.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi David, I ran the RTC-driven test variant on our app but it ended up with a frozen SC and "wait for event" SC instruction when reading the ADC (same as the first issue).

    I had some compilation errors with SDK 7.10.00.98 and the TI Simple Peripheral app, I am trying to resolve them.

    I managed to run one successful test with our reduced, almost empty SC task that used to freeze when using dig Output, ADC channels, and delay:
    Using our current TI SimpleLink SDK CC13xx CC26xx version 6.10.0.29, I passed successfully an approx. 2 hours run of our reduced SC task that performs the same actions as described in the TI doc: Ultra-Low Power Sensing Applications With CC13x2/CC26x2, A.3 Wake Up and Sleep – Sensor Controller, plus 1 additional action:

    - Periodic SC task that reschedules itself in the Init block and the Execution block by calling: fwScheduleTask(1).
    - In the Execution block calls only:
    fwGenAlertInterrupt(); // Alert interrupt to Main MCU app - SC task's additional action compared to the above A.3
    fwScheduleTask(1); // Schedule the next execution <in ticks>, one tick 100 ms

    The main MCU app performed the same initialisation and start of the SC task as describe in section 6. of the TI doc: Ultra-Low Power Sensing Applications With CC13x2/CC26x2, A.3 Wake Up and Sleep – Sensor Controller, plus 4 additional actions:

    1) Ignore each Alert interrupt. This main MCU core app only does the standard acknowledge of this Alert interrupt:
    scifClearAlertIntSource();
    scifAckAlertEvents();
    2) Have the Power Management Policy enabled as "Standby" in .syscfg
    3) Connect to a mobile app running on a phone during the run
    4) Drive a periodic green device LED to indicate successful runs.

    Attached .syscfg Power section used during the successful test:

  • 0 in reply to Ludmil Kalaydjiyski
    TI__Genius 14005 points

    Hello Ludmil,

    Glad to hear the test with this reduced SC task works fine. Therefore can we say the issue is happening when enabling the ADC and using delays? Have you tried to add more of these features to see which one is generating the issue and narrow it even more? This is still being test in your custom board correct (no TI launchpad)?

    BR,

    David.

  • 0 in reply to David
    Prodigy 200 points

    Hi David, Yes I did the testing described in my last post, with our custom board: no TI launchpad used.
    The SC power settings used were:

    RE: adding new features: I added the 2 digiIO operations: outputs that we had in the code previously to provide power to our voltage sensor. I am running the tests. Will let you know the results on Mon around lunch time.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi David, As agreed: I started adding lines of code to the SC main task to see when the errors will appear:
    1) PASSED a test after running in a 2 hr period OK: This is after adding 2 ON/OFF power operations via digital outputs: gpioClearOutput() and gpioSetOutput(), NO ADC operations used. Periodic Alerts OK gen-d by RTC clock. Our 2nd SC task to monitor battery voltage on the SC was commented out and not used. I did not call any APIs on the SC to explicitly change active <=>low power mode.
    Config:
    Power standby policy enabled on MCU main FW in Release mode (Debug mode also ran OK for around 34 mins and I stopped it, reg-s below).
    SC config: Default pwr mode Active, applied at every SC wake up. Debug trace repeatedly showed:

    SC Alert OK SEC SUBSEC CH2CMP:
    2497 62259200 163647064

    CTL CPUSTAT FETCHSTAT:
    1 1281  1879769235

    1281 in Dec is: 0101 0000 0001  => bits set: HALTED  SLEEP (this is OK)

    2) 2nd test failed when I added ADC operations adcReadFifo(sample): read 4 voltages in total, to the 2 digital outputs: gpioClearOutput() and gpioSetOutput(),
    FAILED after 40 secs with device LEDs OFF and no voltage obtained, SC Periodic Alerts driven/gen-d by RTC clock. Our Battery code commented in SC. SC Periodic alerts stopped after 40 secs. Power standby policy again enabled on MCU main FW in Release mode. SC config: Default pwr mode Active, applied at every SC wake up.

    3) 3rd run same as 2nd run: Failed as above but in 25 mins. FW still in Release mode.

    4) Failed run: Same config as above in Debug mode with log trace: SC alerts lost after about 1 min, error log collected, debug info:
    ERROR: LOST SC Alert SEC SUBSEC CH2CMP:
    91 2197946368 6000080
    CTL CPUSTAT FETCHSTAT:
    1  774  2576351456

    774 decimal in SC CPUSTAT register, this is binary: 0011 0000 0110 => bits set: HALTED WEV
    FETCHSTAT has decimal 2576351456 => hex value: 999000E0       

    SC code in the sce.lst file:
    ;? while(sampleCounter < ADC_MAXIMUM_SAMPLES) {
    /id0149:
    00dc ---- ea10 cmp R6, #16
    00dd ---- a606 bgeu /id0150
    ;? U16 sample;
    ;? adcGenManualTrigger();
    00de ---- 6491 iobset #0, [#IOP_ANAIF_ADCTRIG]
    ;? adcReadFifo(sample);
    00df ---- edb1 wev1 #WEVSEL_ADC_FIFO_NOT_EMPTY
    00e0 ---- 9990 in R1, [#IOP_ANAIF_ADCFIFO]

    I had this error weeks ago and reported it already in this post: CPUSTAT still shows WEV, so it seems to me that is a false "NON_EMPTY" event received by the SC code, that causes a read of an empty buffer and a deadlock/frozen SC state.

    5) Successful run: PASSED: I added my workaround to our SC code: polls via adcGetFifoStatus() and reads via adcPopFifo(sample). I made this according to the TI docs: manual ADCFIFO status check. I re-ran with this workaround and had a good run for 1hr 30 mins, main FW in DEBUG mode. Same config as the prev. run, only changed the read FIFO APIs. 

    I will do more runs and will carry on adding our battery monitor.

  • 0 in reply to Ludmil Kalaydjiyski
    Prodigy 200 points

    Hi David, Thanks for your support.
    We ran more tests, and we think we have a stable Power standby mode on CC2642: no loss of periodic SC alert interrupts. The solution to a stopped and frozen SC because of a blocked TI API adcReadFifo(), seemed to be: instead of adcReadFifo(), poll via adcGetFifoStatus() and read via adcPopFifo(). Such scenario is discussed in the TI SC API documentation.

    As shown above, SC was frozen in the code:
    00df ---- edb1 wev1 #WEVSEL_ADC_FIFO_NOT_EMPTY
    00e0 ---- 9990 in R1, [#IOP_ANAIF_ADCFIFO]

    With our latest configurations, no other scenarios of frozen SC were observed.