This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

CC3200 Timer Issues Using Power Management Framework (cc_timer) Routines

Other Parts Discussed in Thread: CC3200, SYSBIOS

My environment is as follows:

CC3200 running TI-RTOS using the Power Management Framework to manage the application's timers.  The application is capable of running always-on, POWER_POLICY_STANDBY, or POWER_POLICY_HIBERNATE.  When in any of the low power modes the application is scheduled to wake via timer every 2 minutes.  For testing purposes I've updated the timer to generate an interrupt in varying intervals as short as 3 seconds.  In addition to the wakeup timer my application allows for the current use of up to 4 general purpose timers which may be called as needed by the application (all created by cc_timer_create).

The issues I've experience so far are related strictly to the use of the functions encapsulated in cc_timer.c as follows:

1)  Occasionally the system will not awaken via the timer.  The application is still alive and can be awakened via GPIO.  When the system returns to a low-power state it then resumes waking via the timer.  At some future point the system may again not awaken using the timer.  The point at which this happens is random and unpredictable.  This can also happen to any other timer created via the cc_timer_start function.  In this case if a callback function is supplied it will never be called.

After extensive testing I believe what is happening is a scheduled timer expires prior to the point at which it exits the code in cc_timer.  The interrupt generated by the scheduled alarm is essentially lost.  In the attached code I've attempted to isolate where the problem exists bookmarked by CCGI-A comments.

2)  Occasionally my system would completely lock up. When paused in the debugger the code would be in the insert_ordered_expiry function within cc_timer in an infinite loop.  I originally thought there was code missing to handle the case where a sw timer was already encountered in the "in use" list.  However after further research I've determined there should be no duplicate timers on that list and when a sw timer is stopped it should be removed from the list.

In the attached code I've attempted to isolate where the problem exists bookmarked by CCGI-B comments.

3)  When establishing a new timer I've placed code to initialized the flags associated with a specific software timer.  This was not causing any noticeable issues but while researching the other issues I put it in as a safeguard.  This changed is bookmarked by CCGI-C comments.

4)  Cosmetic change to set the sw timer next instance to null when it expires.  More to assist in debugging but is consistent with the state of the object.  This change is bookmarked by CCGI-D comments.

The code I've updated below is not intended to be an end-all solution to the issues described above.  However it is intended to point out where I believe the problems exists and the direction I took to document & attempt to correct the problem (at least for purposes of allowing me to perform system tests which run over the course of several days).

For stress testing purposes I use a TI-RTOS application using 2 general purpose timers created via cc_timer_create along with 1 wakeup timer.  I then create 2 tasks each running a separate timer and put each in a forever loop calling cc_timer_start.  Each timer has it's own callback routine which is posted when complete.  I use different intervals on each timer.  So timer A may be 100 milliseconds while timer B may be set at 800 milliseconds.  Or timer A may be 2 seconds while timer B may be 500 milliseconds.  Unmodified code using cc_timer functions should experience either issue #1 described above.

For testing issue #2 I again employ multiple timers however I stop the execution of one of them before it expires and then restart it (cc_timer_stop, cc_timer_start) repeatedly - using the same handle.  The cycle can be continued until eventually the CC3200 should lock up in the routine indicated.

I view issues #1 & #2 as mission critical errors.  If these have been fixed in something I missed please let me know.  I am unable to authorize any code for production until I obtain the permanent fix for these issues.


//*****************************************************************************
//
// Copyright (C) 2014 Texas Instruments Incorporated - http://www.ti.com/ 
// 
// 
//  Redistribution and use in source and binary forms, with or without 
//  modification, are permitted provided that the following conditions 
//  are met:
//
//    Redistributions of source code must retain the above copyright 
//    notice, this list of conditions and the following disclaimer.
//
//    Redistributions in binary form must reproduce the above copyright
//    notice, this list of conditions and the following disclaimer in the 
//    documentation and/or other materials provided with the   
//    distribution.
//
//    Neither the name of Texas Instruments Incorporated nor the names of
//    its contributors may be used to endorse or promote products derived
//    from this software without specific prior written permission.
//
//  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
//  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
//  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
//  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 
//  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 
//  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
//  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
//  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
//  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
//  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 
//  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//
//*****************************************************************************
//  Maintenance:
//
//  01/11/2015 - CCGI-A - Expired timer scheduled, will never generate interrupt.
//                        Symptom is system enters into hibernate but will never
//                        wake up using timer.  When GPIO used to manually wake
//                        up CC3200 the system will hibernate and subsequently
//                        wake up using timer until hanging again.
// **************************************************************************
//  01/11/2015 - CCGI-B - Code to prevent expired timers from appearing on
//                        the in-use timer list.
//                        Symptom is code will hang up in an infinite loop
//                        in the insert_ordered_expiry function.
// **************************************************************************
//  01/11/2015 - CCGI-C -Code to ensure all processing flags on a new
//                       timer instance are turned off.
//                       Symptom - none.  This is a safeguard to ensure if
//                       a timer bucket is reused the flags are initialized
//                       to their proper value.
// **************************************************************************
//  01/11/2015 - CCGI-D -Cosmetic.  When timer is released NULL out the next
//                       pointer to make debugging easier to read and make
//                       object content consistent with state.


#include "cc_timer.h"

#define U32NS_U16MS(nsec)  (nsec / 1000000)
#define U16MS_U32NS(msec)  (msec * 1000000)

/* Model the HW Timer information */
struct hwt_info {

        struct hw_timer_ops  *ops;       /* Methods to operate HW Timer */
        cc_hndl               hndl;      /* Reference  to real HW Timer */

        struct sw_timer      *used_list; // change to work_list? 
        bool                  hw_64bits;
        
        u32                   source;    /* Source that drives HW Timer */
};

static struct hwt_info hwt_objs[MAX_HWT_PLUG];

static sys_irq_enbl enbl_irqc;
static sys_irq_dsbl dsbl_irqc; 

/*-----------------------------------------------------------------------------
 * Utility functions to access either 32bits or 64bits HW Timer on platform
 *---------------------------------------------------------------------------*/
static struct hwt_info *get_hwt(u32 source)
{
        if((HW_REALTIME_CLK == source) ||
           (HW_MONOTONE_CTR == source)) {
                return hwt_objs + source;
        }

        return NULL;
}

static i32 hwt_start(struct hwt_info *hwt, struct u64_val *expires)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;

        return  hwt->hw_64bits? 
                hwt_ops->start64(hwt->hndl, expires, HW_TIMER_MONOTONE) :
                hwt_ops->start32(hwt->hndl, expires->lo_32, HW_TIMER_MONOTONE);
            
}

static i32 hwt_update(struct hwt_info *hwt, struct u64_val *expires)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;

        return  hwt->hw_64bits?
                hwt_ops->update_exp64(hwt->hndl, expires) :
                hwt_ops->update_exp32(hwt->hndl, expires->lo_32);
}

static i32 hwt_stop(struct hwt_info *hwt)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;

        return hwt_ops->stop(hwt->hndl);
}

static i32 hwt_is_running(struct hwt_info *hwt)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;

        return hwt_ops->is_running(hwt->hndl);
}
#if 0
static i32 hwt_get_remaining(struct hwt_info *hwt, struct u64_val *remaining)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;

        return  hwt->hw_64bits? 
                hwt_ops->get_remaining64(hwt->hndl, remaining) :
                hwt_ops->get_remaining32(hwt->hndl, &remaining->lo_32);

}
#endif
static i32 hwt_get_current(struct hwt_info *hwt, struct u64_val *current)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;

        i32 rv = hwt->hw_64bits?
                hwt_ops->get_current64(hwt->hndl, current) :
                hwt_ops->get_current32(hwt->hndl, &current->lo_32);

        current->hi_32 += hwt_ops->get_rollovers(hwt->hndl);

        return rv;
}

static i32 hwt_get_rollovers(struct hwt_info *hwt)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;
        
        return hwt_ops->get_rollovers(hwt->hndl);
}

static i32 hwt_get_frequency(struct hwt_info *hwt)
{
        struct hw_timer_ops *hwt_ops = hwt->ops;

        return hwt_ops->get_frequency(hwt->hndl);
}

static inline bool hwt_is_64bits(struct hwt_info *hwt)
{
        return hwt->hw_64bits;
}

static inline bool hwt_has_abs_time(struct hwt_info *hwt)
{
        return (HW_REALTIME_CLK == hwt->source)? true : false;
}

/*-----------------------------------------------------------------------------
 * SW Timer structure and utility functions
 *----------------------------------------------------------------------------*/
struct sw_timer {

#define SW_TIMER_PERIODIC   0x00000001
#define SW_TIMER_HW_SCHED   0x00000002
#define SW_TIMER_ALLOCATE   0x00000004

        u32                   flags;       /* Flags to manage SW Timer state */

        struct u64_val        hwt_expires; /* Expiry value: unit is HW count */
        struct u64_val        sw_interval; /* SWT Timeout interval: HW count */ 
        
        cc_cb_fn              timeout_cb;  /* Invoke @ timeout: usr callback */
        cc_hndl               cb_param;

        struct hwt_info      *hwt_obj;     /* Associated Real-world HW Timer */

        struct sw_timer      *next;
};

static inline bool is_periodic(struct sw_timer *swt)
{
        return (swt->flags & SW_TIMER_PERIODIC)? true : false;
}

static inline void set_periodic(struct sw_timer *swt, bool periodic)
{
        if(periodic)
                swt->flags |=  SW_TIMER_PERIODIC;
        else
                swt->flags &= ~SW_TIMER_PERIODIC;
}

static inline bool is_scheduled(struct sw_timer *swt)
{
        return (swt->flags & SW_TIMER_HW_SCHED)? true : false;
}

static inline void set_scheduled(struct sw_timer *swt, bool sched)
{
        if(sched) 
                swt->flags |=  SW_TIMER_HW_SCHED;
        else
                swt->flags &= ~SW_TIMER_HW_SCHED;
}
#if 0
static inline void set_user_status(struct sw_timer *swt, bool used)
{
        if(used)
                swt->flags |=  SW_TIMER_ALLOCATE;
        else
                swt->flags &= ~SW_TIMER_ALLOCATE;
}
#endif
static inline void set_alloc_status(struct sw_timer *swt, bool alloc)
{
        if(alloc)
                swt->flags = SW_TIMER_ALLOCATE;
        else
                swt->flags = 0;
}

static inline bool is_alloc_only(struct sw_timer *swt)
{
        return (swt->flags == SW_TIMER_ALLOCATE)? true : false;
}

static inline bool has_started(struct sw_timer *swt)
{
        return  (swt->flags &  SW_TIMER_ALLOCATE) && 
                (swt->flags != SW_TIMER_ALLOCATE)? true : false;
}

/*-----------------------------------------------------------------------------
 * 64bit Math utility functions
 *----------------------------------------------------------------------------*/

/* Returns: 1 for val1 > val2; 0 for val1 = val2; -1 for val1 < val2 */
static i32 cmp_u32(u32 val1, u32 val2)
{
        i32 rv = -1;

        if(val1 == val2)
                rv = 0;
        else if(val1 > val2)
                rv = 1;

        return rv;
}

/* Returns: 1 for val1 > val2; 0 for val1 = val2; -1 for val1 < val2 */
static i32 cmp_u64(struct u64_val *u64_val1, struct u64_val *u64_val2)
{
        i32 rv = -1;

        if(u64_val1->hi_32 == u64_val2->hi_32)
                rv = cmp_u32(u64_val1->lo_32, u64_val2->lo_32);
        else if(u64_val1->hi_32 > u64_val2->hi_32)
                rv = 1;

        return rv;
}

#define NSEC_VAL_FOR_1SEC   1000000000

static i32 calc_ticks_interval_u32(u64 addendum, struct u64_val *sw_interval)
{
        u64 tmp64 = addendum + sw_interval->lo_32;

        if(0xFFFFFFFF < tmp64)
                return -1;  /* Exceeds 32bit value */
        
        sw_interval->lo_32 = tmp64 & 0xFFFFFFFF;
        sw_interval->hi_32 = 0;

        return 0;
}

static i32 calc_ticks_interval_u64(u64 addendum, struct u64_val *sw_interval)
{
        u64 tmp64 = addendum + sw_interval->lo_32;
        
        sw_interval->lo_32 =  tmp64 & 0xFFFFFFFF;
        sw_interval->hi_32 = (tmp64 >> 32) & 0xFFFFFFFF;
        
        return 0;
}

static 
i32 calc_ticks_interval(struct hwt_info *hwt, struct u64_time *time_u64, 
                        struct u64_val *sw_interval)
                       
{
        u64 tmp64 = 0;
        u32 hwt_freq = hwt_get_frequency(hwt);
        
        if(time_u64->nsec > (NSEC_VAL_FOR_1SEC - 1))
                return -1; /* Value exceeds a sec */

        tmp64 = time_u64->nsec * hwt_freq;
        sw_interval->lo_32 = tmp64 / NSEC_VAL_FOR_1SEC;

        return  hwt_is_64bits(hwt)? 
                calc_ticks_interval_u64(((u64)time_u64->secs) * hwt_freq,
                                        sw_interval) :
                calc_ticks_interval_u32(((u64)time_u64->secs) * hwt_freq,
                                        sw_interval);
}

static void calc_hwt_expiry_nsec_u32(struct u64_val *hwt_current,
									 struct u64_val *sw_interval,
                                     struct u64_val *hwt_expires)
{
		/* 1 sec overflow check: v1 nsec + v2 nsec > 1000000000 nanoseconds */
        if(hwt_current->lo_32 + sw_interval->lo_32 >= NSEC_VAL_FOR_1SEC) {
                u32 exceeds = hwt_current->lo_32 + sw_interval->lo_32 - NSEC_VAL_FOR_1SEC;
                hwt_expires->lo_32  = exceeds;
                hwt_expires->hi_32 += 1;                        /* Carry flag */
        } else {
                /* Simple 32bits addition without any carry flag and overflow */
                hwt_expires->lo_32  = hwt_current->lo_32 +  sw_interval->lo_32;
        }

        return;
}

static void calc_hwt_expiry_secs_u64(struct u64_val *hwt_current, 
                                     struct u64_val *sw_interval, 
                                     struct u64_val *hwt_expires)
{
        calc_hwt_expiry_nsec_u32(hwt_current, sw_interval, hwt_expires);
        
        hwt_expires->hi_32 += hwt_current->hi_32 + sw_interval->hi_32;

        return;
}

static void calc_hwt_expiry_secs(struct hwt_info *hwt,
                                 struct u64_val  *hwt_current, 
                                 struct u64_val  *sw_interval,
                                 struct u64_val  *hwt_expires)
                                 
{
        hwt_is_64bits(hwt)? 
                calc_hwt_expiry_secs_u64(hwt_current, sw_interval, hwt_expires) :
                calc_hwt_expiry_nsec_u32(hwt_current, sw_interval, hwt_expires);

        return;
}

static void calc_hwt_expiry_tick_u32(struct u64_val *hwt_current, 
                                     struct u64_val *sw_interval, 
                                     struct u64_val *hwt_expires)
{
        /* 32bit overflow check: v1 + v2 > 0xFFFFFFFF => v1 > 0xFFFFFFFF - v2 */
        if(hwt_current->lo_32 > ~sw_interval->lo_32) {
                /* exceeds by => v1 + v2 - (0xFFFFFFFF + 1) = v1 - (~v2 + 1)  */
                u32 exceeds = hwt_current->lo_32 - ~sw_interval->lo_32 - 1;
                hwt_expires->lo_32  = exceeds;
                hwt_expires->hi_32 += 1;                        /* Carry flag */
        } else {
                /* Simple 32bits addition without any carry flag and overflow */
                hwt_expires->lo_32  = hwt_current->lo_32 +  sw_interval->lo_32;
        }
        
        return;
}

static void calc_hwt_expiry_tick_u64(struct u64_val *hwt_current, 
                                     struct u64_val *sw_interval, 
                                     struct u64_val *hwt_expires)
{
        calc_hwt_expiry_tick_u32(hwt_current, sw_interval, hwt_expires);

        hwt_expires->hi_32 += hwt_current->hi_32 + sw_interval->hi_32;

        return;
}

static void calc_hwt_expiry_tick(struct hwt_info *hwt,
                                 struct u64_val  *hwt_current, 
                                 struct u64_val  *sw_interval,
                                 struct u64_val  *hwt_expires)
{
        hwt_is_64bits(hwt)? 
                calc_hwt_expiry_tick_u64(hwt_current, sw_interval,
                                         hwt_expires):
                calc_hwt_expiry_tick_u32(hwt_current, sw_interval,
                                         hwt_expires);

        return;
}

static void calc_hwt_expiry(struct hwt_info *hwt,
                           struct u64_val  *hwt_current, 
                           struct u64_val  *sw_interval,
                           struct u64_val  *hwt_expires)
{
        hwt_expires->hi_32 = hwt_get_rollovers(hwt);

        hwt_has_abs_time(hwt)?
                calc_hwt_expiry_secs(hwt, hwt_current, sw_interval,
                                     hwt_expires):
                calc_hwt_expiry_tick(hwt, hwt_current, sw_interval,
                                     hwt_expires);

        return;
}

/*-----------------------------------------------------------------------------
 * Core SW Timer servics
 *----------------------------------------------------------------------------*/
static void insert_ordered_expiry(struct sw_timer *elem, struct sw_timer **list)
{
        struct sw_timer *node = *list, *prev = NULL;

        if(NULL == node) {  /* First user request for the  HW timer */
                *list = elem;
                goto sort_insert_elem_exit1;
        }

        while(node) {        /* There is atleast one active request */
                if(0 > cmp_u64(&elem->hwt_expires, &node->hwt_expires)) {
                        /* 'elem' expires earlier than this 'node'  */
                        elem->next = node; 
                        
                        if(NULL == prev) {
                                *list  = elem; /* New first element */
                                set_scheduled(node, false);
                        }
                        else
                                prev->next = elem; /* Intermediary  */

                        /* 'elem' has been placed in 'list', quit.. */
                        goto sort_insert_elem_exit1;
                }

                prev = node;
                node = node->next;
        }

        if(NULL == node) 
                prev->next = elem; /* Farthest in time for schedule */

 sort_insert_elem_exit1:
        return;
}

static i32 sched_timer_if_new(struct sw_timer *head)
{
        i32 rv = 0;
        struct hwt_info *hwt = head->hwt_obj;

        if(true == is_scheduled(head)) 
                goto sched_timer_if_new_exit;
        
        /* If HW timer running, then update it othewise start it */        
        rv = hwt_is_running(hwt)?
                hwt_update(hwt, &head->hwt_expires) :
                hwt_start(hwt, &head->hwt_expires);

        if(0 == rv)
                set_scheduled(head, true);
        
 sched_timer_if_new_exit:
        return rv;
}

static i32 setup_timer_exec(struct sw_timer *swt, u32 options) 
{
        /* Step 1: Use absolute expiry time, if provided. Otherwise .....
           Step 2: Calculate absolute expiry time in terms of HW counter.
           
           TODO: Seems that providing both flags and time_u64 may be redundant
        */

        struct hwt_info *hwt = swt->hwt_obj;

        if(options & OPT_TIME_ABS_VALUE) {
                /* Step 1: Use absolute time, if it has been provided */
                swt->hwt_expires.hi_32 = swt->sw_interval.hi_32;
                swt->hwt_expires.lo_32 = swt->sw_interval.lo_32;
        } else {
                /* Step 2. Calculate HW expiry time using interval .. */
                struct u64_val hwt_current = {0, 0}; /* initial value */

                if((hwt_is_running(hwt) || hwt_has_abs_time(hwt)) && 
                   (-1 == hwt_get_current(hwt, &hwt_current)))
                                return -1;

                calc_hwt_expiry(hwt, &hwt_current, 
                                &swt->sw_interval,
                                &swt->hwt_expires);
        }
        
        insert_ordered_expiry(swt, &hwt->used_list);

        sched_timer_if_new(hwt->used_list);

        return 0;
}

static i32 setup_timer_interval(struct sw_timer *swt, struct u64_time *time_u64)
{
        struct hwt_info *hwt = swt->hwt_obj;

        if(false == hwt_has_abs_time(hwt)) {
                /* Convert time specified by user into ticks for HW Timer */
                if(-1 == calc_ticks_interval(hwt, time_u64, &swt->sw_interval))
                        return -1;
        } else {
                /* If HWT has support for real-time, work with user info */
                swt->sw_interval.hi_32 = time_u64->secs;
                swt->sw_interval.lo_32 = time_u64->nsec;
        }
        
        return 0;
}

static i32 timer_start(struct sw_timer *swt, struct u64_time *time_u64,
                       u32 options)
{
        if(false == is_alloc_only(swt))
                return -1; /* Not only allocated but started as well */
        
        if((-1 == setup_timer_interval(swt, time_u64))  ||
           (-1 == setup_timer_exec(swt, options)))
                return -1;

        set_periodic(swt, options & OPT_TIMER_PERIODIC ? true : false);

        return 0;
}

static bool has_valid_opts(struct hwt_info *hwt, u32 options)
{
        /* Implementtion: A periodic timer can't work with absolute time */
        if((OPT_TIMER_PERIODIC & options) && (OPT_TIME_ABS_VALUE & options))
                return false; 

        /* A request for absolute time has to be supported on HW */
        if((options & OPT_TIME_ABS_VALUE) && !hwt_has_abs_time(hwt))
                return false;

        return true;
}

// **************************************************************************
// * CCGI-A * BEGIN * Code block to minimize potential of expired timer
//                    not generating interrupt
static void process_timer_expiry(struct hwt_info *hwt);
// * CCGI-A * END   *
// **************************************************************************

i32 cc_timer_start(cc_hndl hndl, struct u64_time *time_u64, u32 options)
{
        struct sw_timer *swt = (struct sw_timer*) hndl;
        struct u64_time time64_ref = {0, 0}; /* init */
        // **************************************************************************
        // * CCGI-A * BEGIN * Code block to minimize potential of expired timer
        //                    not generating interrupt
        int iTimerExpired = -1;							//  Flag to be used to determine if last SW timer expired before routine exit
        struct hwt_info *sHWTimer;						//  Pointer to current HW Timer in use
        struct u64_val sHWT_ExitClock;					//	Field used to hold HW Timer value
        struct sw_timer *head;							//  Pointer to head of list of active SW timers
        // * CCGI-A * END   *
        // **************************************************************************
        u32 intr_mask;
        i32 rv = -1;
        
        if((NULL == swt) || 
           (false == has_valid_opts(swt->hwt_obj, options)))
                goto cc_timer_start_exit1;
        
#define IS_LRT_TIMER(hwt) hwt_has_abs_time(hwt) /* LRT:: Low Resolution Timer */
        
        time64_ref.secs = time_u64->secs;
        time64_ref.nsec = time_u64->nsec;

        if(IS_LRT_TIMER(swt->hwt_obj)) {
                /* reducing the precision to milliseconds to avoid floating point ops */
                time64_ref.nsec = U16MS_U32NS(U32NS_U16MS(time_u64->nsec));
        }

        intr_mask = dsbl_irqc();
        rv = timer_start(swt, &time64_ref, options);
        enbl_irqc(intr_mask);

        // **************************************************************************
        // * CCGI-A * BEGIN * Code block to minimize potential of expired timer
        //                    not generating interrupt
        sHWTimer = swt->hwt_obj;                              				//  Establish addressability to HW timer in use

        head = sHWTimer->used_list;											//  Establish addressability to active SW timers

        while (head && iTimerExpired) {							 			//  At least 1 SW timer must be active, check until we find SW timer not yet expired
        	hwt_get_current(sHWTimer, &sHWT_ExitClock);						//  Get current HWT clock

        	iTimerExpired = cmp_u64(&sHWT_ExitClock, &head->hwt_expires);   //  Compare HWT clock against next SW timer expiration (0=they equal, 1=HW timer > expiration)

        	if (iTimerExpired >= 0) {						 				//  If HWT >= SW expiration then go ahead and process it now
        		process_timer_expiry(sHWTimer);
        	} else {
        		iTimerExpired = 0;											//  If HWT < SW expiration then set RC to exit routine
        	}

            head = sHWTimer->used_list;										//  Re-establish addressability to active SW timers in case 1 or more have expired
        }
        // * CCGI-A * END   *
        // **************************************************************************

cc_timer_start_exit1:
        
        return rv;
} 

static i32 remove_elem(struct sw_timer *elem, struct sw_timer **list)
{
        struct sw_timer *node = *list, *prev = NULL;
        i32 rv = -1;

        if(NULL == node)
                return rv;

        while(node) {
                if(elem == node) {
                        if(NULL == prev)
                                *list      = node->next;
                        else
                                prev->next = node->next; 

                        
                        // **************************************************************************
                        //  CCGI-D * BEGIN * Costmetic.
                                                elem->next = NULL;
                        //  CCGI-D * END   *
                        // **************************************************************************
                        rv = 0;
                        break;  /* Exit while loop */
                }
                
                prev = node;
                node = node->next;
        }

        return rv;
}

static i32 timer_stop(struct sw_timer *swt)
{
        struct hwt_info *hwt = swt->hwt_obj;
        // **************************************************************************
        // * CCGI-A * BEGIN * Code block to minimize potential of expired timer
        //                    not generating interrupt
        int iTimerExpired = -1;							//  Flag to be used to determine if last SW timer expired before routine exit
        struct hwt_info *sHWTimer;						//  Pointer to current HW Timer in use
        struct u64_val sHWT_ExitClock;					//	Field used to hold HW Timer value
        struct sw_timer *head;							//  Pointer to head of list of active SW timers
        // * CCGI-A * END   *
        // **************************************************************************

        if(0 != remove_elem(swt, &hwt->used_list)) {
                return -1;
        }

        set_scheduled(swt, false);
        set_periodic(swt, false);

        if(NULL != hwt->used_list)
       	// **************************************************************************
       	// * CCGI-A * BEGIN * Code block to minimize potential of expired timer
       	//                    not generating interrupt
        {
       	// * CCGI-A * END   *
       	// **************************************************************************
            sched_timer_if_new(hwt->used_list);

            // **************************************************************************
            // * CCGI-A * BEGIN * Code block to minimize potential of expired timer
            //                    not generating interrupt
        	sHWTimer = swt->hwt_obj;                            					//  Establish addressability to HW timer in use

        	head = sHWTimer->used_list;												//  Establish addressability to active SW timers

        	while (head && iTimerExpired) {											//  At least 1 SW timer must be active, check until we find SW timer not yet expired
        		hwt_get_current(sHWTimer, &sHWT_ExitClock);							//  Get current HWT clock

        		iTimerExpired = cmp_u64(&sHWT_ExitClock, &head->hwt_expires);    	//  Compare HWT clock against next SW timer expiration (0=they equal, 1=HW timer > expiration)

        		if (iTimerExpired >= 0) {											//  If HWT >= SW expiration then go ahead and process it now
        			process_timer_expiry(sHWTimer);
        		} else {
        			iTimerExpired = 0;												//  If HWT < SW expiration then set RC to exit routine
        		}

        		head = sHWTimer->used_list;											//  Re-establish addressability to active SW timers in case 1 or more have expired
        	}
        }
        // * CCGI-A * END   *
        // **************************************************************************
        else
                hwt_stop(hwt); /* No pending request, stop HW */

        return 0;
}

i32 cc_timer_stop(cc_hndl hndl)
{
        struct sw_timer *swt = (struct sw_timer*) hndl;
        u32 intr_mask;
        i32 rv = -1;

        intr_mask = dsbl_irqc();
        if(swt && has_started(swt)) {
                rv = timer_stop(swt);
        }
		// **************************************************************************
        // * CCGI-B * BEGIN * Code block to prevent expired timers from appearing on
        //                    the in-use timer list.
		else {
        	remove_elem(swt, &swt->hwt_obj->used_list);
		}
        // * CCGI-B * END   *
		// **************************************************************************
        enbl_irqc(intr_mask);

        return rv;
}

static void handle_expired_timer(struct sw_timer *swt)
{
        set_scheduled(swt, false);

        /* Invoke user's callback routine */
        if(swt->timeout_cb)
                swt->timeout_cb(swt->cb_param);
        
        /* Period timer: Set-up next iteration */
        if(true == is_periodic(swt))
                setup_timer_exec(swt, 0);

        return;
}

/* Called in the interrupt context */
static void process_timer_expiry(struct hwt_info *hwt)
{
        struct sw_timer *head = hwt->used_list;
        
        while(head) { /* Run through list to process all expired timers */
                struct u64_val hwt_current;

                if(0 != hwt_get_current(head->hwt_obj, &hwt_current))
                        goto handle_timer_expiry_exit1;
                
                if(0 > cmp_u64(&hwt_current, &head->hwt_expires))
                        break; /* Timer is yet to reach expiry, so quit */

                remove_elem(head, &hwt->used_list);

                handle_expired_timer(head);

                head = hwt->used_list;
        }
        
        if(head)
                sched_timer_if_new(head);
        else
                hwt_stop(hwt);
        
 handle_timer_expiry_exit1:
        return;
}

/* To be invoked by HAL */
static void timer_isr(cc_hndl cb_param)
{
        struct hwt_info *hwt  = (struct hwt_info*) cb_param;
        // **************************************************************************
        // * CCGI-A * BEGIN * Code block to minimize potential of expired timer
        //                    not generating interrupt
        struct sw_timer *head = hwt->used_list;					//  Pointer to head of list of active SW timers
        struct u64_val sHWT_ExitClock;							//	Field used to hold HW Timer value
        int iTimerExpired = -1;									//  Flag to be used to determine if last SW timer expired before routine exit
        // * CCGI-A * END   *
        // **************************************************************************

        if(NULL != hwt)
       	// **************************************************************************
       	// * CCGI-A * BEGIN * Code block to minimize potential of expired timer
       	//                    not generating interrupt
        {
       	// * CCGI-A * END   *
       	// **************************************************************************
                process_timer_expiry(hwt);

                // **************************************************************************
                // * CCGI-A * BEGIN * Code block to minimize potential of expired timer
                //                    not generating interrupt
                head = hwt->used_list;													//  Establish addressability to active SW timers

                while (head && iTimerExpired) {											//  At least 1 SW timer must be active, check until we find SW timer not yet expired
                	hwt_get_current(hwt, &sHWT_ExitClock);								//  Get current HWT clock

                	iTimerExpired = cmp_u64(&sHWT_ExitClock, &head->hwt_expires);		//  Compare HWT clock against next SW timer expiration (0=they equal, 1=HW timer > expiration)

                	if (iTimerExpired >= 0) {											//  If HWT >= SW expiration then go ahead and process it now
                		process_timer_expiry(hwt);
                	} else {
                		iTimerExpired = 0;												//  If HWT < SW expiration then set RC to exit routine
                	}

                    head = hwt->used_list;												//  Re-establish addressability to active SW timers in case 1 or more have expired
                }
        }
        // * CCGI-A * END   *
        // **************************************************************************

        return;
}

/*-----------------------------------------------------------------------------
 * SW Timer Service Initialization
 *----------------------------------------------------------------------------*/
static struct sw_timer  sw_timers[MAX_SW_TIMER];
static struct sw_timer *free_list = NULL;

cc_hndl cc_timer_create(struct cc_timer_cfg *cfg)
{
        struct sw_timer *swt = NULL;
        struct hwt_info *hwt = NULL;
        u32 intr_mask;        

        hwt = get_hwt(cfg->source);
        if((NULL == hwt) || (NULL == hwt->ops))
                goto cc_timer_create_exit1;

        intr_mask = dsbl_irqc();  /* Disable interrupts */

        swt = free_list;
        if(NULL == swt)
                goto cc_timer_create_exit2;

        free_list = swt->next;
        swt->next = NULL;

	// **************************************************************************
        // * CCGI-C * BEGIN * Code block to ensure all processing flags on a new
	//                    timer instance are turned off.
        swt->flags = 0x00;
        // * CCGI-C * END   *
	// **************************************************************************
        set_alloc_status(swt, true);
        swt->hwt_obj    = hwt;
        swt->timeout_cb = cfg->timeout_cb;
        swt->cb_param   = cfg->cb_param;

 cc_timer_create_exit2:
        enbl_irqc(intr_mask); /* Enable  interrupts */

 cc_timer_create_exit1:

        return (cc_hndl) swt;
}

i32 cc_timer_delete(cc_hndl hndl)
{
        struct sw_timer *swt = (struct sw_timer*) hndl;
        u32 intr_mask;

        if(NULL == swt || !is_alloc_only(swt))
                return -1;

        set_alloc_status(swt, false);
        memset(swt, 0, sizeof(swt));
        swt->hwt_obj = NULL;
        swt->next    = NULL;
        
        intr_mask = dsbl_irqc();
        swt->next = free_list;
        free_list = swt;
        enbl_irqc(intr_mask);

        return 0;
}

static i32 sw_timers_init(void)
{
        i32 i = 0;

        memset(sw_timers, 0, sizeof(sw_timers));

        for(i = 0; i < MAX_SW_TIMER; i++) {
                struct sw_timer *swt = sw_timers + i;
                swt->next = free_list;
                free_list = swt;
        }

        return 0;
}

static i32 hw_timers_init(void)
{
        i32 i = 0;
        i32 max_val = MAX_HWT_PLUG - 1;
        
        if((HW_REALTIME_CLK > max_val) || (HW_MONOTONE_CTR > max_val))
                return -1;
        
        for(i = 0; i < MAX_HWT_PLUG; i++) {
                struct hwt_info *hwt = hwt_objs + i;

                hwt->ops       = NULL;
                hwt->hndl      = NULL;
                hwt->used_list = NULL;
                hwt->hw_64bits = false;
                hwt->source    = i;
        }
        
        return 0;
}

i32 cc_timer_module_init(struct cc_timer_setup *timer_setup)
{
        if((NULL == timer_setup->enbl_irqc) || 
                (NULL == timer_setup->dsbl_irqc)) {
                return -1;
        }
        enbl_irqc = timer_setup->enbl_irqc;
        dsbl_irqc = timer_setup->dsbl_irqc;

        return sw_timers_init() || hw_timers_init();
}

i32 cc_timer_register_hwt_ops(u32 source,  cc_hndl hwt_hndl, 
                              struct hw_timer_ops *hwt_ops)
{
        struct hwt_info *hwt = get_hwt(source);
        bool op64;
        
        if((NULL == hwt) || (NULL != hwt->ops) || (NULL == hwt_hndl)) 
                return -1; /* HWT ops already initialized */

#define HAS_HWT_OPS(f1, f2, f3, f4)                                     \
        (hwt_ops->f1 && hwt_ops->f2 && hwt_ops->f3 && hwt_ops->f4)?  true : false
        
        op64 = HAS_HWT_OPS(start64, update_exp64, get_remaining64,
                           get_current64);

        /* Set of ops specific to either 32bit or 64bit must be available */
        if(!(HAS_HWT_OPS(start32, update_exp32, get_remaining32, get_current32) ^ 
             op64))
                return -1;
        
        /* Set of ops common to both 32bit or 64bit must be availalbe */
        if(!HAS_HWT_OPS(stop, is_running, get_rollovers, get_frequency) ||
           !hwt_ops->register_cb)
                return -1;

        /* Install callback function in HW Timer */
        if(-1 == hwt_ops->register_cb(hwt_hndl, timer_isr, hwt))
                return -1;

        hwt->ops       = hwt_ops;
        hwt->hndl      = hwt_hndl;
        hwt->hw_64bits = op64;

        return 0;
}

  • Hi Craig,

    Let me check and get back to you.

    Thanks and Regards,
    Praveen
  • Note the statement updated to:

    remove_elem(swt, &swt->hwt_obj->used_list);

    should read:

    rv = remove_elem(swt, &swt->hwt_obj->used_list);

    I'll post any other changes if I find additional issues.
  • Hi Craig,

    Thanks for the detailed post!

    There were some bug fixes to the timer infrastructure that we did add, in response to your multiple timer lockup issue - captured at http://processors.wiki.ti.com/index.php/CC32xx_Summary_of_Known_Issues 

  • Hi Craig,

    My post wasnt complete. Could you help respond to the below queries?

    1. In the usecase, could it be possible that there were some general purpose timers scheduled to expire when the system entered LPDS?
    2. Is the issue seen when the system doesnt enter low power modes - by disabling PM framework by invoking cc_app_putoff_pm?
    3. Would it be possible to share a minimal subset of your application just sufficient to recreate the issue?

    Thanks.

    Best regards,
    Naveen
  • 1)  Normally I try and ensure all GP timers are either expired or stopped before allowing entry into a low power mode.  From what I've seen the power management system allows for up to 6 timers and has a scheduled alarm set for the first of the 6 to expire.  The wake timer shares this bank of timers and once a timer has been scheduled power management can not distinguish between a GP timer and a wake timer.  What I have seen is as I reduce the wake interval (down to 1 to 2 seconds) there seems to be a higher likelihood of an interrupt missing.  The easiest way to see this however is to have a TI-RTOS app with multiple timers running concurrently and all crossing each others expiration.  I originally added code to tally how many start, update, stops and interrupts were being encountered.  At some point my program would stop (because the first timer to fail will cause the remaining 5 to never be scheduled).  I would then manually go to a cc_start_timer and set it as the next step and step into cc_timer routine then inspect the variables I tallied.

    2)  Yes, the timer test I had did not exercise low power mode at all.  It was simply my application changed to do a series of FOREVER loops before it actually got to the application code.  There were some tasks running in the background but they were not the cause of the lockup.  As stated in #1 above I actually could monitor what cc_timer was doing.  I did catch a few situations where I captured the real time clock just after the routine and it virtually equaled the expire value for the alarm.

    3)  Probably not.  But what I could do is possibly spend some time to create a custom testing application along with a "instrumented" version of cc_timer.  You should be able to recreate what I did by having multiple tasks do what I did.  First create a bank of timers (so they are created once).  Have at least 1 "load" task doing something in the background.  Then have at least 2 tasks starting timers and waiting for them to expire at different (short) intervals. 

  • Hi Craig,

    Thanks for your reply!

    Do allow us some time to recreate and debug the issue. Your help with a custom test application will definitely help in recreating the issue faster and hence expedite the debug.

    Best regards,

    Naveen

  • Naveen,

    I've attached a sample application which contains a set of 2 timer different timer tests using 3 tasks within the application.  When run all you're looking for is for the timer to hang (which indicates there is a problem).  When no bugs are present the timers should simply run continuously until the CC3200 is manually reset.

    This does require a ti_rtos application and I've included the one I setup for the test.  If you choose to use your own the following statements will have to be added to the cfg.script to account for the idle task:

    var Idle = xdc.useModule('ti.sysbios.knl.Idle');

    Idle.addFunc('&Main_Power_Idle');

    var Task = xdc.useModule('ti.sysbios.knl.Task');

    Task.idleTaskStackSize = 3072;

    Note:  I've included my version of cc_timer.c in the application.  Your first step should be to move that file off to the side and then recompile the application.  This does not use any low power mode so there's no issues with simply loading it out of CCS. 

    Also:  Since there are 2 separate tests it may be easier to run one at a time.  In that case start out by commenting out the Task_create statements for TimerTask1 & TimerTask2 and only run TimerTask3.  When you're done with the TimerTask3 test do the reverse and run only TimerTask1 & TimerTask2.  When all bugs have been fixed then run them all and let it go for at least a day.

    I'm pretty sure I already know the answer to this but is there any way TI might consider compensating us for the effort to isolate the issues and put together testing routines?  I've spent a lot of time isolating issues in the SDK and the boss isn't pleased.

    TI_Timer_Lockup_Reported.zip

  • Hi Craig,Many thanks for sharing the application!
    Regarding the 2 tests, we have the following updates:
    1. Test 1 - (TimerTask1 and TimerTask2) - We are able to recreate the issue and are working on providing a fix for the same. I will keep you posted on this.
    2. Test 2 - TimerTask3 - The changes to cc_timer.c mentioned in http://processors.wiki.ti.com/index.php/CC32xx_Summary_of_Known_Issues seems to be helping. Could you try checking incorporating the mentioned changes?
    Best regards,Naveen
  • Naveen,

    Thanks for your reply.  The TimerTask3 changes outlined in the summary of known issues does appear to run.  I look forward to receiving the other fixes.

    Craig

  • Hi Craig,
    We think we have it resolved and have initiated longer stability checks.
    We will share the changes early next week.Best regards,Naveen
  • Hi Craig,
    May we contact you at the email address you provided in your E2E profile?
    Best regards,Naveen
  • Naveen,

    Feel free to contact me anytime at the email address on file for my account.

    Thanks,

    Craig