This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM263P4-Q1: FreeRTOS CPU Load Calculation Settling Time

Part Number: AM263P4-Q1

Tool/software:

Hello,

I'm reading the value of TaskP_loadGetTotalCpuLoad() in two cores at 1s intervals. configGENERATE_RUN_TIME_STATS=1. The FreeRTOS is from SDK 09_02_00_56.

Core 0 has FreeRTOS with some CAN code but not much else. Core 3 has only FreeRTOS with some tasks scheduled.

The problem is that the CPU load calculation is heavily filtered. It takes about a minute to reach a steady value. Is there any way I can tweak it to respond more quickly? I don't mind a bit of noise.

Here's the FreeRTOS config file:

/*
 *  Copyright (C) 2018-2021 Texas Instruments Incorporated
 *
 *  Redistribution and use in source and binary forms, with or without
 *  modification, are permitted provided that the following conditions
 *  are met:
 *
 *    Redistributions of source code must retain the above copyright
 *    notice, this list of conditions and the following disclaimer.
 *
 *    Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the
 *    distribution.
 *
 *    Neither the name of Texas Instruments Incorporated nor the names of
 *    its contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
 *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
 *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
 *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
 *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
 *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */


#ifndef TI_FREERTOS_CONFIG_H
#define TI_FREERTOS_CONFIG_H

#ifdef __cplusplus
extern "C"
{
#endif

#include <kernel/dpl/DebugP.h>

/*-----------------------------------------------------------
 * Application specific definitions.
 *
 * These definitions should be adjusted for your particular hardware and
 * application requirements.
 *
 * THESE PARAMETERS ARE DESCRIBED WITHIN THE 'CONFIGURATION' SECTION OF THE
 * FreeRTOS API DOCUMENTATION AVAILABLE ON THE FreeRTOS.org WEB SITE AND IN THE
 * FreeRTOS REFERENCE MANUAL.
 *----------------------------------------------------------*/

/* Keep below as 1 if the most optimized task switching latency is needed.
 * This disables tracing, logging, assert and other error checks.
 * So unless every last cycle of task switching is important leave this as 0.
 *
 * This is not a FreeRTOS defined config and is defined by TI to quickly switch
 * between optimized and not-so-optimized config
 */
#define configOPTIMIZE_FOR_LATENCY              (0)

#define configUSE_PREEMPTION					(1)
#define configUSE_PORT_OPTIMISED_TASK_SELECTION	(0)
#define configUSE_TICKLESS_IDLE                 (1)
#define configUSE_IDLE_HOOK                     (1)  /* when 1, make sure to implement void vApplicationIdleHook(void) as the hook function */
#define configUSE_MALLOC_FAILED_HOOK            (0)
#define configUSE_DAEMON_TASK_STARTUP_HOOK      (0)
#define configUSE_TICK_HOOK                     (0)
#define configCPU_CLOCK_HZ                      (0) /* NOT USED in TI ports */
#define configSYSTICK_CLOCK_HZ                  (0) /* NOT USED in TI ports */
#define configTICK_RATE_HZ                      (10000)
#define configMAX_PRIORITIES                    (32)
#define configMINIMAL_STACK_SIZE                (1024) /* in units of configSTACK_DEPTH_TYPE, not bytes */
#define configMAX_TASK_NAME_LEN                 (32)
#define configUSE_TRACE_FACILITY                (1)
#if (configOPTIMIZE_FOR_LATENCY==0)
#define configUSE_STATS_FORMATTING_FUNCTIONS    (1)
#else
#define configUSE_STATS_FORMATTING_FUNCTIONS    (0)
#endif
#define configUSE_16_BIT_TICKS                  (0)
#define configIDLE_SHOULD_YIELD                 (1)
#define configUSE_TASK_NOTIFICATIONS            (1)
#define configTASK_NOTIFICATION_ARRAY_ENTRIES   (1)
#define configUSE_MUTEXES                       (1)
#define configUSE_RECURSIVE_MUTEXES             (1)
#define configUSE_COUNTING_SEMAPHORES           (1)
#define configUSE_ALTERNATIVE_API               (0)

/* when = 1, Need to provied below,
 *    void vApplicationStackOverflowHook( TaskHandle_t xTask,
 *           char *pcTaskName );
 */
#if (configOPTIMIZE_FOR_LATENCY==0)
#define configCHECK_FOR_STACK_OVERFLOW          (1)
#else
#define configCHECK_FOR_STACK_OVERFLOW          (0)
#endif
#define configQUEUE_REGISTRY_SIZE               (32)
#define configUSE_QUEUE_SETS                    (0)
#define configUSE_TIME_SLICING                  (0) /* keep as 0 to get same functionality as SysBIOS6 */
#define configUSE_NEWLIB_REENTRANT              (0)
#define configENABLE_BACKWARD_COMPATIBILITY     (1)
#define configNUM_THREAD_LOCAL_STORAGE_POINTERS (4)
#define configSTACK_DEPTH_TYPE                  UBaseType_t
#define configMESSAGE_BUFFER_LENGTH_TYPE        size_t
#define configSUPPORT_STATIC_ALLOCATION         (1) /* when = 1, need to provide below,
                                                     *
                                                     * void vApplicationGetTimerTaskMemory( StaticTask_t **ppxTimerTaskTCBBuffer,
                                                     *      StackType_t **ppxTimerTaskStackBuffer,
                                                     *      uint32_t *pulTimerTaskStackSize );
                                                     *
                                                     * void vApplicationGetIdleTaskMemory( StaticTask_t **ppxIdleTaskTCBBuffer,
                                                     *      StackType_t **ppxIdleTaskStackBuffer,
                                                     *      uint32_t *pulIdleTaskStackSize );
                                                     */
#define configSUPPORT_DYNAMIC_ALLOCATION        (1)
#define configTOTAL_HEAP_SIZE                   (4*1024) /* not used when heap_3.c is the selected heap */
#define configAPPLICATION_ALLOCATED_HEAP        (0)

/* run-time stats config */
#if (configOPTIMIZE_FOR_LATENCY==0)
#define configGENERATE_RUN_TIME_STATS           (1)
#else
#define configGENERATE_RUN_TIME_STATS           (0)
#endif
void vPortConfigTimerForRunTimeStats();
#define portCONFIGURE_TIMER_FOR_RUN_TIME_STATS() vPortConfigTimerForRunTimeStats()
uint32_t uiPortGetRunTimeCounterValue();
#define portGET_RUN_TIME_COUNTER_VALUE()        uiPortGetRunTimeCounterValue()

/* co-routine related config */
#define configUSE_CO_ROUTINES                   (0)
#define configMAX_CO_ROUTINE_PRIORITIES         (0)

/* timer related config */
#define configUSE_TIMERS                        (1)
#define configTIMER_TASK_PRIORITY               (configMAX_PRIORITIES - 1)
#define configTIMER_QUEUE_LENGTH                (16)
#define configTIMER_TASK_STACK_DEPTH            (256)

/* used in M4F & R5F, not applicable in C66x, A53 */
#define configMAX_SYSCALL_INTERRUPT_PRIORITY    (0x4U)

/* used in M4F, not applicable in R5F, C66x, A53 */
#define configKERNEL_INTERRUPT_PRIORITY         (configMAX_SYSCALL_INTERRUPT_PRIORITY)
#define configMAX_API_CALL_INTERRUPT_PRIORITY   (configMAX_SYSCALL_INTERRUPT_PRIORITY)

#if (configOPTIMIZE_FOR_LATENCY==0)
#define configASSERT(x)                        DebugP_assert( (uint32_t)(x))
#endif

/* MPU aware FreeRTOS, not supported as TI */
#define configINCLUDE_APPLICATION_DEFINED_PRIVILEGED_FUNCTIONS  (0)
#define configTOTAL_MPU_REGIONS                                 (0)
#undef  configTEX_S_C_B_FLASH
#undef  configTEX_S_C_B_SRAM
#define configENFORCE_SYSTEM_CALLS_FROM_KERNEL_ONLY             (0)

/* defines need for FreeRTOS+POSIX*/
#define configUSE_POSIX_ERRNO           (1)
#define configUSE_APPLICATION_TASK_TAG  (1)

/* include specific functions */
#define INCLUDE_vTaskDelete             (1)
#define INCLUDE_vTaskDelay              (1)
#define INCLUDE_vTaskSuspend            (1)
#define INCLUDE_xTimerDelete            (1)
#define INCLUDE_vSemaphoreDelete        (1)
#define INCLUDE_xTimerPendFunctionCall  (1)
#define INCLUDE_xTaskGetIdleTaskHandle  (1)
#define INCLUDE_xTaskDelayUntil  (1)

#ifdef __cplusplus
}
#endif

#endif /* TI_FREERTOS_CONFIG_H */

Thank you.

  • Hi Kier,

    1. The only difference I see in the standard FreeRTOSConfig.h file and the one you shared above is the tick rate increased to 10,000. Was the behavior same with the reduced tick rate?

    2. Are there any other configuration changes made in the FreeRTOS Kernel compared to the Standard SDK configuration?

    3. I ran some tests on my end, I modified the MCAN Interrupt loopback example on the AM263Px-LP to print the CPU load every 2 seconds while MCAN communication as well as after the communication. (with tick rate of 1000 as provided in standard SDK)

    Regards,
    Shaunak

  • Hi Shaunak,

    Thank you for the reply and verifying at your end.

    I compared our configuration with the SDK config and can see two differences:


    1) Our project uses TICKLESS_IDLE where the SDK does not.

    2) As you mentioned, our project uses the higher TICK_RATE_HZ.


    Do you think 1) may affect the CPU load calculation?

    Anyway, I shall reverse both of these and see if this improves the response.

  • Hi kier,

    Anyway, I shall reverse both of these and see if this improves the response.

    Did reverting this help?

    Regards,
    Shaunak

  • Sorry, I haven't had chance to check it yet.

  • Hi Shaunak,

    I changed the tick rate to the same as the SDK (1000Hz) and switched off TICLESS_IDLE (whatever that is) but it makes no difference:



    Also, it doesn't seem to be related to the frequency of calling TaskP_loadGetTotalCpuLoad(). If I call it at 20ms, it makes no difference which is what we'd expect I suppose.

    So now I have the same RTOS configuration as the SDK apart from the following:

    INCLUDE_xTaskDelayUntil = 1


    I can't switch this off because otherwise our OS scheduler simply won't work.

    Do you have any other ideas please?

  • Hi Kier,

    On evaluating "TaskP_loadUpdateAll" and "TaskP_calcCpuLoad" function which is called by the "TaskP_loadGetTotalCpuLoad" function, I do see some filtering taking place

    The accumulated runtime seems to be continuously growing which might cause slow CPU load calculation (taking time to stabilize)

    The accTotalTime variable keeps growing over time, meaning new updates have a smaller relative impact. The difference (delta) is calculated based on the last recorded time. If updates happen infrequently, large delta values create a “lag” before the new load stabilizes. 

    I'm not really sure if I can spend a lot of bandwidth on this right now due to new release activities., plus I haven't been able to reproduce it in my previous tests.

    Can you try if these methods work for you?

    1. Reset accTotalTime Periodically so CPU load is calculated for shorter averages

    if (gTaskP_ctrl.accTotalTime > SOME_THRESHOLD) {
        gTaskP_ctrl.accTotalTime = 0;
    }

    The accRunTime and accTotalTime variables are accumulated over time, and the delta values are calculated based on the difference between the current and previous values of the run time counters. This means that the accRunTime and accTotalTime variables are effectively low-pass filters, smoothing out the variations in the run time counters over time. As a result, the cpuLoad calculation in the TaskP_loadGetTotalCpuLoad function is based on these filtered values, which can cause the CPU load to appear higher than it actually is, especially during transient periods of high activity.


    2.The high initial CPU load could be due to the fact that the idle time of the cores is not immediately available or is not accurately calculated at the start of the application, resulting in a higher calculated CPU load. As the system runs and the idle time is accurately calculated, the CPU load stabilizes to a lower value.

    Regards,
    Shaunak

  • Hi Shaunak,

    Many thanks for your investigation.

    FYI, I noticed that the problem gets progressively worse.

    I programmed a large step change in CPU load (approx 50%) that I could control externally. In the plot below, the initial CPU load is around 4%. I then switch on artificial high CPU load at ~4s and the load heads upwards, then off at ~30s then on again at ~60s. The filtering gets harsher as time goes by. I imagine that after a few minutes, CPU load calculation is practically stuck.

    I don't think this continuous accumulation is necessary. It just needs to accumulate between each CPU load call. At the end of each call, the counters can (and should in my view) be reset.

    Taking your lead, I noticed that the function TaskP_loadResetAll(), in which the relevant counters are reset, is never called in the SDK implementation. I figured that the after each load calculation window (TaskP_LOAD_UPDATE_WINDOW_MSEC), that calling TaskP_loadResetAll() would do the trick:

    And it certainly does. Now I get an almost instantaneous change in calculated CPU load (to ~50%) for my actual step change in CPU load:

    From here it looks like the call TaskP_loadResetAll() is missing in the SDK implementation.

  • Hi Kier,

    I don't think this continuous accumulation is necessary. It just needs to accumulate between each CPU load call. At the end of each call, the counters can (and should in my view) be reset.

    Agreed.

    FYI, I noticed that the problem gets progressively worse.

    I programmed a large step change in CPU load (approx 50%) that I could control externally. In the plot below, the initial CPU load is around 4%. I then switch on artificial high CPU load at ~4s and the load heads upwards, then off at ~30s then on again at ~60s. The filtering gets harsher as time goes by. I imagine that after a few minutes, CPU load calculation is practically stuck.

    Thanks for testing and sharing this.

    From here it looks like the call TaskP_loadResetAll() is missing in the SDK implementation.

    Let me talk to some other FreeRTOS experts who might know why this is the case. 

    Does calling TaskP_loadResetAll() at the end of load calculation window fix the filtering issue you were observing originally?

    Regards,
    Shaunak

  • Hi Shaunak,

    Perhaps it is the applicaton's responsbility to call TaskP_loadResetAll()?

    I have now reverted FreeRTOS and added this in my application:

    Of course I get similar (good) results:

    The API says the following:

    It's not clear from this description that, without calling this, the CPU load response will gradually grind to a halt. Perhaps that needs clarification?

    Anyway, the problem is solved! Thanks for your help.

  • Hi Kier,

    Thanks for sharing the results and being patient throughout this debug. Based on what some of our other FreeRTOS experts said, this needs to be called in Application itself whenever it measures CPU load and that is the reason it is not a part of the TaskP kernel code.

    In all MCU+ SDK examples, it is called in the applications itself. You can search how the CPU load is being calculated and how TaskP_loadResetAll() is called in SDK examples.

    Regards,
    Shaunak

  • Hi Shaunak,

    I've no doubt it is in the examples and that was my oversight. But I still think there is scope for improvement of the documentation. I think we have established that it is essential to call TaskP_loadResetAll(). If that is the case, your API documentation should mention that. It would have saved much of our time.

    Kier.

  • But I still think there is scope for improvement of the documentation

    Agreed, I will update the documentation to mention this.

    Regards,
    Shaunak