This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

[FAQ] Where can I find details on SMP with SYS/BIOS?

Where can I find information about SMP (Symmetric Multi-Processing) that is in SYS/BIOS?

  • Supported Devices

    TargetSupported TI SOC devices
    Dual-core Cortex-M3/M4 subsystem OMAP44xx, OMAP54xx, DM81xx, DRA74x (Jacinto 6), TDA2x (ADAS), TDA3x, AM57xx (Sitara)
    Multi-core Cortex-A15 subsystem OMAP54xx, DRA74x (Jacinto 6), TDA2x (ADAS), AM572x/AM574x (Sitara), Keystone2

    Task Scheduling in SMP mode

    Scheduling Rule

    • The Highest Priority ready task is always running
    • The N Highest Priority ready tasks will be running at any given time provided task affinities are not used (where N is the number of cores)

    The rule that the N highest priority ready tasks will be running at any given time may get violated if the application has a mix of don't care and fixed affinity tasks. However, the rule that the highest priority ready task is always running is never violated and guaranteed to be always true.

    Improving cache performance in SMP mode

    An application can leverage the ability to set the affinity of a task to come up with a scheme that allows certain tasks to always run on a given core and hence benefit from a warm cache. The ability to set the priority of a task can also be leveraged to minimize interference from other tasks. Here's an example scheme that demonstrates this idea:

    • Assume we have a 4 core system and a total of 16 different priorities.
    • The 16 priorities can be split by the application into 2 bands. A lower priority band of 8 priorities (i.e. priority 1-8) reserved for don't care affinity tasks and a higher priority band of 7 priorities (i.e. priority 9-15) reserved for fixed affinity tasks.
    • Say we have 6 tasks in our application of which 2 tasks are timing critical and we want to ensure they always have a warm L1 cache. We can assign the time critical tasks a priority from the higher priority band (i.e. priority 9-15) and an affinity of 3 (bound to Core 3) respectively. The remaining 4 tasks can be assigned a priority from the lower priority band and a don't care affinity. This scheme will allow the 2 timing critical tasks to always run on Core 3 thereby ensuring good cache performance. Since these 2 tasks have a priority belonging to the higher priority band, the other less critical tasks cannot interfere (It is however possible that both these tasks block at which time one of the other tasks may run on this core and thrash part of the L1 cache). The remaining 4 tasks are don't care affinity and can therefore run on any core that is idle.

    Getting started

    Install the latest SYS/BIOS product (SYS/BIOS Download pages) and the corresponding XDC Tools (XDCTools Download pages).

    Configuring SMP/BIOS

    SMP/BIOS needs to be configured before applications can be compiled and linked with it. Configuration is primarily used for specifying which modules to include and setting default values for the many tunable parameters in SMP/BIOS. The input to the configuration tool, configuro, is a text file with a .cfg extension written in JavaScript (ECMAScript). This file can be authored using a text editor or graphically generated using a plugin in CCS/Eclipse. The output of configuration is a set of options for the C compiler to be used when compiling user applications, and a set of linker options for use when linking applications.

    In order to build a SYS/BIOS application to run in SMP mode:

    • Enable SMP mode by adding the following to your config script:
    var BIOS = xdc.useModule('ti.sysbios.BIOS');
    BIOS.smpEnabled = true;
    • Use the new SMP-friendly modules for logging and CIO:
    ti.sysbios.smp.LoggerBuf  (Replaces xdc.runtime.LoggerBuf)
    ti.sysbios.smp.SysMin     (Replaces xdc.runtime.SysMin)
    ti.sysbios.smp.SysStd     (Replaces xdc.runtime.SysStd)

    Cortex-A15 specific configuration settings:

    • A SMP application built for Keystone2 devices uses sKernel commands to wake-up all secondary cores by default. If running the application in CCS, then a gel file can be used to wake-up secondary cores instead of using sKernel commands. The "Core.useSkernelCmd" configuration parameter can be set to false (true by default) to stop using sKernel commands. If this parameter is set to false, the secondary cores wait in a WFE loop for core 0's signal to continue with startup.
    var Core = xdc.useModule('');
    Core.useSkernelCmd = false; /* Set to false if running in CCS and using gel files to wake-up secondary cores */
    • SYS/BIOS 6.41.03 adds N core SMP support (i.e. supports SMP operation on 2 or more cores). Older versions of SYS/BIOS (6.41.00, 6.41.01 and 6.41.02) only support dual core SMP operation.
      • If building a SMP application for Keystone2 platform (evmTCI6636K2H) and using SYS/BIOS 6.41.00, 6.41.01 or 6.41.02, then only dual core support is available.Therefore, the Core.numCores config param needs to be set to 2 (default is 4 for some Keystone2 devices):
    var Core = xdc.useModule('');
    Core.numCores = 2;
    • If building a SMP application for Keystone2 platform (evmTCI6636K2H), the Timestamp module's SupportProxy needs to be changed to 'timer64.TimestampProvider'. This change is required due to a bug in the timestamp delegate module name for Keystone2 Cortex-A15 in SYS/BIOS 6.41.00. This has been fixed in all newer releases.
    var Timestamp = xdc.useModule('xdc.runtime.Timestamp');
    Timestamp.SupportProxy = xdc.useModule('ti.sysbios.timers.timer64.TimestampProvider');

    Example CCS Projects

    • Example CCS Projects (created with CCS v6.0.1) for Cortex-A15 based devices
      • Example Project 1 - This example demonstrates how to use Task_setAffinity() and how to configure SMP/BIOS.
      • Example Project 2 - This example demonstrates how to configure Hwi affinity to distribute interrupts between the different cores and how to configure SMP/BIOS.

    Important Things To Consider

    • With SMP/BIOS, tasks with different priorities may be running simultaneously.
    • With SMP/BIOS, a Swi can be running on one core AT THE SAME TIME a task is running on the other.
    • By default, tasks can run on either core.
      • To reduce the number of variables during the initial stages of porting your application, it may be very helpful to constrain all the tasks to run on a single core until basic functionality has been established. To force task affinity for ALL tasks to core 0, add the following:
         Task.defaultAffinity = 0;
      • To constrain a single task to run on a particular core, set the new "affinity" Task instance config parameter to the core id of the core you want it to run on:
         Task_Params tskParams;
         tskParams.affinity = 0; /* Force to run on Core 0 */
         myTask = Task_create(myTaskFunc, &tskParams, NULL);
    • main() runs on Core 0.
    • The Hwi_disable/restore() APIs that are often used to protect critical code sections work across BOTH cores. Invoking Hwi_disable() on one core effectively disables interrupts on both cores.
    • All interrupts run on Core 0 by default.
    • Swis run on Core 0 regardless of which core they are posted on.

    CCS Debugging


    What is the Hwi stack size on each core and how can it be modified?

    The Hwi stack size is derived from "Program.stack" on all cores. On Cortex-M3/M4 specifically, Core 1's Hwi stack size can also be controlled via the Core module's core1HwiStackSize field. Such a special field is not available on other devices supporting SMP mode and it is recommended to modify the Hwi stack size using "Program.stack" on all devices.

    How do I measure CPU Load when running in SMP mode?

    An application that needs to measure CPU Load in SMP mode can use the SYS/BIOS Load module (ti.sysbios.utils.Load) to do so. The Load module can be used the same way it is used in non-SMP mode with the exception that the Global CPU load reported by the Load module is not correct when running in SMP mode. In order to measure the per core CPU Load, the below logic can be used:

    • Measure the idle task load on each core and subtract it from 100. This will give the CPU Load on a given core.
       #include <ti/sysbios/knl/Task.h>
       #include <ti/sysbios/utils/Load.h>
       #include <xdc/runtime/System.h>
       func() {
           UInt load;
           Load_stat stat;
           Task_Handle idlTskHandle;
           idleTskHandle = Task_getIdleTaskHandle(coreId);
           Load_getTaskLoad(idlTskHandle, &stat);
           load = Load_calculateLoad(&stat);
           System_printf("Load = %d\n", (100-load));

    In addition to the problem with the Global CPU load in SMP mode, there are 2 other issues (tracked through bug Id SDOCM00115771) that will be fixed in future releases of SYS/BIOS. These issues can cause the Load module to report an incorrect idle task load. These can be worked around as follows:

    • Call Load_reset() from main() before calling BIOS_start().
    • Register an idle function on Core 1, 2, ... that calls Load_updateCurrentThreadTimer().
    • .cfg:
       var Idle = xdc.useModule('ti.sysbios.knl.Idle');
       Idle.addCoreFunc ('&idle_func', 1);
    • .c:
      Void idle_func()

    Why does C/Assembly step over and step return not work when grouping cores as Sync Groups?

    This is due to a known issue (tracked through bug Id SDSCM00051817). In order to work around this issue, “Halt the target before any debugger accesses” option under "Auto Run and Launch Options" should be selected for each core in the Sync Group. This issue will be fixed in CCS v6.2 release.

    My SMP application on a Keystone2 board is hung at ti_sysbios_family_arm_a15_smp_Core_resetKeystone2__I() routine. Why is this?

    One of 2 reasons maybe causing the A15 cores to hang at boot time. The first possibility (more likely when running in CCS) is that there is no sKernel but the kernel needs to use sKernel commands to wake-up the secondary cores. Please see "Configuring SMP/BIOS" section above for more info on how to disable sKernel commands and resolve this. A second possibility is that the SMP bit in ACTLR is not set. This can be set through a gel script in CCS. If using u-boot & sKernel, this will be done by the bootloader.

    My SMP application on a Keystone2 board is hung at Core_startCoreXKeystone2() routine on Core0 and Core_startup() routine on all other cores. Why is that?

    This may happen if one of the cores is running in secure mode. In order to resolve this problem, switch all cores to non-secure mode before loading and running the application. In CCS environment, a gel function can be used to switch the cores to secure mode.

    Gel function to change Cortex-A15 security mode to non-secure:

    hotmenu enterNonSecureMode() { int status;
    Disable_MMU(); Disable_Caches();
    GEL_TextOut("Enabling non-secure access to cp10 and cp11\n"); status = REG_CTXA15_CP15_C1_NSACR; status |= 0x00000C00; status &= 0x7FFFFFFF; REG_CTXA15_CP15_C1_NSACR = status; GEL_TextOut("Enabled non-secure access to cp10 and cp11\n");
    GEL_TextOut("Enabling SMP bit in ACTLR\n"); status = REG_CTXA15_CP15_C1_ACTLR; status |= 0x40; REG_CTXA15_CP15_C1_ACTLR = status; GEL_TextOut("Enabled SMP bit in ACTLR\n");
    GEL_TextOut("Entering NonSecure Mode\n"); status = REG_CTXA15_CP15_C1_SCR; status |= 0x1; REG_CTXA15_CP15_C1_SCR = status; GEL_TextOut("Entered NonSecure Mode\n");
    // Ensure MMU and Caches are disabled after changing security mode Disable_MMU(); Disable_Caches(); }
    define MMU_ON 0x1
    define MMU_OFF ~MMU_ON
    hotmenu Disable_MMU() {
       int status;
       GEL_TextOut("Disabling MMU\n");
       status = REG_CTXA15_CP15_C1_SCTLR;
       status &= MMU_OFF;
       REG_CTXA15_CP15_C1_SCTLR = status;
    define ICACHE_ON 0x1000
    define DCACHE_ON 0x4
    hotmenu Disable_Caches() {
       int status;
       GEL_TextOut("Disabling Caches\n");
       status = REG_CTXA15_CP15_C1_SCTLR;
       status &= DCACHE_OFF;
       status &= ICACHE_OFF;
       REG_CTXA15_CP15_C1_SCTLR = status;
       // invalidate entire instruction cache
       GEL_TextOut("Invalidate Instruction Caches\n");
       REG_CTXA15_CP15_C7_ICIALLU = 1;