This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM2634: Spinlock Lock Register is presumably read by two cores simultaneously

Part Number: AM2634

Hello, 

I am working on a project with multiple cores configured and I am using the Spinlock module of the device to achieve managed resource access.

I use a single spinlock register like a mutex, to control the execution of a critical section of code which all four cores need execute but not parallelly. The code in all four cores is identical and is this:

	#if(CORE_ID == CSL_CORE_ID_R5FSS0_0)
	/* Unlock the spinlock register to reset it */
	Spinlock_unlock(CSL_SPINLOCK0_BASE, INIT_ZERO_MUTEX);
	#endif
	for (;;)
	{
		int32_t status = Spinlock_lock(CSL_SPINLOCK0_BASE, INIT_ZERO_MUTEX);
		if (status == SPINLOCK_LOCK_STATUS_FREE)
		{
			break;
		}
	}
	/* Critical section start */
	/* This function shall be used to init driver pre-init level elements*/
	sysm_init_zero();
	/* Critical section end */
	Spinlock_unlock(CSL_SPINLOCK0_BASE, INIT_ZERO_MUTEX);

The problem is that when I put a breakpoint at the break statement, in each of the four cores I see multiple cores hitting that breakpoint.

My test sequence is:

  1. Reset all cores
  2. Reload the program to all cores
  3. Resume on all cores with the breakpoint enabled in each of them

 In most occasions, more than one breakpoints are hit, meaning that more than one cores have seemingly succeeded in reading a SPINLOCK_LOCK_STATUS_FREE status in the same Lock Register (INIT_ZERO_MUTEX which is defined as 0).

The only configuration that seems to perform more predictably is the one below, where there is an additional Spinlock_lock call in the beginning of this sequence to ensure that the spinlock is locked before any core get to the busy waiting stage. I have tested this multiple times and it always, so far, follows the predicted pattern of hitting the break statement breakpoint in the various cores in sequence and only after the previous core called Spinlock_unlock.

	Spinlock_lock(CSL_SPINLOCK0_BASE, INIT_ZERO_MUTEX);
	#if(CORE_ID == CSL_CORE_ID_R5FSS0_0)
	/* Unlock the spinlock register to reset it */
	Spinlock_unlock(CSL_SPINLOCK0_BASE, INIT_ZERO_MUTEX);
	#endif
	for (;;)
	{
		int32_t status = Spinlock_lock(CSL_SPINLOCK0_BASE, INIT_ZERO_MUTEX);
		if (status == SPINLOCK_LOCK_STATUS_FREE)
		{
			break;
		}
	}
	/* Critical section start */
	/* This function shall be used to init driver pre-init level elements*/
	sysm_init_zero();
	/* Critical section end */
	Spinlock_unlock(CSL_SPINLOCK0_BASE, INIT_ZERO_MUTEX);

The design for the above mutual exclusion mechanism has been based on these statements in the TRM:

I interpreted this statement as:

Only a single resource (one of the cores in this particular case) can read the same lock register at any given time and then in any sequential read the register will return TAKEN (1). 

However, what I observe indicates either of these scenarios:

  1. Two or more cores read the same lock register simultaneously and return a status of 0 (FREE), or
  2. A core is reading the lock register but the lock register is not set to one before another core gets access to it and therefore it is possible that two cores can read a status of FREE without writing 0 to the register.

Questions:

  1. Is my interpretation of how the Spinlock module works correct?
  2. Is there anything else that I might be missing about this module and/or my design?
  3. Is my improved configuration more correct for some reason and can it be trusted that it will always behave according to the design?

Thanks,

Manos

  • Hello Manos,

    Can you please refer ipc_spinlock_sharedmem example. I have tried to replicate your issue by adding breakpoint at line 81 in this example but only one core is able to get spinlock and other core is waiting.

    Best regards,
    Gunjan

  • Hi Gunjan,

    Thanks for your reply. I had a look at your recommended example but it diverges significantly from what I am trying to do introducing RTOS semaphores which can potentially alter the behaviour significantly. 

    If you could please test this example instead and make sure to modify the code, so that both cores are using the same lock register, i.e. both cores lock and unlock lockNum1. This is almost identical to my setup and desired configuration.

    Thanks,

    Manos

  • Hello Manos,

    Sure, I'll try to replicate it using your suggested spinlock example. Please expect a reply by 30th may.

    Thanks,
    Gunjan

  • Just an update from my side:

    Regarding the improved configuration, with the additional Spinlock_lock call before the main body, it behaves exactly as expected and consistently, i.e. each of the cores takes ownership of the "mutex" in sequence and executes the critical section.

    However, it only does so while debugging and hitting breakpoints at the break statement exiting the infinite busy waiting loop. If I disable the breakpoints and just let the target run through this code it ends up in a state where two of the cores have executed the critical section, unlocked the "mutex" and moved on and the third is seeing the "mutex" as still locked and is stuck waiting for it to be unlocked.

  • Hello Manos,

    Can you share your updated files?

    Thanks,
    Gunjan

  • Hello Gunjan,

    Are you referring to the updates that I made to the simple spinlock example?

    If so, here they are:

    /*
     *  Copyright (C) 2022-23 Texas Instruments Incorporated
     *
     *  Redistribution and use in source and binary forms, with or without
     *  modification, are permitted provided that the following conditions
     *  are met:
     *
     *    Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     *    Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the
     *    distribution.
     *
     *    Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     */
    
    #include <stdlib.h>
    #include "ti_drivers_config.h"
    #include "ti_board_config.h"
    #include "ti_board_open_close.h"
    #include "ti_drivers_open_close.h"
    #include <drivers/spinlock.h>
    #include <kernel/dpl/ClockP.h>
    #include <drivers/hw_include/cslr.h>
    
    #define lockNum1 0
    #define lockNum2 1
    #define DELAY_TIME 1
    
    int main(void)
    {
        System_init();
        Board_init();
    
        Drivers_open();
        Board_driversOpen();
    
        int loopcount = 5;
    
        while(loopcount--)
        {
            int32_t  status;
    
            /* Spin till lock is acquired */
            while(1U)
            {
                status = Spinlock_lock(CSL_SPINLOCK0_BASE, lockNum1);
                if(status == SPINLOCK_LOCK_STATUS_FREE)
                {
                    break;  /* Free and taken */
                }
            }
    
            /*
             * enter critical section
             */
    
            DebugP_log("hello core 1 \r\n");
            ClockP_sleep(DELAY_TIME);
            Spinlock_unlock(CSL_SPINLOCK0_BASE, lockNum1);
        }
    
        Board_driversClose();
        Drivers_close();
    
        Board_deinit();
        System_deinit();
        return 0;
    }
    
    /*
     *  Copyright (C) 2022-23 Texas Instruments Incorporated
     *
     *  Redistribution and use in source and binary forms, with or without
     *  modification, are permitted provided that the following conditions
     *  are met:
     *
     *    Redistributions of source code must retain the above copyright
     *    notice, this list of conditions and the following disclaimer.
     *
     *    Redistributions in binary form must reproduce the above copyright
     *    notice, this list of conditions and the following disclaimer in the
     *    documentation and/or other materials provided with the
     *    distribution.
     *
     *    Neither the name of Texas Instruments Incorporated nor the names of
     *    its contributors may be used to endorse or promote products derived
     *    from this software without specific prior written permission.
     *
     *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
     *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
     *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
     *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
     *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
     *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
     *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
     *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
     *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
     *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
     *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
     */
    #include <stdlib.h>
    #include "ti_drivers_config.h"
    #include "ti_board_config.h"
    #include "ti_board_open_close.h"
    #include "ti_drivers_open_close.h"
    #include <drivers/spinlock.h>
    #include <kernel/dpl/ClockP.h>
    #include <drivers/hw_include/cslr.h>
    
    #define lockNum1 0
    #define lockNum2 1
    #define DELAY_TIME 1
    
    int main(void)
    {
        System_init();
        Board_init();
    
        Drivers_open();
        Board_driversOpen();
    
        Spinlock_unlock(CSL_SPINLOCK0_BASE, lockNum1);
    
        int loopcount = 5;
    
        while(loopcount--)
        {
    
            int32_t  status;
            /* Spin till lock is acquired */
            while(1U)
            {
                status = Spinlock_lock(CSL_SPINLOCK0_BASE, lockNum1);
                if(status == SPINLOCK_LOCK_STATUS_FREE)
                {
                    break;  /* Free and taken */
                }
            }
    
            /*
             * enter critical section
             */
    
            DebugP_log("hello core 0 \r\n");
    
            ClockP_sleep(DELAY_TIME);
    
            Spinlock_unlock(CSL_SPINLOCK0_BASE, lockNum1);
    
            ClockP_sleep(DELAY_TIME);
    
        }
    
        DebugP_log("All Test Cases passed!");
    
        Board_driversClose();
        Drivers_close();
    
        Board_deinit();
        System_deinit();
    
        return 0;
    }
    

    Regards,

    Manos

  • Hello Manos,

    Thanks for sharing the code. I have removed extra line 53: Spinlock_unlock(CSL_SPINLOCK0_BASE, lockNum1); from your code, because I don't think we need to unlock it even before using it. 
    These are my outputs after removing extra unlock at start, and it matches with expected flow. 

    [Cortex_R5_0] hello core 1 
    [Cortex_R5_1] hello core 0 
    [Cortex_R5_0] hello core 1 
    [Cortex_R5_1] hello core 0 
    [Cortex_R5_0] hello core 1 
    [Cortex_R5_1] hello core 0 
    [Cortex_R5_0] hello core 1 
    [Cortex_R5_1] hello core 0 
    [Cortex_R5_0] hello core 1 
    [Cortex_R5_1] hello core 0 
    All Test Cases passed!


    Best Regards,
    Gunjan

  • Hi Gunjan,

    Yes you are correct. This project works correctly with my changes. 

    I have removed extra line 53: Spinlock_unlock(CSL_SPINLOCK0_BASE, lockNum1); from your code, because I don't think we need to unlock it even before using it. 

    The extra unlock is there to reset the spinlock register in case the previous debugging session ended abruptly and therefore the spinlock was left in an uncontrolled state. Since we are not doing a HW reset every time I have observed that this additional unlock helps with ensuring the whole sequence will at least start.

    I additionally tested with three cores enabled and it still works fine. 

    I think I figured out what the difference is between this example project and the real project I am working on. System_init() and Board_init() are called before calling any Spinlock APIs. My assumption is that the Spinlock peripheral depends on the clock initialized by Dpl_init() which is the whole reason I want to use the spinlock/mutex anyway. Please see here for the related issue with Dpl_init().

    I tried to force the error in the example spinlock project by moving the 

        System_init();
        Board_init();
    
        Drivers_open();
        Board_driversOpen();
    section in the critical section after acquiring the mutex. But it still seems to work fine. However, when trying to debug it, it hits breakpoints at the break statement in two cores at the same time.

    This seems so strange. Any insight? 

    Edit: The only other difference I can think of is that the example code is for Nortos whereas my project is built for FreeRtos. Is the SDK implementation different among the two versions?

    Regards,

    Manos

  • Hello Manos,

    You are able to acquire spinlock in 2 cores when you added breakpoints, this might be because of that extra unlock at start of code.
    If you are using extra unlock then it might also happen that previous debug doesn't ended abruptly and spinlock is free before starting program in all cores.
    1. Now you loaded and started program.
    2. Core 2 acquired spinlock and hits breakpoint at line 61.
    3. Core 0 hits line 50 and unlocks spinlock (acquired by core2).
    4. Core 0 hits line 61.

    I don't think adding extra unlock at start is good choice due to this scenario.

    Thanks and Best Regards,
    Gunjan

  • Hi Gunjan,

    Yes, I came to the same conclusion yesterday. The additional unlock is the root cause of this issue.

    However, the scenario that the spinlock register does not get reset for a number of reasons is still possible, even if unlikely. Is there any other "out of the box" solution that also considers such cases?

    I guess the right tool for this would be a semaphore instead of a mutex because a mutex is much stricter in the required sequence of events, i.e. at first the mutex lock is free, then someone acquires it, then releases it so that someone else can acquire it. I think that in my use case, the assumption that we will always start with the spinlock free is probably 100% accurate as at that point in the application there are no interrupts enabled or anything that could disrupt this sequence.

    Thanks,

    Manos

  • Hello Manos,

    You can refer section "8.2.3.1 Spinlock Software Reset" in AM263 TRM for reset information of spinlocks.

    Best Regards,
    Gunjan