Thread scheduling problems on Omap L138

Yoel,

I'm on the IPC team, so I'm interested in knowing why using Notify breaks your program.

I took a look at your SPI post. As I understand it, you are using OMAP-L138. Your program is running on the ARM. You have two devices on SPI bus. Each device raises interrupt when data is ready. In your ISR, you read the data from the bus. Have I got this correct?

The problem comes when the second device raises an interrupt while the previous ISR is still running. Correct?

I would suggest using a single Swi to actually read the data from the SPI bus. Each Hwi would schedule the Swi to run by calling Swi_inc. This way, the second interrupt would simply post the currently running Swi but not preempt it.

In your Swi, you would call Swi_getTrigger. This returns how many times the Swi was incremented (i.e. posted). Your Swi would then loop that many times.

In other words, if your Swi is currently running and reading data on SPI Bus 0, when the second interrupt occurs, your Hwi would temporarily preempt the Swi but only to increment the Swi count and post it. When the Hwi unwinds, your currently running Swi would resume. When the Swi finishes, it would be invoked again (immediately) to read the data from SPI Bus 1.

It is helpful to understand that when the Swi is invoked, SYS/BIOS will latch the current Swi count and make this available to Swi_getTrigger. It atomically, resets the Swi count. So, if both Hwi's ran before the Swi had a chance to start, then the Swi will be invoked once with a count of two. However, if the Swi starts before the second Hwi, then this will result in the Swi running twice, each time with a count of one.

You will probably need to pass some information from the Hwi to the Swi. For example, which SPI bus is ready. I recommend writing your own Queue object for this. Don't use the SYS/BIOS Queue module. Your private queue object can be an array of integers (maybe 4 elements), with a head and tail index. The Hwi writes an integer at the tail value and then increments the tail. The Swi reads an integer at the head value and then increments the head. This way, you don't need any locking or protection. The tail is modified only by the Hwi, and the head is modified only by the Swi. The Swi is guaranteed as many values in the queue as the Swi trigger count. If you make the size of your queue a power of two, then you can make use of every element in the array.

Here is some sample code for the queue implementation.

/* error values */
#define My_E_FAILURE  0xFF000000
#define My_E_OVERFLOW 0xFF000001
#define My_E_NOEVENT  0xFF000002
#define My_S_SUCCESS  0

/* queue size, must be a power of 2 */
#define My_QSZ (1 << 2)

/* queue structure */
typedef struct 
{
    UInt   queue[My_QSZ];   /* queue */
    Bits32 head;            /* dequeue from head pointer */
    Bits32 tail;            /* enqueue at tail pointer */
    UInt   error;           /* error flag */
} My_Queue;   

UInt My_dequeue(My_Queue *Q)
{
    UInt event;

    /* check for queue error */
    if (Q->error >= My_E_FAILURE) {
        event = Q->error;
    }
    else {
        if (Q->head == Q->tail) {
            /* queue should not be empty */
            Q->error = My_E_NOEVENT;
            event = Q->error;
        }
        else {
            /* remove next event from queue */
            event = Q->queue[Q->head];
            Q->head++;
        }
    }

    return (event);
}

Void My_enqueue(My_Queue *Q, UInt event)
{
    /* check if queue is full */
    if ((Q->head + My_QSZ) == Q->tail) {
        Q->error = My_E_OVERFLOW;
    }
    else {
        /* append event to queue */
        Q->queue[Q->tail % My_QSZ] = event;
        Q->tail++;
    }
}

/* initialize the queue */
My_Queue myQ;
myQ.head = 0;
myQ.tail = 0;
myQ.error = My_S_SUCCESS;

Now, would you give me some details on your usage of IPC.

Thanks
~Ramsey

Hello Ramsey,

Thank you for the prompt and elaborated reply.

Couple of things:

1. I use only SPI BUS 0.
2. I use 2 SWI's to handle the transmission (Every Hwi calls for a different SWI) - You can see that code in the end of the previous post.

One of the devices stops streaming at some point probably because I didn't ask for data and it thinks the comm. is dead, the problem I'm having is that I don't know why I didn't read the data (I can see that in the logic analyzer).

What you said about using the same SWI twice is interesting, I will try that, even tough I don't know why using 2 SWI's with the same priority isn't the same (does the HWI preempts the current SWI? or is it going into the schedule?).

I use notify as a sync between the ARM and the DSP, like the notify example, each processor has a callback function that posts a semaphore and then the other knows it can go on. If I don't call notify on the DSP side, streaming is good but If I do, I guess that somewhere the callback function interrupt with the normal scheduling and then I miss a SPI transmission.

This is a shot in the dark because I don't know the SYSBios scheduling very well buy I can see that whenever I do something that interrupts the task or the SWI, the streaming stops (Semaphore_pend for more than 0 ms, Task sleep, notify etc.).

p.s - on the DSP side, I can do whatever I want as long as it doesn't involve with the ARM core (s.a notify).

Does this make any sense to you?

Thanks,
Yoel

Yoel,

Using two Swi's at the same priority is equivalent to using one Swi and calling Swi_inc() twice. I don't think you need to try using one Swi.

In addition, I forgot a detail regarding the queue protection. If you have two Hwi's where each one uses the same queue, then you need to protect the queue from concurrent access. The best way to do this is to use the Hwi mask to prevent the Hwi's from preempting each other. In other words, when creating each Hwi, set the mask to block the other Hwi.

So, back to the real issue. I'm not exactly clear on the data flow. Does the sensor raise a data ready interrupt on its own, or does the ARM request data? I'm not able to understand this part.

What is the data rate you expect from each sensor?

You indicate that calling Task_sleep(0) is okay but calling Task_sleep(1) breaks the application. This is very puzzling. In the first case, the task would simply spin and the idle task would never be invoked. In the second case, the task would be sleeping almost always and the idle task would be spinning almost continuously. I don't see how either of these could effect the interrupts.

Re: Hwi vs Swi preemption. A Hwi will always preempt a Swi. And a Swi will always preempt a Task.

Re: IPC. So, when the DSP calls Notify_send(), and the notify callback is invoked on the ARM, this causes the ARM to miss a sensor sample. Have I got this correct?

What does the notify callback do? Can you try it with an empty notify callback. Does this still break the application?

~Ramsey

Hi Ramsey,

Using MaskingOption_BITMASK is not supported on Omap L138 (according to the compiler).

The flow is as follows:

1. Start sensor streaming
2. Wait for data ready (HWI - Interrupt from the sensor)
3. In HWI ISR call SWI to read data from sensor
4. In SWI ISR read 2 words from sensor
5. Wait for next data (HWI - Interrupt from the sensor)

Each SPI is supposed to service 2 words every 100us (4 bytes).

The notify call back does a semaphore_post for the main task but even when I commented this line, the same thing happened.

Yoel

I never tried Task_sleep(0) but Semaphore_pend(0), In my understanding, the task doesn't wait so I can use a flag instead which doesn't really help, this causes the task to run endlessly and starving all the other tasks, I don't know why pending on a semaphore interrupt the HWI servicing.

Yoel,

I occurred to me that your application seems to miss a data sample whenever another interrupt is enabled. The notify event coming from the DSP would raise an interrupt. Adding a task sleep will cause your task to wake on the timer interrupt for the SYS/BIOS clock. Although, the SYS/BIOS timer interrupt is probably always running, it only seems to impact your application when it causes a scheduling change. Maybe because the timer interrupt runs for a very short duration when there is no work to be done.

So, I don't think IPC is the root cause issue here. Maybe a SYS/BIOS expert should take over. I think the issue has to do with the interrupt handling. It seems to me that if an interrupt occurs while your SWI is reading data from the SPI bus, it causes data loss. Maybe you can share some details on how you are reading the SPI bus. I would expect the SPI controller to have a hardware accumulator to shift in the bits as they are available. Once a word has been collected, it would be written into a hardware FIFO and raise an interrupt to the processor. Your SWI would simply read the data sample from the FIFO.

If this is not the case, and your SWI is reading bits on the bus by busy waiting on a clock signal, then any interruption might account for data loss. If so, then perhaps you need to block all other interrupts while the SWI is reading the data. As a test, you could try to disable interrupts during your SWI.

Re: interrupt masking. Try using MaskingOption_LOWER instead of MaskingOption_BITMASK. You could make your SPI interrupt highest priority. This would prevent your SPI Hwi from begin interrupted, but it will not prevent your SWI from being preempted. You need to disable interrupts to make your SWI non-preempt-able.

Your SPI data rate of 4 bytes / 100us seem quite high. Is this really necessary? Can you reduce the data rate? Maybe even just for testing purposes.

Re: pending on a semaphore. You indicate that if the task blocks (by pending on a semaphore) this will cause data loss. My only guess is that when the task is scheduled to run, there might be some time spent context switching with interrupts disabled. The added latency might cause data loss depending on how you are reading the SPI data.

~Ramsey

Hi Ramsey,

I tried Hwi_disable when entering the SWI and Hwi_enable on the way out but the problem persist (again, it worsen when I add the pending on notify semaphore). So I guess that the Hwi isn't the problem (or is it...).

The data rate is Set by the sensor and it is constant. Here is how I handle the SPI:

// First HWI 
void GPIOBank2Isr(void)
{
	/* Clears the system interrupt status of GPIO in AINTC.*/
	IntSystemStatusClear(SYS_INT_GPIOB2);

	if(GPIOPinIntStatus(SOC_GPIO_0_REGS, SENS0_VD_PIN) == GPIO_INT_PEND)
	{
		/* Clears the Interrupt Status of the correct pin in GPIO.*/
		GPIOPinIntClear(SOC_GPIO_0_REGS, SENS0_VD_PIN);
		if(isAvail0)
		{
			Swi_post(swi0);
		}
	}
}
// Second HWI
void GPIOBank0Isr(void)
{
	/* Clears the system interrupt status of GPIO in AINTC.*/
	IntSystemStatusClear(SYS_INT_GPIOB0);

	if(GPIOPinIntStatus(SOC_GPIO_0_REGS, SENS1_VD_PIN) == GPIO_INT_PEND)
	{
		/* Clears the Interrupt Status of the correct pin in GPIO.*/
		GPIOPinIntClear(SOC_GPIO_0_REGS, SENS1_VD_PIN);
		if(isAvail1)
		{
			Swi_post(swi1);
		}
	}
}
// First SWI (swi0) ISR
Void Read0(Void)
{
	Hwi_disable();
	short *buff = (short*)CalRxBuffA;
	int current = idxA % totalLength;
	readA = length;

	SPIDat1Config(SENS0_SPI_REGS, (SPI_CSHOLD |SPI_DATA_FORMAT0), (1<<SENS0_CS));
	SPIReceiveData(SENS0_SPI_REGS, length, &buff[current], &readA, &idxA);
	SPIDat1Config(SENS0_SPI_REGS, (SPI_DATA_FORMAT0), (1<<SENS0_CS));

	Hwi_enable();
}

// Second SWI (swi1) ISR
Void Read1(Void)
{
	Hwi_disable();
	short *buff = (short*)CalRxBuffB;
	int current = idxB % totalLength;
	readB = length;

	SPIDat1Config(SENS1_SPI_REGS, (SPI_CSHOLD | SPI_DATA_FORMAT0), (1<<SENS1_CS));
	SPIReceiveData(SENS1_SPI_REGS, length, &buff[current], &readB, &idxB);
	SPIDat1Config(SENS1_SPI_REGS, (SPI_DATA_FORMAT0), (1<<SENS1_CS));

	Hwi_enable();
}

/*
 ** Get Data from SPI slave
 **
*/
void SPIReceiveData(unsigned int baseAdd, unsigned short length, short *buffer, UInt8 *read, UInt64 *idx)
{
	int i;
	for(i=0;i<length;i++)
	{
		while(!(HWREG(baseAdd + SPI_SPIFLG) & SPI_SPIFLG_TXINTFLG));
		SPITransmitData1(baseAdd, 0);
		while(!(HWREG(baseAdd + SPI_SPIFLG) & SPI_SPIFLG_RXINTFLG));
		*buffer = (short)SPIDataReceive(baseAdd);
	    buffer++;
	    (*read)--;
	    (*idx)++;
	}
}

How can we notify a SYS\BIOS expert on this matter?

Thanks for the help!

Yoel

Yoel,

Thanks for posting the code. I will talk with the SYS/BIOS team.

Re: Hwi_disable. I should have suggested using Hwi_disable/Hwi_restore. That is the typical way to enable/restore interrupts.

Where are you getting the SPI library from? Do you have source code for that? It seems clear that SPIReceiveData() is busy waiting. The way I see it, it waits for any transmit operation to complete. Then it sends a request to the sensor to receive data. It then waits for the data to be sent (from sensor to MCU). When the data is available, it reads it and stores the data in a buffer. Have I got this correct?

From our discussion, it seems that if an interrupt occurs during this process, the current data sample will be lost. Perhaps this point needs to be verified. I suggest you enable some Hwi logging events and add your own log events to SPIReceiveData. Set a breakpoint shortly after a data sample has been lost. Then inspect the logs and see if indeed an interrupt occurred while in this function. Look through the logs to see if any interrupts occur while reading good samples. If so, then maybe we are on the wrong path.

~Ramsey

Hi Yeol,

just curious, where are CalRxBuffA, CalRxBuffB, and length in your Swis defined? Are those static globals and do they ever change?

Ramsey,

Thanks for the disable\restore tip.

About the SPI procedure, I tried several methods, one of them was using the SPI interrupt but this was the fastest of them all and if I call HWI_disable, there shouldn't be any preemption.

About the logging part, I don't know when I'm loosing a sample, the SPI procedure doesn't happen and then I stop getting interrupts from the sensor but I will try to log everything maybe it can give us a clearer look at what's hapenning.

Tom, regarding your question, the buffers are static and located in L2 memory (0x8004000).

Yoel

The masking option LOWER isn't supported on OMAP-L138, according to the docs.  However BITMASK definitely is.

The problem with BITMASK is that you need to make sure you've masked your own interrupt as well as any others you want to pre-empt.  If you don't, your HWI can get called again halfway through, and HWIs generally aren't re-entrant so this is a Bad Thing.

Alternatively, you could use ALL if you don't mind that HWI disabling all other HWIs for the duration of the HWI.  Which I suspect you don't.

As Ramsey says, definitely don't disable HWIs during the SWI, because you're guaranteed to lose HWIs that way.

I disagree with Ramsey that 4 words in 100us over SPI is "fast".  However you definitely should check what clock rate you're using.  The OMAP-L138 can manage a maximum of 50MHz as a SPI master or 25MHz as a SPI slave.  Your ADC will have some timing limits too, and you'll need to check that.

Hi Graham,

Here is the error I get when trying to use bitmask:

About SPI frequency, 10MHz is the sensor max.

Yoel

Strange, because it worked fine for me, also on same BIOS.  I even tested it to make sure it was correctly pre-empting the lower-priority interrupt.  (I'm now using Hwi_plug for these interrupts, but that's another story.)

I do notice that you're missing the disable and restore masks though.  The whole point of BITMASK is that you're going to tell the BIOS which interrupts to disable, and you haven't there.  The mask values are the same as you'd use for setting the IER register flags, so setting bit 4 disables interrupt 4, setting bit 10 disables interrupt 10, and so on.  Don't forget to also disable the interrupt for this ISR as well.  I've no idea what the BIOS defaults to if you miss it out, but I would not rely on it being what you want. :)

Oh, and I don't think "priority" works for HWIs on the L138, from my reading of the docs.

Ah.I see what you've done...

You're using the generic Hwi library.  Don't do that, because it's not great.  If you select the hardware-specific one instead (which will be called "C64x") then things will probably start working.  In particular, the BITMASK option should start working, because that's tied to the hardware.

Edit: Just checked my CFG file.  You want to edit the Hwi declaration (in the "cfg script" tab, not in the GUI) to read

var Hwi = xdc.useModule('ti.sysbios.family.c64p.Hwi');

That'll pick up the hardware-specific Hwi module, without you needing to change any of the rest of your config file.

Hey Graham,

The SPI devices run on the ARM9 core, this module is for the DSP core if I am not mistaken.
I tried using device specific:

var Hwi = xdc.useModule('ti.sysbios.family.arm.da830.Hwi');

But, A. It doesn't support bit masking also. B. Doesn't solve the problem.

Another point for that matter, after the last changes (I think the major one is the Hwi disable\restore), the problem hardly returns when I run in emulation mode but it keeps on coming when running from SPI Flash1 in standalone mode.

Ah - my apologies, didn't spot that you were on the ARM core.

So, no solution?

Is there another form of support that can help?

Hey there...

I have some new\old insights on that matter.

When I work only with 1 core (ARM) I can get the sensors running without stop.
The problems start to happen when I use "Semaphore_pend".

When I work with 2 cores (IPC notify) no matter what I do over the DSP side (Empty endless while loop), one of the sensors stops streaming.

Task_sleep isn't a problem, probably never was.

Any thoughts?

Thanks,
Yoel

I'm a little confused about the application changes associated with you using "Semaphore_pend"?

What does that mean exactly?

What does the task that is blocked on the semaphore do when it is unblocked by the Semaphore_post() call?

Is that code thread-safe?

Alan

Hi Alan,

I want to clarify what's happening in the software now that I made millions of trials for different situations.

The are 2 devices streaming data at the same rate (15FPS*2KB) which is triggered by a GPIO interrupt in SPI protocol (10Mhz).
When the ARM core works alone (DSP not connected or halted), the data stream is excellent.

Semaphore_pend and Task_sleep doesn't bother me anymore, it was a coincidence.

I tried using IPC notify but that stops the data stream.
I tried setting SYSCFG0_CHIPSIG to implement custom notify and it works only if there isn't much code running on the DSP side.

In the ARM core, the data stream isn't supposed to be affected with the thread synchronization, it is handled by HWI's and SWI's (thus everything works when DSP is off) what I don't understand is why when I run more code on the DSP (the ARM core gets a notification in less frequent times) it interrupts with the data stream...

I use SYSCFG_CHIPINT1 to access the ARM and SYSCFG_CHIPINT3 to access the DSP, SYSCFG_CHIPINT3 is also connected to the ARM AINTC, maybe this confuses it or something but I don't have an extra interrupt except for the NMI which causes exceptions when I use it.

I guess that I made things more complicated than clarified I'm just frustrated and trying to give as much info as I can.

Thanks.
Yoel

Edit:  when I add a simple "NOP" loop for half a second this doesn't disrupt the data streaming but when I use code from the DSP libraries (s.a. DSPLib, VLIB, IMGLIB etc.) it does even if it's for shorter periods.

Yoel,

I'm sorry this is still not working for you. I appreciate your patients.

So, you have isolated IPC once again? Is that correct? Without IPC, all is well on the ARM but with IPC and using Notify, it causes data loss on the ARM. Is this correct?

As you probably already know, IPC uses the SYSCFG module to raise interrupts between the ARM and DSP. There are four interrupt lines sourced from the SYSCFG module:

SYSCFG_CHIPINT0: goes to ARM            IPC: DSP to ARM (line 0)
SYSCFG_CHIPINT1: goes to ARM            IPC: DSP to ARM (line 1)
SYSCFG_CHIPINT2: goes to DSP and ARM    IPC: ARM to DSP (line 0)
SYSCFG_CHIPINT3: goes to DSP and ARM    IPC: ARM to DSP (line 1)

As you can see, CHIPINT2 and CHIPINT3 go to both processors. The intent is for these to be used by the ARM to signal the DSP. They are typically not enabled in the ARM's interrupt controller. Enabling both interrupts is for debugging purposes. Or perhaps to synchronize both processors by invoking an ISR simultaneously on both of them.

I've indicated above how IPC assigns these interrupts. By default, we only enable Line 0. There is a configuration parameter to enable Line 1. Add the following to your application config script to enable the second line.

var NotifySetup = xdc.useModule('ti.sdo.ipc.family.da830.NotifySetup');
NotifySetup.useSecondLine = true;

I'm wondering if there is an issue with this configuration. Would you use CCS and open ROV to inspect the assigned interrupts and to see if the second line is enabled.

Also check the ARM AINTC ESR1 register (0xFFFE E300) and verify that bits 30 and 31 are clear.

Finally, does the data loss occur only when sending an IPC Notify event from DSP to ARM? I assume that sending an event from ARM to DSP does not cause a problem.

Thanks
~Ramsey

Hi Ramsey,

I actually removed the IPC from the projects, I now set the CHIPINT myself. I tried again with the IPC and 2 interrupt lines but the problem remains the same as without the IPC (without the IPC it lasts longer when the DSP is turned off, in fact, it hasn't broke the streaming for me but with the IPC it did a minute later).

In the ROV I see for the ARM interrupts 28,29 and for the DSP events 5, 67.

As for ARM AINTC ESR1 register (0xFFFE E300), I can't see nothing in the memory browser (my MMU is turned off, is that related?).

Actually, the problem happens even when the DSP is on IDLE loop. It doesn't happen when the DSP is off or in HALT.

Thanks,
Yoel

Edit: 

What I see now is really weird. I dis-attached the DSP from the ARM almost completely (Only DSP wakeup in the ARM code), now, the ARM isn't interrupted by the DSP and visa verca. I wouldn't expect to see any interference but whenever I do more than simple computations on the DSP it stopps the streaming on the ARM part, if I do idle loops or halt the DSP, the stream goes without interference.

Is there something special in functions like: 
DSPF_sp_fftSPxSP
DSPF_sp_mat_mul_cplx

Why should the DSP interfere with the ARM business? Is there a way to seperate them completely and maybe use UART for communication?

Yoel,

OK, so you have completely removed IPC from your project. Not interrupts are exchange between ARM and DSP. If DSP only runs idle loop, then data streaming on ARM is good. But if the DSP calls either of the following two functions, it breaks data streaming on ARM.

DSPF_sp_fftSPxSP
DSPF_sp_mat_mul_cplx

Have I got this correct?

I downloaded the dsplib_c674x_1_03_00_01 release. Is this the same release you are using?

I'm not familiar with this code base, but I took a quick look. I don't see any obvious issue with the source code. But, since you have source code for these functions, I suggest trying the following experiment.

Build your DSP program with one of the above functions. Setup the program such that it breaks the data streaming on the ARM. Now, comment out parts of the dsplib function you are calling. Maybe start by commenting out the entire function and verify that data streaming on ARM is okay. Try to narrow down what part of the dsplib function is causing the failure. Once you can identify this, maybe it will shed some light onto the issue.

~Ramsey

Yoel,

Sorry for the delay. I don't know of any reason moving your code and stack sections to L3 memory would fix the problem. But it might indicate that you originally had a memory map issue. Maybe in your old memory map, the ARM and DSP memory maps overlapped? That would certainly explain why running the DSP would cause problems on the ARM.

~Ramsey