F28M35H52C: IPC communication stops working

Nir,

Thanks for reaching out to the E2E forum.  I'm going to ask some clarifying questions:

You mention working with IPC interrupts for a few years, do you mean that a specific device has been working for some time and is now not behaving the same way?  Or do you mean that you have encountered this behavior recently during development/change in FW?

I can't think if a reason that a working device would suddenly stop communicating between its 2 cores.  Perhaps if one of the cores got stalled in another ISR?  I would think that the internal watchdog would eventually cause a reset if that were the case.

Will look for more information on the above from you.

Best,

Matthew

Hello Mathew, thank you for the reply, 

Yes same device is working for years and now not. I am constantly making firmware updates but this problem is hard to reproduce so this is why I am reaching out to see if you can help me understand what can cause this.

Yea, the software kept working, just no interrupts from the IPC...

any ideas ?

Nir,

This is device is somewhat unique, in that there is exists an IPC(Inter Processor Communications) in HW and interrupt, but I'm also aware of a module in TI RTOS package also called IPC that serves a similar purpose, but is a SW library that is applicable to this and other devices with an M4 core.  Can you comment if this issue is with the device HW or the RTOS SW lib?  Just want to make sure I'm on the same page

Best,

Matthew

Hi Matthew,

I am not working with RTOS. I have 2 projects (ARM+DSP) with no RTOS.

The problem I have is with the communication between cores (IPC).

I was able to reproduce the bug, and I can see that the MTOC PUT buffer is full. 

and once full messages cannot be sent.

I am not sure though why the buffer stays full and not being cleared...

any thoughts ?

I added the code for the IpcPut function which returns STATUS_FAIL:

unsigned short
IpcPut (volatile tIpcController *psController, tIpcMessage *psMessage,
unsigned short bBlock)
{
unsigned short writeIndex;
unsigned short readIndex;
unsigned short returnStatus = STATUS_PASS;

writeIndex = *(psController->pusPutWriteIndex);
readIndex = *(psController->pusPutReadIndex);

// Wait until Put Buffer slot is free
while (((writeIndex + 1) & MAX_BUFFER_INDEX) == readIndex)
{
// If designated as a "Blocking" function, and Put buffer is full,
// return immediately with fail status.
if (!bBlock)
{
returnStatus = STATUS_FAIL;   <--------------------------------------   Code goes here and therefore messages are not sent anymore
break;
}

readIndex = *(psController->pusPutReadIndex);
}

if (returnStatus != STATUS_FAIL)
{
// When slot is free, Write Message to PutBuffer, update PutWriteIndex,
// and set M3 IPC INT Flag
psController->psPutBuffer[writeIndex] = *psMessage;

writeIndex = (writeIndex + 1) & MAX_BUFFER_INDEX;
*(psController->pusPutWriteIndex) = writeIndex;

HWREG(MTOCIPC_BASE + IPC_O_MTOCIPCSET) |= psController->ulPutFlag;
}

return returnStatus;
}

Any thoughts ?

Thank you !

Hi Matthew,

I found out that the IPC interrupts stops working on the receiving side (DSP) of the IPC messages.

The interrupt (flag) is not acknowledged (not cleared). I am not sure why, in the ISR I clear the flag and it works all the time.

for some reason once in a while it is not cleared so interrupts from this group stops working. 

Nir,

There are 2 IPC Interrupt vectors on the C28x side, I've C/P the example ISR we provide, could you compare to your ISR and note any differences?  This is from a file in our examples called ctom_ipcdrivers_c28.c

You can just reply back with your code for this as well if that is OK with you.  Assuming the M3 is still sending IPC messages, the typical reasons we stop seeing interrupts for any peripheral is a race condition with re-enabling the interrupt and the ACK bit.  This can happen if there is alot of code in the ISR that delays the ACK/re-enable. 

One thing we could try is to put the re-enables at the beginning of the ISR and see if that fixes things.

Best,

Matthew

//*****************************************************************************
// MtoC IPC INT1 Interrupt Handler -
// Handles writes into M3 addresses as a result of read commands to the C28.
//*****************************************************************************
__interrupt void
MtoCIPC1IntHandler (void)
{
    tIpcMessage sMessage;

    // Continue processing messages as long as MtoC GetBuffer1 is full
    while (IpcGet(&g_sIpcController1, &sMessage,
                  DISABLE_BLOCKING) != STATUS_FAIL)
    {
        switch (sMessage.ulcommand)
        {
        case IPC_DATA_WRITE:
            IPCMtoCDataWrite(&sMessage);
            break;
        default:
            ErrorFlag = 1;
            break;
        }
    }

    // Acknowledge IPC INT1 Flag and PIE to receive more interrupts from group
    // 11
    CtoMIpcRegs.MTOCIPCACK.bit.IPC1 = 1;
    PieCtrlRegs.PIEACK.all = PIEACK_GROUP11;
}

//*****************************************************************************
// MtoC IPC INT2 Interrupt Handler -
// Should never reach this ISR. This is an optional placeholder for
// g_sIpcController2.
//*****************************************************************************
__interrupt void
MtoCIPC2IntHandler (void)
{
    // Should never reach here - Placeholder for Debug

    // Acknowledge IPC INT2 Flag and PIE to receive more interrupts from group
    // 11
    CtoMIpcRegs.MTOCIPCACK.bit.IPC2 = 1;
    PieCtrlRegs.PIEACK.all = PIEACK_GROUP11;
}

Hi Matthew,

Thank you so much for the help.

After receiving the ISR code that you sent I reviewed again my ISR code and apparently I was using interrupt priorities with nesting. I understood that this can be a problem and removed it. I don't really need it. it now works good for few days straight.

my code looked like this before:

__interrupt void MtoCIPC1IntHandler (void)
{
tIpcMessage sMessage;

/* Use this to modify interrupt priority */
volatile Uint16 TempPIEIER = PieCtrlRegs.PIEIER11.all;

DINT;
IER |= M_INT11;
IER &= MINT11; // Set "global" priority
PieCtrlRegs.PIEIER11.all &= MG111; // Set "group" priority
//PieCtrlRegs.PIEACK.all = 0xFFFF; // Enable PIE interrupts
CtoMIpcRegs.MTOCIPCACK.bit.IPC1 = 1;
PieCtrlRegs.PIEACK.all = PIEACK_GROUP11;
__asm(" NOP");
EINT;

// Continue processing messages as long as MtoC GetBuffer1 is full
while (IpcGet (&g_sIpcController1, &sMessage,
DISABLE_BLOCKING)!= STATUS_FAIL)
{
switch (sMessage.ulcommand)
{
case IPC_BLOCK_WRITE:
IPCMtoCBlockWrite(&sMessage);

// my code here

break;
case IPC_BLOCK_READ:
IPCMtoCBlockRead(&sMessage);
break;
case IPC_SET_BITS:
IPCMtoCSetBits(&sMessage);
break;
case IPC_CLEAR_BITS:
IPCMtoCClearBits(&sMessage);
break;
default:
break;
}
}

//CtoMIpcRegs.MTOCIPCACK.bit.IPC1 = 1;
//PieCtrlRegs.PIEACK.all = PIEACK_GROUP11;

DINT;
PieCtrlRegs.PIEIER11.all = TempPIEIER;
EINT;
}

So thank you so much for that !

I am sorry. But now after many hours of straight sun, I have a similar issue in the other side. I stop getting interrupts in the ARM side of the IPC.

my code there looks like this:

static void CtoMIPC2IntHandler (void)
{
tIpcMessage sMessage;

// Continue processing messages as long as MtoC GetBuffer2 is full
while (IpcGet(&g_sIpcController2, &sMessage,
DISABLE_BLOCKING) != STATUS_FAIL)
{
switch (sMessage.ulcommand)
{
case IPC_BLOCK_WRITE:
IPCCtoMBlockWrite(&sMessage);

// my code here
break;
case IPC_BLOCK_READ:
IPCCtoMBlockRead(&sMessage);
break;
case IPC_SET_BITS_PROTECTED:
IPCCtoMSetBits_Protected(&sMessage); // Processes
// IPCCtoMReqMemAccess()
// function
break;
case IPC_CLEAR_BITS_PROTECTED:
IPCCtoMClearBits_Protected(&sMessage); // Processes
// IPCCtoMReqMemAccess()
// function
break;
default:
break;
}
}

// Acknowledge IPC INT2 Flag
HWREG(MTOCIPC_BASE + IPC_O_CTOMIPCACK) |= IPC_CTOMIPCACK_IPC2;
}

Any thoughts ?

Hi Matthew,

Thank you so much for the help.

After receiving the ISR code that you sent I reviewed again my ISR code and apparently I was using interrupt priorities with nesting. I understood that this can be a problem and removed it. I don't really need it. it now works good for few days straight.

my code looked like this before:

__interrupt void MtoCIPC1IntHandler (void)
{
tIpcMessage sMessage;

/* Use this to modify interrupt priority */
volatile Uint16 TempPIEIER = PieCtrlRegs.PIEIER11.all;

DINT;
IER |= M_INT11;
IER &= MINT11; // Set "global" priority
PieCtrlRegs.PIEIER11.all &= MG111; // Set "group" priority
//PieCtrlRegs.PIEACK.all = 0xFFFF; // Enable PIE interrupts
CtoMIpcRegs.MTOCIPCACK.bit.IPC1 = 1;
PieCtrlRegs.PIEACK.all = PIEACK_GROUP11;
__asm(" NOP");
EINT;

// Continue processing messages as long as MtoC GetBuffer1 is full
while (IpcGet (&g_sIpcController1, &sMessage,
DISABLE_BLOCKING)!= STATUS_FAIL)
{
switch (sMessage.ulcommand)
{
case IPC_BLOCK_WRITE:
IPCMtoCBlockWrite(&sMessage);

// my code here


break;
case IPC_BLOCK_READ:
IPCMtoCBlockRead(&sMessage);
break;
case IPC_SET_BITS:
IPCMtoCSetBits(&sMessage);
break;
case IPC_CLEAR_BITS:
IPCMtoCClearBits(&sMessage);
break;
default:
break;
}
}

//CtoMIpcRegs.MTOCIPCACK.bit.IPC1 = 1;
//PieCtrlRegs.PIEACK.all = PIEACK_GROUP11;

DINT;
PieCtrlRegs.PIEIER11.all = TempPIEIER;
EINT;
}

So thank you so much for that !

I am sorry. But now after many hours of straight sun, I have a similar issue in the other side. I stop getting interrupts in the ARM side of the IPC.

my code there looks like this:

static void CtoMIPC2IntHandler (void)
{
tIpcMessage sMessage;

// Continue processing messages as long as MtoC GetBuffer2 is full
while (IpcGet(&g_sIpcController2, &sMessage,
DISABLE_BLOCKING) != STATUS_FAIL)
{
switch (sMessage.ulcommand)
{
case IPC_BLOCK_WRITE:
IPCCtoMBlockWrite(&sMessage);

// my code here
break;
case IPC_BLOCK_READ:
IPCCtoMBlockRead(&sMessage);
break;
case IPC_SET_BITS_PROTECTED:
IPCCtoMSetBits_Protected(&sMessage); // Processes
// IPCCtoMReqMemAccess()
// function
break;
case IPC_CLEAR_BITS_PROTECTED:
IPCCtoMClearBits_Protected(&sMessage); // Processes
// IPCCtoMReqMemAccess()
// function
break;
default:
break;
}
}


// Acknowledge IPC INT2 Flag
HWREG(MTOCIPC_BASE + IPC_O_CTOMIPCACK) |= IPC_CTOMIPCACK_IPC2;
}

Any thoughts  ?

Nir,

Glad that you were able to get the C side working normally, nesting ISRs can prove to be challenging with async type events in a system.

I'm not as well versed in the NVIC interrupt controller on the M3 side, and will need to loop in some other to take a look.  Please give me a day or so to get some more folks to reply.

Best,

Matthew

Nir,

I spoke to a colleague, and he suggested that the ARM NVIC can natively nest ISRs, he suggested to make sure the C28x is still generating from its end if we aren't seeing on the M3 side.

Best,

Matthew

Hi Matthew,

Can you expand on what you wrote please ? I am not sure I understood everything.

Nir,

Matthew is out of the office until July 8th. To restate his last post:  It seems like the managing of interrupt priorities was a suspected cause of faulty interrupt reception on the C28x side because the issue had gone away after removing the software-based interrupt priority code. For the ARM side, the NVIC uses native hardware support for interrupt priority handling, so we would not expect as much of the same problem for the ARM.

How are you implementing the PUT buffer? Are you enumerating through different sets of IPC locations? If so, are the multiple interrupts pointing to the same ISR?

Do you have any mechanisms in place to detect (and service) interrupt overflows? I think that both the C28x and ARM are susceptible to dropped interrupt triggers in case of overflows. For example, if two (or more) IPC interrupts arrive while the CPU is busy executing other tasks / ISRs, the CPU will only be aware of the first IPC interrupt when it eventually runs the ISR.

For quickly reproducible interrupt problems, you can toggle a GPIO when the IPC flag is set and a second GPIO when the IPC ISR is executing. If you see two flag set toggles before the ISR executes, you would expect an overrun condition.

For a slow-burn situation that you are describing, it might make sense to record the local CPU timer count for each IPC interrupt trigger and pass an enumerated identification (like 1, 2, 3, etc) using an unused mailbox or global RAM. The remote CPU can record the ID in a debug buffer when servicing the interrupt and check for enumeration gaps. When an interrupt fails to trigger, the buffer of CPU time stamps can help indicate if there is a minimum delay threshold required between IPC triggers.

-Tommy