C6678 PCIe MSI interrupt going off twice per MSI trigger

Peter Bonham

Hi,

I am currently developing a Root Complex side application in which an EP on the other end of the bus sends an MSI interrupt causing the RC to pull some data across the link. Having set up the PCIe module using refactored code from the TI PCIe example I am having a problem whereby the MSI from the EP triggers the MIS HWI twice, which is not the expected behaviour.

I have added a counter to the interrupt function which is being incremented twice and the event that I am firing from the HWI is also double-firing. As expected the counter is incremented twice before the task which prints it to the console processes the event, suggesting that the HWI is running twice before the event task can run.

See code below, it looks a little esoteric at first, but this is because it is split over multiple files, and the owning file of the PCIE driver is going to trigger some DMA from HWI context, whilst printing a message to the screen from task context.

static volatile uint32_t s_HwiCount = 0;

void PcieDriver_MsiHwi(UArg arg)
{

    pcieRegisters_t setRegs;
    pcieMsiIrqStatusReg_t msiIrqStatus[1];
    pcieIrqEOIReg_t irqEOI;

    (*s_MsiCallback)(MSI_HWI_SUCCESS); // note that this is re-defined from Event_Id_01
    s_HwiCount++;

    memset(&irqEOI, 0, sizeof(irqEOI));
    memset(&setRegs, 0, sizeof(setRegs));
    memset(&msiIrqStatus, 0, sizeof(msiIrqStatus));

    msiIrqStatus[0].msiIrqStatus = 0x1;
    irqEOI.EOI = 0x4;

    setRegs.msiIrqStatus[0] = &msiIrqStatus[0];
    setRegs.irqEOI = &irqEOI;

    Pcie_writeRegs(s_ModuleHandle, pcie_LOCATION_LOCAL, &setRegs);
}

uint32_t PcieDriver_GetMsiHwiCount(void)
{
    return s_HwiCount;
}

// File PcieComms.c, includes PcieDriver.h ...

static void PcieComms_MsiCallbackHandler(uint8_t resetSuccess)
{
    Event_post(s_MsiEvent,resetSuccess);
}

/**
 * Task for MSI reception debug
 */
void PcieComms_MsiWaitTask(UArg a0, UArg a1)
{
    uint32_t msiReceivedEvent;

    while (1)
    {
        msiReceivedEvent = Event_pend(s_MsiEvent,
                   Event_Id_NONE,
                   MSI_HWI_FAIL + MSI_HWI_SUCCESS,
                   BIOS_WAIT_FOREVER);

        System_printf("MSI interrupt %d received.\n", PcieDriver_GetMsiHwiCount());

        if (msiReceivedEvent &  MSI_HWI_SUCCESS)
        {
            System_printf("MSI HWI has reset MSI interrupt!\n");
        }
    }
}

In addition I have modified the code such that the MSI is "spoofed" by another task, rather than trigger it from the EP, and disabled the EP side MSI trigger send.

void MsiMakerTask(UArg a0, UArg a1)
{
    unsigned int *pMSIReg = (unsigned int*) 0x21800100;

    while (1)
    {
        //write to MSI_IRQ_RAW_STATUS REG
        System_printf("Spoofing MSI...\n");
        *pMSIReg = 0xfffffff1;
        Task_sleep(1000);
    }
}

The console output (initialisation removed) is as follows:

[C66xx_0] RC: Link is up...
[C66xx_0] EP: Link is up.
[C66xx_0] Spoofing MSI...
[C66xx_0] MSI interrupt 2 received.
[C66xx_0] MSI HWI has reset MSI interrupt!
[C66xx_0] Spoofing MSI...
[C66xx_0] MSI interrupt 4 received.
[C66xx_0] MSI HWI has reset MSI interrupt!
[C66xx_0] MSI interrupt 4 received.
[C66xx_0] MSI HWI has reset MSI interrupt!
[C66xx_0] Spoofing MSI...
[C66xx_0] MSI interrupt 6 received.
[C66xx_0] MSI HWI has reset MSI interrupt!
[C66xx_0] MSI interrupt 6 received.
[C66xx_0] MSI HWI has reset MSI interrupt!

Am I failing to reset the MSI interrupt received registers properly, or is there something about the way that I am triggerring the HWI or MSI?

Any help would be greatly appreciated.

over 12 years ago

0 Steven Ji over 12 years ago

TI__Genius 12065 points

Peter,

Since you are triggering the MSI interrupt in a while loop and there are lots of printf output, I am wondering if the issue is due to the timing between the interrupt triggering, ISR and status checking.

Could you please trigger the MSI for some determined number of times (for example, only triggered once) or add another counter for the triggering to see if the MSI response counter (s_HwiCount) is the same as the triggering counter or 2x times please?

We would like to confirm what really happened is correlated to what had been printed out.

Or if the issue is not related to the timing/BIOS usage, could you please attach one example project to reproduce the issue please? We can have a simplified version of project to test MSI generation/service part only. Thanks.

0 Gordon Deane over 12 years ago in reply to Steven Ji

Expert 1255 points

I'm a colleague of Peter's. The code above was from a quick prototype a third colleage wrote on an EVM to evaluate PCI performance. When I came to implement the code in our real application, I instinctively disliked using the heavyweight PCIe LLD calls involved and re-wrote it using CSL calls as below. (I like the LLD, just not for an interrupt context).

Interestingly, when I did this, the double-interrupt disappeared. I am sure the two methods are doing the same register writes, but the original version will have had a longer spacing between the two writes. I'm speculating that this gap caused the MSI to re-trigger somehow. Anyway, it's no longer an issue.

Regards,

Gordon

New routine:

#include <ti/csl/cslr_pciess_app.h>
volatile CSL_Pciess_appRegs * const the_pciessAppRegisters = (volatile CSL_Pciess_appRegs *)CSL_PCIE_CONFIG_REGS;
...
void PcieDriver_MsiHwi(UArg arg)
{

    ,,,
    the_pciessAppRegisters->MSIX_IRQ[0].MSI_IRQ_STATUS = 0x1;
    the_pciessAppRegisters->IRQ_EOI = 0x4;
}

Processors

Processors forum

C6678 PCIe MSI interrupt going off twice per MSI trigger