RTOS/TMS320C6670: Missing MSI

One more observation. It looks that all 4 MSIs of event 4 became nonfunctional. I tried vector 24 with no luck. However, when I wrote 4 (as event 4 for MSI interrupts 0, 8, 16, 24) to IRQ_EOI - immediately whole machinery was unblocked. I do explicitly write to EOI in my ISR:

void pcie_isr( xdc_UArg arg )
{
    volatile CSL_Pciess_appRegs * const pcieAppRegs = (volatile CSL_Pciess_appRegs *)CSL_PCIE_CONFIG_REGS;
    u_int32 core_num = CSL_chipReadReg(CSL_CHIP_DNUM);
    u_int32 evt_idx, evt_num, msi_vector;
    int bit;


    /*********************************************************************
     * When using PCIe LLD to clear interrupt in PCIe subsytem it seems
     * that ISR is fired twice per trigger. Multiple reports on the issue
     * found on e2e. The workaround so far is to use direct register
     * manipulations.
     **********************************************************************/

    /*
     * Refer Table 2-10 in SPRUGS6:
     * Interrupt        Interrupt Description
     * Event Number
     *    4             MSI interrupts 0,  8, 16, 24 (EP/RC modes)
     *    5             MSI interrupts 1,  9, 17, 25 (EP/RC modes)
     *    6             MSI interrupts 2, 10, 18, 26 (EP/RC modes)
     *    7             MSI interrupts 3, 11, 19, 27 (EP/RC modes)
     *    8             MSI Interrupts 4, 12, 20, 28 (EP/RC modes)
     *    9             MSI Interrupts 5, 13, 21, 29 (EP/RC modes)
     *    10            MSI Interrupts 6, 14, 22, 30 (EP/RC modes)
     *    11            MSI Interrupts 7, 15, 23, 31 (EP/RC modes)
     *
     *    Core0 receives MSI 0-8-16-24 and 4-12-20-28 --> events 4 and 8
     *    Core1 receives MSI 1-9-17-25 and 5-13-21-29 --> events 5 and 9
     *
     * Do not confuse these event Id with interrupt controller event ids,
     * which are 17 and 18 for events 4, 5, 6, 7 and 8, 9, 10, 11
     * respectively. PCIESS event number is needed to access MSIX_IRQ,
     * MSI_IRQ_STATUS registers, which are array of [8]. Also this event
     * number is needed to signal End Of Interrupt (EOI).
     * Core number is found reading register. HWI for events 4, 5, 6, 7
     * receives arg = 0, while HWI for events 8, 9, 10, 11 receives arg = 4.
     * That is defined through SYS/BIOS config. For register arrays indexing
     * we use evt_idx, which is 0..7 for events 4..11.
     */
    evt_idx = core_num + arg;
    evt_num = evt_idx  + 4;

    //#if DEBUG_PCIE && DEBUG_HIGH
    //Log_print2( Diags_USER1, "pcie_isr(): core_num=%d, arg = %d", core_num, arg );
    //#endif

    /**********************************************************************
     * There are 8 MSIX_IRQ registers, we index them with evt_idx
     * found just above. Every register is 4 bit field. Each bit matches
     * MSI vectors 0, 8, 16, 24 in register[0], vectors 1, 9, 17, 25 in
     * register[1] and so on. Here we find the first vector being set,
     * giving priority to lower vectors.
     * Writing 1 to appropriate bit clears that interrupt status.
     * If there are more pending interrupt vectors, they will get processed
     * in the next invocation of the HWI.
     **********************************************************************/
    for ( bit = 1; bit <= 8; bit <<= 1 )
    {
        if ( bit & pcieAppRegs->MSIX_IRQ[evt_idx].MSI_IRQ_STATUS )
        {
            //pcieAppRegs->MSIX_IRQ[evt_idx].MSI_IRQ_STATUS = bit;
            break;
        }
    }

    /*
     * Calculate MSI vector depending on bit found in MSI_IRQ_STATUS
     * and PCIe event index.
     */
    switch ( bit )
    {
    case 1:
        msi_vector = evt_idx;
        break;
    case 2:
        msi_vector = evt_idx +  8;
        break;
    case 4:
        msi_vector = evt_idx + 16;
        break;
    case 8:
        msi_vector = evt_idx + 24;
        break;
    default:
        #if DEBUG_PCIE && DEBUG_HIGH
        Log_print1( Diags_USER1, "pcie_isr(): invalid bit = 0x%X ", bit );
        #endif
        return;
    }

    #if DEBUG_PCIE && DEBUG_HIGH
    Log_print3( Diags_USER1, "pcie_isr on Core%d: event %d, MSI vector %d ", core_num, evt_num, msi_vector );
    #endif

    /*
     * Check whether callback routine is registered and call it if so.
     * Note, that callback functions is still executed in HWI context.
     * For larger code it might be reasonable to post SWI from callback routine.
     */
    if ( NULL != msi_cb_fcn[msi_vector] )
    {
        msi_cb_fcn[msi_vector]();
    }
    #if DEBUG_PCIE && DEBUG_HIGH
    else
    {
        Log_print1( Diags_USER1, "pcie_isr(): no callback routine for MSI vector %d ", msi_vector );
    }
    #endif
    /*************************************************************************
     * End Of Interrupt must be written explicitly. Note, that MSI interrupt
     * event number for interrupt vector is from PCIESS perspective, i.e
     * {4..11} (ref. SPRUGS6D, Table 2-10). Not to be confused with
     * CorePack Primary Interrupt event number (ref. SPRS689D, Table 7-38),
     * where event number 17 is PCIEXpress_MSI_INTn. This number is used to
     * define HWI trigger source.
     *************************************************************************/
    pcieAppRegs->MSIX_IRQ[evt_idx].MSI_IRQ_STATUS = bit;
    pcieAppRegs->IRQ_EOI = evt_num;

    return;
}

Please advise, what could be a reason, that write to EOI from ISR did not make proper effect sometimes.

Thanks.

Yet another observation. It seems the trouble happens when more than one bit is set in status register. Then I observe really weird thing:

31 119 706,    "pcie_isr(): core_num=0, arg = 0, msi_irq_status = 0x9, bit = 0x1",      // 0x9=1001 - two bits set
31 119 706,    "pcie_isr on Core0: event 4, MSI vector 0 ",                             // 0x1 bit selected for processing
31 119 712,        msi_iqr_status = 0x8,                                                // After clearing 0x1 just before leaving ISR 0x8 remains in the register
31 119 713,    "pcie_isr(): core_num=0, arg = 0, msi_irq_status = 0x0, bit = 0x10",     // BANG! ISR invoked, but there is no bit in MSI0_IRQ_STATUS

It looks that MSI0_IRQ_STATUS was lost and ISR invoked with no pending interrupt.