This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

K2GICE: PCI-e MSI Hwi interupts

Part Number: K2GICE
Other Parts Discussed in Thread: SYSBIOS

Hi

Im trying to figure out how to enable MSI interupts to my code. I feel like i have pretty ok understanding of what i have to do after reading Creating Hwi objects from here http://www.ti.com/lit/ug/spruex3t/spruex3t.pdf and Interupt support from here http://www.ti.com/lit/ug/sprugs6d/sprugs6d.pdf . I anyhow wasnt abble to find information how to translate pciess interupt event number to interupt number (id) needed on creating hwi object. How does this step work.

Im using k2gIce as an rc and fpga as ep. Link has been tested and memory writes and reads work.

-Tommi Mehtonen

  • Hi, Tommi,

    Is the PCIe RC configured by ARM or DSP? and if ARM, is it using Linux or RTOS?

    Rex

  • Hi

    PCIe RC is configured by DSP and RTOS. Arm is not used at all in this configuration.

    Tommi

  • Hi, Tommi,

    Got you. I'll have DSP PCIe expert to help you.

    Rex

  • Hi,

    From K2G TRM, Table 9-53. CIC Input Events Mapping, those are the PCIE system event number:

    224 PCIE_INT0 PCIe interrupt 0
    225 PCIE_INT1 PCIe interrupt 1
    226 PCIE_INT2 PCIe interrupt 2
    227 PCIE_INT3 PCIe interrupt 3
    228 PCIE_INT4 PCIe interrupt 4
    229 PCIE_INT5 PCIe interrupt 5
    230 PCIE_INT6 PCIe interrupt 6
    231 PCIE_INT7 PCIe interrupt 7
    232 PCIE_INT8 PCIe interrupt 8
    233 PCIE_INT9 PCIe interrupt 9
    234 PCIE_INT10 PCIe interrupt 10
    235 PCIE_INT11 PCIe interrupt 11
    236 PCIE_INT12 PCIe interrupt 12
    237 PCIE_INT13 PCIe interrupt 13

    For PCIE user guide: 

    0 PCIe express legacy interrupt mode - INTA (RC mode only)
    1 PCIe express legacy interrupt mode - INTB (RC mode only)
    2 PCIe express legacy interrupt mode - INTC (RC mode only)
    3 PCIe express legacy interrupt mode - INTD (RC mode only)
    4 MSI interrupts 0, 8, 16, 24 (EP/RC modes)
    5 MSI interrupts 1, 9, 17, 25 (EP/RC modes)
    6 MSI interrupts 2, 10, 18, 26 (EP/RC modes)
    7 MSI interrupts 3, 11, 19, 27 (EP/RC modes)
    8 MSI Interrupts 4, 12, 20, 28 (EP/RC modes)
    9 MSI Interrupts 5, 13, 21, 29 (EP/RC modes)
    10 MSI Interrupts 6, 14, 22, 30 (EP/RC modes)
    11 MSI Interrupts 7, 15, 23, 31 (EP/RC modes)
    12 Error Interrupts
    [0] System error (OR of fatal, nonfatal, correctable errors) (RC mode only)
    [1] PCIe fatal error (RC mode only)
    [2] PCIe non-fatal error (RC mode only)
    [3] PCIe correctable error (RC mode only)
    [4] AXI Error due to fatal condition in AXI bridge (EP/RC modes)
    [5] PCIe advanced error (RC mode only)
    13 Power management and reset event interrupts
    [0] Power management turn-off message interrupt (EP mode only)
    [1] Power management ack message interrupt (RC mode only)
    [2] Power management event interrupt (RC mode only)
    [3] Link request reset interrupt (hot reset or link down) (RC mode only)

    If you want to use MSI-0, you need PCIE_INT4, that is system event 228. To configure HWI, you may refer to how we did for DDR ECC interrupt on K2G, where system event is 198 (DDR3_ERR EMIF error interrupt). The code is under pdk_k2g_1_0_xx\packages\ti\csl\example\ecc\ecc_test_app. There is 1-bit ECC ISR function isrEmifSecErr(), you can trace back how this ISR is setup and registered.

    Regards, Eric

  • Hi

    I looked around examples but i keep sunning on problem when creating the hwi. Im using modified pcie example as an base and i am trying to create the interupt after setting upt the link and testing it with following code:

          Hwi_Handle hwi0;
          Hwi_Params hwiParams;
          Error_Block eb;
          Error_init(&eb);
          Hwi_Params_init(&hwiParams);
          hwiParams.arg = 5;
    
          hwi0 = Hwi_create(1, pcie_reader, &hwiParams, &eb);
          Hwi_eventMap(1, 17);
    
          if (hwi0 == NULL) {
           System_abort("Hwi create failed");
          }
          PCIE_logPrintf("Hwi done \n");

    event number 17 is based on chapter 6.2.2.1 DSP Interrupt Sources on 66ak2gx's trm. Code compiles, but on creation i get following error message:

    ti.sysbios.family.c64p.Hwi: line 189: E_invalidIntNum: Invalid interrupt number: intr# 1
    Hwi create failed

    I also get warning:

    Description Resource Path Location Type
    #169-D argument of type "void (*)(void)" is incompatible with parameter of type "ti_sysbios_interfaces_IHwi_FuncPtr" pcie_sample.c /DSP_pcie_platform line 1178 C/C++ Problem

    for Hwi_create line.

    Same error occurs if i try to make the interrupt on main before BIOS_start()

    My includes are:

    #include "pcie_sample.h"
    #include <ti/drv/pcie/soc/pcie_soc.h>
    #include <stdint.h>
    #include "ti/board/board.h"
    #include <ti/csl/csl_bootcfgAux.h>
    #include <ti/csl/csl_xmcAux.h>
    #include <ti/csl/csl_serdes_pcie.h>
    #include <ti/csl/csl_pscAux.h>
    #define PCIE_REV0_HW
    #include <ti/csl/csl_cacheAux.h>
    #include <ti/csl/csl_chip.h>
    #include <ti/sysbios/knl/Clock.h>
    #include <ti/sysbios/timers/timer64/Timer.h>
    #include <xdc/runtime/Error.h>
    
    and on pcie_sample.h
    
    /* C Standard library include */
    #include <string.h>
    
    /* XDC include */
    #include <xdc/std.h>
    #include <xdc/runtime/System.h>
    
    /* BIOS include */
    #include <ti/sysbios/BIOS.h>
    #include <ti/sysbios/knl/Task.h>
    #include <ti/sysbios/family/c64p/EventCombiner.h>
    #include <ti/sysbios/family/c64p/Hwi.h>
    #include <ti/sysbios/knl/Event.h>
    
    /* CSL include */
    #include <ti/csl/cslr_device.h>
    
    /* PCIE LLD include */
    #include <ti/drv/pcie/pcie.h>
    
    #include "PCIeEDMA.h"
    #include <xdc/runtime/System.h>
    #include <ti/drv/uart/UART_stdio.h>
    

  • Hi,

    ti.sysbios.family.c64p.Hwi: line 189: E_invalidIntNum: Invalid interrupt number: intr# 1 =======> Please try to use an interrupt number between 4 and 15 (inclusive) for this.

    Regards, Eric

  • Hi

    Got the interupt done with intNum 8:

      Hwi_Handle hwi0;
      Hwi_Params hwiParams;
      Error_Block eb;
      Error_init(&eb);
      Hwi_Params_init(&hwiParams);
      hwiParams.arg = 5;
    
      hwi0 = Hwi_create(8, pcie_reader, &hwiParams, &eb);
      Hwi_eventMap(8, 228);
    
      if (hwi0 == NULL) {
       System_abort("Hwi create failed");
      }
      PCIE_logPrintf("Hwi done \n");
    
    
    
      BIOS_start();

    I tried eventmap() with both 228 and 17. On ROV it shows:

    Should the eventId/Eventbumber be visible there.

    -Tommi

  • Hi

    Had time to work on this one again and found the page https://processors.wiki.ti.com/index.php/Configuring_Interrupts_on_Keystone_Devices. I added line "  hwiParams.eventId = 17; which seems to do the trick as i got timer interups working from that one using the table http://www.ti.com/lit/ug/spruhy8i/spruhy8i.pdf page 626.

    Anyhow MSI interupt itself does not seem to work. Im not sure if the problem is on dsp or fpga side. Is there need to enable MSI interupts for pcie separately or are those enabled automatically.

    -Tommi

  • answering to myself yes, by writing the register MSI0_IRQ_ENABLE_SET 

  • Hi

    Now iv have gotten my interupts working when in manyally write 1 to register PCIE_APPLICATION_PVIE_MSI0_IRQ_STATUS_RAW (0x21800100). Anyhow interrupts form the fpga dont come thought. Im thinking that this would be memory translation problem.

    What thse value should be on the pcie_sample.h and is there any other memory translation registers i should set up or way of debugging this situation.

    /* Inbound Base Address for PCIe RC */
    #define PCIE_IB_LO_ADDR_RC 0x21800100
    #define PCIE_IB_HI_ADDR_RC 0

    /* PCIE address space for MSI */
    #define PCIE_PCIE_MSI_BASE (0x00000000U)
    #define PCIE_PCIE_MSI_OFF (0x00000000U)

  • Hi,

    The logical is right. In the K2G side, are you able to dump 0x2180_0300 to 0x0340 for me to double check the inbound set up? Also, for the FPGA side, are you able to access the K2G's application register region (that is, 0x2180_0000)?

    Regards, Eric

  • Also, please let me know the BAR0-BAR5 (0x21801010 - 1024) for a double check.

    Regards, Eric

  • Hi

    Here are those registers:

    Here is also the endpoints configuration space where i noticed weard behavior. Im configuring the endpoint msi config registers with following code, but registers dont seem to correspond to correct values. Bar1 write there works fine.

    pcieRet_e pcieEnableInerupts_EP(Pcie_Handle handle) {
        pcieRet_e              retVal;
        pcieRegisters_t setRegs;
        pcieRegisters_t getRegs;
        pcieMsiIrqEnableSetReg_t msiIrqEnableSet[2];
        pcieStatusCmdReg_t statusCmd;
        pcieMsiCapReg_t msiCap;
        pcieIrqEOIReg_t irqEOI;
        pcieMsiIrqStatusReg_t msiIrqStatus[2];
        pcieMsiLo32Reg_t msiLo32;
        pcieMsiDataReg_t msiData;
    
        memset(&setRegs,0 ,sizeof(setRegs));
        memset(&getRegs,0 ,sizeof(getRegs));
        memset(&msiIrqEnableSet,0 ,sizeof(msiIrqEnableSet));
        memset(&statusCmd,0 ,sizeof(statusCmd));
        memset(&msiCap,0 ,sizeof(msiCap));
        memset(&irqEOI,0 ,sizeof(irqEOI));
        memset(&msiIrqStatus,0 ,sizeof(msiIrqStatus));
        memset(&msiLo32,0 ,sizeof(msiLo32));
        memset(&msiData,0 ,sizeof(msiData));
    
        getRegs.msiIrqEnableSet[0] = &msiIrqEnableSet[0];
        getRegs.msiIrqEnableSet[1] = &msiIrqEnableSet[1];
    
        getRegs.msiCap = &msiCap;
        getRegs.statusCmd = &statusCmd;
    
        if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs)) != pcie_RET_OK)
        {
          PCIE_logPrintf ("pcieEnableInerupts_test fail\n");
          return retVal;
        }
    
        msiIrqStatus[0].msiIrqStatus = 0xF;
        msiIrqEnableSet[0].msiIrqEnSet =  0xF;
    
        msiCap.msiEn = 1;
        msiCap.en64bit = 1;
        msiCap.multMsgEn = 0;
        msiCap.multMsgCap = 0;
        msiLo32.addr = 0x100;
        msiData.data = 0x1;
    
    
        statusCmd.dis = 1;
        statusCmd.memSp = 1;
        statusCmd.busMs =1;
    
        setRegs.msiIrqStatus[0] = &msiIrqStatus[0];
        setRegs.msiIrqEnableSet[0] = &msiIrqEnableSet[0];
        setRegs.msiCap = &msiCap;
        setRegs.statusCmd = &statusCmd;
        setRegs.msiLo32 = &msiLo32;
        setRegs.msiData = &msiData;
    
        if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs)) != pcie_RET_OK)
        {
          PCIE_logPrintf ("pcieEnableInerupts_test fail\n");
          return retVal;
        }
    
        irqEOI.EOI = 0x4;
        setRegs.irqEOI = &irqEOI;
    
        if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs)) != pcie_RET_OK)
        {
          PCIE_logPrintf ("pcieEnableInerupts_test fail on EIO\n");
          return retVal;
        }
    
        PCIE_logPrintf("Enterupts enabled on EP\n");
        return pcie_RET_OK;
    
    
    }

    Solution seems to be close, but i still havent figured everything out. Have i understood correct that endpoint makes the memory write for interupt to pcie-memoreyspace address specified in MSI message address (upper+lower) which im trying to set with register msiLo32 here in my code and then with bar and inbound translation i should get that write to hit address where register PCIE_MSI0_IRQ_STATUS_RAW is located.

    Br.

    Tommi Mehtonen

  • Hi,

    From the FPGA side, are you able to look into the DSP's PCIE application register? I thought BAR0 can do this automatically. In the DSP side, 0x2180_030c register, can you try to change this from 0x0 to 0x2180_0000 to see if it works?

    Also, you mentioned to generate MSI by write MSI 0 Raw Interrupt Status Register (offset 0x100), this is for debug purpose. There is another register offset 0x54 MSI_IRQ. If you are able to access the application register region from FPGA, writing to 0x100 or 0x54 should be same to generate MSI.

    Those two registers are specific to Keystone PCIE IP for interrupt generation. 

    The standard way to generate MSI is what you said "i understood correct that endpoint makes the memory write for interupt to pcie-memoreyspace address specified in MSI message address (upper+lower) which im trying to set with register msiLo32 here in my code". But it is not targeted to the RC's side PCIE_MSI0_IRQ_STATUS_RAW.  

    Please see PCIE user guide:  http://www.ti.com/lit/ug/sprugs6d/sprugs6d.pdf 2.14.2.2 MSI Interrupt Generation in EP Mode 

    Regards, Eric

  • Hi

    I managed to track down the problem to register writes which dont seem to work correctly. Im using following code to write ep msi config registers, but it does not work. When writing the registers manually i got the interrupt to work. Now just for sake of cleaning the code i would like to know why this does not work.

    pcieRet_e pcieEnableInerupts_EP(Pcie_Handle handle) {
        pcieRet_e              retVal;
        pcieRegisters_t setRegs;
        pcieRegisters_t getRegs;
        pcieMsiIrqEnableSetReg_t msiIrqEnableSet[2];
        pcieStatusCmdReg_t statusCmd;
        pcieMsiCapReg_t msiCap;
        pcieIrqEOIReg_t irqEOI;
        pcieMsiIrqStatusReg_t msiIrqStatus[2];
        pcieMsiLo32Reg_t msiLo32;
        pcieMsiUp32Reg_t msiUp32;
        pcieMsiDataReg_t msiData;
    
        memset(&setRegs,0 ,sizeof(setRegs));
        memset(&getRegs,0 ,sizeof(getRegs));
        memset(&msiIrqEnableSet,0 ,sizeof(msiIrqEnableSet));
        memset(&statusCmd,0 ,sizeof(statusCmd));
        memset(&msiCap,0 ,sizeof(msiCap));
        memset(&irqEOI,0 ,sizeof(irqEOI));
        memset(&msiIrqStatus,0 ,sizeof(msiIrqStatus));
        memset(&msiLo32,0 ,sizeof(msiLo32));
        memset(&msiUp32,0 ,sizeof(msiUp32));
        memset(&msiData,0 ,sizeof(msiData));
    
        getRegs.msiIrqEnableSet[0] = &msiIrqEnableSet[0];
        getRegs.msiIrqEnableSet[1] = &msiIrqEnableSet[1];
    
        getRegs.msiCap = &msiCap;
        getRegs.statusCmd = &statusCmd;
    
        if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs)) != pcie_RET_OK)
        {
          PCIE_logPrintf ("pcieEnableInerupts_test fail\n");
          return retVal;
        }
    
        msiIrqStatus[0].msiIrqStatus = 0xF;
        msiIrqEnableSet[0].msiIrqEnSet =  0xF;
    
        msiCap.msiEn = 1;
        msiCap.en64bit = 1;
        msiCap.multMsgEn = 0;
        msiCap.multMsgCap = 0;
        msiLo32.addr = 0x21800054;
        msiUp32.addr = 0x0;
        msiData.data = 0x0;
    
    
        statusCmd.dis = 1;
        statusCmd.memSp = 1;
        statusCmd.busMs =1;
    
        setRegs.msiIrqStatus[0] = &msiIrqStatus[0];
        setRegs.msiIrqEnableSet[0] = &msiIrqEnableSet[0];
        setRegs.msiCap = &msiCap;
        setRegs.statusCmd = &statusCmd;
        setRegs.msiLo32 = &msiLo32;
        setRegs.msiUp32 = &msiUp32;
        setRegs.msiData = &msiData;
    
        if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs)) != pcie_RET_OK)
        {
          PCIE_logPrintf ("pcieEnableInerupts_test fail\n");
          return retVal;
        }
    
        irqEOI.EOI = 0x4;
        setRegs.irqEOI = &irqEOI;
    
        if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs)) != pcie_RET_OK)
        {
          PCIE_logPrintf ("pcieEnableInerupts_test fail on EIO\n");
          return retVal;
        }
    
        PCIE_logPrintf("Enterupts enabled on EP\n");
        return pcie_RET_OK;
    
    
    }

    Manually i write registers like this:

          *((volatile uint16_t *) 0x2180204a)=0x1; //enable msi0
          *((volatile uint32_t *) 0x2180204c)=0x21800054; //set address low32
          *((volatile uint32_t *) 0x21802050)=0x0; //set address high32
          *((volatile uint16_t *) 0x21802054)=0x0; //set msi data reg

    Memorory before msi setup:

    after memset code

    after manual set

    anyhow huge thanks for your help, now this is allready atleast somewhat working.

    Tommi

  • Tommi,

    Great to know the MSI worked!

    I saw you used:

         *((volatile uint16_t *) 0x2180204a)=0x1; //enable msi0
          *((volatile uint32_t *) 0x2180204c)=0x21800054; //set address low32
          *((volatile uint32_t *) 0x21802050)=0x0; //set address high32
          *((volatile uint16_t *) 0x21802054)=0x0; //set msi data reg


    How did you get those offset 0x1048, 0x104c, 0x1050, 0x1054? That is from FPGA PCIE register manual? In TI PCIE user guide: 

    1050h MSI_CAP MSI Capabilities Register Section 3.6.2
    1054h MSI_LOW32 MSI Lower 32 Bits register Section 3.6.3
    1058h MSI_UP32 MSI Upper 32 Bits register Section 3.6.4
    105Ch MSI_DATA MSI Data Register (Offset is 0x1058 if 64-bit addressing not enabled) Section 3.6.5

    You see the offset are different for TI PCIE and your FPGA device for the MSI_CAP. That is the reason why TI memset method doesn't work. I looked at the PCIE specification. The configuration space register offset 0x0000 to 0x003c are standard across all the devices. Offset 0x34 is CAP_PTR and TI pointing to 0x40 "First Capability Pointer. By default, it points to Power Management Capability structure. Writable from internal bus interface.". The FPGA 0x34 also points to 0x40 offset, but it seems that is MSI region.

    Regards, Eric 

  • Hi

    Thanks for pointing those out. I now found out that someone had same problem with TI and xilinx combination here.

    "Long story short, both Ti's and Xilinx's way to define config space layout is standard compliant. Just incompatible with each other. Normally, operating system should traverse through capabilities linked list starting from CapPtr and build its own view of remote device capabilities. TI has hardcoded their Chip Support Library and Low Level Driver to work against another TI device. One have to develop own code to match config space layout in Xilinx FPGA."

    https://forums.xilinx.com/t5/PCIe-and-CPM/PCIe-endpoint-config-space-layout/td-p/591123

    So different register layout is here preventing the usage of that kind register writing in this case.

    Big thanks for your help. I think this case is now finally resolved.

    Tommi

  • Tommi,

    Thanks sharing this link! I also just realized the configuration space register layout are different between TI and Xilinx. For TI, offset 0x34 CAP_PTR pointing to 0x40. In 0x40, bit field PM_NEXT_PTR pointing to 0x50 that is MSI_CAP.

    In the FPGA, offset 0x34 CAP_PTR still pointing to 0x40. In 0x40, however bit field PM_NEXT_PTR pointing to 0x48 that is MSI_CAP. 

    Regards, Eric