About MSI functionality of PCIe

Hi Steven,

Thanks for your valuable information.
Now, I am trying to debug MSI interrupt handling as you described in 5th item above. Here is what I am doing:

In the cfg file, I am creating hardware interrupt statically:

var hwi0Params = newHwi.Params();
hwi0Params.instance.name = "hwi0";
hwi0Params.arg = 1;
hwi0Params.enableInt = false;
hwi0Params.eventId = 17;
hwi0Params.priority = 4;
Program.global.hwi0 = Hwi.create(5, "&hwi_isr", hwi0Params);

In my source code;

extern "C"
{
     Void hwi_isr(UArg arg)
     {
          //breakpoint is here; waiting for interrupt
          int deneme =0;
     }
}

/*
* ======== taskFxn ========
*/

Void taskFxn(UArg a0, UArg a1)
{
     System_printf("enter taskFxn()\n");
     Task_sleep(10);

      //write to MSI_IRQ_RAW_STATUS REG
      unsigned int *pMSIReg = (unsigned int*) 0x21800100;
     *pMSIReg = 0xfffffff1;

      System_printf("exit taskFxn()\n");
}

/*
* ======== main ========
*/
Void main()
{

Task_Handle task;
Error_Block eb;

System_printf( "enter main()\n");

Error_init(&eb);

task = Task_create(taskFxn, NULL, &eb);

if (task == NULL) {

System_printf("Task_create() failed!\n");

BIOS_exit(0);

}

Hwi_enableInterrupt(5);

//! initialize PCIe Module

if( PCIeModuleIntf->Initialize() == SUCCESS )
{
      //platform_write ("PCIe Module successfully initialized \n");
}
else
{
     //platform_write ("Failed to initialize PCIe Module \n");
}

BIOS_start();

/* enable interrupts and start SYS/BIOS */

}

I put a breakpoint in the  hwi_isr function, but PC never comes to that line. Is there anything I am missing at the code above ?
Do I need any extra thing to do for enabling MSI in RC side ?

Regards,
Koray.

Hi Steven,

I noticed that I forgot to enable MSI 0 interrrupt by setting MSI0_IRQ_ENABLE_SET register.
I did this modification and now interrupt handler works.

Regards,
koray.

Hi Steven,

Now, I am facing with a different problem.
Although I can get isr running when I set MSI_IRQ status or MSI0_IRQ_STATUS_RAW register manully, can't get same isr running with an EP at the other side. Here is what I am doing :

1. I am running default PCI Init procedure.
2. I am setting RC's BAR0 register to 0x20000000. I don't do any extra thing to enable inbound translation from RC side.
3. I set EP's INTX_DIS bit in STATUS_COMMAND register as 1 to disable legacy interrupt.
4. I read EP's MSI capability register.
   I read CAP_ID as 0x05h which is the required value. 
   I set MSI Enable bit as 1 to set MSI functionality.
   I set other fields of EP MSI Control register as 0.
5. I set EP's MSI Address low register as 0x20000054.
6. I set EP's MSI Address high register as 0.
7. I set EP's MSI Data register as 0 to invoke MSI 0 interrupt  when RC gets MSI TLP packet.

At the EP side, there is an FPGA IP core which is a Spartan-6 Integrated End-Point Block for PCIe (V1.4).
After setting these registers, I did a check-up if they keep the values that I adjusted,  by reading DSP's PCIe address space 0 regions of 
0x21801010 (RC BAR 0 register), 
0x21802004 (EP Cmd Register), 
0x21802048 (EP MSI Control Register), 
0x2180204C (EP MSI Addr Low Register), 
0x21802050 (EP MSI Addr High Register),  and
0x21802054 (EP MSI Data Register). And all the values were correct.

My questions are:

1. Do I miss anything at the procedure above ?
2. At first, I tried to set DSP's BAR 0 to the value of 0x21800000 as you suggested. But I couldn't. The BAR 0 register doesn't let me change the bits after Most Significant nibble. It changes to 0x20000000 even if I try to write 0x21800000. Why can't I change the bits of DSP's BAR 0 register other than the Most Significant nibble ?
3. Even BAR 0 register is adjusted to 0x20000000, I think that RC should accept incoming MSI TLP packets if their address is 0x20000054. But it seems not. Do you have any idea about the cause of this ?
4. FPGA IP core doesn't have any monitor signals that enables us to analyse MSI packets using ChipScope tool. We only see if DSP is setting EP's MSI Enable bit and some 'Configured Interrrupt Signals'. And they seem to be the same with IP Core's User Guide. 
   But still I am not sure if FPGA core sends packets correctly. Could you suggest a way to verify if FPGA sends packets correctly ? 
   

Regards,
Koray.

Koray,

1. MSI interrupt is basically a memory write transaction. First you need to make sure both RC and EP could initialize and accept the memory access by enabling the BUS_MS and MEM_SP bits in Status and Command Register on both RC and EP sides.

Then you need to do the memory write on EP side (write 0x0 to FPGA PCIe data space location which could have the PCIe address as 0x20000054 over the PCIe link). Please make sure the MSI write packet is transferring over the PCIe link with the PCIe address as 0x20000054, which could be accepted by RC BAR0. On FPGA side, it may not be necessary to just write to 0x20000054. I am not sure if FPGA PCIe core has the Outbound translation mechanism same as C66x DSP. But you need to check if the PCIe address (after translation or other mechanism in FPGA) of the memory write is 0x20000054.

2. The BAR0 could have large window size initialized in the BAR0 Mask Register, so most of the bits are masked and could not be changed in BAR0 register. You can refer to the BAR Mask Register section in the PCIe user's guide to setup the BAR0 mask register. But it may not be necessary if the memory access is working.

For #3 and #4, I think instead of the MSI interrupt write, you can try some regular memory transaction first between FPGA and DSP to make sure EP could do the memory access to RC correctly. You can try to setup the BAR1 in RC and do the regular memory transactions from EP to see if it is working first. For BAR1, you may need to setup the inbound translation on RC side to direct PCIe packets to the expected memory location.

Please let us know if you could make any progress. Thanks.

Sincerely,

Steven

Hi Steven,

We resolved the problem.
We noticed that somehow we were running FPGA with wrong code(interrupt functionality of FPGA IP Core were disabled).
It is because of there are different groups of people working with same board.

After correcting the FPGA code, now I can see my isr works.

And about your suggestions:

1. BUS_MS and MEM_SP bits in Status and Command Register on both RC and EP sides were set to 1.
2. It is not possible to do outbound translation from EP side(From EP to RC) for Xilinx IP core. It can only send completion TLP packets as response to RC's memory read requests.
3. I thought BAR Mask registers were only for configuration of EP's BAR registers during enumaration. Not for RC's own BAR registers. But after your suggestion, I checked it again, saw that RC's Bar registers also may have been configured.

Thank you very much for your help,
Regards,
Koray.

Dear Koray,

I am using SPARTAN6 FPGA as EP and I want the EP to interrupt C6678 (RC) once FPGA collects a block of data...you mentioned in step 2 above that EP can't initiate outbound transaction!...so how to interrupt the RC from the EP?

Regards,

Murad

Hi Murad,

EP can send MSI(Message Signaled Interrupt) to RC. No problem on that.
MSI is not an outbound transaction.

regards,
koray.

Thank you Koray for your prompt response!

So in the FPGA I need to enable MSI and somehow in the fpga I can send an MSI interrupt...great!

Do you mind give me an idea on hiw to write the fpga code to do so...i have ok knowledge of both pcie and fpga but managed to get the RC to perform outbound memory read successfully 

Regards,

Murad