This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Can only enable 1 MSI with PCIe

I am trying to write a kernel module that will handle multiple different MSI interrupts generated from a single EP. During boot I see this:

[   16.741897] keystone-pcie: pcie - number of legacy irqs = 4
[   16.741940] keystone-pcie: pcie - number of MSI host irqs = 8, msi_irqs = 32

I do not necessarily need 32 MSI irqs but why does it show MSI host irqs = 8?

In my kernel module, I call pci_enable_msi_block(dev, 32) and it returns 4 to me. I assume this means that I should then call pci_enable_msi_block(dev, 4) in order to enable 4 MSI interrupts, unfortunately this call returns 1. Which I assume means that the device can only enable 1 MSI interrupts (why would it return 4 before?). I then call pci_enable_block(dev, 1) and sucesfully enable 1 MSI. How do I go about enabling more MSI? Am I correct that since I am enabling host side interrupts, it should not matter how many MSI are enabled on the EP?

 

lspci -v before the kernel module is inserted:

00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Memory at <ignored> (32-bit, non-prefetchable)
        Memory at <ignored> (32-bit, prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 50000000-501fffff
        Prefetchable memory behind bridge: 50200000-503fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport

01:00.0 Memory controller: Xilinx Corporation Device 7022
        Subsystem: Xilinx Corporation Device 0007
        Flags: fast devsel
        Memory at 50000000 (32-bit, non-prefetchable) [disabled] [size=1M]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/4 Maskable- 64bit+
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting

 

lspci -v after kernel module is inserted:

00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Memory at <ignored> (32-bit, non-prefetchable)
        Memory at <ignored> (32-bit, prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 50000000-501fffff
        Prefetchable memory behind bridge: 50200000-503fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport

01:00.0 Memory controller: Xilinx Corporation Device 7022
        Subsystem: Xilinx Corporation Device 0007
        Flags: bus master, fast devsel, latency 0, IRQ 576
        Memory at 50000000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=4/4 Maskable- 64bit+
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: Custom_Module

 Any help would be appreciated, this is kind of putting development at a standstill due to our need for multiple MSI interrupts. I have also attatched the relevent code from my kernel module if that is needed.

5734.module.c

  • I see in /driver/pci/msi.c there is a comment in default_setup_msi_irqs() that says

    /*
      * If an architecture wants to support multiple MSI, it needs to
      * override arch_setup_msi_irqs()
      */

    Which does not happen. Is there support for an arch_setup_msi_irqs() function for the keystone II so that I can enable multiple MSI?

  • Hi Bill,

    We are working on your query. We will get back to you.

    Thanks.

  • Has there been any resolution to this issue? Only being able to use 1 MSI could be a very debilitating issue.

  • Will someone please respond to this issue?

  • I am assuming no one has bothered to look into this. As you can see from this post:

    http://e2e.ti.com/support/embedded/linux/f/354/p/349084/1261525.aspx#1261525

    There is record of issues with the Keystone II PCIe driver. Can you put one of your people on it and get it fixed. I am fairly confident that it is a TI issue (just like I was in the above post).

    I have been working around this issue on our project for months now, but it would make it so much easier if it worked as it was supposed to.

  • What TI device is it? What version of Linux kernel?

    Regards, Eric

  • Keystone II 6638 EVMK2X Running 3.8.4 kernel source compiled with keystone2_defconfig

  • Nothing?

  • It has been a month an a half since I answered this. Any resolution? Me and my team have lost many weeks debugging errors that you have had in both your PCIe source, and your IPC source. Please get back to me on this issue.
  • Is there any solution to that?
  • Shehzad,

    I have not tried this but it sounds like a reasonable solution. 

    e2e.ti.com/.../424995

  • thnx Bill,

    I have tried this but this doesn't work.  So what else you did to support multiple MSI or . My code is as:



    for (i = 0; i < 4 ; i++) { printk(KERN_DEBUG "enabling MSI %d\n", i+1); ret = pci_enable_msi_block(pdev,1); printk(KERN_DEBUG "enabed MSI %d\n", i+1); irq_num[i] =pdev->irq; printk(KERN_DEBUG "irq%d has line number = %d\n", i+1, pdev->irq); if(ret != 0) { printk(KERN_DEBUG "Unable to allocate MSI 1-16 return value is %d\n", ret); return ret; } else { printk(KERN_DEBUG "Successfully allocated MSI %d \n",i+1); } Output: enabling MSI 1 enabed MSI 1 irq1 has line number = 576 Successfully allocated MSI 1 enabling MSI 2 [ Crash].

  • I multiplexed multiple interrupts onto a single MSI. I was able to do this because the hardware end point was an FPGA and I could configure the FPGA to set a flag with which interrupt it was. Clearly this will not work with many hardware platforms, and also is not as efficient as multiple MSI (since I have to read a register over the PCIe bus to identify which interrupt it was).

    It is surprising to me that the code you presented above crashes, I haven't looked deep into the code, but I believe that pci_enable_msi_block() should just return a negative value if it fails. Maybe try just pci_enable_msi()? Also take a look at drivers/pci/host/pcie-keystone.c that is TI's PCIe driver implementation. One more note, make sure you Device tree provides support for more than 1 MSI.
  • Hi Bill thnx for your help.

    When I press "lspci -v" I get this

    00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01) (prog-if 00 [Normal decode])

           Flags: bus master, fast devsel, latency 0

           Memory at <ignored> (32-bit, non-prefetchable)

           Memory at <ignored> (32-bit, prefetchable)

           Bus: primary=00, secondary=01, subordinate=01, sec-latency=0

           I/O behind bridge: 00001000-00001fff

           Memory behind bridge: 50000000-501fffff

           Prefetchable memory behind bridge: 50200000-503fffff

           Capabilities: [40] Power Management version 3

           Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+

           Capabilities: [70] Express Root Port (Slot-), MSI 00

           Capabilities: [100] Advanced Error Reporting

    It shows there is only one MSI for Root port. But when I see device tree it shows 8 MSI's. What does that mean? Also I tried with pci_enable_msi() but I got the same result.