Can only enable 1 MSI with PCIe

Bill

I am trying to write a kernel module that will handle multiple different MSI interrupts generated from a single EP. During boot I see this:

[ 16.741897] keystone-pcie: pcie - number of legacy irqs = 4
[ 16.741940] keystone-pcie: pcie - number of MSI host irqs = 8, msi_irqs = 32

I do not necessarily need 32 MSI irqs but why does it show MSI host irqs = 8?

In my kernel module, I call pci_enable_msi_block(dev, 32) and it returns 4 to me. I assume this means that I should then call pci_enable_msi_block(dev, 4) in order to enable 4 MSI interrupts, unfortunately this call returns 1. Which I assume means that the device can only enable 1 MSI interrupts (why would it return 4 before?). I then call pci_enable_block(dev, 1) and sucesfully enable 1 MSI. How do I go about enabling more MSI? Am I correct that since I am enabling host side interrupts, it should not matter how many MSI are enabled on the EP?

lspci -v before the kernel module is inserted:

00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Memory at <ignored> (32-bit, non-prefetchable)
        Memory at <ignored> (32-bit, prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 50000000-501fffff
        Prefetchable memory behind bridge: 50200000-503fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport

01:00.0 Memory controller: Xilinx Corporation Device 7022
        Subsystem: Xilinx Corporation Device 0007
        Flags: fast devsel
        Memory at 50000000 (32-bit, non-prefetchable) [disabled] [size=1M]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable- Count=1/4 Maskable- 64bit+
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting

lspci -v after kernel module is inserted:

01:00.0 Memory controller: Xilinx Corporation Device 7022
        Subsystem: Xilinx Corporation Device 0007
        Flags: bus master, fast devsel, latency 0, IRQ 576
        Memory at 50000000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [80] Power Management version 3
        Capabilities: [90] MSI: Enable+ Count=4/4 Maskable- 64bit+
        Capabilities: [c0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: Custom_Module

Any help would be appreciated, this is kind of putting development at a standstill due to our need for multiple MSI interrupts. I have also attatched the relevent code from my kernel module if that is needed.

5734.module.c

over 11 years ago

0 Bill over 11 years ago

Intellectual 645 points

I see in /driver/pci/msi.c there is a comment in default_setup_msi_irqs() that says

/*
* If an architecture wants to support multiple MSI, it needs to
* override arch_setup_msi_irqs()
*/

Which does not happen. Is there support for an arch_setup_msi_irqs() function for the keystone II so that I can enable multiple MSI?

0 Raja over 11 years ago in reply to Bill

TI__Guru* 81335 points

Hi Bill,

We are working on your query. We will get back to you.

Thanks.

0 Bill over 11 years ago in reply to Raja

Intellectual 645 points

Has there been any resolution to this issue? Only being able to use 1 MSI could be a very debilitating issue.

0 Bill over 11 years ago in reply to Bill

Intellectual 645 points

Will someone please respond to this issue?

0 Bill over 11 years ago in reply to Bill

Intellectual 645 points

I am assuming no one has bothered to look into this. As you can see from this post:

http://e2e.ti.com/support/embedded/linux/f/354/p/349084/1261525.aspx#1261525

There is record of issues with the Keystone II PCIe driver. Can you put one of your people on it and get it fixed. I am fairly confident that it is a TI issue (just like I was in the above post).

I have been working around this issue on our project for months now, but it would make it so much easier if it worked as it was supposed to.

0 lding over 11 years ago in reply to Bill

TI__Guru* 95265 points

What TI device is it? What version of Linux kernel?

Regards, Eric

0 Bill over 11 years ago in reply to lding

Intellectual 645 points

Keystone II 6638 EVMK2X Running 3.8.4 kernel source compiled with keystone2_defconfig

0 Bill over 10 years ago in reply to Bill

Intellectual 645 points

Nothing?

0 Bill over 10 years ago in reply to lding

Intellectual 645 points

It has been a month an a half since I answered this. Any resolution? Me and my team have lost many weeks debugging errors that you have had in both your PCIe source, and your IPC source. Please get back to me on this issue.

0 Shehzad Ahmed over 9 years ago in reply to Bill

Intellectual 260 points

Is there any solution to that?

0 Bill over 9 years ago in reply to Shehzad Ahmed

Intellectual 645 points

Shehzad,

I have not tried this but it sounds like a reasonable solution.

e2e.ti.com/.../424995

0 Shehzad Ahmed over 9 years ago in reply to Bill

Intellectual 260 points

thnx Bill,

I have tried this but this doesn't work. So what else you did to support multiple MSI or . My code is as:



for (i = 0; i < 4 ; i++)

   {

     printk(KERN_DEBUG "enabling MSI %d\n", i+1);

     ret = pci_enable_msi_block(pdev,1);

     printk(KERN_DEBUG "enabed MSI %d\n", i+1);

     irq_num[i] =pdev->irq;

     printk(KERN_DEBUG "irq%d has line number = %d\n", i+1, pdev->irq);

     if(ret != 0)

     {

       printk(KERN_DEBUG "Unable to allocate MSI 1-16 return value is %d\n", ret);

       return ret;

     }

     else

     {

      printk(KERN_DEBUG "Successfully allocated MSI %d \n",i+1);

     }

Output:

enabling MSI 1

enabed MSI 1

irq1 has line number = 576

Successfully allocated MSI 1

enabling MSI 2

[ Crash].

0 Bill over 9 years ago in reply to Shehzad Ahmed

Intellectual 645 points

I multiplexed multiple interrupts onto a single MSI. I was able to do this because the hardware end point was an FPGA and I could configure the FPGA to set a flag with which interrupt it was. Clearly this will not work with many hardware platforms, and also is not as efficient as multiple MSI (since I have to read a register over the PCIe bus to identify which interrupt it was).

It is surprising to me that the code you presented above crashes, I haven't looked deep into the code, but I believe that pci_enable_msi_block() should just return a negative value if it fails. Maybe try just pci_enable_msi()? Also take a look at drivers/pci/host/pcie-keystone.c that is TI's PCIe driver implementation. One more note, make sure you Device tree provides support for more than 1 MSI.

0 Shehzad Ahmed over 9 years ago in reply to Bill

Intellectual 260 points

Hi Bill thnx for your help.

When I press "lspci -v" I get this

00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01) (prog-if 00 [Normal decode])

Flags: bus master, fast devsel, latency 0

Memory at <ignored> (32-bit, non-prefetchable)

Memory at <ignored> (32-bit, prefetchable)

Bus: primary=00, secondary=01, subordinate=01, sec-latency=0

I/O behind bridge: 00001000-00001fff

Memory behind bridge: 50000000-501fffff

Prefetchable memory behind bridge: 50200000-503fffff

Capabilities: [40] Power Management version 3

Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+

Capabilities: [70] Express Root Port (Slot-), MSI 00

Capabilities: [100] Advanced Error Reporting

It shows there is only one MSI for Root port. But when I see device tree it shows 8 MSI's. What does that mean? Also I tried with pci_enable_msi() but I got the same result.

Processors

Processors forum

Can only enable 1 MSI with PCIe