PCIe interrupt - linux_host_loader - C6670

Aymeric,

For the problem, both legacy PCI interrupt and MSI interrupt in PCIe module are in-band message, which is similar to memory write over PCIe link to the other end. There is no physical interrupt pins connected to PCIe bus, which is different from the conventional PCI bus.

And we could not enable both legacy interrupt and MSI interrupt at the same time. Only one mechanism could be enabled on EP and this example chooses to use legacy interrupt.

Do you know if the host can enter into the ISR_Handler () please? 

The host code will register IRQ for C66x PCIe interrupt:

printk("Registering the irq %d ...\n", irqNo);
request_irq(irqNo, ISR_handler, IRQF_SHARED, "TI 667x PCIE", &dummy);

And the host should be able to enter into the ISR handler if EP sends interrupt to host.

And may I ask if the EP_IRQ_STATUS (0x2180006C) register on EP shows 0x1 after the EP sends interrupt to host ((PCIE_EP_IRQ_SET) = 0x1)?

The host ISR handler will check the IRQ STATUS first (status = HAL_CheckPciInterrupt()). If no flag found on EP side, the host ISR will not trigger EDMA transfer.

Another thing is there is 1sec delay on the host side waiting for DSP ISR to generate interrupt to host.

/* Waiting DSP ISR handler to process the interrupt from DSP and then generates an interrupt to host
Waiting for host ISR handler to process the interrupt from DSP before program exits */
mdelay(1000);

I am not sure if it is too short in your case. But ideally the delay should be long enough. 

For the question, it is possible to use MSI interrupt or use PCIe write to modify EDMA registers to trigger EDMA interrupt on DSP. 

This example just chooses to use legacy interrupt. But the users could use other mechanism in their real applications.

Hey steven,

Thanks for your prompt answer, it is highly appreciated.

As far as I can see the interrupt subroutine is never serviced by the host. The PCIE_EP_IRQ_SET is set to 1 but I assume the legacy interrupt assert message is never received on the host side.

The delay is fine (1s for dsp to revert 2 Meg of data is a HUGE time) and I can tell because if I call HAL_readDMA from outside the interrupt subroutine, all is going ok.

I will talk to the HW engineer to see if the CPI switch between the DSP and the host could cause this problem. But any other suggestion that you have from the point of view of software will be appreciated.

ex: check some PCI registers or check linux to see if counters for interrupt increases.

Aymeric

Aymeric,

Do you mean the PCIE_EP_IRQ_STATUS (0x2180006c) is set to 1 as well?

You can add one "printk()" or other output indicator in "ISR_handler()" before checking "if (status ==1)". If the host could print out something means it could enter into the ISR, but the status is not equal to 1.

If the ISR in host could not be triggered, could you compare the debug info of your EDMA-Interrupt example execution with the example output in the README file as below please, especially for the interrupt/IRQ info?

[ 86.781006] Finding the device....
[ 86.781020] Found TI device
[ 86.781021] TI device: vendor=0x104c, dev=0xb005, irq=0x0000000b
[ 86.781022] Reading the BAR areas....
[ 86.781537] Enabling the device....
[ 86.781550] pci 0000:04:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 86.781555] pci 0000:04:00.0: setting latency timer to 64
[ 86.781565] Access PCIE application register ....
[ 86.781568] Registering the irq 11 ...
[ 86.781583] Allocating consistent memory ...
[ 86.788306] Boot entry address is 0x 82e300
[ 86.791065] Total 5 sections, 0xf358 bytes of data written to core 0
[ 88.846147] Write DMA to DSP ...
[ 88.858308] Generating interrupt to DSP ...
[ 88.982125] Interrupt 11 received from DSP
[ 88.982126] Read DMA from DSP ...
[ 88.997980] DMA test passed!
[ 89.870917] DMA write throughput is: 328.38 MB/s
[ 89.870918] DMA read throughput is: 341.64 MB/s
[ 89.870919] Freeing consistent memory ...

Hey Steven,

Thanks for your help, here is a full log of what is happening and I hope you will get the info you are looking for:

At boot time, all the register we are looking at are all 0.

then I run the example that I provided you with the log.

1) I never received the interrupt on the host otherwise the following would have happen:

   printk("ISR_handler with Interrupt %d received\n", irq);

   if (status == 1)
   {
      printk("Interrupt %d received from DSP\n", irq);
      printk("Read DMA from DSP ...\n");

2) the interrupts seem to be registered properly

  5:          0          0   IO-APIC-edge      TI 667x PCIE
 10:          0          0   IO-APIC-edge      TI 667x PCIE

3) you seem to be surprised that EP_IRQ_STATUS was to 1 but , 0x64, 0c66, 0x6c are all set to one. According to the doc, it seems logical as the interrupt is asserted. I am not sure what "function 0" is in "1 = Interrupt for function 0 is asserted" but it looks ok.

4) all the other registers seems to be ok. I even check that all MSI were disabled (0x100 to 0x180) and the capabilities show 0x00807005 which indicates that the MSI capability is not even enabled.

Not sure where to look anymore. According to the HW guy, the PCI switch is set in transparent mode so it should not stop the message from going through.

Is there a quick way to replace the PCI_LEGACY_INT by MSI_INTERRUPT and see if it is working (code on DSP and host)?

Aymeric

Hey Steven,

I was hopping to get some feedback from you earlier but I am now trying to port the existing TI example to use the msi interrupts. However, I have some problem understanding and the documentation (section 2.13.4 and 2.13.5.2) is not really explicit.

You have been quite good at explaining things so I hope you can help again or refer me to some other examples or notes.

this is what I am doing and I know I must be doing:

1) before enumeration, inside the IBL Setting bit [10] of Command Register (0x004h in the configuration space)  to disable Legacy interrupt. And set the bit [16] of the MSI Capability Register Set (0x050h in configuration space), to enable MSI capabilities.

2) then, still before the enumeration finishes I read the following registers: PCIE_REG_MSG_MSI_LOW32, PCIE_REG_MSG_MSI_HI32, PCIE_REG_MSG_MSI_DATA

then, I am not sure how to do the following:

--------------------------

4. Determine the number of MSI vectors allocated (and the number requested) to
the device ====> I think I would like 32 possible channels
5. Depending upon the number of MSI interrupts allocated, issue a memory write
transaction with the address the same as MSI Address Register and the data the
same as the MSI Data Register. In the data, the LSBs can be modified to reflect the
appropriate MSI event that needs to be notified to root complex
6. The memory write transaction can also be optionally routed through the
outbound address translation interface if the destination PCIe address is not
directly accessible.

----------------------

Supposing that the code in the dsp runs on core 0 and I want to be able to "generate" MSI interrupt from the RC. Assuming that since MSI capability is now enable, how do I generate an interrupt? Documentation says that writing to MSI0_IRQ_STATUS_RAW should only be done for debug purpose.

How will I intercept the interrupt on the dsp? And what should the event ID? Should it be 17 or 18 (C6670)

----

    /* Open the INTC Module for Vector ID: 4 and Event ID:?????
     *     Refer to the interrupt architecture and mapping document for the Event ID  (INTC0_OUT3)*/
    vectId = CSL_INTC_VECTID_4;
    hTest = CSL_intcOpen (&intcObj, ???????, &vectId , NULL);
    if (hTest == NULL)
    {
       printDebug("Error: GEM-INTC Open failed\n\r",NO_COMMENT);
       XMC8320_CLIENT_WriteErrorField(4);
       return;
    }

    /* Register an call-back handler which is invoked when the event occurs. */
    EventRecord.handler = &test_isr_handler;
    EventRecord.arg = 0;
    if (CSL_intcPlugEventHandler(hTest,&EventRecord) != CSL_SOK)
    {
      printDebug("Error: GEM-INTC Plug event handler failed\n\r",NO_COMMENT);
      return;
    }

    /* Enabling the events. */
    if (CSL_intcHwControl(hTest,CSL_INTC_CMD_EVTENABLE, NULL) != CSL_SOK)
    {
        printDebug("Error: GEM-INTC CSL_INTC_CMD_EVTENABLE command failed\n\r",NO_COMMENT);
      return;
    }
-----

Finally the DSP now needs to send a MSI interrupt to the RC, is writing 0 in MSI Interrupt IRQ Register enough to trigger this message?

Thanks in advance

Aymeric

Aymeric,

Sorry that I could not reply earlier. I was checking with our software team on this interrupt example. In their setup, the Linux host could get interrupt from DSP, so the code on DSP side should be fine. But I am not sure if anything special needs to be configured on the Linux host side in order to receive the PCIe legacy interrupt correctly since I am not quite familiar with Linux kernel setup.

Regarding to your MSI interrupt questions, 

1. You are correct that we can disable legacy interrupt by clearing INTX_DIS (bit10) in STATUS_COMMAND register (0x21801004) and enable MSI interrupt by setting MSI_EN (bit16) in MSI_CAP register (0x21801050).

And if DSP needs to receive MSI interrupt from Linux host, we also need to enable MSI vector (MSIn_IRQ_EN field) in MSIn_IRQ_ENABLE_SET register. For example, if both MSI_0 and MSI_8 are being used, we need to set MSI0_IRQ_EN_SET=0011b.

If Linux host (RC) needs to receive the MSI interrupt from DSP (EP), the RC side may need to enable the MSI reception similar as EP (discussed above). But coding may be dependent on the Linux driver. 

2. I think those registers are assigned to EP by RC during enumeration. So you can read those registers on EP just before you want to trigger MSI interrupt from EP to RC (host).

For 4, 5, 6, there are some posts discussing about the similar question (between two C667x devices):

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/p/194406/841403.aspx#841403

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/166202.aspx

For the DSP and Linux host example, please take a look at the link below, which has the PCIe example for DSPC-8681 device (four C6678 EVM) working with Linux (has Windows driver as well) . 

http://support.advantech.com.tw/Support/SearchResult.aspx?keyword=DSPC-8681&searchtabs=BIOS,Certificate,Datasheet,Driver,Firmware,Manual,Online%20Training,Specification,Utility,FAQ,Installation,Software%20API,Software%20API%20Manual&select_tab=Driver

I think there is one MSI interrupt example for both PC host and DSP in the example "Lightning_PCIE\examples\ipc\". 

You can take a look at the user guide below:

http://support.advantech.com.tw/Support/SearchResult.aspx?keyword=DSPC-8681&searchtabs=BIOS,Certificate,Datasheet,Driver,Firmware,Manual,Online%20Training,Specification,Utility,FAQ,Installation,Software%20API,Software%20API%20Manual&select_tab=Manual

Hope they could help as well.

For the interrupt controller setup on C6670, there is one typo in the current data manual that event 18 should be "PCIEXpress_MSI_INTn+4" instead of "PCIEXpress_MSI_INTn+1" (which will be fixed in the next data manual release).

So we have 8 MSI interrupt from PCIe module in C6670, event 17 "PCIEXpress_MSI_INTn" means Core0 (n=0) will accept MSI_INT0 (with MSI vectors 0/8/16/24) and event 18 means Core0 will accept MSI_INT4 as well (with MSI vectors 4/12/20/28). So we have 8 vectors assigned to each Core (4 cores with total 32 MSI vectors).

In C6678, we have only event 17, so only 4 vectors assigned to each core (8 cores with total 32 MSI vectors).

And another thing is in C667x device, the PCIe MSI interrupt is connected directly to CorePac interrupt controller, while the legacy interrupt is routed to chip-level interrupt controller (CIC) and then to CorePac INTC. So in the MSI interrupt setup, we do just need to setup CorePac INTC only.

In your example, if RC wants to generate MSI interrupt to DSP using MSI vector 0, we have have event ID #17 in the code:

hTest = CSL_intcOpen (&intcObj, 17, &vectId , NULL);

Similary, if RC want to generate MSI interrupt to DSP (C6670) using MSI vector 4, we have have event ID #18 in the code.

If RC wants to generate MSI interrupt to EP (although not specified by PCIe standard protocol), the RC could write MSI vector value into MSI_IRQ register of EP, which is specific to C667x device. We should have some discussion in the E2E post I listed above.

If EP wants to generate MSI interrupt to RC (Linux host), please take a look at the PCIe example for DSPC-8681 device (link posted above) to see how the MSI could be triggered on Linux side. 

Thanks so much. You provided all the answers to my questions. I really like the way you pass on your knowledge, this is a very important skill to have.

It will take me a few days to implement the following (there is a lot of information) and I will get back to this post mid next week and close it (or ask more concrete questions)

Aymeric

Hey steven,

Sorry it took me such a long time to get back to you. Thanks to your answers, I was able to make the MSI interrupts work both ways. I will provide you what I did as perhpas it will help the community,

From host to target (linux -> C6670), it was pretty simple as it is very similar to legacy interrupts. As long as you clear the MSI interrupt in the isr by doing the following: DEVICE_REG32_W(IRQ_EOI_ADDRESS, PCIE_MSI0_INTERRUPT_EVENT); where  PCIE_MSI0_INTERRUPT_EVENT = 4, all is going well.

The other way around was a bit more difficult and I would like to inform you that the "Lightning_PCIE" example is not a good source for target -> host. I found this note on the internet http://www.pzk-agro.com/0321156307_ch09lev1sec5.html which was what I was looking for. At the end, interrupting the host is again an outbound write to the address that linux provides you with the data that linux provides you. This is how I got it to work:

   int dataVal , addVal;

   SSP_UINT32 writeAddress; /* the address to write to in DSP side */
   SSP_UINT32 baseAddress;  /* use for the outbound translation */

/* here are the information that the host (RC) has provided me when I enabled the MSI interrupt*/
    dataVal = DEVICE_REG32_R(MSI_DATA_ADDRESS);
    addVal = DEVICE_REG32_R(MSI_LOW32_ADDRESS);

    /* to generate an interrupt to the host, this must be done:
     * issue a memory write transaction with the address the same
     * as MSI Address Register and the data the same as the MSI Data
     * Register. In the data, the LSBs can be modified to reflect the
     * appropriate MSI event that needs to be notified to root complex => see query at the end
     */

    baseAddress = addVal&PCIE_1MB_BITMASK;
   DEVICE_REG32_W(OB_OFFSET_INDEX_ADDRESS(0), (baseAddress|0x1)); /* enable outbound translation */
   DEVICE_REG32_W(OB_OFFSET_HI_ADDRESS(0), 0); /* not used as it is a 32 bit address */

   writeAddress = PCIE_DATA_ADDRESS + (addVal & ~PCIE_1MB_BITMASK);
   
    /* clear PCIE interrupt status*/
    DEVICE_REG32_W(MSI0_IRQ_STATUS_ADDRESS, regVal);

    /* clear the interrupt */
    DEVICE_REG32_W(IRQ_EOI_ADDRESS, PCIE_MSI0_INTERRUPT_EVENT);

    /* generate interrupt to host */
    DEVICE_REG32_W(writeAddress, dataVal);

And this worked. When you know what to do (thanks again), it is not that complicated.

So I have one more question to complete this query: the MSI Capabilities Register provides two sections, the MULT_MSG_EN and the MULT_MSG_CAP. As an end point, should I advertise how many MSI interrupt I can handle in MULT_MSG_CAP; Let say 32.

Then the RC, when enabling the MSI functionalities would let me know how many it can handle by setting MULT_MSG_EN (let say 4)?

Therefore, I would write dataVal , dataVal+1, dataVal+2, dataVal+3 at the provided address to generate four different interrupts?

Thanks

Aymeric

The

  /* clear PCIE interrupt status*/
    DEVICE_REG32_W(MSI0_IRQ_STATUS_ADDRESS, regVal);

    /* clear the interrupt */
    DEVICE_REG32_W(IRQ_EOI_ADDRESS, PCIE_MSI0_INTERRUPT_EVENT);

operations in the middle of this description (great BTW), are not required to send an interrupt from the EP to RC, since those operations only apply to interrupt events coming in to the EP.

Am I missing something?

- Andrew

Andrew,

I think you are right that those operations are not necessary to generate interrupt.

They are necessary after receiving the interrupt to clear the previous interrupt status in order to accept the same event later.

Great. That is what I thought. Thanks for the prompt reply.

- Andrew