Data Transfer over PCIe prevents further MSI Interrupts.

Charles Leech

Hello,

I'm trying to perform a loopback data transfer across a PCIe link, initiated and controlled by MSI interrupts. My code is based on the pdk example project.

I can successfully fire the first interrupt from EP to RC to cause the RC to transmit the data. This data is received by the EP.
However, when I try to fire another interrupt to signal to the RC that the EP is sending the data back, the interrupt does not cross the link and get received by the RC.

I am using the following lines of code to configure OB translation to point to the RC BAR0, write the vector to the MSI_IRQ register to trigger the interrupt and then point the OB trans back to its previous value to enable data transfer:

/*Move Outbound Translation to trigger MSI; 0x7000_0000 */
pcieObTransCfg(0x70000000, PCIE_OB_HI_ADDR_S, PCIE_OB_REGION_S);

System_printf("Interrupt Sent\n");
*((volatile uint32_t *) 0x60000054) = 0x0;

/*Move Outbound Translation back to sending data; 0x9000_0000*/
pcieObTransCfg(0x90000000, PCIE_OB_HI_ADDR_S, PCIE_OB_REGION_S);

I was wondering whether the method of directly writing a value to an address, using the PCIe data space, is not the best approach

Is it possible that when a data transfer occurs, this address is displaced to a higher location or some other change in its function?

The ISR reset and all other initialisation code on the RC is fault free as the data transfer can be delayed behind multiple interrupts and they all pass. The program only falters if the interrupt occurs after a data transfer.

Any suggestions of either and alternative system to trigger the interrupt across the link or reason for this phenomenon occurring would be appreciated.

Thx
Charles

over 12 years ago

0 Steven Ji over 12 years ago

TI__Genius 12065 points

Charles,

First of all, I think you just omit the "handle" input for the "pcieObTransCfg()" in your post, but you have it in your real code, right?

1. You shared the code for the first interrupt generation from EP to RC and I assume you would repeat the sequence again for the second MSI generation, is it correct?

2. I assume you take care of the MSIn_IRQ_STATUS and IRQ_EOI registers in the ISR for re-triggering the MSI. Could you confirm that as well?

3. In EP, instead of using single OB translation set and switching it back and forth, we could use multiple OB translations sets and map one to the BAR0 of RC for MSI and map another one for the BAR1 of RC for data transfer.

For example, if you set the OB_SIZE to be 0x0 which means it is 1MB between each outbound translation region. Then any access to the PCIe data space region 0x6000_0000 to 0x600F_FFFF (first 1MB region) will be translated with PCIe address defined in Outbound translation register set 0 (OB_OFFSET_INDEX0 / OB_OFFSET0_HI).

And any access to the PCIe data space region 0x6010_0000 to 0x601F_FFFF (second 1MB region) will be translated with PCIe address defined in Outbound translation register set 1 (OB_OFFSET_INDEX1 / OB_OFFSET1_HI).

So you could map OB set 0 to BAR0 of RC and map OB set 1 to BAR1 of RC (or opposite, no restriction for the order) and you could access both data and application registers regions from EP to RC.

For the Outbound translation, please refer to the PCIe user case document (SPRABK8) for more details.

0 Charles Leech over 12 years ago in reply to Steven Ji

Prodigy 200 points

Steven,

I have attached the projects I am working on, at the point when I posted my questions, so you can see my situation more clearly. Its probably easier than trying to talk about what I have done.

2063.PCIE_MSI_Transfer.zip

When I introduced a task structure to the project, I made the handle global for ease of testing. It is still used for the Pcie_cfgObOffset and Pcie_readRegs/Pcie_writeRegs function calls in the pcieObTransCfg function.
The three lines I posted above are in the EPReady function and are called every time I want to trigger the interrupt.
Upon exiting the hwi_isr, the function runs msiIrqStatus[0].msiIrqStatus = 0x1 and irqEOI.EOI = 0x4 as you suggested in the following thread and that appears to work fine.

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/181633.aspx

In the meantime, I will try the system you suggest with multiple OB translation sets and get back to you if that works. I may have questions for you about how to tell if that is working if its doesn't immediately solve the interrupt re0triggering problem.

Thx
Charles

0 Charles Leech over 12 years ago in reply to Charles Leech

Prodigy 200 points

I have successfully configured the two translation regions as you suggested:

RC BAR0 with inbound address 0x8000_0000 handles MSI communication in translation region 1 (8MB regions).
RC BAR1 with inbound address 0x9000_0000, outbound address 0x7000_0000 handles data transfer from EP to RC in translation region 0.
EP BAR1 with inbound address 0x7000_0000, outbound address 0x9000_0000 handles data transfer from RC to EP in translation region 0.
RC has OB translation for region 0 to send data to EP.
EP has OB translation for region 0 to send data to RC.
EP has OB translation for region 1 to send MSI interrupts to RC.

I now use *((volatile uint32_t *) 0x60800054) = 0x0 to trigger the interrupt.

This code works but still only as it did before where interrupts are fired successfully until a data transfer occurs.

Is there a way to direct data transfer to a certain translation region other than using direct addressing, such as with the Pcie_getMemSpaceRange function? This seems to map the pcieBase pointer to 0x6000_0000 by default, presumably because it is the beginning address of the pcie data space.

Thanks
Charles

0 Steven Ji over 12 years ago in reply to Charles Leech

TI__Genius 12065 points

Charles,

I ran your example code in my setup. I think the issue is because of the interrupt flag variable "launched" in the RC project.

It is stored in L2SRAM and is being cached by the core. The code may not get the correct status (0 or 1) based on this variable.

If you add the "volatile" in front of the definition as follows, the MSI+Data transfer should work.

"uint32_t volatile launched, Echoindex;"

I attached the print out in my setup after the modification. I think all interrupts are being recognized by RC and the test could complete.

Please let us know if it is working for you.

[C66xx_0] Starting link training...
[C66xx_0] Link is up.
[C66xx_0] Interrupt Sent
[C66xx_0] Link is up.
[C66xx_0] Passed 1
[C66xx_0] Passed 2
[C66xx_0] Push data to the EP (then verify that it is echoed back)
[C66xx_0] Data sent to EP
[C66xx_0] Interrupt Sent
[C66xx_0] Waiting for data from the RC
[C66xx_0] End Point received data:
[C66xx_0] EPdataStore.content[0] = 0
[C66xx_0] EPdataStore.content[1] = 1
[C66xx_0] EPdataStore.content[3] = 3
[C66xx_0] EPdataStore.content[7] = 7
[C66xx_0] EPdataStore.content[15] = 15
[C66xx_0] EPdataStore.content[31] = 31
[C66xx_0] EPdataStore.content[63] = 63
[C66xx_0] EPdataStore.content[127] = 127
[C66xx_0] Interrupt Sent
[C66xx_0] End Point sent data to Root Complex
[C66xx_0] Passed 3
[C66xx_0] Waiting for data from the EP
[C66xx_0] Root Complex received data.
[C66xx_0] EchodataStore.content[0] = 0
[C66xx_0] EchodataStore.content[1] = 1
[C66xx_0] EchodataStore.content[3] = 3
[C66xx_0] EchodataStore.content[7] = 7
[C66xx_0] EchodataStore.content[15] = 15
[C66xx_0] EchodataStore.content[31] = 31
[C66xx_0] EchodataStore.content[63] = 63
[C66xx_0] EchodataStore.content[127] = 127
[C66xx_0] Data Verified.
[C66xx_0] Test passed.

0 Charles Leech over 12 years ago in reply to Steven Ji

Prodigy 200 points

Steven,

Thank you very much, that change has worked for me too.

I had in fact just asked a colleague to look through my code as well and they recommended making that alteration.

I was just about to post on here to tell you but it looks like you were on to it and solved it quicker than me.

Now I'm only having problems with my EDMA/QDMA functions :).

Many Thanks

Charles

Processors

Processors forum

Data Transfer over PCIe prevents further MSI Interrupts.