C6678 MSI functionality of PCIe

hi Steven:

    Thanks for your reply. I know the answer to the question. 

Hi Steven,

I am having the same problems as qbliu and I understand your explanation of the process of MSI but I still don't understand how to turn it into lines of code.

My hardware setup is 2 C6678 DSP EVM boards connected over PCIe using the TI Dual EVM Breakout Card. I am using the emulation interface on the board and a target configuration in CCSv5 that specifies both EVMs and allows me to program a core of each EVM with either the RC or EP program. I have had no problems with getting/maintaining a link with this setup.

My program is based on the PCIe example project provided in the pdk for the C6678 but am attempting to incorporate MSI functionality into it to improve on the handshake technique used; where a specific data value at an address indicates a full buffer. I would like to do this with MSI interrupts instead.

I have also read and understood the thread you have linked to which was very helpful for my understanding but does not show what code is required on the end point to trigger the MSI interrupt across the PCIe link. I have tried figuring out a solution by myself including writing directly to addresses, reading BAR values and writing to offsets from them, using pcieWriteRegs using the REMOTE location identifier but I haven't found anything that works.

My code has methods to enable MSI interrupts on the RC and uses the code given on the other thread to define the interrupt statically in the config file and uses the hwi_isr handler. I have confirmed, by triggering the interrupt manually on the RC by writing 0x0000000 to the msiIrq register, that this code works.

I have also followed the steps given by Koray (the ones that are given in the PCIe User guide section 2.13.2.2) to set up MSI on the end point. The only step I haven't managed, which is why I have resorted to posting here, is the part "issue a memory write transaction with the address the same as MSI Address Register and the data the same as the MSI Data Register". When I read each of these registers on the EP the address and data both return 0.

I you do have any example code that you can show that would be the most helpful. If not then some suggestions of the functions/methods to use would also be helpful.

My eventual aim is to test the throughput of the PCIe link using either an MSI or EDMA protocol similar to what you have show in the PCIe Use Case document in section 2.4. I assume from having done those tests you have code that matches what I am trying to write but I suspect you would not be able to provide that to me.

Many Thanks for your help

Charles

Charles,

"issue a memory write transaction with the address the same as MSI Address Register and the data the same as the MSI Data Register".

Those registers in the EP should be filled out by the RC during the enumeration. Since we do not have the enumeration in the PCIe example, their default values are 0s.

They are actually referring to the MSI_IRQ register and the vector value.

For example, if you want to trigger MSI interrupt 0 in RC from EP, you can issue a memory write from EP to RC and the target location is the MSI_IRQ register address (0x21800054 in RC) with the vector value 0x0 (for MSI_0) as the data being written.

The BAR0 is mapped to the Application Registers of PCIe in RC (starting from 0x21800054). So you can just issue the memory transaction from EP with the PCIe address which could be accepted by BAR0 in RC, then the memory transaction will be targeted to the Application Registers in RC (including MSI_IRQ).

For example, if you set BAR0 in RC as 0x70000000 (PCIe address over the PCIe link as you defined) and you should setup the outbound translation in EP to use the same PCIe address (0x70000000) over the link. Then you can write 0x0 to the PCIe data space (e.g. write 0x0 to 0x60000054 in EP). The write will be translated to 0x70000054 over the PCIe link and could be accepted by BAR0 in RC, which is 0x70000000.  SInce BAR0 is mapped to Application Registers region (from 0x21800000), the writing to 0x70000054 (from EP) will be targeted to 0x21800054 in RC and the MSI_IRQ in RC will be written into 0x0, which will trigger MSI_0 in the RC.

All of the above will happen assume you already configure the interrupt controller and enable the  PCIe MSI interrupt in RC correctly (as what you have done in the RC of self-triggering).

You may also need to clear the MSI_0 interrupt and write to the IRQ_EOI register first before re-triggering the same event again. It has been discussed in the following thread as well.

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/p/181633/657546.aspx#657546

Hope it works for you.

Steven,

I am trying to interpret the process you describe in your example. I have been experimenting in my code but have had no success so far.

Would these be the right steps to take?

1. In RC I use Pcie_cfgBar(handle, &bar0Cfg) with bar0Cfg.base = 0x7000000 and bar0Cfg.idx = 0 to set BAR0 in RC as 0x7000000. (I use a copy of what is done in the example project to set BAR1).
   
   
2. In EP I use the pcieObTransCfg method with obAddrLo = 0x70000000  which calls Pcie_cfgObOffset to setp the outbound translation. (Again, I have edited what is done in the example project.)
   
   
3. Then, after all initialisation on both sides, I use *((volatile uint32_t *) 0x60000054) = 0x0; to write 0x0 to the PCIe data space to trigger the RC's interrupt. (As is used to transmit data in the example project).

Is there anything I am missing or that I am doing wrong? I assume all the setup on the RC is correct given I can self-trigger it.

Many Thanks

Charles

Charles,

I hope it was just one typo in your last post, not in your actual code, that for RC it should be bar0Cfg.base = 0x70000000 (not 0x7000000).

If it is not a typo, could you please check the actual registers setup in CCS memory view after you run the modified code on both RC and EP?

EP:

OB_OFFSET_INDEX0(0x21800200) = 0x70000001 (bit_0=1 means outbound translation is enabled)

OB_OFFSET_HI (0x21800204) = 0x0

OB_XLT_EN(bit 1) in CMD_STATUS (0x21800004) = 1

Also please make sure the writing to 0x60000054 is on EP side, not RC.

RC:

BAR0 (0x21801010) = 0x70000000

After the writing from EP to RC, you could also check if MSI0_IRQ_STATUS (0x21800104) = 0x1 on RC side, indicating the MSI0 has been triggered.

MSI0_IRQ_ENABLE_SET (0x21800108) = 0x1as well in RC to enable MSI0 before triggering it.

If you would like, you could also post your code here if it is not working. We can look into it as well.

Steven,

Well I'm not sure what I've changed, as I'm sure my setup was as you described, but after checking my code for the parameters you mention and tidying out some old redundant and commented code, it appears to have started working!
I was using the definitions in the header file to assign the BAR address (0x7000_0000) so a typo shouldn't have been the problem. I think I was just trying to overcomplicate the whole procedure.

My next step is to incorporate some EDMA/DMA/QDMA methods to execute the data transfer

Would you advise using MSI interrupts to control this system? or is there a more formal interrupt system provided by the EDMA module that can operate over PCIe?

Should I open a new thread if I have more questions about using MSI interrupts in conjunction with EDMA?
The following thread appears very similar to my proposition, although it does not elaborate on MSI interrupts at all.
http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/169491.aspx

Thank you very much for your clear explanations.
Charles

Charles,

If I understand your intention correctly, you would like to transfer data from EP to RC and then send message from EP to notify RC that data is ready, instead of keeping polling the final data address in RC.

If so, I think the MSI interrupt should be a good option to send the interrupt over the PCIe link.

EDMA will only generate the interrupt inside of the local device.

You can open a new thread or follow up on any similar thread as well. We should be able to monitor the forum regularly.

hi Steven

           I understand what you said about MSI interrupt.  and i also  have a question about that.

          1 Does "pre-determined address" have to be  the address of MSI_IRQ (0x21800054). Accourding to SECTION 2.13.2 of the document SPRUGS6A,the addresss is 

MSI_LOW32(32bit mode). if not,What role does MSI_LOW32 play?

          2  In my application,my RC is a PC and my EP is 6678. how can i generate a MSI interrupt from 6678(EP) to PC(RC).I can send MSI interrupt from PC to 6678 by writing to 6678's MSI_IRQ throught 6678's Inbound address translation。

Mou,

1. MSI_IRQ (0X21800054) is specific for C667x devices.

If you use C667x PCIe device as RC, the MSI_LOW32 and MSI_UP32 registers in EP will have PCIe addresses by could be pointed to the MSI_IRQ register in RC. The PCIe address may be 64bit (using both MSI_LOW32 and MSI_UP32) or 32bit (MSI_LOW32). The values in MSI_LOW32 and MSI_UP32 could be anything but after the outbound translation in EP or inbound translation in RC, writing to this PCIe address should be equal to writing to the MSI_IRQ in RC, in order to be able to generate MSI interrupt from EP to RC.

If you use C667x PCIe device as EP, the MSI_LOW32 and MSI_UP32 values could be something assigned by the RC/host, which are pointed to the similar MSI IRQ register in the RC/host. The register name and memory map address of that MSI register in RC/host may be different from MSI_IRQ (0x21800054) in C667x devices. But the functionality should be similar (RC will generate MSI interrupt when EP send memory write to this register).

2. As I mentioned above, the PC(RC) may have the MSI register with similar functionality of MSI_IRQ in C667x as RC. The MSI_LOW32, MSI_UP32 and MSI_DATA registers are standard PCIe registers. Usually the PCIe driver running the PC (RC/host) will assign the MSI values to those MSI registers in EP during the enumeration process. And EP could read those MSI registers first to get knowledge of where (MSI_LOW/UP) and what values (MSI_DATA) to be used in order to generate MSI interrupt to RC.

So you should refer to the documentation of the PCIe chip in your PC or the PCIe driver to see details. Usually, the EP just need to read those MSI registers and issue the memory write to the address mentioned in MSI_UP32/MSI_LOW32 with values in MSI_DATA to generate interrupt to RC.

Hi steven,

thanks a  lot for your reply.

I solve this problem 

Our RC is a PC whose operating system is  windows XP,and accourding macrosoft'MSDN ,XP doesn't support MSI interrupt..Only VISTA and above support it.

So,wocan only use legacy interrupt.