QUESTIONS on 6678 PCIe multi message msi dma and inbound translation(steven ji help me!!!)

1. In the original example, I think your understanding is correct. Could you confirm you are using the example on Core0 or other cores please? 0x10800000 is only for Core0 L2, but 0x00800000 is the local L2 for every core.

And bits[7:0] of IB_OFFSET register are reserved as you can see if the PCIe user guide. It seems fine to have 0x8000_0000 as offset. But I am not sure if anything else changed. Could you please check IB registers (IB_Start_Lo/HI, IB_Offset, IB_BAR0) directly to see what values are being programmed please?

In your own inbound translation, the formula is correct that you could change either inbound LO/HI address or change inbound offset to map the incoming packets to the destination location.

So you can also program IB_START_LO=0x90000000 (matches incoming PCIe address from FPGA) and program IB_OFFSET=0x80000000.

But it is not recommended to have outbound address in RC to be the same as incoming PCIe address from EP (outbound address from EP, 0x90000000 in your example).  It will be better to have different outbound addresses in RC and EP and different BARn addresses in RC and EP (if you prefer to use BAR in RC, which is optional, please refer to BAR registers section in PCIe user guide for details).

2. a

PCIExpress_MSI_INTn is connected to CorePac_n in C6678, which means MSI_0 is connected to Core0 and MSI_1 is connected to Core1, etc.

In each MSI event, there are four vectors such as MSI_0 includes MSI vectors 0,8,16,24. Once you receive the MSI event, in the ISR, you could check the bits in MSI0_IRQ_STATUS register to check which vector has been used in this event. You can set 1 to bits of MSI_IRQ to trigger MSI interrupt with different vectors, such as write 0x0 to trigger MSI vector0 and write 0x100 to trigger vector 8.

2.b

As we mentioned above, you should set corresponding bits in MSI_IRQ to trigger MSI interrupt. If you are using Core0 only in your case, you should set bits 0,8,16 or 24 in MSI_IRQ to trigger the MSI_0 interrupt. MSI_1 is not corrected to Core0, so it seems you are still triggering MSI_0 in your case (not MSI_1 event, so nothing changed in MSI1_IRQ_STATUS).

2.c and d

The HWI cfg setup seems fine and you are able to trigger the interrupt.

The thing is missing that you can try to write different values in MSI_IRQ to trigger different vectors in MSI_0 for Core0 (if it is your case). As we mentioned above, you can check MSI0_IRQ_STATUS to see which vectors is being used and do the actions accordingly.

Hope it could help.

Thank you Steven Ji.

1.about the translation，I'll test for more.

 a. I have been operating on core0, as you say ,the offset only use   bits[7:0], so what's the different between  0x10800000 and 0x00800000? (while using 0x80000000, i can find nothing after the data transation)      

 b. But why  is it not recommended to have outbound address in RC to be the same as incoming PCIe address from EP?

2. I've found out that I can only handle MSI 0,8,16,24 in core0, and we might have some mistakes in the early test. It's very awkward.

    But we have such a problem now.

   while i use 00 to trigger the msi0, i can break into the hwi_isr, but it seems always run twice!  

   my hwi_isr mainly include:

   msiIrqStatus[0].msiIrqStatus =0x1;

   irqEOI.EOI=0x4;

  by setting these two reigsters (using Pcie_write Regs), i think i can clear the status of the interrupt, but why?  run twice?

  so , I add the following code:

 Hwi_clearInterrupt(5);

Hwi_enableInterrupt(5);

It seems to solve the problem ,  one interrupt only triggers  hwi_isr once, but  for further msi0 interrupts, I can see the msi0_irq_status become 1, but the hwi_isr is not triggered.

I don't see the old post use these code:

 Hwi_clearInterrupt(5);

Hwi_enableInterrupt(5);

is there any mistake? and where should i check my code.

Thank you very much!!

1.a

Please take a look at section 3.1.85 in PCIe user guide that I mentioned bits 7:0 are Reserved (not being used) in IB_OFFSET. Bits [31:8] will take effect.

You may want to check the IB_OFFSET_n (for IB_OFFSET0, it is 0x2180030c) values in the memory view to double check it is indeed the same as you plan to program. You can check the IB_STARTn_LO/HI and IB_BARn as well.

If they look fine, you may want to further check the BAR setup/BAR mask (window size) and PCIe address over the link to see if the PCIe address is out of the region of BAR window.

1.b

Normally there could be multiple EPs connected to RC in PCIe hierarchy and the BARs (acceptable PCIe address region) of all the EPs should be OUTSIDE of the acceptable PCIe address region of RC. Otherwise the PCIe packets with PCIe address will match both RC and EP BARs (if you use BAR in RC) which will be confusing.

In your case, if you have the same outbound address of RC and EP, which  means you need to have same BAR setup in RC and EP (since you use BAR in RC). It might be OK in the peer-to-peer transfer, since you only have one EP. But if you have another EP (EP2) and tries to send data to RC, the same data may be accepted by the other EP(EP1) as well since EP1 and RC has the same BAR setup. So it is recommended not to use the same setup for RC and EP outbound transaction.

I think the PDK PCIe example, we are using different setup for RC and EP such as 0x90000000 and 0x70000000.

2.

In the ISR, besides clearing the interrupt status at source level (PCIe in this case), we may need to clear the CorePac INTC event as well.

Below is the CSL example we could add to the ISR:

For BIOS, I am not quite sure. Maybe you could only add "Hwi_clearInterrupt(5);" to see if it could re-trigger the same event.

Thank you ! Steve

1. I've been beset by the problem for too long and totally forgot the main purpose of the address translation.

2.I'll check  your suggestion on ISR as soon as quickly, and response soon!

steven 

I haven't able to check the answer yet, but i have a further question to ask

if I only use core0 ,and  want to use msi 0 and msi 8, should i set   msiCap.multMsgEn to 1 or 3(mean 8)? 

and  should i set the EP with the same multmsgen?

Thanks !

Xing,

I think the "MSI_CAP" register is most likely to be used by the OS PCIe driver during the enumeration. The driver could understand the capability of MSI interrupts in RC and EP based on this register and assign the MSI vectors to EPs accordingly.

I guess in the embedded system (peer-to-peer setup without PCIe enumeration driver) it cold be fine to trigger multiple MSI interrupts even have this MSI_CAP with default setup (support 1 MSI interrupt only). But it is no harm to set this register to support multiple interrupt based on real application.

You may want to give a try since I am not quite sure about this. It will be good to share your observation as well. Thanks.

dear Steven

I've come across a new problem.

while I am using the multiple message interrupt, first the FPGA will send a vector 8 interrupt, then i wll start a dma from FPGA by sending some packets to FPGA via pcie,

then the when this pcie dma finished ,the FPGA will send a interrupt vector 0 to DSP, then the dsp is assumed to do some further execution of the data.  the duration of the

dma is about 30us, and the next vector 8 interrupt will come after the first vector 8 interrupt  for 100us.

but as my hwi_irs, i will check the msi0_irq_status register for 1 and 2.

but as i see(put breakpoint in the hwi_isr) , first i recive a vector 8 , the msi0_irq_status becomes 2 ,and  the dma start , next time i check the register , the value is 3 , it recieve both interrupt. actually I can't hit a time the register is 1 with only a vector 0 interrupt.

is it because the interrupt cannot tell two events about 70us interval? what can i do

Xing,

I think the two subsequent interrupts (vector 8 and vector 0) will come in the interval of 30us, which is the duration of DMA transfer, right?

Once you receive the first interrupt, your ISR should be simple and compact enough to be existed before the next interrupt happens, is it correct?

And in your ISR, the current interrupt received will be cleared in PCIe module as follows, is it correct?

*Msi0_IRQ_status=(uint32_t )0x1 or 0x2; (depending on interrupt event received)

*irq_eoi=(uint32_t)0x4;

sorry 

I've express it not clearly.

The FPGA will send a vector 8 interrupt every 200us, at first DSP only recive a 8 vector and send some data to FPGA to start a DMA,  but then I will recieve the 0 vector(means FPGA has finished the transfer) and vector 8.

It seems that  the interrupt handler just can't tell the two interrupt apart!

and another  phenomenon

from the Chipscope of the FPGA side, we  can see the internal time between the vector 8 interrupt packet(FPGA to DSP) and the data send back to FPGA vary between 400 FPGA cycle to 4000 cycle(it is not accepted).

I only have one task, only have this interrupt !  the hwi_isr only include this data transfer and status clear! why would this internal time vary? the

Xing,

It might be helpful if you could take time stamps on certain points to understand the timing of your data flow. Such as you could get Time_1 when vector 8 triggers DSP interrupt ISR, Time_2 when DSP exists ISR and Time_3 when vector 1 triggers ISR. You would see if 200us is realistic for the vector 8 and vector 1 arrival and if it is appropriate to have data transfer inside ISR.

For the time interval variation, you could see which part causing it by using the time stamp on DSP/FPGA side as well. Hope it could help.