pcie outbound region map

Steven,

your information is very important to me , thanks !

but I still wondering that is this the hardware decide or the software decide?because the data from 0x12300000 does not AUTO transfer to the outbound region 0(It means that the data from 0x12300000 does not the same with the data in outbound region 0).

Mike,

After the data updates in remote device, do you issue another data transfer from PCIe outbound region (such as from 0x60000000) in local device again please?

Do you observe the old data reading back from PCIe outbound region which is mapping to the remote device's memory region (such as 0x12300000) please? Or does PCIe read back some random incorrect data please?

If it is still the old data, it seems like the new data has not been updated correctly in the memory region in remote device. Could you check if you can disable cache in the remote device or manually maintain the cache coherency to make sure the new data has been landed to the desired memory region correctly in remote device first please?

Steven,

I just re-run the transfer function, maybe this is same with 'issue another data transfer' you said.

I observed the data, PCIe just read back some random incorrect data.

so I checked some registers: like PCIE_SERDES_CFGPLL(address 0x02620358) is 0x000001C9(the same with default value) and PCIE_SERDES_STS(address 0x0262015C) is 0x00000201(different from the default value 0x00000001).

Is there any possibility that the register PCIE_SERDES_STS wrong ?

Mike,

PCIE_SERDES_STS=0x201 (i.e. LOSDTCT1=1) means lane 1 is not being connected, but lane 0 is connected (LOSDTCT0=0). Do you have x1 lane or x2 lane connection between DSP and PC please? What is the status of PCIE_SERDES_STS at the beginning before the error happens please? It makes sense if you have only single lane (lane 0) connection between DSP and PC, that lane 1 is not being used.

You can also check the LTSSM_STAT field in DEBUG0 (0x21801728) register to see if it is still equal to 0x11 which means the PCIe link is up and functional between DSP and PC.

For incorrect data issue, you are able to read the data correctly (using EDMA or CorePac in C66x to read PCIe outbound region) for the first time (before data update on PC side), is it correct?

Between your first data transfer and second transfer, did you change any PCIe configurations, such as outbound/inbound mapping, BAR or anything else?

Are you able to read data correctly for two back-to-back data transfer without any changes?

If you are using EDMA in C66x for the data transfer, are you able to do the same testing using CorePac to read the data back from PCIe outbound  region please? It the CorePac reading is working, it means the PCIe setup is fine and something wrong with the EDMA transfer setup (for the re-run). If CorePac reading is not working neither, then probably something wrong with PCIe setup or data update on PC side. 

It will be good if you provide more details of your testing sequence and we need to narrow down what will cause the issue.

Steven,

I not sure which lane connect between DSP and PC.

Before the error happened, the status of PCIE_SERDES_STS is 0x201.

The LTSSM_STAT field in DEBUG0(0x21081728) is still equal to 0x11.

and for incorrect data issue, I am NEVER able to read the data correctly,even though the first time. the data start 0x1230_0000 just donot equal to the data in oubound region 0 even though I mapped.

Let me re-illustrate my problem: I use EDMA to transfer data between PC and DSP, I can transfer data in outbound region 0 to DDR(0x8000_0000) correctly(means inside DSP). but the data reside in outbound region 0(0x6000_0000) DO NOT equal to the data reside in 0x1230_0000(means between PC and DSP).

Mike,

It seems the SerDes status is the same all the time and link is up. So the issue seems not related to the incorrect SerDes configuration.

Based on your description, it seems the issue was due to the outbound mapping in DSP (if enabled) and inbound mapping in PC (if enabled) for the memory region you are testing.

I hope you already look at section 2.7.1 "Outbound Address Translation" in PCIe user guide and section 3.1 "PCIe Outbound Address Translation" in PCIe use case application note.

Please check if you enable or disable the outbound translation in C66x PCIe module (OB_XLT_EN =1 or 0 in CMD_STATUS register). If you want to enable outbound translation and please check your OB_SIZE, OB_OFFSET_INDEXn and OB_OFFSETn_HI registers to see if you translate the PCIe data space address (from 0x60000000) to the targeted PCIe address which could be accepted by PCIe on PC side.

And please check if PC could accept those PCIe packets from EP with PCIe addresses after translation and if you need to enable the inbound translation on PC side to forward those packets to the targeted memory region (such as 0x12300000).

For example, if the PCIe address from EP after outbound translation is already translated to 0x12300000 and so on, then you may not need to enable inbound translation on PC side and just need to make sure PC could accept those PCIe address (from 0x12300000) and the PCIe transactions (write or read) will be applied to that memory region.

So the "automatic transfer" you mentioned before for the DSP PCIe data space (from 0x60000000) and PC memory region (such as 0x12300000) is based on the correct mapping in your setup. In this DSP initialized transfer, you need to make sure the PCIe addressing over the PCIe link can be accepted by PC and is targeted to the correct memory region. Then when you initialize the EDMA transfer on DSP side, the PCIe ports and link will be able to transfer the correct data between DSP and PC as you planned.

Steven,

your information is very important to me, thanks !

I have checked the CMD_STATUS register, and it says 0x07, It seems fine. and I also have looked the user guide you mentioned.

The following list the outbound translation registers configuration(here srcAddr = 0x12300000):

OB_SIZE=0x0;  //OB_SIZE=1MB;

pageBase = srcAddr & 0xFFF00000;
WDC_WriteAddr32(hDev, 0, OB_OFFSET_INDEX(0), pageBase|0x1);

                                            //It means write value pageBase|0x1 to register OB_OFFSET_INDEX(0), same below
WDC_WriteAddr32(hDev, 0, OB_OFFSET_HI(0), 0x0);

based on above, I am sure PCIe address from EP after outbound translation is already translated to 0x12300000.

QUESTIONS:

1) should I configure the inbound translation registers in DSP?

2)"For example, if the PCIe address from EP after outbound translation is already translated to 0x12300000 and so on, then you may not need to enable inbound translation on PC side and just need to make sure PC could accept those PCIe address (from 0x12300000) and the PCIe transactions (write or read) will be applied to that memory region."        

I have no idea how to make sure PC can accept those PCIe address(from 0x12300000)? should I configure the BARn registers on the PC side? If so, From where I can find information about BARn registers on the PC side?

PS: The EVM plugged into the pcie slot on main-board through AMC to pcie adapter, and following list the windriver detected information about the EVM.

Vendor ID: 0x104C, Device ID: 0xB005
Location: Bus 0x1, Slot 0x0, Function 0x0
Memory range [BAR 0]: base 0xFE400000, size 0x100000
Memory range [BAR 1]: base 0xD1400000, size 0x80000
Memory range [BAR 2]: base 0xD1000000, size 0x400000
Memory range [BAR 3]: base 0xD0000000, size 0x1000000
Memory range [BAR 4]: base 0xD1490000, size 0x1000
Memory range [BAR 5]: base 0xD1480000, size 0x10000
Interrupt: IRQ 11
Interrupt Options (supported interrupts):
Message-Signaled Interrupt (MSI)
Level-Sensitive Interrupt

Mike,

1. The "Inbound translation" is used for "inbound transaction" that the remote device starts/initiates the Write/Read transaction to local device (C66x PCIe). The PCIe master port in C66x DSP will forward the transactions to memory without the help of EDMA in DSP. The "Outbound translation" is used in "outbound transaction" that the local device (C66x PCIe) starts/initiates the Write/Read transactions to remote device, which needs to help of EDMA and PCIe slace ports (and PCIe data space starting from 0x60000000).

So for this "outbound transaction" case, you are using EDMA in C66x to start the transaction, only outbound translation in C66x is needed. The inbound translation in DSP is not needed. And the inbound translation in remote device (PC) is optional depends on how you want to map the PCIe packets to PC memory.

2. In general, for RC mode, if the address is in the range configured in the BAR or is outside of the range defined by the three Base/Limit register sets (non-prefetchable memory, prefetchable memory, and I/O), and then that TLP is accepted by RC. Please refer to C66x PCIe user guide section 2.7.3 for details.

But I am not sure where to find the PCIe info on PC/Host side. You may need to refer to the user guide of PC/host PCIe module. 

The code seems correct for the OB setup, but could you also connect JTAG to DSP and check the OB registers directly on DSP side to see what the registers values are after the setup please? We need to double check the PCIe registers are set correctly first.

And what is the EDMA transfer setup please? Is the source address starting from 0x60000000 please?