Reading BAR data using C6678 as PCIe Root Complex with FPGA End Point

Hi Bruce,

TI doesn’t provide PCIE RC enumeration example/driver source. Normally the RC runs some operating system that performs the enumeration.

In the MCSDK PCIe sample example, two DSP EVMs are used to test the PCIe driver. DSP 1 is configured as a Root Complex and DSP 2 is configured as End Point. Once the PCIe link is established, the following sequence of actions will happen:

1. DSP 1 sends data to DSP 2 - DSP 2 waits to receive all the data

2. DSP 2 sends the data back to DSP 1

3. DSP 1 waits to receive all the data

4. DSP 1 verifies if the received data matches the sent data and declares test pass or fail.

Modified the value of PCIE_IB_LO_ADDR_M and PCIE_OB_LO_ADDR_M to PCIe inbound/outbound address. The BAR0 register is fixed to be mapped to PCIe application registers region. So the BAR0 mask setup does not affect BAR0 register. If you want to use 4 configurable BAR registers, please configure BAR1~BAR4 accordingly.

Thanks,

Ganapathi,

Can you provide some more information on the parameters needed for pcieBarCfg_t to map to the BARs I described above? I do not need to explicitly enumerate the BARs since these addresses will be fixed.

To keep it simple, let's take BAR2 as an example. If I use the following code, I am able to successfully configure the BAR, but then the program stops working after that point:

pcieBarCfg_t     barCfg;

barCfg.location = pcie_LOCATION_REMOTE;
barCfg.mode     = pcie_RC_MODE;
barCfg.base     = 0x00000000;
barCfg.prefetch = pcie_BAR_NON_PREF;
barCfg.type     = pcie_BAR_TYPE32;
barCfg.memSpace = pcie_BAR_MEM_IO;
barCfg.idx      = 2;

if ((retVal = Pcie_cfgBar(handle, &barCfg[i])) != pcie_RET_OK)
{
   System_printf ("Failed to configure BAR%d (%d)\n", i, (int)retVal);
   exit(1);
}
else
{
   System_printf("Configured BAR%d successfully!\n", i);
}

I also do not understand how to configure outbound translation when the BAR mappings in the FPGA all start with address 0x0. Is there a way to specify to the function pcieObTransCfg which BAR the mapping corresponds to?

Thanks,
-Bruce

Hi Bruce,

PCIe BAR configuration details are clearly explained in PCIe user guide. Please take took at section 2.7.3 Use of Base Address Registers (BARs) on PCIe user guide.

Also refer PCIe BAR configuration defined in IBL source code.

Function: iblPCIeWorkaround()

Path: “\ti\mcsdk_x_xx_xx_xx\tools\boot_loader\ibl\src\device\c66x\c66xinit.c”

Thanks,

Ganapathi,

I've looked at the PCIe user guide and c66xinit.c source code carefully, but the documentation is not clear on what steps are needed to enumerate and read data from a remote BAR when the C6678 is acting as the RC. If TI doesn’t provide PCIE RC enumeration example/driver source then I need to know who to contact to in order to get in this information.

Here are the steps I am taking and maybe you can tell me what I am missing:

-C6678 DSP eval board and FPGA eval board are connected via a PCIe backplane card.
-Operation of each board in EP mode has been confirmed using a Linux PC.
-C6678 has been set in RC mode, FPGA eval board in EP mode.

-Set the OB_SIZE register to 0x3 and enable outboard address translation in the CMD_STATUS register.

-Set the BAR addresses for BAR0 through BAR4 described described in my first post in the FPGA registers by specifying the pcie_LOCATION_REMOTE option. I assigned these addresses to the BARs: 

0x00000000 to BAR0, 
0x00100000 to BAR1, 
0x00200000 to BAR2, 
0x00300000 to BAR3, 
0x00400000 to BAR4

-Read the BAR addresses back out from the FPGA board and verified that they were set correctly.
-Read the vendor ID and device ID from the FPGA board and verified that they are correct.

-Configured outbound offsets in the pcie_LOCATION_LOCAL registers:

Region 1: LO: 0x01, HI: 0x0
Region 2: LO: 0x00100001, HI: 0x0
Region 3: LO: 0x00200001, HI: 0x0
Region 4: LO: 0x00300001, HI: 0x0
Region 5: LO: 0x00400001, HI: 0x0

-Perform a memcpy from the outbound address regions into a local buffer and print out the buffer. For example if I have a 1024 byte buffer declared as char buff[1024] and I want to read from BAR0 (which is mapped to region 1/address 0x60800000):

memcpy(buff, 0x60800000, 1024);

When I read the memory out of the buffer (or directly from the PCIESS address space), I just get garbage data.

I've also tried to use an EDMA transfer, but that was not successful. I will probably try to use EDMA to optimize the transfer speed at some point, but for now, I just want to verify that I am able to transfer the correct data into memory using memcpy.

-Bruce

Hi Bruce,

The BAR0 register is fixed to be mapped to PCIe application registers region. So the BAR0 mask setup does not affect BAR0 register. If you want to use 4 configurable BAR registers, please configure BAR1~BAR4 accordingly. 

Please go through below wiki for more information.

http://processors.wiki.ti.com/index.php/PCI_Express_(PCIe)_Resource_Wiki_for_Keystone_Devices

Thanks,

Ganapathi,

Sorry for the delay in getting back to you. I've been reconfiguring the code I have to experiment with some different register settings.

I've managed to be able to read data from BAR0 on the remote device, but I still cannot read from BAR1-BAR4. I've been through all of the documentation that TI has provided, but it's still not clear in describing how to configure the C6678 as RC. I see there have been several other posts on this E2E forum with similar questions but they do not seem to conclude with clear resolutions.

It's my understanding that BAR0 is mapped to the PCIe application registers on the local device, but this restriction does not apply to outbound reads to an external device. The data I am reading from BAR0 on the remote device is not application register data.

Can you take a look at the steps I outlined in my October 3rd post and clarify what I am missing or doing wrong?

Thanks,

-Bruce

Ganapathi,

I've made some progress in reading the other BARs. It appears that the outbound address translation needs to be configured before the remote BAR addresses in order to work properly. 

-Bruce

Hi Bruce,

Thanks for your update.

TI doesn’t provide PCIE RC enumeration example/driver source. Normally the RC runs some operating system that performs the enumeration with the following steps.

If C66x device is configured as RC, then CFG_SETUP register is used to specify the target bus/device/function numbers for the target device (switch/EP) and try to read/write the remote device space in MMR (from offset 0x2000 in PCIe MMR space) to generate those configuration requests.

Thanks,

Thanks Eric! I managed to get outbound reads working for all of the BARs. As I mentioned in my reply to Ganapathi, it appears that outbound translation regions need to be configured before the remote BAR addresses are set. I'm not sure if that is particular to the FPGA card I am using as an EP or if that is something that applies to the PCIESS.

The only other issue I'm having is that Region 0 behaves a little strangely. It reads some of the data, but sections that should be all zero are filled with 0xDEADBEEF. The problem went away when I moved the translation to Region 1. I don't need to use all of the regions that are available, so this is not really a problem for me. I just wanted to note it here in case someone is having a similar issue in the future.

-Bruce

Bruce,

Good to know the outbound read worked! "that outbound translation regions need to be configured before the remote BAR addresses are set", I don't see this is required. I often configure or reconfigure OB registers after remote BAR is set, my test setup is between TI PCIE device, not sure if anything particular for this FPGA. 

My think is that, you don't need to program 0x2180_0208 and 0x2180_020c, just load the DSP RC code and run, before executing outbound read, you can set a break point there, so halt DSP and use CCS memory window to change those two registers then run, you should be able to read back the correct results. If you are interested, you can try. You can alos try to program those two to different BAR via CCS memory window, then you should read back different results.

For BAR0, on our DSP, it is used to access remote side's PCIE application registers. For FPGA, I don't know if BAR0 is special as well, you may find it on data manual. Since you can use other BARs than BAR0, good to know it is OK for you.

Regards, Eric   

Hello all,

Bruce, i am having similar issues as your starting point...reading dev id, and link up but cant read data from Spartan6 FPGA...is it possible to share your eorking code please

How important is the choice of the Type0 or Type1 for RC and EP?

As far as I understand this choice depends on the CFG_SETUP.CFG_TYPE setting.

The SPRUGS6D says :

Typically, a RC module uses Type 1 and EP configuration uses Type 0 configuration space.

But in example 

C:\ti\pdk_C6670_1_1_2_6\packages\ti\drv\pcie\example\sample\pcie_sample.c

CFG_TYPE bit of CFG_SETUP registr is set to "0" for RC and for EP. And so RC configured as Type 0 and EP too.

Why?  Is it importently or no?

Should CFG_SETUP.CFG_TYPE be changed  fo RC on"1" (Type1)?

With regards, Alexander.

Hi,

In the MCSDK PCIe sample example,  DSP 1 is configured as a Root Complex and DSP 2 is configured as End Point by own test application. For  test the PCIe driver between two DSP devices.

Thanks,

Hi Bruce,

I am also experimenting on the PCie, I had following question,

From driver point of view, the Endpoint will have same BAR address for data read/write.

For example purpose, let us assume that we have configured address 0x8A000000 as BAR address in configuration space of EP at logical address Bus_id=0, dev_id=0 and fun_id=0.

For Linux OS, when driver requests memory resource for EP 0, the address 0x8A000000 will be returned by the PCIe stack. Now for any read/write for the end-point will through 0x8A000000 from driver.

But PCIe requires the BAR address to be different for outbound transfer(For data write and read request packets) and inbound transfer for inbound packets.

So how is that the same BAR address is maintained for read and write in driver

Thanks in advance