RTOS/TMS320C6678: PCIe BAR Data read from FPGA endpoint

Arun Singh26

Part Number: TMS320C6678

Tool/software: TI-RTOS

Hello,

I am trying to read data using a C6678 customized board using PCIe. The C6678 is the RC and the FPGA is the EP on the PCIe bus.

I am able to perform link training and reading remote device ID and vendor ID, but they are swapped in order, that may be because of endianess mismatch.

I am following the example in the PCIE_exampleProject and also some modifications that I found on the TI E2E forum.

On FPGA side i have configured BAR0 and BAR1 of size 8MB.

On C6678 DSP side i configured outbound translation for region 1 and region 2.

Region 1: LO: 0x00100001, HI: 0x0

Region 2: LO: 0x00200001, HI: 0x0

After that i configured remote BARs of FPGA.

BAR0

pcieBarCfg_t barCfg;

barCfg.location = pcie_LOCATION_REMOTE;

barCfg.mode = pcie_EP_MODE;

barCfg.base = 0x00100000;

barCfg.prefetch = pcie_BAR_NON_PREF;

barCfg.type = pcie_BAR_TYPE32;

barCfg.memSpace = pcie_BAR_MEM_MEM;

barCfg.idx = 0;

BAR1

pcieBarCfg_t barCfg;

barCfg.location = pcie_LOCATION_REMOTE;

barCfg.mode = pcie_EP_MODE;

barCfg.base = 0x00200000;

barCfg.prefetch = pcie_BAR_NON_PREF;

barCfg.type = pcie_BAR_TYPE32;

barCfg.memSpace = pcie_BAR_MEM_MEM;

barCfg.idx = 1;

outbound translation and BAR configuration has been completed successfully.

But when i am trying to read data from region 1 (0x60800000) and region 2 (0x61000000), i am getting data alternates between 0x60800000 and an sequence starting with 0x01 that increases in steps of 2.

Please let me know which steps i am missing out in configuration and data reading.

Thanks

over 8 years ago

0 Cvetolin Shulev-XID over 8 years ago

TI__Guru 65405 points

The team is notified. They will post their feedback directly here.

BR
Tsvetolin Shulev

0 Victor Kazmirenko over 8 years ago

Guru 13202 points

Hello!

You telling you're setting 8MB memory windows, but your bases are 10_0000 and 20_0000, which are 1MB and 2 MB respectively. Spec requires bases be aligned at least at memory window width. So, one may have 2MB window aligned as on 2MB, so on 4 or 8, but 8MB window should be aligned on 8MB. Thus first thing to check is barCfg.base.

0 lding over 8 years ago in reply to Victor Kazmirenko

TI__Guru* 95265 points

Hi,

I thought the PCIE example set the OB_SIZE = 8MB. Please check if you modified this to 1MB. If it is, then you need to read from 0x6010_0000 and 0x6020_0000.

Regards, Eric

0 Arun Singh26 over 8 years ago in reply to Victor Kazmirenko

Prodigy 145 points

Thanks for pointing this out, I reconfigured the remote BARs as below:

BAR0

pcieBarCfg_t barCfg;
barCfg.location = pcie_LOCATION_REMOTE;
barCfg.mode = pcie_EP_MODE;
barCfg.base = 0x00800000;
barCfg.prefetch = pcie_BAR_NON_PREF;
barCfg.type = pcie_BAR_TYPE32;
barCfg.memSpace = pcie_BAR_MEM_MEM;
barCfg.idx = 0;

BAR1
pcieBarCfg_t barCfg;
barCfg.location = pcie_LOCATION_REMOTE;
barCfg.mode = pcie_EP_MODE;
barCfg.base = 0x01000000;
barCfg.prefetch = pcie_BAR_NON_PREF;
barCfg.type = pcie_BAR_TYPE32;
barCfg.memSpace = pcie_BAR_MEM_MEM;
barCfg.idx = 1;

and outbound regions As:

Region 1: LO: 0x00800001, HI: 0x0

Region 2: LO: 0x01000001, HI: 0x0

Outbound translation and BAR configuration has been configured successfully,
and i observed PCI Application and configuration register in memory browser.

*0x21800208 = 0x00800001 (OB region 1 low)
*0x2180020C = 0x0 (OB region 1 Hi)
*0x21800210 = 0x01000001 (OB region 2 low)
*0x21800214 = 0x0 (OB region 2 Hi)

and in remote configuration space
*0x21802010 = 0x00800000 (Remote BAR 1 raw register value)
*0x21802014 = 0x00800000 (Remote BAR 2 raw register value)

When i m trying to read from memory 0x60800000 i am getting alternate data 0x60800000 and 0.
and When i m trying to read from memory 0x61000000 i am getting alternate data 0x61000000 and 0x60800000.

Remote BAR1 base address i am reading 0x00080000
Remote BAR2 base address i am reading 0x00100000

Can you please point out what mistake i am doing?

Thanks

0 Victor Kazmirenko over 8 years ago in reply to Arun Singh26

Guru 13202 points

Hello!

It looks like you are doing correct setup, but let us make sure there was no typo.

Arun Singh26 said:

and in remote configuration space
*0x21802010 = 0x00800000 (Remote BAR 1 raw register value)
*0x21802014 = 0x00800000 (Remote BAR 2 raw register value)

When i m trying to read from memory 0x60800000 i am getting alternate data 0x60800000 and 0.
and When i m trying to read from memory 0x61000000 i am getting alternate data 0x61000000 and 0x60800000.

Remote BAR1 base address i am reading 0x00080000
Remote BAR2 base address i am reading 0x00100000

I see discrepancies in the first two lines, should be 0x0080_0000 and 0x0100_0000. And that does not match last two lines of your statement.

In the original post you mentioned Device / Vendor IDs were swapped somehow, is it still observed? What you see and what was programmed on FPGA side?

May we know, what is your ability to debug FPGA? If you are with Xilinx, do you have programming cable and programming header on the board? Are you familiar with ChipScope?

0 Arun Singh26 over 8 years ago in reply to Victor Kazmirenko

Prodigy 145 points

hey!
Let me answer your questions one by one:

This is not Typo error, first two lines i am referring from memory browser window, and then i am reading the registers through read reg command.

getRegs_rem.vndDevId = &vend_test_rem;
getRegs_rem.type0BarIdx = &type0BarIdx_rem;
if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs_rem)) != pcie_RET_OK)
{
System_printf ("Remote Vedor ID read failed!\n");
}

Then i am observing structure type0Baridx_rem, in that the base address is 0x00080000 (base address is raw register[31:20] ).

Same thing with vendor ID and device ID, when i am observing in memory browser (*0x21802000) their order is correct (Device ID in higher 16 bits), but in structure vend_test_rem their values are swapped(vendor id in place of device ID and vice versa) .

FPGA Device = kintex 7
Tool = Xilinx vivado
Yes i have programming cable and programming header on the board, i am familiar with Chipscope but i used that on xilinx ISE, in vivado it may take some time.

I think from FPGA side i am getting data in correct sequence, only thing is with interpretation.

0 Victor Kazmirenko over 8 years ago in reply to Arun Singh26

Guru 13202 points

OK, it seems that at least with Vnd/Dev ID we see the same. I've just checked in my setup, for

parameter  [15:0] CFG_VEN_ID                        = 16'h10EE,
parameter  [15:0] CFG_DEV_ID                        = 16'h0007,

I see

======> Remote configuration =====>
VndDevId: raw = 0x710ee, Vnd = 0x7, Dev = 0x10ee

I think I've traced the reason and it looks as CSL error to me, but we'd better hear developers too.

In \TI\pdk_C6670_1_1_2_6\packages\ti\drv\pcie\src\pciecfg.c the very first function is

pcieRet_e pcie_read_vndDevId_reg
(
  CSL_Pcie_cfg_space_endpointRegs   *baseAddr, 
  pcieVndDevIdReg_t *reg
)
{
  uint32_t val = reg->raw = baseAddr->VENDOR_DEVICE_ID;

  pcie_getbits(val, CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_VENDOR_ID,  reg->vndId);
  pcie_getbits(val, CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_DEVICE_ID,  reg->devId);

  return pcie_RET_OK;
} /* pcie_read_vndDevId_reg */

pcie_getbits is simple macro as

#define pcie_getbits(val,field,final_result) \
  final_result = (val & field##_MASK) >> field##_SHIFT;

and finally in TI\pdk_C6670_1_1_2_6\packages\ti\csl\cslr_pcie_cfg_space_endpoint.h we see:

/* VENDOR_DEVICE_ID */

#define CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_VENDOR_ID_MASK (0xFFFF0000u)
#define CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_VENDOR_ID_SHIFT (0x00000010u)
#define CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_VENDOR_ID_RESETVAL (0x00008888u)

#define CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_DEVICE_ID_MASK (0x0000FFFFu)
#define CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_DEVICE_ID_SHIFT (0x00000000u)
#define CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_DEVICE_ID_RESETVAL (0x0000104Cu)

#define CSL_PCIE_CFG_SPACE_ENDPOINT_VENDOR_DEVICE_ID_RESETVAL (0x8888104Cu)

Mask and shift suggest that VndID was expected in upper halfword, same as reset value of 0x8888 was put there.

I believe you may disregard vendor / device ID swap for a while, I see exactly same swap and that makes no diagnostic importance to your case.

Now closer to configuration. In my case I have

#define FPGA_BAR0_BASE      0x00000000
#define FPGA_BAR1_BASE      0x08000000

When I look at FPGA with memory browser I see

0x21802000	000710EE	00100446	05800000	00000000	00000000
0x21802014	08000000	00000000	00000000	00000000	00000000

Note, BAR0 at 0x21802010 reads as zero, BAR1 at 0x21802014 reads as 08000000, exactly same as I configured. However in your case we see

Arun Singh26 said:

*0x21802010 = 0x00800000 (Remote BAR 1 raw register value)
*0x21802014 = 0x00800000 (Remote BAR 2 raw register value)

where both BARs reads same, while BAR1 should be barCfg.base = 0x01000000.

When I asked about FPGA debugging, I would plug on transaction interface and monitor bar_hit signal along with trn_rd and trn_td. Knowing TLP formal you can easily decode addresses in read requests and payload of returned completers. But Vivado, I perfectly understand you, let's try to proceed step by step at DSP side first.

0 Victor Kazmirenko over 8 years ago in reply to Victor Kazmirenko

Guru 13202 points

Because you used indices 0 and 1, remote BARs are BAR0 and BAR1, and yes, their location is at 0x21802010 and 0x21802014, they are just mistakenly referred as BAR1 and BAR2. However, their values should match what you have programmed, but right now they do not.

0 Arun Singh26 over 8 years ago in reply to Victor Kazmirenko

Prodigy 145 points

sorry, i didn't check about this typo error, i am actually getting correct value at those addresses, actually when i reading those register by Pcie_readRegs command, in that structure, base address is only different. But now i can assume that may be because of CSL bug, only thing that matters is register raw value, which is correct, still i am getting garbage value when i tried to read BAR registers.

*0x21802010 = 0x00800000 (Remote BAR 0 raw register value)
*0x21802014 = 0x01000000 (Remote BAR 1 raw register value)

If i am correct upto this stage, what are the other possibilities of getting garbage data, and additional thing to notice is that value at (0x60800000) is same register address.

can you tell me what should be remote command and status register(0x21802004) value, in remote configuration space.

0 Arun Singh26 over 8 years ago in reply to lding

Prodigy 145 points

Hi lding,
Kindly look into recent updates in my previous posts, and point out what mistake i am doing in data read.

0 Victor Kazmirenko over 8 years ago in reply to Arun Singh26

Guru 13202 points

Remote BARs seems OK.

My value of remote Status and Command Register is 0X00100446 which is:

 1 MEM_SP
 2 BUS_MS
 6 PAR_ERR_RESP
10 INTX_DIS
20 CAP_LIST

Nothing special, and I disabled legacy INTx because we use MSIs.

Here is my reading of the FPGA config space:

0x21802000	000710EE	00100446	05800000	00000000	00000000	08000000	00000000	00000000
0x21802020	00000000	00000000	00000000	000710EE	00000000	00000040	00000000	000000FF
0x21802040	7E034801	00000008	00DB5805	20000054	00000000	00000000	00010010	00008FC0
0x21802060	00001810	0003F411	00110000	00000000	00000000	00000000	00000000	00000000
0x21802080	00000000	00000000	00000000	00000000	00000000	00000000	00000000	00000000
0x218020A0	00000000	00000000	00000000	00000000	00000000	00000000	00000000	00000000
0x218020C0	00000000	00000000	00000000	00000000	00000000	00000000	00000000	00000000
0x218020E0	00000000	00000000	00000000	00000000	00000000	00000000	00000000	00000000
0x21802100	00010003	01000A35	00000001	00000000	00000000	00000000	00000000	00000000

Note, I had to use MSI, thus my settings might be somewhat different together with bases. However, I don't see, what else could be screwed here.

May we know, what kind of application is on FPGA side? What data you expect at your BARs?

0 Arun Singh26 over 8 years ago in reply to Victor Kazmirenko

Prodigy 145 points

hi,
My value of remote Status and Command Register is 0X00100000, i think this is the prime concern.
On FPGA Side i did basic PCIe core initialization in which i configured two bars of 8MB size.
I want to loopback the data first then i will develop further, I am trying to write data from DSP and then read the same data.

0 Victor Kazmirenko over 8 years ago in reply to Arun Singh26

Guru 13202 points

Hello,

Indeed, it is.

Let me guess: you make configuration of remote BARs, but you do not explicitly configure Status and Command register. If so, you stepped same trap as I did 3 years ago.

TI examples are technically correct. However, as RC, so EP do configure their config registers on their own, and that's easy to do as they are processors. With FPGA the situation is a bit different. The logic behind FPGA's EP operation is to get ready and serve configuration requests from RC. Thus, it is responsibility of RC, DSP in your case to configure remote Status and Command register. Config space layout of FPGA matches that of DSP, so you can use something like:

pcieRegisters_t         setRegs;
pcieStatusCmdReg_t      StatusCmd;

memset( &setRegs,   0, sizeof(setRegs)   );
memset( &StatusCmd, 0, sizeof(StatusCmd) );

setRegs.statusCmd = &StatusCmd;
Pcie_readRegs(handle, pcie_LOCATION_REMOTE, &setRegs);

//StatusCmd.busMs = 1;    // Enable bus mastering
//StatusCmd.dis   = 1;    // Set DIS to disable Legacy Intr --> use MSI
//StatusCmd.resp  = 1;
StatusCmd.memSp = 1;

Pcie_writeRegs(handle, pcie_LOCATION_REMOTE, &setRegs);

To my understanding, setting MEM_SP bit is required for device to respond memory accesses. For your reference I left commented field making possible busmastering and disabling legacy INTx in favour of MSI. IF you ever need them, be cautious, layout of pcie capabilities in FPGA may differ from DSP.

ANd just in case, I remember, basic PIO design for Spartan 6 was providing some small memory as target of PIO access. Do you have such a memory in your design&

0 Arun Singh26 over 8 years ago in reply to Victor Kazmirenko

Prodigy 145 points

Hello,

Actually that command and status register was in my mind, so what i did i ran the same DSP code in EP mode and then checked the register value, it was 0X00100146. So i simply wrote that on remote configuration command register. But still that problem persists in my case. Actually in that DSP example code both DSP are configured in same manner and their Command register value is exactly same.
Actually data at that PCIe bar is same as that of address space value (0x60800000), does it give any hint?

Yes, i do have access to that memory as i am using same example in FPGA case.

Thanks

0 Victor Kazmirenko over 8 years ago in reply to Arun Singh26

Guru 13202 points

Well, if you have configured FPGA's status command register, it looks like that's all we can do on DSP side.
As to the strange readings you have, it's hard for me to judge. You may think, that DWORDs of incoming TLP are sliced one by one, and third word of header carrying address is latched some wrong way instead of data. I'm afraid, its time to plug in ChipScope.

0 user4248043 over 8 years ago in reply to Victor Kazmirenko

Prodigy 210 points

I get the same error, and I tracked this post for several days.

My EP is a Kintex-7 running with "7 Series FPGAs Integrated Block for PCI Express v3.2 ".

Hope someone can fix and share.

Fullscreen 7384.log20180330.txt Download

[C66xx_0] **********************************************
*             PCIe Test Start                *
*                RC mode                     *
**********************************************

Version #: 0x01000003; string PCIE LLD Revision: 01.00.00.03:Feb 27 2018:08:21:39

Power domain is already enabled.  You probably re-ran without device reset (which is OK)
PCIe Power Up.
PLL configured.
agan>> Config OB region 0 OK
agan>> Config OB region 1 OK
Starting link training...
Link is up.
bar size[1]: raw: fff00000, 00100000
1111bar[0][0]: 00000000
1111bar[1][800000]: 00800000
handle rc mode test.
vid raw: 702210ee, devId = 10ee, vndId = 7022
revid raw: 5800000, classCode = 58000, revId = 0
bar[0]: raw:00000000 base 00000000 prefetch 00 type 0 [memory]
bar[1]: raw:00800000 base 00080000 prefetch 00 type 0 [memory]
bar[2]: raw:00000000 base 00000000 prefetch 00 type 0 [memory]
bar[3]: raw:00000000 base 00000000 prefetch 00 type 0 [memory]
bar[4]: raw:00000000 base 00000000 prefetch 00 type 0 [memory]
bar[5]: raw:00000000 base 00000000 prefetch 00 type 0 [memory]
write base address = 60800000
ADDR[60800000]: 00000000
ADDR[60800004]: 20b00000
ADDR[60800008]: 00000002
ADDR[6080000c]: 20b00000
ADDR[60800010]: 00000004
ADDR[60800014]: 20b00000
ADDR[60800018]: 00000006
ADDR[6080001c]: 20b00000
ADDR[60800020]: 00000008
ADDR[60800024]: 20b00000
ADDR[60800028]: 0000000a
ADDR[6080002c]: 20b00000
ADDR[60800030]: 0000000c
ADDR[60800034]: 20b00000
ADDR[60800038]: 0000000e
ADDR[6080003c]: 20b00000
ADDR[60800040]: 00000010
ADDR[60800044]: 20b00000
ADDR[60800048]: 00000012
ADDR[6080004c]: 20b00000
ADDR[60800050]: 00000014
ADDR[60800054]: 20b00000
ADDR[60800058]: 00000016
ADDR[6080005c]: 20b00000
ADDR[60800060]: 00000018
ADDR[60800064]: 20b00000
ADDR[60800068]: 0000001a
ADDR[6080006c]: 20b00000
ADDR[60800070]: 0000001c
ADDR[60800074]: 20b00000
ADDR[60800078]: 0000001e
ADDR[6080007c]: 20b00000
ADDR[60800080]: 00000020
ADDR[60800084]: 20b00000
ADDR[60800088]: 00000022
ADDR[6080008c]: 20b00000
ADDR[60800090]: 00000024
ADDR[60800094]: 20b00000
ADDR[60800098]: 00000026
ADDR[6080009c]: 20b00000

0 Victor Kazmirenko over 8 years ago in reply to user4248043

Guru 13202 points

Hello!

If you don't mind share your pcie setup code, I could run it on my machine just to make sure it works with proven design.

0 lding over 8 years ago in reply to Victor Kazmirenko

TI__Guru* 95265 points

rrlagic,
Thanks for helping for this topic and share many of your experience!

Arun and user4248043,
Can you provide a PCIE register dump from 0x2180_0000 to 0x2180_3000 using CCS memory window? Thanks!

Regards, Eric

0 Arun Singh26 over 8 years ago in reply to lding

Prodigy 145 points

Hello,
Thanks for replying to my queries.
I will provide you the code and ccs memory window on coming monday as rightnow i dnt hv access to my workstation.

Thanks

0 lding over 8 years ago in reply to Arun Singh26

TI__Guru* 95265 points

Any news?

Processors

Processors forum

RTOS/TMS320C6678: PCIe BAR Data read from FPGA endpoint