PCIe link up c6678

On DSP side, could you confirm what changes you have made in the PCIe LLD example for your testing please?

From your current description, I think the following items have been modified:

pcieMode_e PcieModeGbl = pcie_EP_MODE; --> pcieMode_e PcieModeGbl = pcie_RC_MODE;

gen2.lnEn = 1; --> gen2.lnEn = 2;

Have you changed anything else please?

And could you please check the "DEBUG0" register (0x21801728) during the link up training process to see what values in the "LTSSM_STATE" field?

It should be "0x11" after the link up. Otherwise, it will be other values based on the decoding in Table A-1 in the PCIe user guide.

On the FPGA side, could you confirm if the it has been setup as EP correctly please?

If it is possible, you can try to connect two DSPs together and use the original PCIe LLD example to see if you could get the link up. 

It could rule out the variable of FPGA and software (LLD) and narrow down the issue to your testing setup (peer-to-peer connection).

Hi Steven,

I have made no further changes in my testing.

The register “Debug0” has the value 0x00004A03 during the linkung up.

I could not confirm that the FPGA work correctly as EP. I have tested the functionality only in a testbench yet. There it works fine. But I will look up again the FPGA design.

Further I have no ability to connect two DSPs together or to connect one to a PC to test the PCIe LLD example to see if I get the link up.

Regards serg05

It looks like the PCIe link training is stuck at "Polling.Compliance" status (last 4 bits 0x03 in DEBUG0 register).

It might be at least one Lane that detected a receiver during "Detect" status has never detected an exit from the Electrical Idle state on its receiver since entering "Polling.Active" status.

It indicates the presence of a passive test load, such as a resistor, on at lease one Lane, which will force all Lanes to enter Polling.Compliance.

Could you please refer to the C6678 Hardware Design Guide or the FPGA hardware design guide to see if the proper register/capacitor are being used on the DSP and FPGA PCIe link on your board?

Hi,

I solve my last problem. The PCIe clock was to worse that the pll in the fpga don't lock correctly.

Now I have a link from DSP as RC to FPGA as EP. I wanted to send one word to bar1 at address 88 to switch on/off the led on the board. But this don't work and this lead me to the issue of what did I wrong.

On the DSP side I have configured the Outbound address( PCIE_OB_LO_ADDR_M 0x70000000,PCIE_OB_HI_ADDR_M 0), Inbound address(PCIE_IB_LO_ADDR_M 0x90000000,PCIE_IB_HI_ADDR_M 0) and the EP FPGA

(

pcieRet_e retVal;

pcieRegisters_t getRegs, setRegs;

pcieCfgTransReg_t cfgTrans;

pcieVndDevIdReg_t idReg;

memset (&idReg, 0, sizeof(idReg));

memset(&cfgTrans, 0, sizeof(cfgTrans));

memset (&getRegs, 0, sizeof(getRegs));

getRegs.cfgTrans = &cfgTrans;

if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)

{

System_printf ("Read CFG_SETUP register failed!\n");

}

pcieBarCfg_t barCfg;

/* Configure Address Translation */

barCfg.location = pcie_LOCATION_REMOTE;

barCfg.mode = pcie_EP_MODE;

barCfg.base = 0x70000000;

barCfg.prefetch = pcie_BAR_PREF;

barCfg.type = pcie_BAR_TYPE64;

barCfg.memSpace = pcie_BAR_MEM_MEM;

barCfg.idx = 0;

if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK)

{

System_printf ("Failed to configure BAR!\n");

exit(1);

}

Afterwards I controlled the settings

volatile UInt32 *epCfg = (UInt32 *)0x21802000;

int n=0;

for(n = 0; n < 116; n+=4)

{

System_printf("offset: 0x%x : %08x %08x %08x %08x\n", n*4, epCfg[n+0], epCfg[n+1], epCfg[n+2], epCfg[n+3]);

}

The output was

[C66xx_0] offset: 0x0 : 000710ee 00100000 05000000 00000000

[C66xx_0] offset: 0x10 : 70000004 00000000 00000000 00000000

[C66xx_0] offset: 0x20 : 00000000 00000000 00000000 000710ee

[C66xx_0] offset: 0x30 : 00000000 00000040 00000000 000001ff

[C66xx_0] offset: 0x40 : 00034801 00000000 01806005 00000000

[C66xx_0] offset: 0x50 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x60 : 00010010 00008fea 00002810 0003f421

[C66xx_0] offset: 0x70 : 10210000 00000000 00000000 00000000

[C66xx_0] offset: 0x80 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x90 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0xa0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0xb0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0xc0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0xd0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0xe0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0xf0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x100 : 00010003 00000000 00000000 14410001

[C66xx_0] offset: 0x110 : 00000000 00000000 00062010 00000000

[C66xx_0] offset: 0x120 : 00002000 00000000 00000000 00000000

[C66xx_0] offset: 0x130 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x140 : 00000000 10010004 00000000 00000000

[C66xx_0] offset: 0x150 : 00000000 00010002 00000000 00000000

[C66xx_0] offset: 0x160 : 00000000 00004000 800000ff 00000000

[C66xx_0] offset: 0x170 : 00004000 01000000 00020000 00000000

[C66xx_0] offset: 0x180 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x190 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x1a0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x1b0 : 00000000 00000000 00000000 00000000

[C66xx_0] offset: 0x1c0 : 00000000 00000000 00000000 00000000

I wonder why the address of bar1 at 0x10 has the value of 70000004 and not 70000000! But further to sending words to bar1.

*(UInt32 *)0x70000088=0x000000FF;

volatile UInt32 *bar1remote = (UInt32 *)0x70000000;

for(n = 0; n < 116; n+=4)

{

System_printf("offset: 0x%x : %08x %08x %08x %08x\n", n*4, bar1remote[n+0], bar1remote[n+1], bar1remote[n+2], bar1remote[n+3]);

}

I write to the outbound address to send 0x000000FF to bar1 of ep.

The leds aren't switched on and if I try to read in bar1 of ep I get only one value 000001c0.

Regards serg05

Hi Steven,

thank you for you help! It works all. I have written on the wrong address.

Regards serg05

 i have a similar problem with you 

you said "The PCIe clock was to worse that the pll in the fpga don't lock correctly."wolud you please share something about how to solve it ?

thank you

Hi,

I don't know which FPGA you use. I worked with Xilinx IP Core V5-Block Plus for PCI Express. There it was possible to activate the debug mode “GTP_DEBUG” and control if the PLL is locked and the clocks are right.

If the PLL don't lock control

• if you used DC coupling capacitors an the clock lines.

• the voltage level on the clock line.

• Control the input jitter, it should be less than 300 ppm

I hope I could help you

good luck!

Regards serg05

dear serg05:

i don't quite solve the link up problem, but there is a big chance to link up now ,so  i want to carry on with the register read/write and data transfer.

in my board ,i connect a 6678 with a v6 fpga  using  PCIE, while yours is a v5 FPGA.

now, i use the "PCIE_exampleProject" for test ,and after link up,  the 

pcieReadDeviceIDReg()

pcieReadPMStatusReg()

pcieEnumEpConfig()

pcieReadLnkStatCtrlReg()

are used to read remote(fpga) register ,but the result is not quite right ,a followed:

[C66xx_0] Location: 1 vndDevId: 0x0

[C66xx_0] Location: 1 vndDevId: 0x0

[C66xx_0] Location: 1 PMEnable: 0x1

[C66xx_0] Location: 1 pmeStatus: 0x0 pmeReqID: 0x98fc

[C66xx_0] Location: 1 aspmL1: 0x0 l1entry: 0x0 l0sentry: 0x0 commnf: 0x86 numfts: 0x98 ackfreq: 0xf0

[C66xx_0] CFG SETUP before enumeration: bus 0, device 0, func 0, type 0

[C66xx_0] Location: 0 BusEnable: 0x1  Memory bit is: 0x1

[C66xx_0] Location: 1 BusEnable: 0x0  Memory bit is: 0x0

[C66xx_0] Done enumeration FAILED

[C66xx_0] LnkBWStat: 0x0, dllactive: 0x0, sltClk: 0x1, LnkTrn: 0x0, commonClk: 0x0, activeLnkPM: 0x0

[C66xx_0] ti.sysbios.heaps.HeapMem: line 311: assertion failure: A_invalidFree: Invalid free

[C66xx_0] xdc.runtime.Error.raise: terminating execution

 i have two question :

1. how do you  read the fpga side register, does it have the same structure mapped to the dsp "PCIe Remote Configuration Space"?

2. how to write data to fpga via pcie?  write data into PCIe data field?(0x60000000-0x6fffffff)after link up ?

Thank you  in advanced!

Hi,

1. how do you  read the fpga side register, does it have the same structure mapped to the dsp "PCIe Remote Configuration Space"?

At first you have to configure outbound address translation with the function pcieObTransCfg (handle, PCIE_OB_LO_ADDR_M, PCIE_OB_HI_ADDR_M, PCIE_OB_REGION_M)

after that you can read the register with the function Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs)

1. how to write data to fpga via pcie?  write data into PCIe data field?(0x60000000-0x6fffffff)after link up ?

If you have set outbound address translation for example on the value PCIE_OB_LO_ADDR_M = 0x70000000 and PCIE_OB_HI_ADDR_M = 0x00000000 than after write in address field of dsp (0x60000000-0x6fffffff) you will write in pcie address field 0x70000000 - 0x7fffffff.

I hope I could help you

good luck

sometimes   i can link up on both sides,and i read the remote reigster out just like your method

and the result came out like this:

[C66xx_0] local sideoffset:0x0:8888104c  00100146  00000001  00010000 

[C66xx_0] remote sideoffset:0x0:8e2810a2  00873680  21802000  00000000 

[C66xx_0] offset:0x10:8e2810a2  00873680  21802000  00000004 

[C66xx_0] offset:0x20:8e2810a2  00873680  21802000  00000008 

[C66xx_0] offset:0x30:8e2810a2  00873680  21802000  0000000c 

[C66xx_0] offset:0x40:8e2810a2  00873680  21802000  00000010 

[C66xx_0] offset:0x50:8e2810a2  00873680  21802000  00000014 

[C66xx_0] offset:0x60:8e2810a2  00873680  21802000  00000018 

[C66xx_0] offset:0x70:8e2810a2  00873680  21802000  0000001c 

[C66xx_0] offset:0x80:8e2810a2  00873680  21802000  00000020 

[C66xx_0] offset:0x90:8e2810a2  00873680  21802000  00000024 

[C66xx_0] offset:0xa0:8e2810a2  00873680  21802000  00000028 

[C66xx_0] offset:0xb0:8e2810a2  00873680  21802000  0000002c 

[C66xx_0] offset:0xc0:8e2810a2  00873680  21802000  00000030 

[C66xx_0] offset:0xd0:8e2810a2  00873680  21802000  00000034 

[C66xx_0] offset:0xe0:8e2810a2  00873680  21802000  00000038 

[C66xx_0] offset:0xf0:8e2810a2  00873680  21802000  0000003c 

[C66xx_0] offset:0x100:8e2810a2  00873680  21802000  00000040 

[C66xx_0] offset:0x110:8e2810a2  00873680  21802000  00000044 

[C66xx_0] offset:0x120:8e2810a2  00873680  21802000  00000048 

[C66xx_0] offset:0x130:8e2810a2  00873680  21802000  0000004c 

[C66xx_0] offset:0x140:8e2810a2  00873680  21802000  00000050 

[C66xx_0] offset:0x150:8e2810a2  00873680  21802000  00000054 

[C66xx_0] offset:0x160:8e2810a2  00873680  21802000  00000058 

[C66xx_0] offset:0x170:8e2810a2  00873680  21802000  0000005c 

[C66xx_0] offset:0x180:8e2810a2  00873680  21802000  00000060 

[C66xx_0] offset:0x190:8e2810a2  00873680  21802000  00000064 

[C66xx_0] offset:0x1a0:8e2810a2  00873680  21802000  00000068 

[C66xx_0] offset:0x1b0:8e2810a2  00873680  21802000  0000006c 

[C66xx_0] offset:0x1c0:8e2810a2  00873680  21802000  00000070 

i think it make no sense

and to be honest ,i don't quite get it in these sentence,

If the PLL don't lock control

• if you used DC coupling capacitors an the clock lines.

• the voltage level on the clock line.

• Control the input jitter, it should be less than 300 ppm

i can hardly have a link up between my dsp and fpga,

only in a few times, i can succeed.

 can you explain it for me further more ?

thank you !

Hi serg05,

we have the same design, c6678(rc)  Fpga(EP),but i can not write date to FPGA from PCIE.

    volatile UInt32 *epCfg = (UInt32 *)0x21802000;

    int n=0;

    for(n = 0; n < 116; n+=4)
    {

      System_printf("offset: 0x%x : %08x %08x %08x %08x\n", n*4, epCfg[n+0], epCfg[n+1], epCfg[n+2], epCfg[n+3]);

    }

[C66xx_0] offset: 0x0 : 000710ee 00100000 05800000 00000000
[C66xx_0] offset: 0x10 : 70000000 00000000 00000000 00000000
[C66xx_0] offset: 0x20 : 00000000 00000000 00000000 000710ee
[C66xx_0] offset: 0x30 : 00000000 00000040 00000000 000001ff
[C66xx_0] offset: 0x40 : 7e034801 00000008 00805805 00000000
[C66xx_0] offset: 0x50 : 00000000 00000000 00010010 00008fc0
[C66xx_0] offset: 0x60 : 00002810 0003f411 00110000 00000000
[C66xx_0] offset: 0x70 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x80 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x90 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0xa0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0xb0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0xc0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0xd0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0xe0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0xf0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x100 : 00010003 01000a35 00000001 00000000
[C66xx_0] offset: 0x110 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x120 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x130 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x140 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x150 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x160 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x170 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x180 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x190 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x1a0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x1b0 : 00000000 00000000 00000000 00000000
[C66xx_0] offset: 0x1c0 : 00000000 00000000 00000000 00000000

can you give me your code on c6678,email : zengxing_ok@163.com

thank you!

Regards zengxing