This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PCIe link troubles on C6672

I have a custom C6672 board (running as the RC) communicating with a separate board with an FPGA (running as the EP), connected using a laser wire PCIe interface. 

If the FPGA board is powered up first, I can always get a good PCIe link established.  I can cycle power on the DSP board, leaving the FPGA board powered, and the link comes up again, every time.

If, on the other hand, the DSP board is powered first, followed by the FPGA board, I can NEVER establish a link. 

Also, when running under the debugger:  I power the FPGA board first, then the DSP.  I load the debugger and the first time I run I get a good link.  If I exit and re-start the debugger, without power cycling the DSP, I can not get a successful link.

Since the FPGA board always re-links without power-cycling, as long as the DSP board is power-cycled, I'm assuming the problem isn't on the FPGA side.  I realize this is a big assumption.

I've tried to add a power down routine that is called if the link fails:

pcieRet_e pciePowerDownCfg (void)
{
 /* Turn off the PCIe power domain */
 if (CSL_PSC_getPowerDomainState (CSL_PSC_PD_PCIEX) != PSC_PDSTATE_OFF)
  {
 /* Enable the domain */
    CSL_PSC_disablePowerDomain (CSL_PSC_PD_PCIEX);
 /* Enable MDCTL */
    CSL_PSC_setModuleNextState (CSL_PSC_LPSC_PCIEX, PSC_MODSTATE_DISABLE);
 
 /* Apply the domain */
    CSL_PSC_startStateTransition (CSL_PSC_PD_PCIEX);
 /* Wait for it to finish */
 while (! CSL_PSC_isStateTransitionDone (CSL_PSC_PD_PCIEX));
  }
 else
  {
    System_printf ("Power domain is already disabled.\n");
  }
}
 
I was under the impression that this would reset the PCIe registers.  It appears to power down the PCIe, but after re-powering and going through setup I still fail link training.
Apparently, there is something in the register set that I'm not re-initializing properly, but I can't figure out what.  Any suggestions would be greatly appreciated.
  • Does the FPGA also provide the common PCIe reference clock to DSP please?

    It will make sense if there is some dependency between FPGA and DSP, which requires FPGA to power up first.

    When you power up DSP first, do you see the LOCK bit in PCIE_SERDES_STS (0x0262015c) register is 1 to indicate the PLL has been locked please?

    And what is the LTSSM_STATUS bit field in DEBUG0 (0x21801728) register when the link is not up? 

    Is the LTSSM_EN bit in CMD_STATUS register of DSP still be 1 when you power up the FPGA later please?

    In the debugger case, you could dump the PCIe registers space in both of the link up case and link down case to compare if any bits not being initialized correctly.

    Please note that in link-down case, we could not access remote register space (starting from 0x21800000+0x2000). So it is better to just capture the application registers (0x21800000~0x21800FFF) and the local configuration registers (0x21801000~0x21801FFF) in both cases for comparison.