This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Problem on C6678 to work as root complex in GEN2 x2

Hi,

I modified the CSL PCIe example to create a C6678 RC design. It works fine with my FPGA endpoint when the RC was configured in GEN1 mode with 1 lane only. That's the default setting of the reference design. When I changed the RC to work in GEN2 mode with 2 lanes. It didn't work. Linkup failed. 

Here is the major change on the code. Modifed the value of PL_LINK_CTRL and PL_GEN2 registers.

pcieRet_e pcieCfgRC(Pcie_Handle handle)
{

.......

.......

/* Setting PL_GEN2 */
    memset (&setRegs, 0, sizeof(setRegs));
    gen2.numFts = 0xF;
    gen2.dirSpd = 1; // 0 -> not to gen2, 1 -> gen2
    gen2.lnEn   = 2; // 1 -> 1xlane, 2 -> 2xlane
    setRegs.gen2 = &gen2;
    
    lnkCtrl.lnkMode = 3;
    setRegs.lnkCtrl = &lnkCtrl;

    if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    {
        System_printf ("SET GEN2 register failed!\n");
        return retVal;
    }

.....

.....

}

Any idea??? Any working example code for RC+GEN2+2lanes???

Watson

  • Hi Watson,

    Welcome to the TI E2E forum. I hope you will find many good answers here and in the TI.com documents and in the TI Wiki Pages (for processor issues). Be sure to search those for helpful information and to browse for the questions others may have asked on similar topics (e2e.ti.com).

    Please go through below wiki for PCIe related FAQs and Resources.

    http://processors.wiki.ti.com/index.php/PCI_Express_(PCIe)_Resource_Wiki_for_Keystone_Devices

    We will get back to you on the above query shortly. Thank you for your patience.

  • Hi,

    No need to update LNK_MODE on your code, the reset value of this register field is 0x3. For more information refer pcie user guide.

    Try the below code for your testing.
    /* Setting PL_GEN2 */
    memset (&setRegs, 0, sizeof(setRegs));
    gen2.numFts = 0xF;
    gen2.dirSpd = 0;
    gen2.lnEn = 2; // 1 -> 1xlane, 2 -> 2xlane
    setRegs.gen2 = &gen2;

    Have you confirm your end point device(FPGA) supports PCIe gen2+x2?

    Thanks,
  • Ganapathi

    Thanks for your quick reply.

    1. My endpoint supports GEN2 x 4. It works well in PC.

    2. "No need to update LNK_MODE on your code, the reset value of this register field is 0x3"
    I think it's no harm to set it again in my code. However, when I add the below two lines, the link up process can still finish (GEN1 x1 detected) but RC hangs up. RC hangs up at the point when I try to read endpoint vendor ID and device ID.
    ----------- add these two lines ------------------------
    lnkCtrl.lnkMode = 3;
    setRegs.lnkCtrl = &lnkCtrl;
    --------------------------------------------------------------

    I also read few registers just after the linkup (for debug purpose)
    debug0 = 0x1004a11
    debug1 = 0x8000410
    linkStatCtrl = 0x10210008

    Why?

    3. Try the below code for your testing.
    /* Setting PL_GEN2 */
    memset (&setRegs, 0, sizeof(setRegs));
    gen2.numFts = 0xF;
    gen2.dirSpd = 0;
    gen2.lnEn = 2; // 1 -> 1xlane, 2 -> 2xlane
    setRegs.gen2 = &gen2;

    RC can finish linkup process with GEN1 x2. In this case, I can read endpoint vendor ID and device ID successfully. Also, read and write endpoint memory successfully. But I need GEN2 x2. That's my goal. How to make GEN2 happen?

    Some more info:
    - I'm using TMDXEVM6678L_EVM from Advantech.
    - Code Composer Studio Version: 5.5.0.00077
    - pdk_C6678_1_1_2_6
    - mcsdk_2_01_02_06

    Watson
  • Hi,

    Refer section "5 PCIe Programming Example" on PCIe Use Cases for KeyStone Devices Application Report document. It should be help you for configure GEN1 and GEN2.
    www.ti.com/.../sprabk8.pdf

    Thanks,
  • Ganapathi,

    Of course I have read those available doc from TI about C6678 PCIe including sprabk8.pdf. I found from the doc that it's necessary to program PL_LINK_CTRL register for GEN2. I followed but have the problem mentioned in my earlier post. 

    I'm attaching my code here. Please help to point out the fault.

    pcieRootComplex.c
    /* C Standard library include */
    #include <string.h>
    #include <cerrno>
    #include <stdio.h>
    #include <stdlib.h>
    
    
    /* CSL include */
    #include <ti/csl/csl_bootcfgAux.h>
    #include <ti/csl/csl_cacheAux.h>
    #include <ti/csl/csl_chip.h>
    #include <ti/csl/csl_xmcAux.h>
    
    
    #include <ti/csl/csl_pscAux.h>
    #include <ti/csl/cslr_device.h>
    #include <ti/csl/csl_edma3.h>
    #include <ti/csl/csl_edma3Aux.h>
    #include <ti/csl/csl_cacheAux.h>
    
    /* PCIE LLD include */
    #include <ti/drv/pcie/pcie.h>
    
    
    /* Platfrom library include */
    #include "ti\platform\platform.h"
    #include "ti\platform\resource_mgr.h"
    #include <ti/platform/evmc6678l/platform_lib/include/types.h>
    #include <ti/platform/evmc6678l/platform_lib/include/evmc66x_fpga.h>
    #include <ti/platform/evmc6678l/platform_lib/include/evmc66x_gpio.h>
    #include <ti/platform/platform.h>
    
    
    /* PCIe DMA */
    #include "pciedma.h"
    
    /*************************************************************************
    ** PCIE defines
    **************************************************************************
    */
    
    /* Size of application buffers */
    #define PCIE_BUFSIZE_APP 256
    
    /* Write pattern */
    #define PCIE_WR_PATTERN 0xFACEFACE
    
    /* In this example all addresses are 32bit */
    /* Outbound Base Address for PCIe Master */
    #define PCIE_OB_LO_ADDR_M    0x70000000
    #define PCIE_OB_HI_ADDR_M    0
    
    /* Inbound  Base Address for PCIe Master */
    #define PCIE_IB_LO_ADDR_M    0x90000000
    #define PCIE_IB_HI_ADDR_M    0
    
    /* Outbound Base Address for PCIe Slave */
    #define PCIE_OB_LO_ADDR_S    PCIE_IB_LO_ADDR_M
    #define PCIE_OB_HI_ADDR_S    0
    
    /* Inbound  Base Address for PCIe Slave */
    #define PCIE_IB_LO_ADDR_S    PCIE_OB_LO_ADDR_M
    #define PCIE_IB_HI_ADDR_S    0
    
    /* BAR mask */
    #define PCIE_BAR_MASK         0x0FFFFFFF
    
    /* BAR Index PCie*/
    #define PCIE_BAR_IDX_M         1
    #define PCIE_BAR_IDX_S         1
    
    /* PCIe Regions used in the example */
    #define PCIE_IB_REGION_M    0
    #define PCIE_OB_REGION_M    0
    #define PCIE_IB_REGION_S    0
    #define PCIE_OB_REGION_S    0
    
    
    /////////////
    #define DEVICE_REG32_W(x,y)   *(volatile uint32_t *)(x)=(y)
    #define DEVICE_REG32_R(x)    (*(volatile uint32_t *)(x))
    
    #define DDR_TEST_START                 	0x80000000
    #define DDR_TEST_END                   	0x80400000
    #define BOOT_UART_BAUDRATE            	115200
    #define PCIEXpress_Legacy_INTA        	50
    #define PCIE_IRQ_EOI                   	0x21800050
    #define PCIE_EP_IRQ_SET		           	0x21800064
    #define LEGACY_A_IRQ_STATUS_RAW       	0x21800180
    #define LEGACY_A_IRQ_STATUS   			0x21800184
    #define LEGACY_A_IRQ_ENABLE_SET       	0x21800188
    #define LEGACY_A_IRQ_ENABLE_CLR       	0x2180018c
    ////////////
    
    /*************************************************************************
    ** Global variables
    **************************************************************************
    */
    #pragma DATA_SECTION(pcieBuf, ".pcieBufSec")
    /* Cache coherence: Align must be a multiple of cache line size (L2=128 bytes, L1=64 bytes) to operate with cache enabled. */
    /* Aligning to 256 bytes because the PCIe inbound offset register masks the last 8bits of the buffer address  */
    #pragma DATA_ALIGN(pcieBuf, PCIE_BUFSIZE_APP*2)
    /* last element in the buffer is a marker that indicates the buffer status: full/empty */
    
    struct pcieBuf_s {
    	volatile uint32_t srcBuf[PCIE_BUFSIZE_APP/4];
    	volatile uint32_t dstBuf[PCIE_BUFSIZE_APP/4];
    } pcieBuf;
    
    
    /* Does not need to be aligned (even for cache) since it is only accessed locally */
    //uint32_t srcBuf[PCIE_BUFSIZE_APP];
    
    extern volatile unsigned int cregister TSCL;
    
    /* Global config variable that controls
       the PCIe mode. It is global so it can be poked
       from CCS. It should be set either to EP or RC. */
    pcieMode_e PcieModeGbl = pcie_RC_MODE;
    
    
    /*************************************************************************
    ** Functions prototypes
    **************************************************************************
    */
    void edma_test (void);
    Int32 edmaRc2Ep (Int32 instNum, Uint8 channelNum, void *srcBuff, void *dstBuff, Int32 trSize);
    void initInterruptSystem (void);
    void setBenchmark(int32_t pinNum);
    void clrBenchmark(int32_t pinNum);
    
    /*************************************************************************
    ** Functions
    **************************************************************************
    */
    
    void cycleDelay (uint32_t count);
    
    /* Use regular "printf" when not using BIOS */
    //#include <xdc/runtime/System.h>
    #define System_printf printf
    
    /* OSAL functions for Platform Library */
    uint8_t *Osal_platformMalloc (uint32_t num_bytes, uint32_t alignment)
    {
        return malloc(num_bytes);
    }
    
    void Osal_platformFree (uint8_t *dataPtr, uint32_t num_bytes)
    {
        /* Free up the memory */
        if (dataPtr)
        {
            free(dataPtr);
        }
    }
    
    void Osal_platformSpiCsEnter(void)
    {
        /* Get the hardware semaphore.
         *
         * Acquire Multi core CPPI synchronization lock
         */
        while ((CSL_semAcquireDirect (PLATFORM_SPI_HW_SEM)) == 0);
    
        return;
    }
    
    void Osal_platformSpiCsExit (void)
    {
        /* Release the hardware semaphore
         *
         * Release multi-core lock.
         */
        CSL_semReleaseSemaphore (PLATFORM_SPI_HW_SEM);
    
        return;
    }
    
    
    
    /*****************************************************************************
     * Function: Power domain configuration
     ****************************************************************************/
    pcieRet_e pciePowerCfg(void)
    {
    	/* Turn on the PCIe power domain */
    	if (CSL_PSC_getPowerDomainState (CSL_PSC_PD_PCIEX) != PSC_PDSTATE_ON) {
    		/* Enable the domain */
    		CSL_PSC_enablePowerDomain (CSL_PSC_PD_PCIEX);
    		/* Enable MDCTL */
    		CSL_PSC_setModuleNextState (CSL_PSC_LPSC_PCIEX, PSC_MODSTATE_ENABLE);
    		/* Apply the domain */
    		CSL_PSC_startStateTransition (CSL_PSC_PD_PCIEX);
    		/* Wait for it to finish */
    		while (! CSL_PSC_isStateTransitionDone (CSL_PSC_PD_PCIEX));
    	} else {
    		System_printf ("Power domain is already enabled.  You probably re-ran without device reset (which is OK)\n");
    	}
    
      return pcie_RET_OK;
    }
    
    /*****************************************************************************
     * Function: Utility function to introduce delay
     ****************************************************************************/
    void cycleDelay (uint32_t count)
    {
      uint32_t sat;
    
      if (count <= 0)
        return;
    
      sat = TSCL + count;
      while (TSCL < sat);
    }
    
    /*****************************************************************************
     * Function: Serdes configuration
     ****************************************************************************/
    pcieRet_e pcieSerdesCfg(void)
    {
      uint16_t cfg;
    
      CSL_BootCfgUnlockKicker();
    
      /* Provide PLL reference clock to SERDES inside PCIESS
         Program PLL settings and enable PLL from PCIe SERDES.*/
      cfg = 0x01C9; /* value based on PCIe userguide */
      CSL_BootCfgSetPCIEConfigPLL(cfg);
    
      CSL_BootCfgLockKicker();
    
      /*Wait for PLL to lock (3000 CLKIN1 cycles) */
      cycleDelay(10000);
    
      return pcie_RET_OK;
    }
    
    /*****************************************************************************
     * Function: Enable/Disable LTSSM (Link Training)
     ****************************************************************************/
    pcieRet_e pcieLtssmCtrl(Pcie_Handle handle, uint8_t enable)
    {
      pcieCmdStatusReg_t    cmdStatus;
      pcieRegisters_t       setRegs;
      pcieRegisters_t       getRegs;
      pcieRet_e retVal;
    
      memset (&cmdStatus,    0, sizeof(cmdStatus));
      memset (&setRegs,      0, sizeof(setRegs));
      memset (&getRegs,      0, sizeof(getRegs));
    
      getRegs.cmdStatus = &cmdStatus;
      if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
      {
        System_printf ("Read CMD STATUS register failed!\n");
        return retVal;
      }
    
      if(enable)
        cmdStatus.ltssmEn = 1;
      else
        cmdStatus.ltssmEn = 0;
    
      setRegs.cmdStatus = &cmdStatus;
    
      if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
      {
        System_printf ("SET CMD STATUS register failed!\n");
        return retVal;
      }
    
      return pcie_RET_OK;
    }
    
    
    /*****************************************************************************
     * Function: Configure PCIe in Root Complex Mode
     ****************************************************************************/
    pcieRet_e pcieCfgRC(Pcie_Handle handle)
    {
    	pcieRet_e retVal;
    
    	pcieCmdStatusReg_t     	cmdStatus;
    	pcieObSizeReg_t        	obSize;
    	pcieGen2Reg_t          	gen2;
    	pcieType1Bar32bitIdx_t 	type1Bar32bitIdx;
    	pcieStatusCmdReg_t     	statusCmd;
    	pcieDevStatCtrlReg_t   	devStatCtrl;
    	pcieAccrReg_t          	accr;
    	pcieLnkCtrlReg_t     	lnkCtrl;
    
    	pcieRegisters_t        	setRegs;
    	pcieRegisters_t        	getRegs;
    
    	memset (&cmdStatus,        0, sizeof(cmdStatus));
    	memset (&obSize,           0, sizeof(obSize));
    	memset (&gen2,             0, sizeof(gen2));
    	memset (&type1Bar32bitIdx, 0, sizeof(type1Bar32bitIdx));
    	memset (&statusCmd,        0, sizeof(statusCmd));
    	memset (&devStatCtrl,      0, sizeof(devStatCtrl));
    	memset (&accr,             0, sizeof(accr));
    	memset (&lnkCtrl,          0, sizeof(lnkCtrl));
    
    	/*Disable link training*/
    	if ((retVal = pcieLtssmCtrl(handle, FALSE)) != pcie_RET_OK)
    	{
    		System_printf ("Failed to disable Link Training!\n");
    		return retVal;
    	}
    
    	/* Configure the size of the translation regions */
    	memset (&setRegs, 0, sizeof(setRegs));
    	memset (&getRegs, 0, sizeof(getRegs));
    
    	obSize.size = pcie_OB_SIZE_8MB;
    	setRegs.obSize = &obSize;
    
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET OB_SIZE register failed!\n");
    		return retVal;
    	}
    
    	/* Setting PL_GEN2 */
    	memset (&setRegs, 0, sizeof(setRegs));
    	gen2.numFts = 0xF;
    	gen2.dirSpd = 0; // 0 -> not to gen2, 1 -> gen2 
    	gen2.lnEn   = 2; // 1 -> 1xlane, 2 -> 2xlane
    	setRegs.gen2 = &gen2;
    	
    	//lnkCtrl.lnkMode = 3;
    	//setRegs.lnkCtrl = &lnkCtrl;
    
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET GEN2 register failed!\n");
    		return retVal;
    	}
    
    	/* Configure BAR Masks */
    	/* First need to enable writing on BAR mask registers */
    	memset (&setRegs, 0, sizeof(setRegs));
    	memset (&getRegs, 0, sizeof(getRegs));
    	memset (&cmdStatus,  0, sizeof(cmdStatus));
    
    	getRegs.cmdStatus = &cmdStatus;
    	if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
    	{
    		System_printf ("Read CMD STATUS register failed!\n");
    		return retVal;
    	}
    	cmdStatus.dbi = 1;
    	setRegs.cmdStatus = &cmdStatus;
    
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET CMD STATUS register failed!\n");
    		return retVal;
    	}
    
    	/* Configure Masks*/
    	memset (&setRegs, 0, sizeof(setRegs));
    	type1Bar32bitIdx.reg.reg32 = PCIE_BAR_MASK;
    	setRegs.type1Bar32bitIdx = &type1Bar32bitIdx;
    
    	/* BAR 0 */
    	type1Bar32bitIdx.idx = 0; /* configure BAR 0*/
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET BAR MASK register failed!\n");
    		return retVal;
    	}
    
    	/* BAR 1 */
    	type1Bar32bitIdx.idx = 1; /* configure BAR 1*/
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET BAR MASK register failed!\n");
    		return retVal;
    	}
    
    	/* Disable writing on BAR Masks */
    	memset (&setRegs, 0, sizeof(setRegs));
    	memset (&getRegs, 0, sizeof(getRegs));
    	memset (&cmdStatus,  0, sizeof(cmdStatus));
    
    	getRegs.cmdStatus = &cmdStatus;
    	if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
    	{
    		System_printf ("Read CMD STATUS register failed!\n");
    		return retVal;
    	}
    	cmdStatus.dbi = 0;
    	setRegs.cmdStatus = &cmdStatus;
    
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET CMD STATUS register failed!\n");
    		return retVal;
    	}
    
    	/* Enable memory access and mastership of the bus */
    	memset (&setRegs, 0, sizeof(setRegs));
    	memset (&getRegs, 0, sizeof(getRegs));
    
    	getRegs.statusCmd = &statusCmd;
    	if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
    	{
    		System_printf ("Read Status Comand register failed!\n");
    		return retVal;
    	}
    	statusCmd.memSp  = 1;
    	statusCmd.busMs  = 1;
    	statusCmd.resp   = 1;
    	statusCmd.serrEn = 1;
    	setRegs.statusCmd = &statusCmd;
    
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET Status Command register failed!\n");
    		return retVal;
    	}
    
    	/* Enable Error Reporting */
    	memset (&setRegs, 0, sizeof(setRegs));
    	memset (&getRegs, 0, sizeof(getRegs));
    
    	getRegs.devStatCtrl = &devStatCtrl;
    	if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
    	{
    		System_printf ("Regad Device Status Control register failed!\n");
    		return retVal;
    	}
    
    	devStatCtrl.reqRp = 1;
    	devStatCtrl.fatalErRp = 1;
    	devStatCtrl.nFatalErRp = 1;
    	devStatCtrl.corErRp = 1;
    	setRegs.devStatCtrl = &devStatCtrl;
    
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET Device Status Control register failed!\n");
    		return retVal;
    	}
    
    	/* Enable ECRC */
    	memset (&setRegs, 0, sizeof(setRegs));
    
    	accr.chkEn=1;
    	accr.chkCap=1;
    	accr.genEn=1;
    	accr.genCap=1;
    	setRegs.accr = &accr;
    
    	if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
    	{
    		System_printf ("SET ACCR register failed!\n");
    		return retVal;
    	}
    
    	return pcie_RET_OK;
    }
    
    
    
    /*****************************************************************************
     * Function: Configure and enable Outbound Address Translation
     ****************************************************************************/
    pcieRet_e pcieObTransCfg(Pcie_Handle handle, uint32_t obAddrLo, uint32_t obAddrHi, uint8_t region)
    {
    
      pcieRet_e retVal;
    
      pcieRegisters_t      setRegs;
      pcieRegisters_t      getRegs;
    
      pcieCmdStatusReg_t   cmdStatus;
    
      memset (&setRegs,   0, sizeof(setRegs));
      memset (&getRegs,   0, sizeof(getRegs));
      memset (&cmdStatus, 0, sizeof(cmdStatus));
    
      /* Set outbound offset registers */
      if ((retVal = Pcie_cfgObOffset(handle, obAddrLo, obAddrHi, region)) != pcie_RET_OK)
      {
        System_printf ("Failed to configure ObOffset registers!\n");
        return retVal;
      }
    
      /*enable Outbound address translation*/
      memset (&setRegs,    0, sizeof(setRegs));
      memset (&getRegs,    0, sizeof(getRegs));
    
      getRegs.cmdStatus = &cmdStatus;
      if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
      {
        System_printf ("Read CMD STATUS register failed!\n");
        return retVal;
      }
      cmdStatus.obXltEn = 1;
      setRegs.cmdStatus = &cmdStatus;
    
      if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
      {
        System_printf ("SET CMD STATUS register failed!\n");
        return retVal;
      }
    
      return pcie_RET_OK;
    }
    
    
    /*****************************************************************************
     * Function: Configure and enable Inbound Address Translation
     ****************************************************************************/
    pcieRet_e pcieIbTransCfg(Pcie_Handle handle, pcieIbTransCfg_t *ibCfg)
    {
      pcieRet_e retVal;
    
      pcieRegisters_t      setRegs;
      pcieRegisters_t      getRegs;
    
      pcieCmdStatusReg_t   cmdStatus;
    
      memset (&setRegs,   0, sizeof(setRegs));
      memset (&getRegs,   0, sizeof(getRegs));
      memset (&cmdStatus, 0, sizeof(cmdStatus));
    
      /* Set outbound offset registers */
      if ((retVal = Pcie_cfgIbTrans(handle, ibCfg)) != pcie_RET_OK)
      {
        System_printf ("Failed to configure Inbound Translation registers!\n");
        return retVal;
      }
    
      /*enable Inbound address translation*/
      memset (&setRegs,    0, sizeof(setRegs));
      memset (&getRegs,    0, sizeof(getRegs));
    
      getRegs.cmdStatus = &cmdStatus;
      if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
      {
        System_printf ("Read CMD STATUS register failed!\n");
        return retVal;
      }
      cmdStatus.ibXltEn = 1;
      setRegs.cmdStatus = &cmdStatus;
    
      if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
      {
        System_printf ("SET CMD STATUS register failed!\n");
        return retVal;
      }
    
      return pcie_RET_OK;
    }
    
    
    /*****************************************************************************
     * Function: Initialize application buffers
     ****************************************************************************/
    void pcieInitAppBuf(void)
    {
      uint32_t i;
    
      for (i=0; i<PCIE_BUFSIZE_APP/4; i++)
      {
        pcieBuf.dstBuf[i] = 0;
        pcieBuf.srcBuf[i] = i;
      }
    
    
    }
    
    /*****************************************************************************
     * Function: Check LTSSM status and wait for the link to be up
     ****************************************************************************/
    void pcieWaitLinkUp(Pcie_Handle handle)
    {
      pcieRegisters_t  getRegs;
      pcieDebug0Reg_t  debug0;
      pcieDebug1Reg_t  debug1;
      pcieLinkStatCtrlReg_t linkStatCtrl; 
      int i;
    
      memset (&debug0,  0, sizeof(debug0));
      memset (&debug1,  0, sizeof(debug1));
      memset (&getRegs, 0, sizeof(getRegs));
    
      uint8_t ltssmState = 0;
    
      while(ltssmState != pcie_LTSSM_L0)
      {
        cycleDelay(100);
        getRegs.debug0 = &debug0;
        if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK)
        {
          System_printf ("Read LTSSM state failed!\n");
          return;
        }
        ltssmState = debug0.ltssmState;
      }
      
      /// testing
      i = 20;
      getRegs.debug0 = &debug0;
      getRegs.debug1 = &debug1;
      getRegs.linkStatCtrl = &linkStatCtrl;
      
      while(i-- > 0){
    	System_printf("linkup testing\n");
    	cycleDelay(100);
        memset (&debug0,  0, sizeof(debug0));
    	memset (&debug1,  0, sizeof(debug1));
    	memset (&linkStatCtrl,  0, sizeof(linkStatCtrl));
        if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK)
        {
          System_printf ("Read LTSSM state failed!\n");
          return;
        }
        System_printf("debug0       0x%x ltssmState 0x%x\n", debug0.raw, debug0.ltssmState);
    	System_printf("debug1       0x%x\n", debug1.raw);
    	System_printf("linkStatCtrl 0x%x\n", linkStatCtrl.raw);
    	
    	
    	
    	
      }
      
      //// end
      
      
    }
    
    /*****************************************************************************
     * Function: Converts a core local L2 address to a global L2 address
     *   Input addr:  L2 address to be converted to global.
     *   return:  uint32_t   Global L2 address
     *****************************************************************************/
    uint32_t pcieConvert_CoreLocal2GlobalAddr (uint32_t  addr)
    {
      uint32_t coreNum;
    
      /* Get the core number. */
      coreNum = CSL_chipReadReg(CSL_CHIP_DNUM);
    
      /* Compute the global address. */
      return ((1 << 28) | (coreNum << 24) | (addr & 0x00ffffff));
    }
    
    
    /*****************************************************************************
     * Function: Configurate EP after linkup
     ****************************************************************************/
    void pcieUserConfigEP(Pcie_Handle handle)
    {
    	pcieRegisters_t  		getRegs;
    	pcieRegisters_t    		setRegs;
    	pcieVndDevIdReg_t  		vndDevId;
    	pcieCfgTransReg_t		cfgTrans;
    	pcieType0Bar32bitIdx_t 	type0Bar32bitIdx;
    	pcieObOffsetLoReg_t 	obOffsetLo;
    	pcieObOffsetHiReg_t 	obOffsetHi;
    	
    	pcieSubIdReg_t       	subId;                   /**< @brief Subsystem ID*/
    	pcieExpRomReg_t        	expRom;                  /**< @brief Expansion ROM base addr*/
    	pcieCapPtrReg_t        	capPtr;                  /**< @brief Capabilities Pointer*/
    	pcieIntPinReg_t       	intPin;                  /**< @brief Interrupt Pin*/
    	
    	pcieStatusCmdReg_t    	statusCmd;               /**< @brief Status Command*/
    	pcieRevIdReg_t       	revId;                   /**< @brief Class code and Revision ID*/
    	
    	pcieBarCfg_t     		barCfg;
    	
    	int busNum, devNum;
    	//int i;
    	int index;
    
    	/* Check RC configuration */
    	
    	//// Read RC DeviceId and VendorId
    	memset (&vndDevId,  0, sizeof(vndDevId));
    	memset (&getRegs, 0, sizeof(getRegs));
    	getRegs.vndDevId = &vndDevId;
    	if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK)
    	{
    	  System_printf ("Read RC VendorId and DevicId failed!\n");
    	  return;
    	}
    	System_printf ("VendorId and DeviceId swapped in PCIe API (???), manual swap here\n");
    	System_printf ("RC Raw 0x%x Vendor ID 0x%x Device ID 0x%x\n", vndDevId.raw, vndDevId.devId, vndDevId.vndId);
    	
    	//// Read RC BAR1
    	memset (&type0Bar32bitIdx, 0, sizeof(type0Bar32bitIdx));
    	memset (&getRegs, 0, sizeof(getRegs));
    	type0Bar32bitIdx.idx = 1;
    	getRegs.type0Bar32bitIdx = &type0Bar32bitIdx;
    	if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK)
    	{
    	  System_printf ("Read RC BAR1 failed!\n");
    	  return;
    	}
    	System_printf("RC BAR1 0x%x\n", type0Bar32bitIdx.reg.reg32);
    
    	//// Read RC obOffset low and high address
    	memset (&getRegs, 0, sizeof(getRegs));
    	memset (&obOffsetLo, 0, sizeof(obOffsetLo));
    	memset (&obOffsetHi, 0, sizeof(obOffsetHi));
    	index = 0;
    	getRegs.obOffsetLo[index] = &obOffsetLo;
    	getRegs.obOffsetHi[index] = &obOffsetHi;
    	if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK)
    	{
    	  System_printf ("Read RC obOffset%d failed!\n", index);
    	  return;
    	}
    	System_printf("RC obOffset%d low 0x%x high 0x%x\n", index, obOffsetLo.raw, obOffsetHi.raw);
    	
    	/* Set BusNum:DevNum:Function of RC (this may not need) */
    	memset (&setRegs, 0, sizeof(setRegs));
    	devNum = 0;
    	busNum = 0;
    
    	//for(busNum=0; busNum<256; busNum++){
    	for(devNum=0; devNum<1; devNum++){
    		//// Set RC bus:devnum:function
    		cfgTrans.type = 0;
    		cfgTrans.bus = busNum;
    		cfgTrans.device = devNum;
    		cfgTrans.func = 0;
    		setRegs.cfgTrans = &cfgTrans;
    
    		if (Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs) != pcie_RET_OK)
    		{
    			System_printf ("EP CFG_SETUP failed!\n");
    			return;
    		}
    
    		//// Read EP VendorId and DeviceId
    		memset (&getRegs, 0, sizeof(getRegs));
    		memset (&vndDevId,  0, sizeof(vndDevId));
    		getRegs.vndDevId = &vndDevId;
    		if (Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs) != pcie_RET_OK)
    		{
    		  System_printf ("Read EP VendorId and DeviceId failed!\n");
    		  return;
    		}
    		System_printf ("VendorId and DeviceId swapped in PCIe API (???), manual swap here\n");
    		System_printf ("EP [%d %d %d] Raw 0x%x Vendor ID 0x%x Device ID 0x%x\n", busNum, devNum, 0, vndDevId.raw, vndDevId.devId, vndDevId.vndId);
    	}
    
    	/* Configure EP BAR0 */
    	//// Write EP BAR0 mask first
    	memset (&type0Bar32bitIdx, 0, sizeof(type0Bar32bitIdx));
    	memset (&setRegs, 0, sizeof(setRegs));
    	type0Bar32bitIdx.reg.reg32 = 0xffffffff; // mask
    	type0Bar32bitIdx.idx = 0; // BAR0
    	setRegs.type0Bar32bitIdx = &type0Bar32bitIdx;
    	if (Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs) != pcie_RET_OK)
    	{
    		System_printf ("Write EP BAR0 mask failed!\n");
    		return;
    	}
    	
    	memset (&type0Bar32bitIdx, 0, sizeof(type0Bar32bitIdx));
    	memset (&getRegs, 0, sizeof(getRegs));
    	type0Bar32bitIdx.idx = 0; // BAR0
    	getRegs.type0Bar32bitIdx = &type0Bar32bitIdx;
    	if (Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs) != pcie_RET_OK)
    	{
    	  System_printf ("Read EP BAR0 size failed!\n");
    	  return;
    	}
    	System_printf("EP BAR0 size 0x%x\n", ~(type0Bar32bitIdx.reg.reg32 & ~0xf)+1 );
    	
    	/* Configure EP */
    	//// Set BAR0
        barCfg.location = pcie_LOCATION_REMOTE;
        barCfg.mode     = pcie_EP_MODE;
        barCfg.base     = PCIE_IB_LO_ADDR_S;
        barCfg.prefetch = pcie_BAR_NON_PREF;
        barCfg.type     = pcie_BAR_TYPE32;
        barCfg.memSpace = pcie_BAR_MEM_MEM;
        barCfg.idx      = 0;
    
        if (Pcie_cfgBar(handle, &barCfg) != pcie_RET_OK)
        {
          System_printf ("Failed to configure EP BAR0\n");
          return;
        }
    	
    	//// Set EP statusCmd
    	memset (&setRegs, 0, sizeof(setRegs));
    	memset (&statusCmd, 0, sizeof(statusCmd));
    	memset (&intPin, 0, sizeof(intPin));
    	
    	statusCmd.memSp = 1;
    	statusCmd.busMs  = 1;
    	statusCmd.resp   = 1;
    	statusCmd.serrEn = 1;
    	setRegs.statusCmd = &statusCmd;
    	
    	intPin.intPin = 1;
    	intPin.intLine = 0xa;
    	setRegs.intPin = &intPin;
    	
    	if (Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs) != pcie_RET_OK)
    	{
    		System_printf ("Set EP statusCmd failed!\n");
    		return;
    	}
    	
    	//// Read EP BAR0 for verification
    	memset (&getRegs, 0, sizeof(getRegs));
    	memset (&type0Bar32bitIdx, 0, sizeof(type0Bar32bitIdx));
    	type0Bar32bitIdx.idx = 0; // BAR0
    	getRegs.type0Bar32bitIdx = &type0Bar32bitIdx;
    	if (Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs) != pcie_RET_OK)
    	{
    	  System_printf ("Read EP BAR0 failed!\n");
    	  return;
    	}
    	System_printf("EP BAR0 0x%x\n", type0Bar32bitIdx.reg.reg32);
    	
    	//// Read EP config space
    	memset (&getRegs, 0, sizeof(getRegs));
    	memset (&subId, 0, sizeof(subId));
    	memset (&expRom, 0, sizeof(expRom));
    	memset (&capPtr, 0, sizeof(capPtr));
    	memset (&intPin, 0, sizeof(intPin));
    	memset (&statusCmd, 0, sizeof(statusCmd));
    	memset (&revId, 0, sizeof(revId));
    	
    	getRegs.subId = &subId;
    	getRegs.expRom = &expRom;
    	getRegs.capPtr = &capPtr;
    	getRegs.intPin = &intPin;
    	getRegs.statusCmd = &statusCmd;
    	getRegs.revId = &revId;
    	
    	if (Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &getRegs) != pcie_RET_OK)
    	{
    	  System_printf ("Read EP config space failed!\n");
    	  return;
    	}
    	
    	System_printf("EP subId 0x%x expRom 0x%x capPtr 0x%x intPin 0x%x intLine 0x%x\n", subId.raw, expRom.raw, capPtr.raw, intPin.intPin, intPin.intLine);
    	System_printf("EP statusCmd 0x%x revId 0x%x\n", statusCmd.raw, revId.raw);
      
    }
    
    
    /*****************************************************************************
     * Function: Main
     ****************************************************************************/
    void main (void)
    {
    	TSCL = 1;
    	uint16_t lock=0;
    
    	pcieRet_e        	retVal;
    	pcieIbTransCfg_t 	ibCfg;
    	pcieBarCfg_t     	barCfg;
    	Pcie_Handle      	handle = NULL;
    	void             	*pcieBase;
    	void 				*pcieEpMailBox;
    	uint32_t    		i;
    	uint8_t 			regValue;
    
    
    
    	System_printf ("**********************************************\n");
    	System_printf ("*             PCIe Test Start                *\n");
    
    	if(PcieModeGbl == pcie_RC_MODE)
    	System_printf ("*                RC mode                     *\n");
    	else
    	System_printf ("*                EP mode                     *\n");
    
    	System_printf ("**********************************************\n\n");
    
    	System_printf("Version #: 0x%08x; string %s\n\n", Pcie_getVersion(), Pcie_getVersionStr());
    	System_printf("fpga version 0x%x\n", getFpgaDevID());
    	System_printf("board version 0x%x\n", getBoardVersion());
    
    	gpioInit();
    	initInterruptSystem();
    	
    	fpgaControlUserLEDs(FPGA_USER_LED0, FPGA_LED_OFF);
    	fpgaControlUserLEDs(FPGA_USER_LED2, FPGA_LED_OFF);
    
    	// Note: tried to use refclk from clk synthzier and pcie slot, same problem on GEN2 linkup
    	fpgaWriteConfigurationRegister(0x50, 1); // Use refclk from PCIe SLOT
    	fpgaReadConfigurationRegister(0x50, &regValue);
    	System_printf("read 0x50 0x%x\n", regValue);
    
    	/* Initialize application buffers */
    	pcieInitAppBuf();
    
    	/* Configure SERDES*/
    	if ((retVal = pcieSerdesCfg()) != pcie_RET_OK) {
    		System_printf ("PCIe Serdes config failed (%d)\n", (int)retVal);
    		exit(1);
    	}
    
    	/* Set the PCIe mode*/
    	if ((retVal = Pcie_setMode(PcieModeGbl)) != pcie_RET_OK) {
    		System_printf ("Set PCIe Mode failed (%d)\n", (int)retVal);
    		exit(1);
    	}
    
    	/* Power up PCIe Module */
    	if ((retVal = pciePowerCfg()) != pcie_RET_OK) {
    		System_printf ("PCIe Power Up failed (%d)\n", (int)retVal);
    		exit(1);
    	}
    
    	System_printf ("PCIe Power Up.\n");
    
    	/* Wait until the PCIe SERDES PLL locks */
    	while (!lock)
    	{
    		CSL_BootCfgGetPCIEPLLLock(&lock);
    	}
    
    	if ((retVal = Pcie_open(0, &handle)) != pcie_RET_OK)
    	{
    		System_printf ("Open failed (%d)\n", (int)retVal);
    		exit(1);
    	}
    
    	System_printf ("PLL configured.\n");
    
    	if(PcieModeGbl == pcie_RC_MODE)
    	{
    		/* Configure application registers for Root Complex*/
    		if ((retVal = pcieCfgRC(handle)) != pcie_RET_OK)
    		{
    		  System_printf ("Failed to configure PCIe in RC mode (%d)\n", (int)retVal);
    		  exit(1);
    		}
    
    		/* Configure Address Translation */
    
    		barCfg.location = pcie_LOCATION_LOCAL;
    		barCfg.mode     = pcie_RC_MODE;
    		barCfg.base     = PCIE_IB_LO_ADDR_M;
    		barCfg.prefetch = pcie_BAR_NON_PREF;
    		barCfg.type     = pcie_BAR_TYPE32;
    		barCfg.memSpace = pcie_BAR_MEM_MEM;
    		barCfg.idx      = PCIE_BAR_IDX_M;
    
    		if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK)
    		{
    		  System_printf ("Failed to configure BAR (%d)\n", (int)retVal);
    		  exit(1);
    		}
    
    		ibCfg.ibBar         = PCIE_BAR_IDX_M; /* Match BAR that was configured above*/
    		ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_M;
    		ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_M;
    		ibCfg.ibOffsetAddr  = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)pcieBuf.srcBuf);
    		ibCfg.region        = PCIE_IB_REGION_M;
    
    		System_printf("Inbound buffer global addr 0x%x\n", ibCfg.ibOffsetAddr);
    		
    		if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK)
    		{
    			System_printf ("Failed to configure Inbound Translation (%d)\n", (int)retVal);
    			exit(1);
    		}
    		else
    		{
    			System_printf ("Successfully configured Inbound Translation!\n");
    		}
    
    		if ((retVal = pcieObTransCfg (handle, PCIE_OB_LO_ADDR_M, PCIE_OB_HI_ADDR_M, PCIE_OB_REGION_M)) != pcie_RET_OK)
    		{
    			System_printf ("Failed to configure Outbound Address Translation (%d)\n", (int)retVal);
    			exit(1);
    		}
    		else
    		{
    			System_printf ("Successfully configured Outbound Translation!\n");
    		}
    	}
    
    	System_printf ("Starting link training...\n");
    
    	/*Enable link training*/
    	if ((retVal = pcieLtssmCtrl(handle, TRUE)) != pcie_RET_OK)
    	{
    		System_printf ("Failed to Enable Link Training!\n", (int)retVal);
    		exit(1);
    	}
    
    	/* Wait for link to be up */
    	pcieWaitLinkUp(handle);
    
    	System_printf ("Link is up.\n");
    	
    	if(PcieModeGbl == pcie_RC_MODE)
    	{
    		/**********************************************************************/
    		/* Push a single message to the EP then verify that it is echoed back */
    		/**********************************************************************/
    
    		pcieUserConfigEP(handle);
    
    		/* Get EP memory mapped base */
    		if ((retVal = Pcie_getMemSpaceRange (handle, &pcieBase, NULL)) != pcie_RET_OK) {
    			System_printf ("getMemSpaceRange failed\n", (int)retVal);
    			exit(1);
    		}
    
    		/* 
    		** Note: 
    		** Program outbound region to 0 (PCIE_OB_REGION_M) with outbound address 0x70000000 (PCIE_OB_LO_ADDR_M)
    		** So, write to PCIe data region 0x60000000 would use region 0 and use outbound address 0x70000000
    		** Endpoint's BAR address should be programmed to 0x70000000 to receive RC data
    		*/
    		
    		/* PIO transfer */
    		pcieEpMailBox = (void *)((uint32_t)pcieBase + MAILBOX_OFFSET);
    
    		System_printf("pcieEpMailBox %p\n", pcieEpMailBox);
    
    		#if 1
    		for (i=0; i<32; i++)
    		{
    			*((volatile uint32_t *)pcieEpMailBox + i) = i+0xffaa0000;
    		}
    		#endif
    		for (i=0; i<32; i++)
    		{
    			System_printf("PCIe read[%2d] = 0x%8x (addr 0x%8x)\n", i, *((volatile uint32_t *)pcieEpMailBox + i), (volatile uint32_t *)pcieEpMailBox + i);
    		}
    
    		/* EDMA transfer (not working yet) */
    		// Note: ADS EP not support brust access 
    		//edmaRc2Ep (0, 0, (void *)srcBuf.buf, pcieEpMailBox, 256);
    		
    		pcieDmaClose();
    		System_printf ("Root Complex test end\n");
    	}
    
    
      exit(0);
    }
    
    
    void setBenchmark(int32_t pinNum)
    {
    	gpioSetDirection(pinNum, GPIO_OUT);
    	gpioSetOutput(pinNum);
    }
    
    void clrBenchmark(int32_t pinNum)
    {
    	gpioSetDirection(pinNum, GPIO_OUT);
    	gpioClearOutput(pinNum);
    }
    
    
    
    
    
    
    

    Do you have a working reference code for RC GEN2 x2 ??? 

    I need it badly.

    Watson

  • Hi,

    I have tested the both gen configuration on EVM broads, It is working fine.

    Have you done the same gen configuration on FPGA side? Your FPGA change to Gen2 after linkup.

    Thanks,
  • Ganapathi,

    FPGA is hardcoded to support up to GEN2.

    1. Any comment on pcieRootComplex.c? Coding on TI side is ok?

    2. Per your early suggestion, you have this setting for GEN2

    gen2.dirSpd = 0;

    According to datasheet, dirSpd should be set to 1 in order to link to GEN2. Am I right?

    3. May I have the test code you used to check GEN2 linking?

    Watson
  • Hi,

    Yes your understanding is correct. Set dirSpd bit field to 1 in order to link to GEN2.

    I am using MCSDK PCIe example project for testing GEN2 testing between two C6678 EVM.

    Thanks,
  • Ganapathi,

    Any register in C6678 can help me to debug the link up issue?

    Watson
  • Read the LTSSM_STATE debug0 register in DSP side. It should be useful to track the issue, for more information refer Appendix A on PCIe user guide(SPRUGS6D). Insure link training completion and success by observing LTSSM_STATE field in DEBUG0 register change to 0x11.

    Thanks,
  • Ganapathi,

    The problem was solved after changing my FPGA design from 4xlanes to 2xlanes.

    Thanks for your help!

    Watson
  • Glad to here. Thanks for your update.