This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PCIe between K2K EVM 6638 and FPGA board

Hi all,

We are trying to bring up PCIe communication between K2K EVM and a Xilinx FPGA. We are using the example project in the keystone2 pdk. We are able to bring up the link and also able to read the config. area of the FPGA board. We are able to read the vendor, device ID from the FPGA. We then write the BAR address to the FPGA config. space. We use the same BAR address as our outbound address on our EVM board. We write some data into the FPGA but on reading back from the FPGA, we get some unexpected values. I have attached the source file for the project and below is the output we get from the EVM:

[C66xx_0] **********************************************

* PCIe Test Start *

* RC mode *

**********************************************

Version #: 0x02010001; string PCIE LLD Revision: 02.01.00.01:Sep 30 2015:00:14:39

PCIe Power Up.

PLL configured.

Configured for BAR (0)

Successfully configured Inbound Translation!

Successfully configured Outbound Translation!

Starting link training...

PCIe pcieWaitLinkUp...

Debug0 Value #: 0x00820ca8

Link is up.

Device ID is 32802

Vendor ID is 4333

Configured remote BAR0 successfully!

value from PCIe interface 1342177280

value from PCIe interface 1

value from PCIe interface 1342177280

value from PCIe interface 3

value from PCIe interface 1342177280

value from PCIe interface 5

value from PCIe interface 1342177280

value from PCIe interface 7

value from PCIe interface 1342177280

value from PCIe interface 9

value from PCIe interface 1342177280

value from PCIe interface 11

value from PCIe interface 1342177280

value from PCIe interface 13

value from PCIe interface 1342177280

value from PCIe interface 15

value from PCIe interface 1342177280

value from PCIe interface 17

value from PCIe interface 1342177280

value from PCIe interface 19

value from PCIe interface 1342177280

value from PCIe interface 21

value from PCIe interface 1342177280

value from PCIe interface 23

value from PCIe interface 1342177280

value from PCIe interface 25

value from PCIe interface 1342177280

value from PCIe interface 27

value from PCIe interface 1342177280

value from PCIe interface 29

value from PCIe interface 1342177280

value from PCIe interface 31

value from PCIe interface 1342177280

value from PCIe interface 33

value from PCIe interface 1342177280

value from PCIe interface 35

value from PCIe interface 1342177280

value from PCIe interface 37

value from PCIe interface 1342177280

value from PCIe interface 39

Received data = 0

Transmited data = 5

Index = 0.

Test failed.

Please suggest on this issue.

Regards, Yunas

7875.pcie_sample.c
/*  ============================================================================
 *   Copyright (c) Texas Instruments Incorporated 2010-2013
 * 
 *  Redistribution and use in source and binary forms, with or without 
 *  modification, are permitted provided that the following conditions 
 *  are met:
 *
 *    Redistributions of source code must retain the above copyright 
 *    notice, this list of conditions and the following disclaimer.
 *
 *    Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the 
 *    documentation and/or other materials provided with the   
 *    distribution.
 *
 *    Neither the name of Texas Instruments Incorporated nor the names of
 *    its contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
 *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
 *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 
 *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 
 *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
 *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
 *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 
 *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *
*/


/**
 *   @file  pcie_sample.c
 *
 *   @brief   
 *      This is the PCIe example code.   
 *
 */

/** 
 * In the PCIe sample example two Shannon EVMs are used to test the PCIe driver. 
 * As described in the following figure, Shannon 1 is configured as a Root Complex
 * and Shannon 2 is configured as End Point.
 *
 *       Shannon 1                                       Shannon 2
 *   ------------------                             -------------------
 *   |                |                             |                 |
 *   |   Root         |          PCIe Link          |  End Point      |
 *   |   Complex      | <-------------------------->|                 |
 *   |                |                             |                 |
 *   ------------------                             -------------------
 *  
 * Once the PCIe link is established, the following sequence of actions will happen:
 *  - Shannon 1 sends data to Shannon 2
 *  - Shannon 2 waits to receive all the data
 *  - Shannon 2 sends the data back to Shannon 1
 *  - Shannon 1 waits to receive all the data
 *  - Shannon 1 verifies if the received data matches the sent data and declares test pass or fail.
 *
 */
#define DEVICE_REG32_W(x,y)   *(volatile unsigned int *)(x)=(y)
#define DEVICE_REG32_R(x)    (*(volatile unsigned int *)(x))

#define ENABLE_PHYLOOPBACK
#include "pcie_sample.h"
#include <ti/drv/pcie/device/pcie_device.h>

/* Use regular "printf" when not using BIOS */
//#include <xdc/runtime/System.h>
#define System_printf printf
#include <stdio.h>

#include <ti/csl/csl_bootcfgAux.h>
#include <ti/csl/csl_cacheAux.h>
#include <ti/csl/csl_chip.h>
#include <ti/csl/csl_xmcAux.h>


#pragma DATA_SECTION(dstBuf, ".dstBufSec")
/* Cache coherence: Align must be a multiple of cache line size (L2=128 bytes, L1=64 bytes) to operate with cache enabled. */
/* Aligning to 256 bytes because the PCIe inbound offset register masks the last 8bits of the buffer address  */
#pragma DATA_ALIGN(dstBuf, 256)
/* last element in the buffer is a marker that indicates the buffer status: full/empty */
#define PCIE_EXAMPLE_MAX_CACHE_LINE_SIZE 128
#define PCIE_EXAMPLE_UINT32_SIZE           4 /* preprocessor #if requires a real constant, not a sizeof() */

#define PCIE_EXAMPLE_DSTBUF_BYTES ((PCIE_BUFSIZE_APP + 1) * PCIE_EXAMPLE_UINT32_SIZE)
#define PCIE_EXAMPLE_DSTBUF_REM (PCIE_EXAMPLE_DSTBUF_BYTES % PCIE_EXAMPLE_MAX_CACHE_LINE_SIZE)
#define PCIE_EXAMPLE_DSTBUF_PAD (PCIE_EXAMPLE_DSTBUF_REM ? (PCIE_EXAMPLE_MAX_CACHE_LINE_SIZE - PCIE_EXAMPLE_DSTBUF_REM) : 0)
struct dstBuf_s {
  volatile uint32_t buf[PCIE_BUFSIZE_APP + 1];
  /* Cache coherence: Must pad to cache line size in order to enable cacheability */
#if PCIE_EXAMPLE_DSTBUF_PAD
  uint8_t padding[PCIE_EXAMPLE_DSTBUF_PAD];
#endif
} dstBuf;

#define PCIE_EXAMPLE_BUF_EMPTY 0
#define PCIE_EXAMPLE_BUF_FULL  1

/* Does not need to be aligned (even for cache) since it is only accessed locally */
uint32_t srcBuf[PCIE_BUFSIZE_APP];

extern volatile unsigned int cregister TSCL;

/* Global config variable that controls
   the PCIe mode. It is global so it can be poked
   from CCS. It should be set either to EP or RC. */
pcieMode_e PcieModeGbl = pcie_RC_MODE;

#if defined(DEVICE_K2K) || defined(DEVICE_K2H)
/**********************************************************************************
 * FUNCTION PURPOSE: Function for updating register value with mask
 **********************************************************************************
 * DESCRIPTION: Function for updating register value with mask
 **********************************************************************************/ 
static void write_32_mask(volatile uint32_t  base_addr,
                              uint32_t           offset,
                              uint32_t           mask,
                              uint32_t           write_data)
{
    uint32_t read_data, data;
    read_data = (*(volatile uint32_t *)(base_addr + offset));
    data = (write_data & ~mask ) | (read_data & mask);
    (*(uint32_t *) ((base_addr + offset))) = data;
}


void reg_poll(volatile unsigned int addr, volatile unsigned int poll_val, volatile unsigned int masked_bits)
{
  volatile unsigned int read_data;

  read_data = (*(unsigned int *) (addr));
  read_data = read_data & masked_bits;
  while (read_data != (poll_val & masked_bits))
    {
      read_data = (*(unsigned int *) (addr));
      read_data = read_data & masked_bits;
    }
}

void pcieSSDefSbSerdesSetup(volatile uint32_t  base_addr) 
{
    write_32_mask(base_addr, 0x000,0xffff00ff,0x00000800);
    write_32_mask(base_addr, 0x0060,0xff000000,0x00041c5c);
    write_32_mask(base_addr, 0x0064,0x000000ff,0x0343c700);
    write_32_mask(base_addr, 0x006c,0xffffff00,0x00000012);
    write_32_mask(base_addr, 0x0068,0xff00ffff,0x00070000);
    write_32_mask(base_addr, 0x0078,0xffff00ff,0x0000c000);

    write_32_mask(base_addr, 0x0200,0xffffff00,0x00000000);
    write_32_mask(base_addr, 0x0204,0x00ffff00,0x5e000080);
    write_32_mask(base_addr, 0x0208,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0210,0xffffff00,0x00000023);
    write_32_mask(base_addr, 0x0214,0x00ff0000,0x2e003060);
    write_32_mask(base_addr, 0x0218,0x00ffffff,0x76000000);
    write_32_mask(base_addr, 0x022c,0xff00ff00,0x00100002);
    write_32_mask(base_addr, 0x02a0,0x0000ffff,0xffee0000);
    write_32_mask(base_addr, 0x02a4,0xffffff00,0x0000000f);
    write_32_mask(base_addr, 0x0204,0x00ffffff,0x5e000000);
    write_32_mask(base_addr, 0x0208,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0278,0xffff00ff,0x00002000);
    write_32_mask(base_addr, 0x0280,0xff00ff00,0x00280028);
    write_32_mask(base_addr, 0x0284,0x00000000,0x2d0f0385);
    write_32_mask(base_addr, 0x0250,0x00ffffff,0xd0000000);    
    write_32_mask(base_addr, 0x0284,0xffffff00,0x00000085);
    write_32_mask(base_addr, 0x0294,0x00ffffff,0x20000000);
    
    write_32_mask(base_addr, 0x0400,0xffffff00,0x00000000);
    write_32_mask(base_addr, 0x0404,0x00ffff00,0x5e000080);
    write_32_mask(base_addr, 0x0408,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0410,0xffffff00,0x00000023);
    write_32_mask(base_addr, 0x0414,0x00ff0000,0x2e003060);
    write_32_mask(base_addr, 0x0418,0x00ffffff,0x76000000);
    write_32_mask(base_addr, 0x042c,0xff00ff00,0x00100002);
    write_32_mask(base_addr, 0x04a0,0x0000ffff,0xffee0000);
    write_32_mask(base_addr, 0x04a4,0xffffff00,0x0000000f);
    write_32_mask(base_addr, 0x0404,0x00ffffff,0x5e000000);
    write_32_mask(base_addr, 0x0408,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0478,0xffff00ff,0x00002000);
    write_32_mask(base_addr, 0x0480,0xff00ff00,0x00280028);
    write_32_mask(base_addr, 0x0484,0x00000000,0x2d0f0385);
    write_32_mask(base_addr, 0x0450,0x00ffffff,0xd0000000);   
    write_32_mask(base_addr, 0x0494,0x00ffffff,0x20000000);
    
    write_32_mask(base_addr, 0x0604,0xffffff00,0x00000080);
    write_32_mask(base_addr, 0x0600,0xffffff00,0x00000000);
    write_32_mask(base_addr, 0x0604,0x00ffffff,0x5e000000);
    write_32_mask(base_addr, 0x0608,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0610,0xffffff00,0x00000023);
    write_32_mask(base_addr, 0x0614,0x00ff0000,0x2e003060);
    write_32_mask(base_addr, 0x0618,0x00ffffff,0x76000000);
    write_32_mask(base_addr, 0x062c,0xff00ff00,0x00100002);
    write_32_mask(base_addr, 0x06a0,0x0000ffff,0xffee0000);
    write_32_mask(base_addr, 0x06a4,0xffffff00,0x0000000f);
    write_32_mask(base_addr, 0x0604,0x00ffffff,0x5e000000);
    write_32_mask(base_addr, 0x0608,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0678,0xffff00ff,0x00002000);
    write_32_mask(base_addr, 0x0680,0xff00ff00,0x00280028);
    write_32_mask(base_addr, 0x0684,0x00000000,0x2d0f0385);
    write_32_mask(base_addr, 0x0650,0x00ffffff,0xd0000000);
    write_32_mask(base_addr, 0x0694,0x00ffffff,0x20000000);
    
    write_32_mask(base_addr, 0x0800,0xffffff00,0x00000000);
    write_32_mask(base_addr, 0x0804,0x00ffff00,0x5e000080);
    write_32_mask(base_addr, 0x0808,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0810,0xffffff00,0x00000023);
    write_32_mask(base_addr, 0x0814,0x00ff0000,0x2e003060);
    write_32_mask(base_addr, 0x0818,0x00ffffff,0x76000000);
    write_32_mask(base_addr, 0x082c,0xff00ff00,0x00100002);
    write_32_mask(base_addr, 0x08a0,0x0000ffff,0xffee0000);
    write_32_mask(base_addr, 0x08a4,0xffffff00,0x0000000f);
    write_32_mask(base_addr, 0x0804,0x00ffffff,0x5e000000);
    write_32_mask(base_addr, 0x0808,0xffffff00,0x00000006);
    write_32_mask(base_addr, 0x0878,0xffff00ff,0x00002000);
    write_32_mask(base_addr, 0x0880,0xff00ff00,0x00280028);
    write_32_mask(base_addr, 0x0884,0x00000000,0x2d0f0385);
    write_32_mask(base_addr, 0x0850,0x00ffffff,0xd0000000); 
    write_32_mask(base_addr, 0x0894,0x00ffffff,0x20000000);
    
    write_32_mask(base_addr, 0x0a00,0xffff00ff,0x00000100);
    write_32_mask(base_addr, 0x0a08,0xff000000,0x00e12c08);
    write_32_mask(base_addr, 0x0a0c,0xffffff00,0x00000081);
    write_32_mask(base_addr, 0x0a18,0xff00ffff,0x00e80000);
    write_32_mask(base_addr, 0x0a30,0x00ffff00,0x002f2f00);
    write_32_mask(base_addr, 0x0a48,0xff0000ff,0x00e3ce00);
    write_32_mask(base_addr, 0x0a4c,0x0000ffff,0xac820000);
    write_32_mask(base_addr, 0x0a54,0x00ffffff,0xc0000000);
    write_32_mask(base_addr, 0x0a58,0xffff0000,0x00001441);
    write_32_mask(base_addr, 0x0a84,0xffff0000,0x00000301); 
    
    write_32_mask(base_addr, 0x0a8c,0x0000ffff,0x81030000);    
    write_32_mask(base_addr, 0x0a90,0xffff0000,0x00006001);    
    write_32_mask(base_addr, 0x0a94,0x00ffffff,0x01000000);    
    write_32_mask(base_addr, 0x0aa0,0x00ffffff,0x81000000);    
    write_32_mask(base_addr, 0x0abc,0x00ffffff,0xff000000);    
    write_32_mask(base_addr, 0x0ac0,0xffffff00,0x0000008b);

// Added For PHY Loop Back
#ifdef 0
   uint32_t old_val;
   old_val = DEVICE_REG32_R(base_addr + 0x200);
   DEVICE_REG32_W(base_addr + 0x200, ((old_val & 0x0FFFFFFF)|0x40000000));
   old_val = DEVICE_REG32_R(base_addr + 0x400);
   DEVICE_REG32_W(base_addr + 0x400, ((old_val & 0x0FFFFFFF)|0x40000000));
   old_val = DEVICE_REG32_R(base_addr + 0x600);
   DEVICE_REG32_W(base_addr + 0x600, ((old_val & 0x0FFFFFFF)|0x40000000));
   old_val = DEVICE_REG32_R(base_addr + 0x800);
   DEVICE_REG32_W(base_addr + 0x800, ((old_val & 0x0FFFFFFF)|0x40000000));
#endif
    write_32_mask(base_addr, 0x0000,0xffffff00,0x00000003);    
    write_32_mask(base_addr, 0x0a00,0xffffff00,0x0000009f);
}

#endif
/*****************************************************************************
 * Function: Power domain configuration
 ****************************************************************************/
pcieRet_e pciePowerCfg(void)
{
  /* Turn on the PCIe power domain */
  if (CSL_PSC_getPowerDomainState (CSL_PSC_PD_PCIE) != PSC_PDSTATE_ON) {
    /* Enable the domain */
    CSL_PSC_enablePowerDomain (CSL_PSC_PD_PCIE);
    /* Enable MDCTL */
    CSL_PSC_setModuleNextState (CSL_PSC_LPSC_PCIE, PSC_MODSTATE_ENABLE);
    /* Apply the domain */
    CSL_PSC_startStateTransition (CSL_PSC_PD_PCIE);
    /* Wait for it to finish */
    while (! CSL_PSC_isStateTransitionDone (CSL_PSC_PD_PCIE));
  } else {
    System_printf ("Power domain is already enabled.  You probably re-ran without device reset (which is OK)\n");
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Utility function to introduce delay 
 ****************************************************************************/
void cycleDelay (uint32_t count)
{
  uint32_t sat;

  if (count <= 0)
    return;

  sat = TSCL + count;
  while (TSCL < sat);
}

/*****************************************************************************
 * Function: Serdes configuration 
 ****************************************************************************/
pcieRet_e pcieSerdesCfg(void)
{
#if !defined(DEVICE_K2K) && !defined(DEVICE_K2H)
  uint16_t cfg;
#endif
  
#if !defined(DEVICE_K2K) && !defined(DEVICE_K2H)
  /* Provide PLL reference clock to SERDES inside PCIESS
     Program PLL settings and enable PLL from PCIe SERDES.*/
  cfg = 0x01C9; /* value based on PCIe userguide */

  CSL_BootCfgSetPCIEConfigPLL(cfg);
#else /* !DEVICE_K2K && !DEVICE_K2H */
#ifndef  SIMULATOR_SUPPORT
   //CSL_PciessDefSbSerdesSetup();
   pcieSSDefSbSerdesSetup(CSL_PCIE_SERDES_CFG_REGS);
#endif
#endif
  /*Wait for PLL to lock (3000 CLKIN1 cycles) */
  cycleDelay(10000);
  
  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Enable/Disable LTSSM (Link Training)
 ****************************************************************************/
pcieRet_e pcieLtssmCtrl(Pcie_Handle handle, uint8_t enable)
{
  pcieCmdStatusReg_t    cmdStatus;
  pcieRegisters_t       setRegs;
  pcieRegisters_t       getRegs;
  pcieRet_e retVal;

  memset (&cmdStatus,    0, sizeof(cmdStatus));
  memset (&setRegs,      0, sizeof(setRegs));
  memset (&getRegs,      0, sizeof(getRegs));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  
  if(enable)
    cmdStatus.ltssmEn = 1;
  else  
    cmdStatus.ltssmEn = 0;

  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}


/*****************************************************************************
 * Function: Configure PCIe in Root Complex Mode
 ****************************************************************************/
pcieRet_e pcieCfgRC(Pcie_Handle handle)
{
  pcieRet_e retVal;

  pcieCmdStatusReg_t     cmdStatus;
  pcieObSizeReg_t        obSize;
  pcieGen2Reg_t          gen2;
  pcieType1Bar32bitIdx_t type1Bar32bitIdx;
  pcieStatusCmdReg_t     statusCmd;
  pcieDevStatCtrlReg_t   devStatCtrl;
  pcieAccrReg_t          accr;
                 
  pcieRegisters_t        setRegs;
  pcieRegisters_t        getRegs;

  memset (&cmdStatus,        0, sizeof(cmdStatus));
  memset (&obSize,           0, sizeof(obSize));
  memset (&gen2,             0, sizeof(gen2));
  memset (&type1Bar32bitIdx, 0, sizeof(type1Bar32bitIdx));
  memset (&statusCmd,        0, sizeof(statusCmd));
  memset (&devStatCtrl,      0, sizeof(devStatCtrl));
  memset (&accr,             0, sizeof(accr));

  /*Disable link training*/
  if ((retVal = pcieLtssmCtrl(handle, FALSE)) != pcie_RET_OK) 
  {
    System_printf ("Failed to disable Link Training!\n");
    return retVal;
  }
  
  /* Configure the size of the translation regions */   
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  
  obSize.size = pcie_OB_SIZE_8MB;
  setRegs.obSize = &obSize;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET OB_SIZE register failed!\n");
    return retVal;
  }
     
  /* Setting PL_GEN2 */   
  memset (&setRegs, 0, sizeof(setRegs));
  gen2.numFts = 0xF;
  gen2.dirSpd = 0x0;
  gen2.lnEn   = 1;
  setRegs.gen2 = &gen2;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET GEN2 register failed!\n");
    return retVal;
  }
  
  /* Configure BAR Masks */   
  /* First need to enable writing on BAR mask registers */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }
  
  /* Configure Masks*/
  memset (&setRegs, 0, sizeof(setRegs));
  type1Bar32bitIdx.reg.reg32 = PCIE_BAR_MASK;
  setRegs.type1Bar32bitIdx = &type1Bar32bitIdx;

  /* BAR 0 */
  type1Bar32bitIdx.idx = 0; /* configure BAR 0*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* BAR 1 */
  type1Bar32bitIdx.idx = 1; /* configure BAR 1*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* Disable writing on BAR Masks */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 0;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  /* Enable memory access and mastership of the bus */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.statusCmd = &statusCmd;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read Status Comand register failed!\n");
    return retVal;
  }
  statusCmd.memSp  = 1;
  statusCmd.busMs  = 1;
  statusCmd.resp   = 1;
  statusCmd.serrEn = 1;
  setRegs.statusCmd = &statusCmd;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Status Command register failed!\n");
    return retVal;
  }

  /* Enable Error Reporting */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.devStatCtrl = &devStatCtrl;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Regad Device Status Control register failed!\n");
    return retVal;
  }
 
  devStatCtrl.reqRp = 1;
  devStatCtrl.fatalErRp = 1;
  devStatCtrl.nFatalErRp = 1;
  devStatCtrl.corErRp = 1;
  setRegs.devStatCtrl = &devStatCtrl;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Device Status Control register failed!\n");
    return retVal;
  }

  /* Enable ECRC */
  memset (&setRegs, 0, sizeof(setRegs));
  
  accr.chkEn=1;
  accr.chkCap=1;
  accr.genEn=1;
  accr.genCap=1;
  setRegs.accr = &accr;

  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET ACCR register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Configure PCIe in End Point Mode
 ****************************************************************************/
pcieRet_e pcieCfgEP(Pcie_Handle handle)
{
  pcieRet_e retVal;

  pcieCmdStatusReg_t     cmdStatus;
  pcieObSizeReg_t        obSize;
  pcieGen2Reg_t          gen2;
  pcieType0Bar32bitIdx_t type0Bar32bitIdx;
  pcieStatusCmdReg_t     statusCmd;
  pcieDevStatCtrlReg_t   devStatCtrl;
  pcieAccrReg_t          accr;
                 
  pcieRegisters_t        setRegs;
  pcieRegisters_t        getRegs;

  memset (&cmdStatus,        0, sizeof(cmdStatus));
  memset (&obSize,           0, sizeof(obSize));
  memset (&gen2,             0, sizeof(gen2));
  memset (&type0Bar32bitIdx, 0, sizeof(type0Bar32bitIdx));
  memset (&statusCmd,        0, sizeof(statusCmd));
  memset (&devStatCtrl,      0, sizeof(devStatCtrl));
  memset (&accr,             0, sizeof(accr));

  /*Disable link training*/
  if ((retVal = pcieLtssmCtrl(handle, FALSE)) != pcie_RET_OK) 
  {
    System_printf ("Failed to disable Link Training!\n");
    return retVal;
  }
  
  /* Configure the size of the translation regions */   
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  
  obSize.size = pcie_OB_SIZE_8MB;
  setRegs.obSize = &obSize;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET OB_SIZE register failed!\n");
    return retVal;
  }
     
  /* Setting PL_GEN2 */   
  memset (&setRegs, 0, sizeof(setRegs));
  gen2.numFts = 0xF;
  gen2.dirSpd = 0x0;
  gen2.lnEn   = 1;
  setRegs.gen2 = &gen2;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET GEN2 register failed!\n");
    return retVal;
  }
  
  /* Configure BAR Masks */   
  /* First need to enable writing on BAR mask registers */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }
  
  /* Configure Masks*/
  memset (&setRegs, 0, sizeof(setRegs));
  type0Bar32bitIdx.reg.reg32 = PCIE_BAR_MASK;
  setRegs.type0Bar32bitIdx = &type0Bar32bitIdx;

  /* BAR 0 */
  type0Bar32bitIdx.idx = 0; /* configure BAR 0*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* BAR 1 */
  type0Bar32bitIdx.idx = 1; /* configure BAR 1*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* Disable writing on BAR Masks */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 0;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  /* Enable memory access and mastership of the bus */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.statusCmd = &statusCmd;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read Status Comand register failed!\n");
    return retVal;
  }
  statusCmd.memSp  = 1;
  statusCmd.busMs  = 1;
  statusCmd.resp   = 1;
  statusCmd.serrEn = 1;
  setRegs.statusCmd = &statusCmd;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Status Command register failed!\n");
    return retVal;
  }

  /* Enable Error Reporting */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.devStatCtrl = &devStatCtrl;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Regad Device Status Control register failed!\n");
    return retVal;
  }
 
  devStatCtrl.reqRp = 1;
  devStatCtrl.fatalErRp = 1;
  devStatCtrl.nFatalErRp = 1;
  devStatCtrl.corErRp = 1;
  setRegs.devStatCtrl = &devStatCtrl;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Device Status Control register failed!\n");
    return retVal;
  }

  /* Enable ECRC */
  memset (&setRegs, 0, sizeof(setRegs));
  
  accr.chkEn=1;
  accr.chkCap=1;
  accr.genEn=1;
  accr.genCap=1;
  setRegs.accr = &accr;

  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET ACCR register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Configure and enable Outbound Address Translation 
 ****************************************************************************/
pcieRet_e pcieObTransCfg(Pcie_Handle handle, uint32_t obAddrLo, uint32_t obAddrHi, uint8_t region)
{

  pcieRet_e retVal;

  pcieRegisters_t      setRegs;
  pcieRegisters_t      getRegs;

  pcieCmdStatusReg_t   cmdStatus;

  memset (&setRegs,   0, sizeof(setRegs));
  memset (&getRegs,   0, sizeof(getRegs));
  memset (&cmdStatus, 0, sizeof(cmdStatus));

  /* Set outbound offset registers */
  if ((retVal = Pcie_cfgObOffset(handle, obAddrLo, obAddrHi, region)) != pcie_RET_OK) 
  {
    System_printf ("Failed to configure ObOffset registers!\n");
    return retVal;
  }

  /*enable Outbound address translation*/
  memset (&setRegs,    0, sizeof(setRegs));
  memset (&getRegs,    0, sizeof(getRegs));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.obXltEn = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}


/*****************************************************************************
 * Function: Configure and enable Inbound Address Translation 
 ****************************************************************************/
pcieRet_e pcieIbTransCfg(Pcie_Handle handle, pcieIbTransCfg_t *ibCfg)
{
  pcieRet_e retVal;

  pcieRegisters_t      setRegs;
  pcieRegisters_t      getRegs;

  pcieCmdStatusReg_t   cmdStatus;

  memset (&setRegs,   0, sizeof(setRegs));
  memset (&getRegs,   0, sizeof(getRegs));
  memset (&cmdStatus, 0, sizeof(cmdStatus));

  /* Set outbound offset registers */
  if ((retVal = Pcie_cfgIbTrans(handle, ibCfg)) != pcie_RET_OK) 
  {
    System_printf ("Failed to configure Inbound Translation registers!\n");
    return retVal;
  }

  /*enable Inbound address translation*/
  memset (&setRegs,    0, sizeof(setRegs));
  memset (&getRegs,    0, sizeof(getRegs));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.ibXltEn = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Configure PHY Loopback
 ****************************************************************************/
pcieRet_e pcieSerDesLoopback(Pcie_Handle handle)
{
  pcieRet_e retVal;

  pcieRegisters_t      setRegs;
  pcieRegisters_t      getRegs;

  pcieSerdesCfg0Reg_t   serdesCfg0;
  pcieSerdesCfg1Reg_t   serdesCfg1;

  memset (&setRegs,   0, sizeof(setRegs));
  memset (&getRegs,   0, sizeof(getRegs));
  memset (&serdesCfg0, 0, sizeof(serdesCfg0));
  memset (&serdesCfg1, 0, sizeof(serdesCfg1));


  getRegs.serdesCfg0 = &serdesCfg0;
  getRegs.serdesCfg1 = &serdesCfg1;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
  {
    System_printf ("Read SerdesCfg register failed!\n");
    return retVal;
  }
  serdesCfg0.txLoopback = 0x2;
  serdesCfg0.rxLoopback = 0x3;
  serdesCfg0.rxLos = 0;
  serdesCfg1.txLoopback = 0x2;
  serdesCfg1.rxLoopback = 0x3;
  serdesCfg1.rxLos = 0;

  setRegs.serdesCfg0 = &serdesCfg0;
  setRegs.serdesCfg1 = &serdesCfg1;

  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
  {
    System_printf ("SET SerdesCfg PHYLOOPBACK failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Configure POLL ACTIVE STATE
 ****************************************************************************/
pcieRet_e pcieForceLink(Pcie_Handle handle)
{
  pcieRet_e retVal;

  pcieRegisters_t      setRegs;
  pcieRegisters_t      getRegs;

  pciePlForceLinkReg_t   plForceLink;


  memset (&setRegs,   0, sizeof(setRegs));
  memset (&getRegs,   0, sizeof(getRegs));
  memset (&plForceLink, 0, sizeof(plForceLink));


  getRegs.plForceLink = &plForceLink;

  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)
  {
    System_printf ("Read plForceLink register failed!\n");
    return retVal;
  }
  plForceLink.lnkState = 0x2;
  plForceLink.forceLink = 0x1;

  setRegs.plForceLink = &plForceLink;

  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)
  {
    System_printf ("SET plForceLink failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Initialize application buffers
 ****************************************************************************/
void pcieInitAppBuf(void)
{
  uint32_t i;
  
  for (i=0; i<PCIE_BUFSIZE_APP; i++)
  {
    dstBuf.buf[i] = 0;	
    srcBuf[i] = i+5;
  }
  
  dstBuf.buf[PCIE_BUFSIZE_APP] = PCIE_EXAMPLE_BUF_EMPTY;
}

/*****************************************************************************
 * Function: Check LTSSM status and wait for the link to be up
 ****************************************************************************/
void pcieWaitLinkUp(Pcie_Handle handle)
{
  pcieRegisters_t  getRegs;
  pcieDebug0Reg_t  debug0;
  
  memset (&debug0,  0, sizeof(debug0));
  memset (&getRegs, 0, sizeof(getRegs));
  
  uint8_t ltssmState = 0;
  getRegs.debug0 = &debug0;
  if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK)
  {
       System_printf ("Read LTSSM state failed!\n");
       return;
  }
  System_printf ("Debug0 Value #: 0x%08x\n\n",  debug0);
  while(ltssmState != pcie_LTSSM_L0)
  {
    cycleDelay(100);
    getRegs.debug0 = &debug0;
    if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK) 
    {
      System_printf ("Read LTSSM state failed!\n");
      return;
    }
    ltssmState = debug0.ltssmState;
  }
}

//yunas
void readDevIdRemote(Pcie_Handle handle)
{
	 pcieRegisters_t  myRegs;
	 pcieRet_e        retVal = 0;
	 pcieVndDevIdReg_t vendID;


	memset (&myRegs,   0, sizeof(myRegs));
	memset (&vendID, 0, sizeof(vendID));
	myRegs.vndDevId = &vendID;

	   //myRegs.vndDevId = &myRemoteCfgDevAndVendorID;

	   if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
	   {
	        System_printf ("Read REMOTE DEVICE AND VENDOR ID register failed!\n");
	   }
	   System_printf ("Device ID is %d \n", vendID.devId);
	   System_printf ("Vendor ID is %d \n", vendID.vndId);
	   System_printf ("asas!\n");
	   retVal = 0;
#if 0
	   memset (&myRegs,   0, sizeof(myRegs));

	     for(barNum=0; barNum < 1; barNum++)
	     {

	           if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
	           {
	                System_printf ("Read REMOTE BAR CONFIGURATION register failed!\n");
	           }
	     }

	     System_printf ("BAR number values \n", myRegs.vndDevId->vndId, myRegs.vndDevId->devId);
#endif
}


void readBARConfigRemote(Pcie_Handle handle)
{
		pcieRegisters_t  myRegs;
		pcieRet_e        retVal = 0;
		pcieType0BarIdx_t BARID;

		memset (&myRegs,   0, sizeof(myRegs));
		memset (&BARID, 0, sizeof(BARID));
		myRegs.type0BarIdx = &BARID;


		if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
		{
			System_printf ("Read REMOTE BAR ID register failed!\n");
		}

		//System_printf ("Device ID is %d \n", vendID.devId);
		//System_printf ("Vendor ID is %d \n", vendID.vndId);
		System_printf ("asas!\n");

}


void readBARAddressConfigRemote(Pcie_Handle handle, int read_before_config)
{

			pcieRegisters_t  myRegs;
			pcieRet_e        retVal = 0;
			pcieType0Bar32bitIdx_t BARIDx;

			memset (&myRegs,   0, sizeof(myRegs));
			memset (&BARIDx, 0, sizeof(BARIDx));
			BARIDx.idx = 1;
			myRegs.type0Bar32bitIdx = &BARIDx;


	if (1 == read_before_config)
	{
	//	if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
	//	{
	//		System_printf ("Read REMOTE BAR ID register failed!\n");
	//	}
		BARIDx.reg.reg32 = 0xFFFFFFFF;

		myRegs.type0Bar32bitIdx = &BARIDx;

		if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
		{
			System_printf ("Read REMOTE BAR ID register failed!\n");
		}

		memset (&myRegs,   0, sizeof(myRegs));
		memset (&BARIDx, 0, sizeof(BARIDx));
		BARIDx.idx = 1;
		myRegs.type0Bar32bitIdx = &BARIDx;


		if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
		{
			System_printf ("Read REMOTE BAR ID register failed!\n");
		}


		//System_printf ("Device ID is %d \n", vendID.devId);
		//System_printf ("Vendor ID is %d \n", vendID.vndId);
		System_printf ("asas!\n");
	}

	else
	{

		if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
			{
				System_printf ("Read REMOTE BAR ID register failed!\n");
			}
		System_printf ("asas!\n");
	}

}



/*****************************************************************************
 * Function: Converts a core local L2 address to a global L2 address 
 *   Input addr:  L2 address to be converted to global.
 *   return:  uint32_t   Global L2 address
 *****************************************************************************/
uint32_t pcieConvert_CoreLocal2GlobalAddr (uint32_t  addr)
{
  uint32_t coreNum;

  /* Get the core number. */
  coreNum = CSL_chipReadReg(CSL_CHIP_DNUM); 
  
  /* Compute the global address. */
  return ((1 << 28) | (coreNum << 24) | (addr & 0x00ffffff));
}    


/*****************************************************************************
 * Function: Main 
 ****************************************************************************/
void main (void)
{
  TSCL = 1;
#if !defined(DEVICE_K2K) && !defined(DEVICE_K2H)
  uint16_t lock=0;
#endif
  pcieRet_e        retVal;
  pcieIbTransCfg_t ibCfg;
  pcieBarCfg_t     barCfg,barCfg_remote;    //yunas
  Pcie_Handle      handle = NULL;
  void             *pcieBase;
  uint32_t    i;

  /* Unlock kicker once, and don't relock, because its not multicore safe */
  CSL_BootCfgUnlockKicker();

  System_printf ("**********************************************\n");
  System_printf ("*             PCIe Test Start                *\n");
  
  if(PcieModeGbl == pcie_RC_MODE)
    System_printf ("*                RC mode                     *\n");
  else
    System_printf ("*                EP mode                     *\n");
  
  System_printf ("**********************************************\n\n");
  
  System_printf ("Version #: 0x%08x; string %s\n\n", Pcie_getVersion(), Pcie_getVersionStr());
  
  /* Pass device config to LLD */
  if ((retVal = Pcie_init (&pcieInitCfg)) != pcie_RET_OK)
  {
    System_printf ("LLD device configuration failed\n");
    exit(1);
  }

  /* Initialize application buffers */
  pcieInitAppBuf();

  /* Configure SERDES*/
  if ((retVal = pcieSerdesCfg()) != pcie_RET_OK) {
    System_printf ("PCIe Serdes config failed (%d)\n", (int)retVal);
    exit(1);
  }

  /* Set the PCIe mode*/
  if ((retVal = Pcie_setMode(PcieModeGbl)) != pcie_RET_OK) {
    System_printf ("Set PCIe Mode failed (%d)\n", (int)retVal);
    exit(1);
  }
  
  /* Power up PCIe Module */
  if ((retVal = pciePowerCfg()) != pcie_RET_OK) {
    System_printf ("PCIe Power Up failed (%d)\n", (int)retVal);
    exit(1);
  }

  System_printf ("PCIe Power Up.\n");
  
  /* Wait until the PCIe SERDES PLL locks */
#if !defined(DEVICE_K2K) && !defined(DEVICE_K2H)
  while (!lock)
  {
    CSL_BootCfgGetPCIEPLLLock(&lock);
  }                    
#endif /* !DEVICE_K2K && !DEVICE_K2H */

  if ((retVal = Pcie_open(0, &handle)) != pcie_RET_OK) 
  {
    System_printf ("Open failed (%d)\n", (int)retVal);
    exit(1);
  }

  System_printf ("PLL configured.\n");

  if(PcieModeGbl == pcie_RC_MODE)
  {
    /* Configure application registers for Root Complex*/
    if ((retVal = pcieCfgRC(handle)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure PCIe in RC mode (%d)\n", (int)retVal);
      exit(1);
    }

    /* Configure Address Translation */
    
    barCfg.location = pcie_LOCATION_LOCAL;
    barCfg.mode     = pcie_RC_MODE;
    barCfg.base     = PCIE_IB_LO_ADDR_M;
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR_IDX_M;
    
    if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure BAR (%d)\n", (int)retVal);
      exit(1);
    }
    else
    {
    	System_printf("Configured for BAR (%d)\n", (int)retVal);
    }

    ibCfg.ibBar         = PCIE_BAR_IDX_M; /* Match BAR that was configured above*/
    ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_M;
    ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_M;
    ibCfg.ibOffsetAddr  = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)dstBuf.buf);
    ibCfg.region        = PCIE_IB_REGION_M;       

    if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Inbound Translation (%d)\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Inbound Translation!\n");
    }

    if ((retVal = pcieObTransCfg (handle, PCIE_OB_LO_ADDR_M, PCIE_OB_HI_ADDR_M, PCIE_OB_REGION_M)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Outbound Address Translation (%d)\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Outbound Translation!\n");
    }

  }
  else
  {
    /* Configure application registers for End Point*/
    if ((retVal = pcieCfgEP(handle)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure PCIe in EP mode (%d)\n", (int)retVal);
      exit(1);
    }

    /* Configure Address Translation */
    
    barCfg.location = pcie_LOCATION_REMOTE;
    barCfg.mode     = pcie_RC_MODE;
    barCfg.base     = PCIE_IB_LO_ADDR_S;
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR_IDX_S;
    
    if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure BAR!\n");
      exit(1);
    }


    ibCfg.ibBar         = PCIE_BAR_IDX_S; /* Match BAR that was configured above*/
    ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_S;
    ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_S;
    ibCfg.ibOffsetAddr  = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)dstBuf.buf);
    ibCfg.region        = PCIE_IB_REGION_S;       

    if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Inbound Translation!\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Inbound Translation!\n");
    }

    if ((retVal = pcieObTransCfg (handle, PCIE_OB_LO_ADDR_S, PCIE_OB_HI_ADDR_S, PCIE_OB_REGION_S)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Outbound Address Translation!\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Outbound Translation!\n");
    }
  }

#ifdef 0
  System_printf ("Configure PHY Loopback...\n");
  pcieSerDesLoopback(handle);
#endif
  System_printf ("Starting link training...\n");

  /*Enable link training*/
   if ((retVal = pcieLtssmCtrl(handle, TRUE)) != pcie_RET_OK)
  {
    System_printf ("Failed to Enable Link Training!\n", (int)retVal);
    exit(1);
  }
#ifdef 0
   System_printf ("Force PCIe Link state...\n");
   pcieForceLink(handle);
#endif
  /* Wait for link to be up */
  System_printf ("PCIe pcieWaitLinkUp...\n");
  pcieWaitLinkUp(handle);


  System_printf ("Link is up.\n");


  //yunas


  	  readDevIdRemote(handle);

      readBARConfigRemote(handle);

      readBARAddressConfigRemote(handle,1);

      barCfg_remote.location = pcie_LOCATION_REMOTE;
      barCfg_remote.mode     = pcie_RC_MODE;
      barCfg_remote.base     = 0x00800001;
      barCfg_remote.prefetch = pcie_BAR_NON_PREF;
      barCfg_remote.type     = pcie_BAR_TYPE32;
      barCfg_remote.memSpace = pcie_BAR_MEM_MEM;
      barCfg_remote.idx      = 1;

      if ((retVal = Pcie_cfgBar(handle, &barCfg_remote)) != pcie_RET_OK)
      {
         System_printf ("Failed to configure BAR%d\n",(int)retVal);
         exit(1);
      }
      else
      {
         System_printf("Configured remote BAR%d successfully!\n", (int)retVal);
      }


      readBARAddressConfigRemote(handle, 0);
//yunas



  if(PcieModeGbl == pcie_RC_MODE)
  {
    /**********************************************************************/
    /* Push a single message to the EP then verify that it is echoed back */
    /**********************************************************************/

    /* Write from RC to EP                                                */
    if ((retVal = Pcie_getMemSpaceRange (handle, &pcieBase, NULL)) != pcie_RET_OK) {
      System_printf ("getMemSpaceRange failed\n", (int)retVal);
      exit(1);
    }

	  //pcieBase = (void *)0x50000000;

    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
      *((volatile uint32_t *)pcieBase + i) = srcBuf[i];
    }

    *((volatile uint32_t *)pcieBase + PCIE_BUFSIZE_APP) = PCIE_EXAMPLE_BUF_FULL;

#if 0
    do {
        unsigned int key;

        /* Disable Interrupts */
        key = _disable_interrupts();

        /*  Cleanup the prefetch buffer also. */
        CSL_XMC_invalidatePrefetchBuffer();

        CACHE_invL1d ((void *)pcieBase,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);
        CACHE_invL2  ((void *)pcieBase,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

        /* Reenable Interrupts. */
        _restore_interrupts(key);

      } while(*((volatile uint32_t *)pcieBase + 40) != PCIE_EXAMPLE_BUF_FULL);
#endif

    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
    	System_printf ("value from PCIe interface %d\n",*((volatile uint32_t *)pcieBase + i));
    }


    /* Mark that the buffer is full, so EP can process it */


 //   System_printf ("value from PCIe interface %d\n",*((volatile uint32_t *)pcieBase + PCIE_BUFSIZE_APP));
  //  System_printf ("value from PCIe interface %d\n",*((volatile uint32_t *)pcieBase + PCIE_BUFSIZE_APP - 1));
  //  System_printf ("value from PCIe interface %d\n",*((volatile uint32_t *)pcieBase + PCIE_BUFSIZE_APP - 2));
    /* Note on cache coherence: Write back is not necessary because pcieBase is in
       peripheral address space instead of physical memory*/
    
    /* Data sent to EP.
       RC waits for the loopback to be completed and
       receive data back from EP */
#if 0
    do {
      unsigned int key;

      /* Disable Interrupts */
      key = _disable_interrupts();

      /*  Cleanup the prefetch buffer also. */
      CSL_XMC_invalidatePrefetchBuffer();    

      CACHE_invL1d ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);
      CACHE_invL2  ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

      /* Reenable Interrupts. */
      _restore_interrupts(key);

    } while(dstBuf.buf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);
#endif
    /* check all the data */
    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {

      if(dstBuf.buf[i] != srcBuf[i])
      {
        System_printf ("Received data = %d\nTransmited data = %d\nIndex = %d.\n\nTest failed.\n", 
                        dstBuf.buf[i], srcBuf[i], i);
        exit(1);
      }

    }
    
    System_printf ("Root Complex received data.\nTest passed.\n");
  }
  else
  {
    /**********************************************************************/
    /* Wait for a single message from the RC then echo it back            */
    /**********************************************************************/

    /* EP waits for the data received from RC */
    do {
      unsigned int key;

      /* Disable Interrupts */
      key = _disable_interrupts();

      /*  Cleanup the prefetch buffer also. */
      CSL_XMC_invalidatePrefetchBuffer();    

      CACHE_invL1d ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);
      CACHE_invL2  ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

      /* Reenable Interrupts. */
      _restore_interrupts(key);

    } while(dstBuf.buf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);

    System_printf ("End Point received data.\n");

    /* Loopback to RC what was written in the DST buffer.
       Write from EP to RC */
    if ((retVal = Pcie_getMemSpaceRange (handle, &pcieBase, NULL)) != pcie_RET_OK) {
      System_printf ("getMemSpaceRange failed\n", (int)retVal);
      exit(1);
    }

    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
      *((volatile uint32_t *)pcieBase + i) = dstBuf.buf[i];
    }
    
    /* Mark that the buffer is full, so RC can process it */
    *((volatile uint32_t *)pcieBase + PCIE_BUFSIZE_APP) = PCIE_EXAMPLE_BUF_FULL;

    /* Note on cache coherence: Write back is not necessary because pcieBase is in
       peripheral address space instead of physical memory*/

    System_printf ("End Point sent data to Root Complex, completing the loopback.\nEnd of Test.\n");
  }

  exit(0);
}

  • First of all, may I draw your attention, that 1342177280 is 0x5000_0000 and that should mean something to you.

    Next,  starting from line 1309 you are configuring remote PCIe party. Since your SoC runs RC, remote should be EP, but you set its mode as RC (line 1310).

    The very next line

    barCfg_remote.base     = 0x00800001;

    looks strange. At least BAR base address should be multiple of its mem window, thus seeing there 1 in lower position looks strange.

    I wonder, why you configure BAR1 (barCfg_remote.idx      = 1;) rather than BAR0 on remote EP.

    In your code there is OB configuration, but PCIE_OB_LO_ADDR_S, PCIE_OB_HI_ADDR_S, PCIE_OB_REGION_S are not defined in the given file, so I would refrain of guessing there values.

    These are enough to fix first.

    Meanwhile I would suggest to start with BAR0, setup OB translation with addresses of zero for zero region, and se. Then whenever you write to PCIe data address range that would land to zero PCIe address in FPGA. That is good starting point to debug your design.

    And may I ask, what is on FPGA side of your design? Can you plug ChipScope in? If so, I would suggest to monitor bar_hit signal and transaction data.

  • Hi,

    PCI support team is notified. Their feedback will be posted directly here.

    Best Regards,
    Yordan
  • Yunas,

    rrlagic provides very good points! Please try to follow his suggestion to see if any good results. In our TI Keystone devices, PCIE BAR0 is dedicated to the application register, that is why we used from BAR1 in our code example, which was run between two TI devices. For the FPGA side, if BAR0 usage is not dedicated, please try from BAR0 first.

    You can also provide us some PCIE register dump for check-up: from 0x2180_0000 to 0x2180_2FFC.

    Regards, Eric
  • I saw you opened another thread for FPGA with K2H over PCIE, e2e.ti.com/.../543481. And the linux log showed the link came up already? Are you still pursuing K2H with RTOS PDK package?

    Regards, Eric
  • Hello Eric,

    We are able to use PCIe using K2H RTOS PDK package. The problem turned out to be the BAR config on the FPGA. Nothing wrong with the EVM.

    We will soon be doing PCIe throughput test. I hope we get some decent speed. :)

    Thanks for your support and help.

    Regards, Yunas