PCIe to PCIe using two C6657 EVMs

Edward Ewer1

Other Parts Discussed in Thread: TMS320C6657, TMDXEVMPCI

I'm using two TMS320C6657 EVMs with a PCIe backplane powered by an ATX supply. I've built the PCIE_exampleProject, one as RC and one as EP for the two EVMs and have failed to get a reliable PCIe between the two EVMs. I'm using the integrated XDS200 Onboard Emulator for each EVM. Both EVMs are configured for nonBoot mode (Sleep/EMIF16). What I'm seeing isa successful link training reaching the L0 state (0x11) in DEBUG0[4:0] and then I lose the L0 state almost immediately. I do sometimes get a successful run between the two boards but only if I power off everything and rerun - this does not always result in a successful test run of the PCIe example code. In fact, I see the link state return to 0x0 (pcie_LTSSM_DETECT_QUIET) even with a successful run.

If I do a system reset and reload of core0 I cannot get a successful run - The link training gets to the L0 state and quickly goes to recovery and back to quiet.

I have checked registers with the emulator such as the Device control register (EP = 0x00011801, RC = 0x00019801), SERDES_CFG[0/1] 0x000622A0/0x000222A0 - All the register settings are consistent with the PCIE_exampleProject code settings.

This is an ongoing issue that we need resolved so we can move on to our development phase.

We need to determine why the link state is not staying in L0 for any meaningful period. What help can you provide? What questions do you have for me?

Thanks,

Edward

over 12 years ago

0 Tom Johnson 16214 over 12 years ago

TI__Mastermind 46460 points

Edward,

Please provide more information about this motherboard. Does it contain a PCIe root complex? Are there other PCIe endpoints in the chassis or on the motherboard? Does it supply a PCIe compliant 100MHz clock to the add-in cards? If so, is it a spread spectrum clock?

There are multiple e2e threads that deal with PCIe enumeration using KeyStone-I devices. The C6657 EVM and the C6678 EVM are nearly identical. Please review the existing threads on this topic.

Tom

0 Edward Ewer1 over 12 years ago in reply to Tom Johnson 16214

Prodigy 140 points

Tom:

The backplane is not a motherboard; it is just a PCI Express 2-slot card from One Stop Systems (OSS-PCIe-BP-2019) powered by an ATX supply.

http://www.onestopsystems.com/passive_backplanes_2019.php

The 100MHz clock is being supplied by the individual EVMs and the common clock bit is cleared. We don't have a spread spectrum clock as I'm assuming the clock chip on the EVM board is not configured for spread spectrum. I simply have two 6657 EVMs on the 2-slod backplane, one configured as an EP and one configured as RC.

I'm not having an issue with enumeration - I'm having an issue with maintaining a stable link after link training.

They successfully achieve the L0 state when link training is enabled and then both cards lose the link and return to state 0 after a brief period in the recovery state.

I've looked at at specific threads related to this already under the 6657EVM posts.

Edward

0 Tom Johnson 16214 over 12 years ago in reply to Edward Ewer1

TI__Mastermind 46460 points

Edward,

I will request additional support for this. From a hardware perspective, I do not see why it is failing to sustain the link.

Tom

0 Steven Ji over 12 years ago in reply to Edward Ewer1

TI__Genius 12065 points

Edward,

May I ask if you are using the exact PCIe PDK example project for your testing on the two EVMs please, just compiled one in RC mode and another one in EP mode?

If there is any change for the configuration, could you let us know or attach the test case you are using please?

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Steven,

I am using the PCIe PDK 6657_1_1_2_5 project example; one compiled as RC and one compiled as EP. I made no changes to the configuration in the build I'm currently using. I start the EP and then the RC - Both reach the L0 state in link training and then we lose the link before the PCIe transactions (data writes from RC to EP to RC) occur. I don't have the interrupt mapped/enabled but we are getting a LINK_RST_REQ in the PMRST_IRQ_STATUS register at offset 0x1D4 according to the debugger.

Although I don't have any hard numbers with regards to timing, once we achieve the L0 state in link training the LTSSM goes to the recovery state(s) and then to the link quiet state. Again, this consistently happens before the data transfer begins. We cannot maintain a stable L0 state.

Edward

0 Steven Ji over 12 years ago in reply to Edward Ewer1

TI__Genius 12065 points

Edward,

Since we do not have the same backplane in your setup here, I connected the two C6657 EVMs via the Dual EVM Break Out Card. Please find the the break-out card info as follows:

http://processors.wiki.ti.com/index.php/CI_Dual_EVM_Break_Out_Card

One C6657 EVM has the version number as follows:

PCB REV: 17-00132-01

PCA REV: 18-00132-01

Another EVM has the follows:

PCB REV: 17-00132-02

PCA REV: 18-00132-02

With those setup above, I could load the RC and EP PCIe PDK example and passed the testing on both of the boards:

After the testing, I checked the DEBUG0 register on both EP and RC, the LTSSM field is 0x11 on both EVMs.

And I repeated the tests for 3 times without any issue.

So it looks like the PCIe configuration and test case you are using should be fine.

I am not sure if you have some other two board connection platform to try. But It might be some hardware connection or signal integrity issue with your setup, maybe in the backplane or the connection between the PCIe slot and the adapter card.

It will be great if you could replace some components to narrow down the issue.

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Steve:

Seeing that you have success using the Dual EVM Breakout Card is encouraging; however, this is not an option for us as we need to connect to our prototype FPGA design reliably using a PCIe backplane. Perhaps another backplane is an option, I'll have to check with the other team members.

I'm using two EVMs with PCB Rev : 17-00132-02 and PCA REV:18-00132-02. I have questions about the setup of your test.

Where you running with the CCS and the XDS200 Onboard Emulator or similar?

I believe the example sets most of these but I'd like to know what were the switch settings you had for SW3, SW5 and SW9? I'm using (all binary format):

BM_GPIO0 = 0 - Little Endian

BM_GPIO[4:1] = 0000 - Sleep/EMIF16 (Non boot mode)

BM_GPIO[10:5] = 000000

BM_GPIO[13:11] = 011 - 100.00MHz

BM_GPIO[15:14] = 00 - End Point (10 on Root Complex EVM)

BM_GPIO16 - 1 PCIe Module enabled.

Also, This may have no bearing on the PCIe, but, on one EVM I've noticed that the Red (D1) and Blue (D2) LEDs connected to the MSP430 are not active; but, on the other EVM I see activity on power-up with the blue LED remaining in the on state. The VCC3V3_MP is present.

Thanks,

Edward

0 Tom Johnson 16214 over 12 years ago in reply to Edward Ewer1

TI__Mastermind 46460 points

Steven,

Which boot mode are you using? Note that the PCIe clock mux configuration is different in PCIe boot mode versus other modes. SW5.3 only controls the mux when PCIe boot mode is selected. Otherwise, software needs to control the CLK_MUX_SEL. Since Edward is using NOBOOT, this might be part of the problem. Does the example code program this register? See below from the TMDSEVM6657LS Technical Reference Manual Version 2.1 on page 91:

CLK_MUX_SEL: This bit can be updated by the DSP software to drive a
high or low value on the CLK_MUX_SEL pin. The default value of this bit
is over-ridden when PCIe boot mode is selected by SW5.3 (BM_GPIO_10).
0 : CLK_MUX_SEL drives low
1 : CLK_MUX_SEL drives high

When CLK_MUX select is high, it selects mux input 1 which is the external clock from the AMC connector. Alternately, when CLK_MUX select is low, it selects mux input 2 which is the local clock generator. You want this bit set low for local clock use.

Tom

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Steven:

I was looking at the manual for the Dual EVM Breakout Card and it has an option for a PCIe common clock. When you run your test are you using a common PCIe clock from the Dual EVM Breakout Card? This is important because we are using local EVM clocks on each board and DO NOT have a common PCIe clock source when running the PCIe_exampleProject.

We hesitate to point at the PCIe backplane as the potential issue here as we have successfully connected two of our prototype FPGA boards (one as RC, one as EP) and maintained a reliable LTSSM L0 state. When using the two EVMs we achieve the L0 state but lose it immediately as I stated before.

Please confirm whether you are using a common PCIe clock from the EVM breakout card or separate local EVM clocks whe running the PCIe example.

Thanks,

Edward

0 Edward Ewer1 over 12 years ago in reply to Tom Johnson 16214

Prodigy 140 points

Tom:

I tried changing the SW5.3 switch and that resulted in NO PLL lock. So, this doesn't appear to be our issue.

Pending a reply from Steven regarding whether he is using a common PCIe clock from the EVM breakout card or local EVM clocks, I'm leaning towards this as our potential issue.

Thanks,

Edward

0 Tom Johnson 16214 over 12 years ago in reply to Edward Ewer1

TI__Mastermind 46460 points

Edward,

Good, your results prove that your environment is definately running on local clock as that is the only way you would get a PLL lock. We need to see which clock mux setting that Steven was using.

Tom

0 Steven Ji over 12 years ago in reply to Tom Johnson 16214

TI__Genius 12065 points

Edward and Tom,

I do not use the common clock option in the Dual EVM board. The two EVMs are using its own reference clock for PCIe.

The original boot setup I have for the two board setup is as follows:

1 2 3 4 5 6 7 8

SW3: 1 0 0 0 0 0 0 0 (little endian; no boot)

SW5: 0 0 0 1 1 0 0 0 (100MHz PLL)

SW9: 0 0 (PCIe disabled)

CLK_MUX_SEL by default is 0, which chooses to use on board clock for PCIe. The PCIe LLD example does not program this bit. I think it is driven by the FPGA configuration registers but can be over-ridden by the SW5.3 in PCIe boot mode.

The PCIe LLD example will enable the power/clock domain of PCIe module, so we do not need to enable it in the boot mode (SW9.1). It is needed when you are using PCIe boot mode. And the PCIe LLD example will overwrite the PCIe mode after boot as well, so we do not need to specify the PCIe mode in boot pins for this no-boot mode testing.

Then I verified your setup as well:

1 2 3 4 5 6 7 8

SW3: 1 0 0 0 0 0 0 0 (little endian; no boot)

SW5: 0 0 0 1 1 0 0 (0 or 1) (100MHz PLL, 00 for EP, 10 for RC)

SW9: 1 0 (PCIe enabled)

The test still passed on the two EVMs with dual board card setup.

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Steven:

By any chance do you have access to a PCIe backplane? That way, you could duplicate the setup we have here and see if you could get a successful test scenario.

Edward

0 Steven Ji over 12 years ago in reply to Edward Ewer1

TI__Genius 12065 points

Edward,

That would be our first choice, but unfortunately I could not find any backplane at my hand.

Do you have any document or link for the backplane in your setup please?

Maybe we can take a look at the datasheet or schematic to compare with the direct dual board setup on paper first.

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Steven:

I posted the link for the backplane we are using earlier in the thread. Here it is:

http://www.onestopsystems.com/passive_backplanes_2019.php

Edward

0 Edward Ewer1 over 12 years ago in reply to Edward Ewer1

Prodigy 140 points

Steven:

We have run some tests around the PCIe power domain with limited success. We found that if we detected that the PCIe power domain was left on (that debug that says something along the lines of "system didn't power cycle, this is OK"), disabled it then re-enabled it we were able to run the test multiple times without a complete power down. It was not 100% but 3/4 times it would work.

Would the LTSSM enter into any power saver states automatically? The default seems to be that power management is disabled but perhaps there is some automatic entry into power saver states. Could you comment on this possibility?

Thanks,

Edward

0 Edward Ewer1 over 12 years ago in reply to Edward Ewer1

Prodigy 140 points

Steven:

I don't know if this sheds any light on our issue, but, I've been checking the PCIE_SERDES_STS register (0x0262015C) and I'm seeing 0x0000_0209 indicating we have a loss of signal detection (LOSTDTCT0 - bit 3). I'm also seeing bit 9 set indicating a LOSTDTCT1 but we are only configured for x1; so, not sure this matters. However, while we have a link established in the L0 state the register returns 0x0000_0001. bit 0 is LOCK (PLL).

Also, would you be able to provide a register dump of the PCIE Application and Configuration Space registers you have for a successful test run so that we can compare our setups? I'm running the PCIe sample code in pdk_C6657_1_1_2_5.

Thanks,

Edward

0 Steven Ji over 12 years ago in reply to Edward Ewer1

TI__Genius 12065 points

Edward,

I think it is normal to have the 0x209 as the value of PCIE_SERDES_STS before the link up, which indicates no connection for both Lane 0 and Lane 1.

After the link up, the value will be 0x1 if connecting two lanes or 0x201 if only Lane 0 is connected (Lane 1 is still electrical idle).

I attach the register dumps of both RC and EP PCIe registers after testing below:

1145.PCIe_RC_Regs_AfterTest.dat

4087.PCIe_EP_Regs_AfterTest.dat

The dump (from 0x21800000 to 0x21801FFF) is stored in the following format:

Format: Hex

Start Address: 0x21800000

Length:

Specify the number of memory words to read: 0x2000/4

Please refer to the following sequence for the testing I have done in my setup:

1. Power up both EVMs and connect CCS to Core0 (or Core1) on each board

2. Do "System Reset" on the connected CorePac, PC will be pointed to 0x20B00000

3. Load "evmc6657l.gel" file (C:\ti\mcsdk_2_01_02_05\tools\program_evm\gel) on connected CorePac

4. Execute "EVMC6657L Init Functions" -> "Global_Default_Setup" under Scripts menu

5. Load test case to the CorePac (RC on one EVM and EP on the othe EVM)

6. Run the test cases on both EVMs (no matter run RC first or EP first)

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Steve:

I executed your sequence of instructions but the results remain the same.

I'll look over the register dump and compare to our settings.

Edward

0 Edward Ewer1 over 12 years ago in reply to Edward Ewer1

Prodigy 140 points

Steven:

I'm seeing a couple of differences in the register dumps that I'd for which I'd like more information.

0x21800388 - PCS_STATUS - Your value: 0x00001111 - My value 0x00001300 - Not sure if this matters or if it applies to our application.

0x21801700 - PL_ACKTIMER - Your value: 0x01630076 - My value is default: 0x00C00040.

I tried setting the value that you have, 0x01630076, but it didn't produce any different results. Still failing.

Edward

0 Steven Ji over 12 years ago in reply to Edward Ewer1

TI__Genius 12065 points

Edward,

Those registers will have the default values before we initialize the link training (set LTSSM bit to 1 in CMD_STATUS register).

After we enable the link training, the values will change to 0x00001111 for PCS_STATUS (for single lane connection) to indicate the TX/RX enable status at physical layer.

The replay timer and latency timer values will change to what is negotiated during the peer-to-peer hand shaking.

Do you capture those registers dump after you enable the link training (after the following step in the test source code)?

if ((retVal = pcieLtssmCtrl(handle, TRUE)) != pcie_RET_OK)

If so, it looks like the PCIe module could not find the peer at the other end of the link in your setup.

Do you see the same behavior with the AMC-to-SMA breakout cards setup as well?

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Steven:

So, looks like we have resolved our issue of losing the PCIe link.. We had a bad EVM on one side of our RC<->EP setup; in this case, it was the RC. Initially, we would get a PLL lock in the SERDES status register but lose the PLL lock and subsequently lose the L0 link state.

Once we replaced the bad board, we are able to run the PCIe transfers successfully repeatedly.

I am interested in obtaining an example for the LINK_RST_REQ interrupt mapping found in the PMRST_IRQ_STATUS register on the TMS320C6657 DSP.

Thanks,

Edward

0 Steven Ji over 12 years ago in reply to Edward Ewer1

TI__Genius 12065 points

Edward,

Please find the LLD example source file with LINK_RST_REQ interrupt test. You can recompile the LLD project with this new source file for both RC and EP side.

Fullscreen 7140.pcie_sample.c Download

/*  ============================================================================
 *   Copyright (c) Texas Instruments Incorporated 2010
 * 
 *  Redistribution and use in source and binary forms, with or without 
 *  modification, are permitted provided that the following conditions 
 *  are met:
 *
 *    Redistributions of source code must retain the above copyright 
 *    notice, this list of conditions and the following disclaimer.
 *
 *    Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the 
 *    documentation and/or other materials provided with the   
 *    distribution.
 *
 *    Neither the name of Texas Instruments Incorporated nor the names of
 *    its contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
 *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
 *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 
 *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 
 *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
 *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
 *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 
 *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *
*/


/**
 *   @file  pcie_sample.c
 *
 *   @brief   
 *      This is the PCIe example code.   
 *
 */

/** 
 * In the PCIe sample example two Shannon EVMs are used to test the PCIe driver. 
 * As described in the following figure, Shannon 1 is configured as a Root Complex
 * and Shannon 2 is configured as End Point.
 *
 *       Shannon 1                                       Shannon 2
 *   ------------------                             -------------------
 *   |                |                             |                 |
 *   |   Root         |          PCIe Link          |  End Point      |
 *   |   Complex      | <-------------------------->|                 |
 *   |                |                             |                 |
 *   ------------------                             -------------------
 *  
 * Once the PCIe link is established, the following sequence of actions will happen:
 *  - Shannon 1 sends data to Shannon 2
 *  - Shannon 2 waits to receive all the data
 *  - Shannon 2 sends the data back to Shannon 1
 *  - Shannon 1 waits to receive all the data
 *  - Shannon 1 verifies if the received data matches the sent data and declares test pass or fail.
 *
 */
 

#include "pcie_sample.h"

/* Use regular "printf" when not using BIOS */
//#include <xdc/runtime/System.h>
#define System_printf printf
#include <stdio.h>

#include <ti/csl/csl_bootcfgAux.h>
#include <ti/csl/csl_cacheAux.h>
#include <ti/csl/csl_chip.h>
#include <ti/csl/csl_xmcAux.h>

#include <ti/csl/src/intc/csl_intc.h>
#include <ti/csl/tistdtypes.h>
#include <ti/csl/csl_cpIntcAux.h>

#pragma DATA_SECTION(dstBuf, ".dstBufSec")
/* Cache coherence: Align must be a multiple of cache line size (L2=128 bytes, L1=64 bytes) to operate with cache enabled. */
/* Aligning to 256 bytes because the PCIe inbound offset register masks the last 8bits of the buffer address  */
#pragma DATA_ALIGN(dstBuf, 256)
/* last element in the buffer is a marker that indicates the buffer status: full/empty */
#define PCIE_EXAMPLE_MAX_CACHE_LINE_SIZE 128
#define PCIE_EXAMPLE_UINT32_SIZE           4 /* preprocessor #if requires a real constant, not a sizeof() */

#define PCIE_EXAMPLE_DSTBUF_BYTES ((PCIE_BUFSIZE_APP + 1) * PCIE_EXAMPLE_UINT32_SIZE)
#define PCIE_EXAMPLE_DSTBUF_REM (PCIE_EXAMPLE_DSTBUF_BYTES % PCIE_EXAMPLE_MAX_CACHE_LINE_SIZE)
#define PCIE_EXAMPLE_DSTBUF_PAD (PCIE_EXAMPLE_DSTBUF_REM ? (PCIE_EXAMPLE_MAX_CACHE_LINE_SIZE - PCIE_EXAMPLE_DSTBUF_REM) : 0)
struct dstBuf_s {
  volatile uint32_t buf[PCIE_BUFSIZE_APP + 1];
  /* Cache coherence: Must pad to cache line size in order to enable cacheability */
#if PCIE_EXAMPLE_DSTBUF_PAD
  uint8_t padding[PCIE_EXAMPLE_DSTBUF_PAD];
#endif
} dstBuf;

#define PCIE_EXAMPLE_BUF_EMPTY 0
#define PCIE_EXAMPLE_BUF_FULL  1

/* Does not need to be aligned (even for cache) since it is only accessed locally */
uint32_t srcBuf[PCIE_BUFSIZE_APP];

extern volatile unsigned int cregister TSCL;

/* Global config variable that controls
   the PCIe mode. It is global so it can be poked
   from CCS. It should be set either to EP or RC. */
pcieMode_e PcieModeGbl = pcie_EP_MODE;

#define IRQ_EOI					(*(unsigned int*)(0x21800050))
#define PMRST_IRQ_STATUS_RAW	(*(unsigned int*)(0x218001D0))
#define PMRST_IRQ_STATUS		(*(unsigned int*)(0x218001D4))
#define PMRST_ENABLE_SET		(*(unsigned int*)(0x218001D8))
#define PMRST_ENABLE_CLR		(*(unsigned int*)(0x218001DC))
#define LINK_STAT_CTRL			(*(unsigned int*)(0x21801080))
#define CMD_STATUS				(*(unsigned int*)(0x21800004))
#define DEBUG0					(*(unsigned int*)(0x21801728))

/* Intc variable declaration */
CSL_CPINTC_Handle           hnd;
CSL_IntcContext             intcContext;
CSL_IntcEventHandlerRecord  EventHandler[30];
CSL_IntcObj                 intcObj;
CSL_IntcHandle              hTest;
CSL_IntcGlobalEnableState   state;
CSL_IntcEventHandlerRecord  EventRecord;
CSL_IntcParam               vectId;

volatile uint32_t reg_read = 0;
volatile uint32_t intTestNum = 0;
volatile uint32_t intTestDone = 0;

/*****************************************************************************
 * Function: Power domain configuration
 ****************************************************************************/
pcieRet_e pciePowerCfg(void)
{
  /* Turn on the PCIe power domain */
  if (CSL_PSC_getPowerDomainState (CSL_PSC_PD_PCIEX) != PSC_PDSTATE_ON) {
    /* Enable the domain */
    CSL_PSC_enablePowerDomain (CSL_PSC_PD_PCIEX);
    /* Enable MDCTL */
    CSL_PSC_setModuleNextState (CSL_PSC_LPSC_PCIEX, PSC_MODSTATE_ENABLE);
    /* Apply the domain */
    CSL_PSC_startStateTransition (CSL_PSC_PD_PCIEX);
    /* Wait for it to finish */
    while (! CSL_PSC_isStateTransitionDone (CSL_PSC_PD_PCIEX));
  } else {
    System_printf ("Power domain is already enabled.  You probably re-ran without device reset (which is OK)\n");
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Utility function to introduce delay 
 ****************************************************************************/
void cycleDelay (uint32_t count)
{
  uint32_t sat;

  if (count <= 0)
    return;

  sat = TSCL + count;
  while (TSCL < sat);
}

/*****************************************************************************
 * Function: Serdes configuration 
 ****************************************************************************/
pcieRet_e pcieSerdesCfg(void)
{
  uint16_t cfg;
  
  CSL_BootCfgUnlockKicker();

  /* Provide PLL reference clock to SERDES inside PCIESS
     Program PLL settings and enable PLL from PCIe SERDES.*/
  cfg = 0x01C9; /* value based on PCIe userguide */
  CSL_BootCfgSetPCIEConfigPLL(cfg);
  
  CSL_BootCfgLockKicker();

  /*Wait for PLL to lock (3000 CLKIN1 cycles) */
  cycleDelay(10000);
  
  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Enable/Disable LTSSM (Link Training)
 ****************************************************************************/
pcieRet_e pcieLtssmCtrl(Pcie_Handle handle, uint8_t enable)
{
  pcieCmdStatusReg_t    cmdStatus;
  pcieRegisters_t       setRegs;
  pcieRegisters_t       getRegs;
  pcieRet_e retVal;

  memset (&cmdStatus,    0, sizeof(cmdStatus));
  memset (&setRegs,      0, sizeof(setRegs));
  memset (&getRegs,      0, sizeof(getRegs));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  
  if(enable)
    cmdStatus.ltssmEn = 1;
  else  
    cmdStatus.ltssmEn = 0;

  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}


/*****************************************************************************
 * Function: Configure PCIe in Root Complex Mode
 ****************************************************************************/
pcieRet_e pcieCfgRC(Pcie_Handle handle)
{
  pcieRet_e retVal;

  pcieCmdStatusReg_t     cmdStatus;
  pcieObSizeReg_t        obSize;
  pcieGen2Reg_t          gen2;
  pcieType1Bar32bitIdx_t type1Bar32bitIdx;
  pcieStatusCmdReg_t     statusCmd;
  pcieDevStatCtrlReg_t   devStatCtrl;
  pcieAccrReg_t          accr;
                 
  pcieRegisters_t        setRegs;
  pcieRegisters_t        getRegs;

  memset (&cmdStatus,        0, sizeof(cmdStatus));
  memset (&obSize,           0, sizeof(obSize));
  memset (&gen2,             0, sizeof(gen2));
  memset (&type1Bar32bitIdx, 0, sizeof(type1Bar32bitIdx));
  memset (&statusCmd,        0, sizeof(statusCmd));
  memset (&devStatCtrl,      0, sizeof(devStatCtrl));
  memset (&accr,             0, sizeof(accr));

  /*Disable link training*/
  if ((retVal = pcieLtssmCtrl(handle, FALSE)) != pcie_RET_OK) 
  {
    System_printf ("Failed to disable Link Training!\n");
    return retVal;
  }
  
  /* Configure the size of the translation regions */   
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  
  obSize.size = pcie_OB_SIZE_8MB;
  setRegs.obSize = &obSize;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET OB_SIZE register failed!\n");
    return retVal;
  }
     
  /* Setting PL_GEN2 */   
  memset (&setRegs, 0, sizeof(setRegs));
  gen2.numFts = 0xF;
  gen2.dirSpd = 0x0;
  gen2.lnEn   = 1;
  setRegs.gen2 = &gen2;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET GEN2 register failed!\n");
    return retVal;
  }
  
  /* Configure BAR Masks */   
  /* First need to enable writing on BAR mask registers */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }
  
  /* Configure Masks*/
  memset (&setRegs, 0, sizeof(setRegs));
  type1Bar32bitIdx.reg.reg32 = PCIE_BAR_MASK;
  setRegs.type1Bar32bitIdx = &type1Bar32bitIdx;

  /* BAR 0 */
  type1Bar32bitIdx.idx = 0; /* configure BAR 0*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* BAR 1 */
  type1Bar32bitIdx.idx = 1; /* configure BAR 1*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* Disable writing on BAR Masks */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 0;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  /* Enable memory access and mastership of the bus */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.statusCmd = &statusCmd;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read Status Comand register failed!\n");
    return retVal;
  }
  statusCmd.memSp  = 1;
  statusCmd.busMs  = 1;
  statusCmd.resp   = 1;
  statusCmd.serrEn = 1;
  setRegs.statusCmd = &statusCmd;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Status Command register failed!\n");
    return retVal;
  }

  /* Enable Error Reporting */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.devStatCtrl = &devStatCtrl;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Regad Device Status Control register failed!\n");
    return retVal;
  }
 
  devStatCtrl.reqRp = 1;
  devStatCtrl.fatalErRp = 1;
  devStatCtrl.nFatalErRp = 1;
  devStatCtrl.corErRp = 1;
  setRegs.devStatCtrl = &devStatCtrl;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Device Status Control register failed!\n");
    return retVal;
  }

  /* Enable ECRC */
  memset (&setRegs, 0, sizeof(setRegs));
  
  accr.chkEn=1;
  accr.chkCap=1;
  accr.genEn=1;
  accr.genCap=1;
  setRegs.accr = &accr;

  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET ACCR register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Configure PCIe in End Point Mode
 ****************************************************************************/
pcieRet_e pcieCfgEP(Pcie_Handle handle)
{
  pcieRet_e retVal;

  pcieCmdStatusReg_t     cmdStatus;
  pcieObSizeReg_t        obSize;
  pcieGen2Reg_t          gen2;
  pcieType0Bar32bitIdx_t type0Bar32bitIdx;
  pcieStatusCmdReg_t     statusCmd;
  pcieDevStatCtrlReg_t   devStatCtrl;
  pcieAccrReg_t          accr;
                 
  pcieRegisters_t        setRegs;
  pcieRegisters_t        getRegs;

  memset (&cmdStatus,        0, sizeof(cmdStatus));
  memset (&obSize,           0, sizeof(obSize));
  memset (&gen2,             0, sizeof(gen2));
  memset (&type0Bar32bitIdx, 0, sizeof(type0Bar32bitIdx));
  memset (&statusCmd,        0, sizeof(statusCmd));
  memset (&devStatCtrl,      0, sizeof(devStatCtrl));
  memset (&accr,             0, sizeof(accr));

  /*Disable link training*/
  if ((retVal = pcieLtssmCtrl(handle, FALSE)) != pcie_RET_OK) 
  {
    System_printf ("Failed to disable Link Training!\n");
    return retVal;
  }
  
  /* Configure the size of the translation regions */   
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  
  obSize.size = pcie_OB_SIZE_8MB;
  setRegs.obSize = &obSize;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET OB_SIZE register failed!\n");
    return retVal;
  }
     
  /* Setting PL_GEN2 */   
  memset (&setRegs, 0, sizeof(setRegs));
  gen2.numFts = 0xF;
  gen2.dirSpd = 0x0;
  gen2.lnEn   = 1;
  setRegs.gen2 = &gen2;
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET GEN2 register failed!\n");
    return retVal;
  }
  
  /* Configure BAR Masks */   
  /* First need to enable writing on BAR mask registers */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }
  
  /* Configure Masks*/
  memset (&setRegs, 0, sizeof(setRegs));
  type0Bar32bitIdx.reg.reg32 = PCIE_BAR_MASK;
  setRegs.type0Bar32bitIdx = &type0Bar32bitIdx;

  /* BAR 0 */
  type0Bar32bitIdx.idx = 0; /* configure BAR 0*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* BAR 1 */
  type0Bar32bitIdx.idx = 1; /* configure BAR 1*/
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET BAR MASK register failed!\n");
    return retVal;
  }
  
  /* Disable writing on BAR Masks */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));
  memset (&cmdStatus,  0, sizeof(cmdStatus));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.dbi = 0;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  /* Enable memory access and mastership of the bus */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.statusCmd = &statusCmd;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read Status Comand register failed!\n");
    return retVal;
  }
  statusCmd.memSp  = 1;
  statusCmd.busMs  = 1;
  statusCmd.resp   = 1;
  statusCmd.serrEn = 1;
  setRegs.statusCmd = &statusCmd;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Status Command register failed!\n");
    return retVal;
  }

  /* Enable Error Reporting */
  memset (&setRegs, 0, sizeof(setRegs));
  memset (&getRegs, 0, sizeof(getRegs));

  getRegs.devStatCtrl = &devStatCtrl;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Regad Device Status Control register failed!\n");
    return retVal;
  }
 
  devStatCtrl.reqRp = 1;
  devStatCtrl.fatalErRp = 1;
  devStatCtrl.nFatalErRp = 1;
  devStatCtrl.corErRp = 1;
  setRegs.devStatCtrl = &devStatCtrl;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET Device Status Control register failed!\n");
    return retVal;
  }

  /* Enable ECRC */
  memset (&setRegs, 0, sizeof(setRegs));
  
  accr.chkEn=1;
  accr.chkCap=1;
  accr.genEn=1;
  accr.genCap=1;
  setRegs.accr = &accr;

  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET ACCR register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}

/*****************************************************************************
 * Function: Configure and enable Outbound Address Translation 
 ****************************************************************************/
pcieRet_e pcieObTransCfg(Pcie_Handle handle, uint32_t obAddrLo, uint32_t obAddrHi, uint8_t region)
{

  pcieRet_e retVal;

  pcieRegisters_t      setRegs;
  pcieRegisters_t      getRegs;

  pcieCmdStatusReg_t   cmdStatus;

  memset (&setRegs,   0, sizeof(setRegs));
  memset (&getRegs,   0, sizeof(getRegs));
  memset (&cmdStatus, 0, sizeof(cmdStatus));

  /* Set outbound offset registers */
  if ((retVal = Pcie_cfgObOffset(handle, obAddrLo, obAddrHi, region)) != pcie_RET_OK) 
  {
    System_printf ("Failed to configure ObOffset registers!\n");
    return retVal;
  }

  /*enable Outbound address translation*/
  memset (&setRegs,    0, sizeof(setRegs));
  memset (&getRegs,    0, sizeof(getRegs));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.obXltEn = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}


/*****************************************************************************
 * Function: Configure and enable Inbound Address Translation 
 ****************************************************************************/
pcieRet_e pcieIbTransCfg(Pcie_Handle handle, pcieIbTransCfg_t *ibCfg)
{
  pcieRet_e retVal;

  pcieRegisters_t      setRegs;
  pcieRegisters_t      getRegs;

  pcieCmdStatusReg_t   cmdStatus;

  memset (&setRegs,   0, sizeof(setRegs));
  memset (&getRegs,   0, sizeof(getRegs));
  memset (&cmdStatus, 0, sizeof(cmdStatus));

  /* Set outbound offset registers */
  if ((retVal = Pcie_cfgIbTrans(handle, ibCfg)) != pcie_RET_OK) 
  {
    System_printf ("Failed to configure Inbound Translation registers!\n");
    return retVal;
  }

  /*enable Inbound address translation*/
  memset (&setRegs,    0, sizeof(setRegs));
  memset (&getRegs,    0, sizeof(getRegs));

  getRegs.cmdStatus = &cmdStatus;
  if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK) 
  {
    System_printf ("Read CMD STATUS register failed!\n");
    return retVal;
  }
  cmdStatus.ibXltEn = 1;
  setRegs.cmdStatus = &cmdStatus;   
  
  if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK) 
  {
    System_printf ("SET CMD STATUS register failed!\n");
    return retVal;
  }

  return pcie_RET_OK;
}


/*****************************************************************************
 * Function: Initialize application buffers
 ****************************************************************************/
void pcieInitAppBuf(void)
{
  uint32_t i;
  
  for (i=0; i<PCIE_BUFSIZE_APP; i++)
  {
    dstBuf.buf[i] = 0;	
    srcBuf[i] = i;	
  }
  
  dstBuf.buf[PCIE_BUFSIZE_APP] = PCIE_EXAMPLE_BUF_EMPTY;
}

/*****************************************************************************
 * Function: Check LTSSM status and wait for the link to be up
 ****************************************************************************/
void pcieWaitLinkUp(Pcie_Handle handle)
{
  pcieRegisters_t  getRegs;
  pcieDebug0Reg_t  debug0;
  
  memset (&debug0,  0, sizeof(debug0));
  memset (&getRegs, 0, sizeof(getRegs));
  
  uint8_t ltssmState = 0;
 
  while(ltssmState != pcie_LTSSM_L0)
  {
    cycleDelay(100);
    getRegs.debug0 = &debug0;
    if (Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs) != pcie_RET_OK) 
    {
      System_printf ("Read LTSSM state failed!\n");
      return;
    }
    ltssmState = debug0.ltssmState;
  }
}



/*****************************************************************************
 * Function: Converts a core local L2 address to a global L2 address 
 *   Input addr:  L2 address to be converted to global.
 *   return:  uint32_t   Global L2 address
 *****************************************************************************/
uint32_t pcieConvert_CoreLocal2GlobalAddr (uint32_t  addr)
{
  uint32_t coreNum;

  /* Get the core number. */
  coreNum = CSL_chipReadReg(CSL_CHIP_DNUM); 
  
  /* Compute the global address. */
  return ((1 << 28) | (coreNum << 24) | (addr & 0x00ffffff));
}    

/*****************************************************************************
 * Hardware Interrupt Subroutines
 ****************************************************************************/
static void pcie_isr(void* handle) {


	/* Disable host interrupt */
	CSL_CPINTC_disableHostInterrupt (hnd, 40);

	if ((PMRST_IRQ_STATUS_RAW & 0x8) == 0x8)
		System_printf ("Link request reset interrupt triggered!\n");

	/* Initialize link training again */
	CMD_STATUS |= 0x1;		//set LTSSM_EN (bit 0)

	reg_read = DEBUG0;
	/* Wait for link to be up */
	while (reg_read != 0x11)
	{
		reg_read = DEBUG0 & 0x1F;
	}

	intTestDone ++;

	System_printf ("Link is up again.\n");


	/* clear PCIE interrupt */
	PMRST_IRQ_STATUS = 0x8;		//clear link request reset event flag bit
	IRQ_EOI = 0xD; 	//Power Management event is #13(0xD) for EOI (Table 2-10 in PCIe user guide)
    CSL_CPINTC_clearSysInterrupt (hnd, 49);

    /* Enable host interrupt */
    CSL_CPINTC_enableHostInterrupt (hnd, 40);
}

/*****************************************************************************
 * Enable PCIe Link Down Interrupt
 ****************************************************************************/
void pcieEnableInterrupts() {

	PMRST_ENABLE_SET = 0x8;	//Enable LINK_RST_REQ bit
}

/*****************************************************************************
 * Interrupt Controller Setup
 ****************************************************************************/
void intcSetup() {


    /* INTC Configuration */
	System_printf("Debug: CorePac-INTC Configuration...\n");

    /* INTC module initialization */
    intcContext.eventhandlerRecord = EventHandler;
    intcContext.numEvtEntries      = 10;
    if (CSL_intcInit(&intcContext) != CSL_SOK)
    {
    	System_printf("Error: CorePac-INTC initialization failed \n");
        return;
    }

    /* Enable NMIs */
    if (CSL_intcGlobalNmiEnable() != CSL_SOK)
    {
    	System_printf("Error: CorePac-INTC global NMI enable failed \n");
        return;
    }

    /* Enable global interrupts */
    if (CSL_intcGlobalEnable(&state) != CSL_SOK)
    {
    	System_printf("Error: CorePac-INTC global enable failed \n");
        return;
    }

    /* Open the INTC Module for Vector ID: 4 and Event ID: 56 (CIC0_OUT40 in C6657)
     * 	Refer to the interrupt architecture and mapping document for the Event ID  (CIC0_OUT40)*/
    vectId = CSL_INTC_VECTID_4;
    hTest = CSL_intcOpen (&intcObj, 56, &vectId , NULL);
    if (hTest == NULL)
    {
    	System_printf("Error: CorePac-INTC Open failed\n");
        return;
    }

    /* Register an call-back handler which is invoked when the event occurs. */
    EventRecord.handler = &pcie_isr;
    EventRecord.arg = 0;
    if (CSL_intcPlugEventHandler(hTest,&EventRecord) != CSL_SOK)
    {
    	System_printf("Error: CorePac-INTC Plug event handler failed\n");
        return;
    }

    /* Enabling the events. */
    if (CSL_intcHwControl(hTest,CSL_INTC_CMD_EVTENABLE, NULL) != CSL_SOK)
    {
    	System_printf("Error: CorePac-INTC CSL_INTC_CMD_EVTENABLE command failed\n");
        return;
    }

    System_printf("Debug: CorePac-INTC Configuration Completed \n");

    /* CPINTC-0 Configuration */

    System_printf("Debug: CIC-0 Configuration...\n");

    /* Open the handle to the CIC0 Instance */
    hnd = CSL_CPINTC_open(0);
    if (hnd == 0)
    {
    	System_printf("Error: Unable to open CIC0\n");
        return;
    }

    /* Disable all host interrupts. */
    CSL_CPINTC_disableAllHostInterrupt(hnd);

    /* Configure no nesting support in the CIC Module. */
    CSL_CPINTC_setNestingMode (hnd, CPINTC_NO_NESTING);

   	/* We now map System Interrupt 49 (PCIEXpress_PM_INT) to channel 40 (CIC0_OUT40) */
  	CSL_CPINTC_mapSystemIntrToChannel (hnd, 49, 40);

   	/* We now enable system interrupt 49 */
   	CSL_CPINTC_enableSysInterrupt (hnd, 49);

   	/* We enable host interrupts. */
   	CSL_CPINTC_enableHostInterrupt (hnd, 40);

   	/* Enable all host interrupts also. */
   	CSL_CPINTC_enableAllHostInterrupt(hnd);

   	System_printf("Debug: CPINTC-0 Configuration Completed\n");

}

/*****************************************************************************
 * Function: Main 
 ****************************************************************************/
void main (void)
{
  TSCL = 1;
  uint16_t lock=0;

  pcieRet_e        retVal;
  pcieIbTransCfg_t ibCfg;
  pcieBarCfg_t     barCfg;
  Pcie_Handle      handle = NULL;
  void             *pcieBase;
  uint32_t    i;

  System_printf ("**********************************************\n");
  System_printf ("*             PCIe Test Start                *\n");
  
  if(PcieModeGbl == pcie_RC_MODE)
    System_printf ("*                RC mode                     *\n");
  else
    System_printf ("*                EP mode                     *\n");
  
  System_printf ("**********************************************\n\n");
  
  System_printf ("Version #: 0x%08x; string %s\n\n", Pcie_getVersion(), Pcie_getVersionStr());
  
  /* Initialize application buffers */
  pcieInitAppBuf();

  /* Configure SERDES*/
  if ((retVal = pcieSerdesCfg()) != pcie_RET_OK) {
    System_printf ("PCIe Serdes config failed (%d)\n", (int)retVal);
    exit(1);
  }

  /* Set the PCIe mode*/
  if ((retVal = Pcie_setMode(PcieModeGbl)) != pcie_RET_OK) {
    System_printf ("Set PCIe Mode failed (%d)\n", (int)retVal);
    exit(1);
  }
  
  /* Power up PCIe Module */
  if ((retVal = pciePowerCfg()) != pcie_RET_OK) {
    System_printf ("PCIe Power Up failed (%d)\n", (int)retVal);
    exit(1);
  }

  System_printf ("PCIe Power Up.\n");
  
  /* Wait until the PCIe SERDES PLL locks */
  while (!lock)
  {
    CSL_BootCfgGetPCIEPLLLock(&lock);
  }                    

  if ((retVal = Pcie_open(0, &handle)) != pcie_RET_OK) 
  {
    System_printf ("Open failed (%d)\n", (int)retVal);
    exit(1);
  }

  System_printf ("PLL configured.\n");

  if(PcieModeGbl == pcie_RC_MODE)
  {
    /* Configure application registers for Root Complex*/
    if ((retVal = pcieCfgRC(handle)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure PCIe in RC mode (%d)\n", (int)retVal);
      exit(1);
    }

    /* Configure Address Translation */
    
    barCfg.location = pcie_LOCATION_LOCAL;
    barCfg.mode     = pcie_RC_MODE;
    barCfg.base     = PCIE_IB_LO_ADDR_M;
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR_IDX_M;
    
    if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure BAR (%d)\n", (int)retVal);
      exit(1);
    }

    ibCfg.ibBar         = PCIE_BAR_IDX_M; /* Match BAR that was configured above*/
    ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_M;
    ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_M;
    ibCfg.ibOffsetAddr  = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)dstBuf.buf);
    ibCfg.region        = PCIE_IB_REGION_M;       

    if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Inbound Translation (%d)\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Inbound Translation!\n");
    }

    if ((retVal = pcieObTransCfg (handle, PCIE_OB_LO_ADDR_M, PCIE_OB_HI_ADDR_M, PCIE_OB_REGION_M)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Outbound Address Translation (%d)\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Outbound Translation!\n");
    }
  }
  else
  {
    /* Configure application registers for End Point*/
    if ((retVal = pcieCfgEP(handle)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure PCIe in EP mode (%d)\n", (int)retVal);
      exit(1);
    }

    /* Configure Address Translation */
    
    barCfg.location = pcie_LOCATION_LOCAL;
    barCfg.mode     = pcie_EP_MODE;
    barCfg.base     = PCIE_IB_LO_ADDR_S;
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR_IDX_S;
    
    if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure BAR!\n");
      exit(1);
    }

    ibCfg.ibBar         = PCIE_BAR_IDX_S; /* Match BAR that was configured above*/
    ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_S;
    ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_S;
    ibCfg.ibOffsetAddr  = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)dstBuf.buf);
    ibCfg.region        = PCIE_IB_REGION_S;       

    if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Inbound Translation!\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Inbound Translation!\n");
    }

    if ((retVal = pcieObTransCfg (handle, PCIE_OB_LO_ADDR_S, PCIE_OB_HI_ADDR_S, PCIE_OB_REGION_S)) != pcie_RET_OK) 
    {
      System_printf ("Failed to configure Outbound Address Translation!\n", (int)retVal);
      exit(1);
    }
    else
    {
      System_printf ("Successfully configured Outbound Translation!\n");
    }
  }

  /* Enable PCIe interrupt */
  intcSetup();
  pcieEnableInterrupts();


  System_printf ("Starting link training...\n");

  /*Enable link training*/
  if ((retVal = pcieLtssmCtrl(handle, TRUE)) != pcie_RET_OK) 
  {
    System_printf ("Failed to Enable Link Training!\n", (int)retVal);
    exit(1);
  }
  
  /* Wait for link to be up */
  pcieWaitLinkUp(handle);

  System_printf ("Link is up.\n");


  if(PcieModeGbl == pcie_RC_MODE)
  {
    /**********************************************************************/
    /* Push a single message to the EP then verify that it is echoed back */
    /**********************************************************************/

    /* Write from RC to EP                                                */
    if ((retVal = Pcie_getMemSpaceRange (handle, &pcieBase, NULL)) != pcie_RET_OK) {
      System_printf ("getMemSpaceRange failed\n", (int)retVal);
      exit(1);
    }

    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
      *((volatile uint32_t *)pcieBase + i) = srcBuf[i];
    }
    
    /* Mark that the buffer is full, so EP can process it */
    *((volatile uint32_t *)pcieBase + PCIE_BUFSIZE_APP) = PCIE_EXAMPLE_BUF_FULL;

    /* Note on cache coherence: Write back is not necessary because pcieBase is in
       peripheral address space instead of physical memory*/
    
    /* Data sent to EP.
       RC waits for the loopback to be completed and
       receive data back from EP */

    do {
      unsigned int key;

      /* Disable Interrupts */
      key = _disable_interrupts();

      /*  Cleanup the prefetch buffer also. */
      CSL_XMC_invalidatePrefetchBuffer();    

      CACHE_invL1d ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);
      CACHE_invL2  ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

      /* Reenable Interrupts. */
      _restore_interrupts(key);

    } while(dstBuf.buf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);


    /* check all the data */
    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
      if(dstBuf.buf[i] != srcBuf[i])
      {
        System_printf ("Received data = %d\nTransmited data = %d\nIndex = %d.\n\nTest failed.\n", 
                        dstBuf.buf[i], srcBuf[i], i);
        exit(1);
      }
    }
    
    System_printf ("Root Complex received data.\nTest passed.\n");
  }
  else
  {
    /**********************************************************************/
    /* Wait for a single message from the RC then echo it back            */
    /**********************************************************************/

    /* EP waits for the data received from RC */
    do {
      unsigned int key;

      /* Disable Interrupts */
      key = _disable_interrupts();

      /*  Cleanup the prefetch buffer also. */
      CSL_XMC_invalidatePrefetchBuffer();    

      CACHE_invL1d ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);
      CACHE_invL2  ((void *)dstBuf.buf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

      /* Reenable Interrupts. */
      _restore_interrupts(key);

    } while(dstBuf.buf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);

    System_printf ("End Point received data.\n");

    /* Loopback to RC what was written in the DST buffer.
       Write from EP to RC */
    if ((retVal = Pcie_getMemSpaceRange (handle, &pcieBase, NULL)) != pcie_RET_OK) {
      System_printf ("getMemSpaceRange failed\n", (int)retVal);
      exit(1);
    }

    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
      *((volatile uint32_t *)pcieBase + i) = dstBuf.buf[i];
    }
    
    /* Mark that the buffer is full, so RC can process it */
    *((volatile uint32_t *)pcieBase + PCIE_BUFSIZE_APP) = PCIE_EXAMPLE_BUF_FULL;

    /* Note on cache coherence: Write back is not necessary because pcieBase is in
       peripheral address space instead of physical memory*/

    System_printf ("End Point sent data to Root Complex, completing the loopback.\nEnd of Test.\n");
  }

  /* Link down interrupt test */
  intTestNum = 3;	//repeat interrupt test for several times

  if(PcieModeGbl == pcie_RC_MODE)
  {
	  for (i=0; i<intTestNum; i++)
	  {
		  System_printf ("Link down interrupt test #%d...\n", i);

		  /* Disconnect link from RC */
		  CMD_STATUS &= ~0x1;		//clear LTSSM_EN (bit 0)

		  cycleDelay(500000);	//add some delay between each interrupt test
	  }
  }
  else //EP mode
  {
	  while (intTestDone != intTestNum);
  }

  System_printf ("End of Link down interrupt test.\n");

  exit(0);
}

The source code demonstrate the interrupt mapping. RC will disconnect the link and both RC and EP will retrain the link in the ISR to recover the link up. It could be repeated for several times to make sure the interrupt could be re-triggered.

Please find the console output for the example test below.

[C66xx_0] **********************************************
[C66xx_0] * PCIe Test Start *
[C66xx_0] * RC mode *
[C66xx_0] **********************************************
[C66xx_0]
[C66xx_0] Version #: 0x01000003; string PCIE LLD Revision: 01.00.00.03:Nov 5 2012:15:19:39
[C66xx_0]
[C66xx_0] PCIe Power Up.
[C66xx_0] PLL configured.
[C66xx_0] Successfully configured Inbound Translation!
[C66xx_0] Successfully configured Outbound Translation!
[C66xx_0] Debug: CorePac-INTC Configuration...
[C66xx_0] Debug: CorePac-INTC Configuration Completed
[C66xx_0] Debug: CIC-0 Configuration...
[C66xx_0] Debug: CPINTC-0 Configuration Completed
[C66xx_0] Starting link training...
[C66xx_0] Link is up.
[C66xx_0] Root Complex received data.
[C66xx_0] Test passed.
[C66xx_0] Link down interrupt test #0...
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link down interrupt test #1...
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link down interrupt test #2...
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] End of Link down interrupt test.

[C66xx_0] **********************************************
[C66xx_0] * PCIe Test Start *
[C66xx_0] * EP mode *
[C66xx_0] **********************************************
[C66xx_0]
[C66xx_0] Version #: 0x01000003; string PCIE LLD Revision: 01.00.00.03:Nov 5 2012:15:19:39
[C66xx_0]
[C66xx_0] PCIe Power Up.
[C66xx_0] PLL configured.
[C66xx_0] Successfully configured Inbound Translation!
[C66xx_0] Successfully configured Outbound Translation!
[C66xx_0] Debug: CorePac-INTC Configuration...
[C66xx_0] Debug: CorePac-INTC Configuration Completed
[C66xx_0] Debug: CIC-0 Configuration...
[C66xx_0] Debug: CPINTC-0 Configuration Completed
[C66xx_0] Starting link training...
[C66xx_0] Link is up.
[C66xx_0] End Point received data.
[C66xx_0] End Point sent data to Root Complex, completing the loopback.
[C66xx_0] End of Test.
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] End of Link down interrupt test.

0 Steven Ji over 12 years ago in reply to Steven Ji

TI__Genius 12065 points

Please add the CSL INTC library to the project linker configuration since the example is using the INTC CSL APIs:

Add "C:\ti\pdk_C6657_1_1_2_5\packages\ti\csl\lib\ti.csl.intc.ae66" into "Properties" -> "Build" -> "C6000 Linker" ->"File Search Path" -> "Include library file or command file as input"

0 Edward Ewer1 over 12 years ago in reply to Steven Ji

Prodigy 140 points

Thanks Steven for the quick turnaround. This is a big help to me to see an example of mapping a System Interrupt.

I have duplicated your results on my setup using the PCIe backplane.

[C66xx_0] **********************************************
[C66xx_0] *             PCIe Test Start                *
[C66xx_0] *                RC mode                     *
[C66xx_0] **********************************************
[C66xx_0] 
[C66xx_0] Version #: 0x01000003; string PCIE LLD Revision: 01.00.00.03:Nov  5 2012:15:19:39
[C66xx_0] 
[C66xx_0] Power domain is already enabled.  You probably re-ran without device reset (which is OK)
[C66xx_0] PCIe Power Up.
[C66xx_0] PLL configured.
[C66xx_0] Successfully configured Inbound Translation!
[C66xx_0] Successfully configured Outbound Translation!
[C66xx_0] Debug: CorePac-INTC Configuration...
[C66xx_0] Debug: CorePac-INTC Configuration Completed 
[C66xx_0] Debug: CIC-0 Configuration...
[C66xx_0] Debug: CPINTC-0 Configuration Completed
[C66xx_0] Starting link training...
[C66xx_0] Link is up.
[C66xx_0] Root Complex received data.
[C66xx_0] Test passed.
[C66xx_0] Link down interrupt test #0...
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link down interrupt test #1...
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link down interrupt test #2...
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] End of Link down interrupt test.

[C66xx_0] **********************************************
[C66xx_0] *             PCIe Test Start                *
[C66xx_0] *                EP mode                     *
[C66xx_0] **********************************************
[C66xx_0] 
[C66xx_0] Version #: 0x01000003; string PCIE LLD Revision: 01.00.00.03:Nov  5 2012:15:19:39
[C66xx_0] 
[C66xx_0] Power domain is already enabled.  You probably re-ran without device reset (which is OK)
[C66xx_0] PCIe Power Up.
[C66xx_0] PLL configured.
[C66xx_0] Successfully configured Inbound Translation!
[C66xx_0] Successfully configured Outbound Translation!
[C66xx_0] Debug: CorePac-INTC Configuration...
[C66xx_0] Debug: CorePac-INTC Configuration Completed 
[C66xx_0] Debug: CIC-0 Configuration...
[C66xx_0] Debug: CPINTC-0 Configuration Completed
[C66xx_0] Starting link training...
[C66xx_0] Link is up.
[C66xx_0] End Point received data.
[C66xx_0] End Point sent data to Root Complex, completing the loopback.
[C66xx_0] End of Test.
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] Link request reset interrupt triggered!
[C66xx_0] Link is up again.
[C66xx_0] End of Link down interrupt test.

Thanks again,

Edward

0 murad qahwash over 11 years ago in reply to Edward Ewer1

Genius 3770 points

Dear all,

I am about to buy the above backplane from "one stop systems" since I am having similar experiment (C6678 to Spartan 6 EVM) and would like to know if you where able to successfully get the PCIe communication running!

Regards,

Murad

0 Tom Johnson 16214 over 11 years ago in reply to murad qahwash

TI__Mastermind 46460 points

Murad,

TI produces a board called the CI2EVMBOC that allows two C6678 EVMs to be connected at the AMC connector. This provides a complete PCIe to PCIe cross connect. This board is available at https://estore.ti.com/CI2EVMBOC--P2685.aspx and more information is available at http://processors.wiki.ti.com/index.php/CI_Dual_EVM_Break_Out_Card. The examples provided within the MCSDK for PCIe to PCIe operation were verified with this card.

Tom

0 murad qahwash over 11 years ago in reply to Tom Johnson 16214

Genius 3770 points

Thanks Tom,

We actually looked at this board before but the Xilinx FPGA EVK (SP605) has a pcie x1 connector. We already have the AMC to PCIe adaptor from TI (TMDXEVMPCI) and we need a way to connect the two boards together using PCIe x1

Regards,

Murad

0 Rediet Million over 10 years ago in reply to Steven Ji

Prodigy 150 points

Hello Stive

i am working on EVM6657 and windows 7 PC as a host. i am trying to use the PCI interface to transfer large amount of data for DSP. when i connect th TMDXEVMPCI to the PC i just only get "Unknown devices" in device manger . i am new to this PCI thing and i really need your help. my Questions are:

1. how would i know that the Windows PC can enumerate the 6657 device?

2. how can i develop the PCIE driver?

best regard,

Rediet

0 Tom Johnson 16214 over 10 years ago in reply to Rediet Million

TI__Mastermind 46460 points

Rediet,

There are multiple threads that discuss this type of implementation and related debug. Please read through this thread from beginning to end (be sure to click on the little (1) below to see the beginning of the thread. Also research other related threads like this one: http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/160607.aspx.

Tom

Processors

Processors forum

PCIe to PCIe using two C6657 EVMs