This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5706: PCIE Endpoint runtime failure.

Part Number: AM5706
Other Parts Discussed in Thread: SYSBIOS

Tool/software:

Hi 

We are facing run time failure in PCIe link .

Processor :- AM5706

SDK Version :- processor_sdk_rtos_am57xx_08_01_00_09

CCS Version :- Version: 9.3.0.00012 

We have developed an application using the driver provided by TI which is configured as PCIe end point with gen1 speed .

One Intel E3930 processor is used as a Root Complex. But the problem is , run time the PCIE link is getting down and the communication is stopping randomly. While testing we found out the clock from Root Complex is proper even the link is down.

I have attached the register details and source code along with the configuration file for reference , if any other information is required please let me know and I will provide.

/* ================ General configuration ================ */
var enableStaticIP         = 1;


var Edma = xdc.loadPackage ("ti.sdo.edma3.drv.sample");
var drv = xdc.loadPackage ("ti.sdo.edma3.drv");
var rm = xdc.loadPackage ("ti.sdo.edma3.rm");
var Defaults = xdc.useModule('xdc.runtime.Defaults');
var Diags = xdc.useModule('xdc.runtime.Diags');
var Error = xdc.useModule('xdc.runtime.Error');
var Log = xdc.useModule('xdc.runtime.Log');
var LoggerBuf = xdc.useModule('xdc.runtime.LoggerBuf');
var Main = xdc.useModule('xdc.runtime.Main');
var System = xdc.useModule('xdc.runtime.System');
var Text = xdc.useModule('xdc.runtime.Text');
var IntXbar      = xdc.useModule('ti.sysbios.family.shared.vayu.IntXbar');
var Clock = xdc.useModule('ti.sysbios.knl.Clock');
var Timer = xdc.useModule('ti.sysbios.timers.dmtimer.Timer');
var Swi = xdc.useModule('ti.sysbios.knl.Swi');
var Task = xdc.useModule('ti.sysbios.knl.Task');
var Semaphore = xdc.useModule('ti.sysbios.knl.Semaphore');


var BIOS = xdc.useModule('ti.sysbios.BIOS');
var Memory = xdc.useModule('xdc.runtime.Memory')
var SysMin = xdc.useModule('xdc.runtime.SysMin');
var Hwi = xdc.useModule('ti.sysbios.hal.Hwi');
var InitXbar    = xdc.useModule("ti.sysbios.family.shared.vayu.IntXbar");
var HeapMem = xdc.useModule('ti.sysbios.heaps.HeapMem');
var Task = xdc.useModule('ti.sysbios.knl.Task');
var ti_sysbios_hal_Timer = xdc.useModule('ti.sysbios.hal.Timer');
var ti_sysbios_hal_Hwi = xdc.useModule('ti.sysbios.hal.Hwi');
var Cache = xdc.useModule('ti.sysbios.family.arm.a15.Cache');
var SemiHostSupport = xdc.useModule('ti.sysbios.rts.gnu.SemiHostSupport'); 

/* NDK modules */
var Global 			= xdc.useModule('ti.ndk.config.Global');
var Ip 				= xdc.useModule('ti.ndk.config.Ip');
var Tcp             = xdc.useModule('ti.ndk.config.Tcp');
var Udp             = xdc.useModule('ti.ndk.config.Udp');
var Telnet          = xdc.useModule('ti.ndk.config.Telnet');

Tcp.transmitBufSize = 16384;
Tcp.receiveBufSize = 65536;
Tcp.receiveBufLimit = 65536;
Global.pktNumFrameBufs=384;

/*
 * Uncomment this line to globally disable Asserts.
 * All modules inherit the default from the 'Defaults' module.  You
 * can override these defaults on a per-module basis using Module.common$. 
 * Disabling Asserts will save code space and improve runtime performance.
Defaults.common$.diags_ASSERT = Diags.ALWAYS_OFF;
 */
/* 
 * Program.argSize sets the size of the .args section. 
 * The examples don't use command line args so argSize is set to 0.
 */
/*Program.argSize = 0x0;*/

/* System stack size (used by ISRs and Swis) */
Program.stack = 0x20000;

/*
 * Uncomment this line to keep module names from being loaded on the target.
 * The module name strings are placed in the .const section. Setting this
 * parameter to false will save space in the .const section.  Error and
 * Assert messages will contain an "unknown module" prefix instead
 * of the actual module name.
Defaults.common$.namedModule = false;
 */

/*
 * Minimize exit handler array in System.  The System module includes
 * an array of functions that are registered with System_atexit() to be
 * called by System_exit().
 */
System.maxAtexitHandlers = 4;       

/* 
 * Uncomment this line to disable the Error print function.  
 * We lose error information when this is disabled since the errors are
 * not printed.  Disabling the raiseHook will save some code space if
 * your app is not using System_printf() since the Error_print() function
 * calls System_printf().
Error.raiseHook = null;
 */

/* 
 * Uncomment this line to keep Error, Assert, and Log strings from being
 * loaded on the target.  These strings are placed in the .const section.
 * Setting this parameter to false will save space in the .const section.
 * Error, Assert and Log message will print raw ids and args instead of
 * a formatted message.
Text.isLoaded = false;
 */

/*
 * Uncomment this line to disable the output of characters by SysMin
 * when the program exits.  SysMin writes characters to a circular buffer.
 * This buffer can be viewed using the SysMin Output view in ROV.
SysMin.flushAtExit = false;
 */

 System.SupportProxy = SysMin;
/* Circular buffer size for System_printf() */
SysMin.bufSize = 0x200;


/* ================ BIOS configuration ================ */

var BIOS = xdc.useModule('ti.sysbios.BIOS');
/*
 * Build a custom SYS/BIOS library from sources.
 */
BIOS.libType = BIOS.LibType_Custom;



/*
 * The BIOS module will create the default heap for the system.
 * Specify the size of this default heap.
 */
BIOS.heapSize = 0xA0000;
  

/* 
 * Create and install logger for the whole system
 */
var loggerBufParams = new LoggerBuf.Params();
loggerBufParams.numEntries = 16;
var logger0 = LoggerBuf.create(loggerBufParams);
Defaults.common$.logger = logger0;
Main.common$.diags_INFO = Diags.ALWAYS_ON;



if (enableStaticIP)
{
    /* Settings for static IP configuration */
	Ip.ResolveIP = false;
	Ip.CallByIP = false;
	Ip.autoIp = false;
    Ip.address = "172.16.3.47";
    Ip.mask = "255.255.0.0";
    Ip.gatewayIpAddr = "172.16.0.0";
}
else
{
    Ip.dhcpClientMode = Ip.CIS_FLG_IFIDXVALID;
}

Global.ndkTickPeriod = 100;
Global.kernTaskPriLevel = 11;
Global.serviceReportHook = null;
Global.IPv6 = false;

Task.defaultStackSize = 4096;
Task.idleTaskStackSize = 4096;


/* ================ Driver configuration ================ */

var deviceType           = "am571x";
var socType           = "am571x";

var Csl 		= xdc.loadPackage('ti.csl');
Csl.Settings.deviceType  = deviceType;


/* Load the OSAL package */
var osType = "tirtos"
var Osal = xdc.useModule('ti.osal.Settings');
Osal.osType = osType;

/* Load the Board package and set the board name */
var Board = xdc.loadPackage('ti.board');
Board.Settings.boardName = "idkAM571x";

/* Load Profiling package */
var Utils = xdc.loadPackage('ti.utils.profiling');

/* Load and use the PCIE packages */

var Pcie = xdc.loadPackage('ti.drv.pcie');
Pcie.Settings.enableProfiling = true;
/* Enable only if soc-specific library should be used */ 
/* Pcie.Settings.socType = socType; */ /* use soc/am571x/src/pcie_soc.c */


/* Load the EMAC packages */
var Emac = xdc.loadPackage('ti.drv.emac');
Emac.Settings.socType = socType;
/* Load the UART package */
var UART = xdc.loadPackage('ti.drv.uart');

/* Load the I2C package */
var I2C = xdc.loadPackage('ti.drv.i2c');

/* Load the spi package */
var Spi              = xdc.loadPackage('ti.drv.spi');
Spi.Settings.socType = socType;

/* Load the LLD package */
var Gpio = xdc.loadPackage('ti.drv.gpio');


/* ================ Task configuration ================ */

var task0Params = new Task.Params();
task0Params.instance.name = "POST_and_GMC";
task0Params.stackSize = 0xA0000;
//task0Params.priority = 5; //5
Program.global.echo = Task.create("&RSIDM_LRU_App_All_Test", task0Params);

var task2Params = new Task.Params();
task2Params.instance.name = "CBIT";
task2Params.stackSize = 0xA0000;
//task2Params.priority = 5; //2
//Program.global.task2 = Task.create("&RSIDM_LRU_CBIT", task2Params);

var task1Params = new Task.Params();
task1Params.instance.name = "EmacStats";
//task1Params.priority = 5; //7
task1Params.stackSize = 0xA0000;
//Program.global.task1 = Task.create("&app_stats", task1Params);




/*
 * Create the stack Thread Task for our application.
 */
/*
Program.global.Ndktask = Task.create("&RSIDM_LRU_Gmc_App");
Program.global.Ndktask.stackSize = 0xA0000;
Program.global.Ndktask.priority = 7;
*/

var socType     = "am571x";
var Nimu 		= xdc.loadPackage('ti.transport.ndk.nimu');
Nimu.Settings.socType  = socType;

var Ndk     = xdc.loadPackage('ti.ndk.config');
var Global  = xdc.useModule('ti.ndk.config.Global');
/* ================ Cache and MMU configuration ================ */

var Cache1  = xdc.useModule('ti.sysbios.family.arm.a15.Cache');
Cache1.enableCache = true;

/***********************************************
 *              MMU Configuration              *
 ***********************************************/
var Mmu = xdc.useModule('ti.sysbios.family.arm.a15.Mmu');
Mmu.enableMMU = true;

/******************************************************************************
 *
 * SYS/BIOS assigns the following defaults to MAIR0 ATTR0, ATTR1 and ATTR2:
 *
 * ATTR0 -> 0x44 (mark memory region as non-cacheable normal memory)
 * ATTR1 -> 0x04 (mark memory region as device memory, i.e. strongly
 * ordered and non-cacheable)
 * ATTR2 -> 0xFF (mark memory region as normal memory, RW cacheable and
 * RW allocate)
 ******************************************************************************
 */


// descriptor attribute structure
var attrs0 = new Mmu.DescriptorAttrs();
Mmu.initDescAttrsMeta(attrs0);
attrs0.type = Mmu.DescriptorType_BLOCK;    // BLOCK descriptor
attrs0.shareable = 2;                      // sharerable
attrs0.attrIndx = 1;                       // Non-cache, device memory

// Set the descriptor for each entry in the address range
for (var i=0x20000000; i < 0x60000000; i = i + 0x00200000) {
    // Each 'BLOCK' descriptor entry spans a 2MB address range
    Mmu.setSecondLevelDescMeta(i, i, attrs0);
}

// descriptor attribute structure
var attrs1 = new Mmu.DescriptorAttrs();
	
Mmu.initDescAttrsMeta(attrs1);
attrs1.type = Mmu.DescriptorType_BLOCK;    // BLOCK descriptor
attrs1.shareable = 2;                      // sharerable
attrs1.attrIndx = 2;                       // Cached, normal memory

// Set the descriptor for each entry in the address range
for (var i=0x80000000; i < 0xA0000000; i = i + 0x00200000) {
    // Each 'BLOCK' descriptor entry spans a 2MB address range
    Mmu.setSecondLevelDescMeta(i, i, attrs1);
}
  
          
/* ================ Memory sections configuration ================ */
Program.sectMap["BOARD_IO_DELAY_DATA"] = "OCMC_RAM1";
Program.sectMap["BOARD_IO_DELAY_CODE"] = "OCMC_RAM1";


/*
var ti_sysbios_hal_Timer0Params = new ti_sysbios_hal_Timer.Params();
ti_sysbios_hal_Timer0Params.instance.name = "ti_sysbios_hal_Timer0";
ti_sysbios_hal_Timer0Params.period = 10000;
ti_sysbios_hal_Timer0Params.startMode = xdc.module("ti.sysbios.interfaces.ITimer").StartMode_USER;
ti_sysbios_hal_Timer0Params.runMode = xdc.module("ti.sysbios.interfaces.ITimer").RunMode_ONESHOT;
Program.global.ti_sysbios_hal_Timer0 = ti_sysbios_hal_Timer.create(-1, "&app_stats", ti_sysbios_hal_Timer0Params);
*/

Cache.enableCache = true;



//var task0Params0 = new Task.Params();
//task0Params0.instance.name = "task0";
//task0Params.priority = 1;
//Program.global.task0 = Task.create("&RSIDM_LRU_UART_CMD_CBIT", task0Params0);
DEFAULT After Configuration
===========================
0x51000718 --> 0x4000
0x5100071c --> 0x280
0x51000720 --> 0x0
0x51000724 --> 0x1
0x51000728 --> 0x3843811
0x5100072c --> 0x28000010
0x51000730 --> 0xc050
0x51000734 --> 0xc001
0x51000738 --> 0xfffff
0x5100073c --> 0x0
0x51000740 --> 0xf
0x51000744 --> 0x0
0x51000748 --> 0x4021502d
0x5100074c --> 0x215005
0x51000750 --> 0x800000
0x5100080c --> 0x20f
0x51000810 --> 0x0
0x51000814 --> 0x0
0x51000820 --> 0x0
0x51000824 --> 0x0
0x51000888 --> 0x0
0x510008b8 --> 0x0
0x510008bc --> 0x0
0x510008d0 --> 0x0
0x510008d4 --> 0x32
0x51000900 --> 0x1
0x51000904 --> 0x0
0x51000908 --> 0x80000000
0x5100090c --> 0x1000000
0x51000910 --> 0x0
0x51000914 --> 0x1ffffff
0x51000918 --> 0x90000000
0x5100091c --> 0x0
0x51000920 --> 0x0
0x51001000 --> 0x8888104c
0x51001004 --> 0x20100146
0x51001008 --> 0x1
0x5100100c --> 0x0
0x51001010 --> 0x8
0x51001014 --> 0x70000008
0x51001018 --> 0x8
0x5100101c --> 0x8
0x51001020 --> 0x8
0x51001024 --> 0x8
0x51001028 --> 0x0
0x5100102c --> 0x10000
0x51001030 --> 0x0
0x51001034 --> 0x40
0x5100103c --> 0x1ff
0x51001040 --> 0x5a035001
0x51001044 --> 0x0
0x51001070 --> 0x20010
0x51001074 --> 0x5908701
0x51001078 --> 0x1281f
0x5100107c --> 0x433c21
0x51001080 --> 0x10110000
0x51001094 --> 0x1f
0x51001098 --> 0x0
0x5100109c --> 0x2
0x510010a0 --> 0x10002
0x51002010 --> 0x28
0x51002018 --> 0x0
0x51002020 --> 0x7800
0x51002024 --> 0x0
0x51002028 --> 0x0
0x5100202c --> 0x0
0x51002030 --> 0x0
0x51002034 --> 0x0
0x51002038 --> 0x0
0x5100203c --> 0x0
0x51002100 --> 0x0
0x51002104 --> 0x600041
0x51002108 --> 0x0
0x5100210c --> 0x10000
0x51002124 --> 0x0
0x51002128 --> 0x0
0x5100212c --> 0x0
0x51002140 --> 0x0
0x51002144 --> 0x0
0x51002148 --> 0x0
0x51000700 --> 0x1630076
0x51000704 --> 0xffffffff
0x51000708 --> 0x7000004
0x5100070c --> 0x1b0f0f00
0x51000710 --> 0x30120
0x51000714 --> 0x0
0x51000000 --> 0x8888104c
0x51000004 --> 0x20100146
0x51000008 --> 0x1
0x5100000c --> 0x0
0x51000010 --> 0x8
0x51000014 --> 0x70000008
0x51000018 --> 0x8
0x5100001c --> 0x8
0x51000020 --> 0x8
0x51000024 --> 0x8
0x51000028 --> 0x0
0x5100002c --> 0x10000
0x51000030 --> 0x0
0x51000034 --> 0x40
0x5100003c --> 0x1ff
0x51000040 --> 0x5a035001
0x51000044 --> 0x0
0x51000050 --> 0x807005
0x51000054 --> 0x0
0x51000058 --> 0x0
0x5100005c --> 0x0
0x51000070 --> 0x20010
0x51000074 --> 0x5908701
0x51000078 --> 0x1281f
0x5100007c --> 0x433c21
0x51000080 --> 0x10110000
0x51000094 --> 0x1f
0x51000098 --> 0x0
0x5100009c --> 0x2
0x510000a0 --> 0x10002
0x51002000 --> 0x500a7200



After Link Down
===========================
0x51000718 --> 0x4000
0x5100071c --> 0x280
0x51000720 --> 0x0
0x51000724 --> 0x1
0x51000728 --> 0x57e600
0x5100072c --> 0x8200000
0x51000730 --> 0x0
0x51000734 --> 0x0
0x51000738 --> 0x0
0x5100073c --> 0x0
0x51000740 --> 0xf
0x51000744 --> 0x0
0x51000748 --> 0x4021502d
0x5100074c --> 0x215005
0x51000750 --> 0x800000
0x5100080c --> 0x2020f
0x51000810 --> 0x0
0x51000814 --> 0x0
0x51000820 --> 0x0
0x51000824 --> 0x0
0x51000888 --> 0x0
0x510008b8 --> 0x0
0x510008bc --> 0x1
0x510008d0 --> 0x0
0x510008d4 --> 0x32
0x51000900 --> 0x0
0x51000904 --> 0x0
0x51000908 --> 0x0
0x5100090c --> 0x0
0x51000910 --> 0x0
0x51000914 --> 0xfff
0x51000918 --> 0x0
0x5100091c --> 0x0
0x51000920 --> 0x0
0x51001000 --> 0x8888104c
0x51001004 --> 0x100000
0x51001008 --> 0x1
0x5100100c --> 0x0
0x51001010 --> 0x8
0x51001014 --> 0x8
0x51001018 --> 0x8
0x5100101c --> 0x8
0x51001020 --> 0x8
0x51001024 --> 0x8
0x51001028 --> 0x0
0x5100102c --> 0x10000
0x51001030 --> 0x0
0x51001034 --> 0x40
0x5100103c --> 0x1ff
0x51001040 --> 0x5a035001
0x51001044 --> 0x0
0x51001070 --> 0x20010
0x51001074 --> 0x8701
0x51001078 --> 0x2810
0x5100107c --> 0x433c22
0x51001080 --> 0x10110000
0x51001094 --> 0x1f
0x51001098 --> 0x0
0x5100109c --> 0x6
0x510010a0 --> 0x10002
0x51002010 --> 0x28
0x51002018 --> 0x0
0x51002020 --> 0x7800
0x51002024 --> 0x0
0x51002028 --> 0x0
0x5100202c --> 0x0
0x51002030 --> 0x0
0x51002034 --> 0x0
0x51002038 --> 0x0
0x5100203c --> 0x0
0x51002100 --> 0x0
0x51002104 --> 0x600000
0x51002108 --> 0x0
0x5100210c --> 0x0
0x51002124 --> 0x0
0x51002128 --> 0x0
0x5100212c --> 0x0
0x51002140 --> 0x0
0x51002144 --> 0x0
0x51002148 --> 0x0
0x51000700 --> 0xc00040
0x51000704 --> 0xffffffff
0x51000708 --> 0x7000004
0x5100070c --> 0x1b0f0f00
0x51000710 --> 0x30120
0x51000714 --> 0x0
0x51000000 --> 0x8888104c
0x51000004 --> 0x100000
0x51000008 --> 0x1
0x5100000c --> 0x0
0x51000010 --> 0x8
0x51000014 --> 0x8
0x51000018 --> 0x8
0x5100001c --> 0x8
0x51000020 --> 0x8
0x51000024 --> 0x8
0x51000028 --> 0x0
0x5100002c --> 0x10000
0x51000030 --> 0x0
0x51000034 --> 0x40
0x5100003c --> 0x1ff
0x51000040 --> 0x5a035001
0x51000044 --> 0x0
0x51000050 --> 0x807005
0x51000054 --> 0x0
0x51000058 --> 0x0
0x5100005c --> 0x0
0x51000070 --> 0x20010
0x51000074 --> 0x8701
0x51000078 --> 0x2810
0x5100007c --> 0x433c22
0x51000080 --> 0x10110000
0x51000094 --> 0x1f
0x51000098 --> 0x0
0x5100009c --> 0x6
0x510000a0 --> 0x10002
0x51002000 --> 0x500a7200

void Pcie_Init(void)
{

    for(INT32 i=0;i<80;i++)
        for(INT32 j=0;j<99999999;j++);

    /* Add MMU entries for MMR's required for PCIE example */
    Mmu_DescriptorAttrs attrs;
    extern char ti_sysbios_family_arm_a15_Mmu_Module_State_0_secondLevelTableBuf_0__A;
    extern char ti_sysbios_family_arm_a15_Mmu_Module_State_0_secondLevelTableBuf_1__A;
    UINT32 addr0 = (UINT32)&ti_sysbios_family_arm_a15_Mmu_Module_State_0_secondLevelTableBuf_0__A;
    UINT32 addr1 = (UINT32)&ti_sysbios_family_arm_a15_Mmu_Module_State_0_secondLevelTableBuf_1__A;

    Mmu_initDescAttrs(&attrs);

    attrs.type = Mmu_DescriptorType_TABLE;
    attrs.shareable = 0;            /* non-shareable */
    attrs.accPerm = 1;              /* read/write at any privelege level */
    attrs.attrIndx = 0;             /* Use MAIR0 Register Byte 3 for */
    /* determining the memory attributes */
    /* for each MMU entry */


    /* Update the first level table's MMU entry for 0x00000000 with the */
    /* new attributes. */
    Mmu_setFirstLevelDesc((Ptr)0x00000000, (UInt64)addr0, &attrs);
    /* Update the first level table's MMU entry for 0x40000000 with the */
    /* new attributes. */
    Mmu_setFirstLevelDesc((Ptr)0x40000000, (UInt64)addr1, &attrs);


    INT32          deviceNum = 0;
    pcieRet_e        retVal;
    pcieIbTransCfg_t ibCfg;
    pcieBarCfg_t     barCfg;
    Pcie_Handle      handle = NULL;
    void            *pcieBase;
    dstBuf_t        *pciedstBufBase;
    UINT32    i;
    //char             pcieModeResponse;

#if RC3_30_LRU
    INT32          deviceNum_1 = 1;
    Pcie_Handle      handle_1 = NULL;
    void            *pcieBase_1;
#endif


#if defined(PCIE_REV1_HW) || (defined(PCIE_REV2_HW) && defined(BUILD_MPU)) || (defined(PCIE_REV3_HW) && defined(BUILD_MPU))
    SemaphoreP_Handle sem = NULL;
#endif

    PCIE_logPrintf("\n\n");
    PCIE_logPrintf("------------------------------------------------\n");
    PCIE_logPrintf("-----------------PCIe Test----------------------\n");
    PCIE_logPrintf("------------------------------------------------\n");

    /* Get remote buffer out of cache */
    cache_writeback ((void *)&dstBuf, sizeof(dstBuf));
    /* Unlock kicker once, and don't relock, because its not multicore safe */

#ifdef EDMA
    EDMA3_DRV_Handle hEdma = NULL;
    //  hEdma = edmaInit(hEdma);
    //  if (hEdma==NULL) PCIE_logPrintf("ERROR: EDMA handle not initialized!\n");
#endif


#ifndef IO_CONSOLE
    Console_printf ("IO_CONSOLE not defined.  Most output will go to UART\n");
#endif

    PCIE_logPrintf("\nPCIe Started in EP Mode\n");

    /* Pass device config to LLD */
    if ((retVal = Pcie_init (&pcieInitCfg)) != pcie_RET_OK)
    {
        PCIE_logPrintf ("LLD device configuration failed\n");
        exit(1);
    }

    /* Initialize application buffers */
    pcieInitAppBuf();

    /* Power up PCIe Module */
    if ((retVal = pciePowerCfg()) != pcie_RET_OK) {
        PCIE_logPrintf ("PCIe Power Up failed (%d)\n", (INT32)retVal);
        exit(1);
    }

    PCIE_logPrintf ("PCIe Power Up.\n");

    if ((retVal = Pcie_open(deviceNum, &handle)) != pcie_RET_OK)
    {
        PCIE_logPrintf ("Open failed (%d)\n", (INT32)retVal);
        exit(1);
    }

    pcie_handle = handle;


#if RC3_30_LRU
    if ((retVal = Pcie_open(deviceNum_1, &handle_1)) != pcie_RET_OK)
    {
        PCIE_logPrintf ("Open failed (%d)\n", (INT32)retVal);
        exit(1);
    }

    pcie_handle1 = handle_1;

#endif

    /* Configure SERDES*/
    if ((retVal = pcieSerdesCfg()) != pcie_RET_OK) {
        PCIE_logPrintf ("PCIe Serdes config failed (%d)\n", (INT32)retVal);
        exit(1);
    }

    /* Set the PCIe mode*/
    if ((retVal = Pcie_setInterfaceMode(handle, PcieModeGbl)) != pcie_RET_OK) {
        PCIE_logPrintf ("Set PCIe Mode failed (%d)\n", (INT32)retVal);
        exit(1);
    }

#if RC3_30_LRU
    /* Set the PCIe mode*/
    if ((retVal = Pcie_setInterfaceMode(handle_1, PcieModeGbl)) != pcie_RET_OK) {
        PCIE_logPrintf ("Set PCIe Mode failed (%d)\n", (INT32)retVal);
        exit(1);
    }
#endif

    if(PcieModeGbl == pcie_RC_MODE)
    {
        /* Configure application registers for Root Complex*/
        if ((retVal = pcieCfgRC(handle)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure PCIe in RC mode (%d)\n", (INT32)retVal);
            exit(1);
        }

#if !defined(PCIE_REV2_HW) /* RC does not support BAR */
        /* Configure Address Translation */

        barCfg.location = pcie_LOCATION_LOCAL;
        barCfg.mode     = pcie_RC_MODE;
        barCfg.base     = PCIE_IB_LO_ADDR_RC;
        barCfg.prefetch = pcie_BAR_NON_PREF;
        barCfg.type     = pcie_BAR_TYPE32;
        barCfg.memSpace = pcie_BAR_MEM_MEM;
        barCfg.idx      = PCIE_BAR_IDX_RC;

        if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure BAR (%d)\n", (INT32)retVal);
            exit(1);
        }
#endif

        ibCfg.ibBar         = PCIE_BAR_IDX_RC; /* Match BAR that was configured above*/
        ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_RC;
        ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_RC;
        ibCfg.ibOffsetAddr  = (UINT32)pcieConvert_CoreLocal2GlobalAddr ((uintptr_t)dstBuf.buf);
        ibCfg.region        = PCIE_IB_REGION_RC;

        if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure Inbound Translation (%d)\n", (INT32)retVal);
            exit(1);
        }
        else
        {
            PCIE_logPrintf ("Successfully configured Inbound Translation!\n");
        }

        if ((retVal = pcieObTransCfg (handle, PCIE_OB_LO_ADDR_RC, PCIE_OB_HI_ADDR_RC, PCIE_OB_REGION_RC)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure Outbound Address Translation (%d)\n", (INT32)retVal);
            exit(1);
        }
        else
        {
            PCIE_logPrintf ("Successfully configured Outbound Translation!\n");
        }
    }
    else
    {

        /* Configure application registers for End Point*/
        if ((retVal = pcieCfgEP(handle)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure PCIe in EP mode (%d)\n", (INT32)retVal);
            exit(1);
        }

#if RC3_30_LRU
        /* Configure application registers for End Point*/
        if ((retVal = pcieCfgEP(handle_1)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure PCIe in EP mode (%d)\n", (INT32)retVal);
            exit(1);
        }
#endif

        /* Configure Address Translation */

        barCfg.location = pcie_LOCATION_LOCAL;
        barCfg.mode     = pcie_EP_MODE;
        barCfg.base     = PCIE_IB_LO_ADDR_EP;
        barCfg.prefetch = pcie_BAR_NON_PREF;
        barCfg.type     = pcie_BAR_TYPE32;
        barCfg.memSpace = pcie_BAR_MEM_MEM;
        barCfg.idx      = PCIE_BAR_IDX_EP;

        if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure BAR!\n");
            exit(1);
        }
#if RC3_30_LRU
        if ((retVal = Pcie_cfgBar(handle_1, &barCfg)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure BAR!\n");
            exit(1);
        }
#endif

        ibCfg.ibBar         = PCIE_BAR_IDX_EP; /* Match BAR that was configured above*/
        ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_EP;
        ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_EP;
        ibCfg.ibOffsetAddr  = (UINT32)pcieConvert_CoreLocal2GlobalAddr ((uintptr_t)dstBuf.buf);
        ibCfg.region        = PCIE_IB_REGION_EP;

        if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure Inbound Translation (%d)!\n", (INT32)retVal);
            exit(1);
        }
        else
        {
            //      PCIE_logPrintf ("Successfully configured Inbound Translation!\n");
        }
#if RC3_30_LRU
        if ((retVal = pcieIbTransCfg(handle_1, &ibCfg)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure Inbound Translation (%d)!\n", (INT32)retVal);
            exit(1);
        }
        else
        {
            //      PCIE_logPrintf ("Successfully configured Inbound Translation!\n");
        }
#endif


        if ((retVal = pcieObTransCfg (handle, PCIE_OB_LO_ADDR_EP, PCIE_OB_HI_ADDR_EP, PCIE_OB_REGION_EP)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure Outbound Address Translation(%d)!\n", (INT32)retVal);
            exit(1);
        }
        else
        {
            //      PCIE_logPrintf ("Successfully configured Outbound Translation!\n");
        }
#if RC3_30_LRU
        if ((retVal = pcieObTransCfg (handle_1, PCIE_OB_LO_ADDR_EP, PCIE_OB_HI_ADDR_EP, PCIE_OB_REGION_EP)) != pcie_RET_OK)
        {
            PCIE_logPrintf ("Failed to configure Outbound Address Translation(%d)!\n", (INT32)retVal);
            exit(1);
        }
        else
        {
            //      PCIE_logPrintf ("Successfully configured Outbound Translation!\n");
        }
#endif

    }

    /* Configure/limit number of lanes */
    pcieSetLanes (handle);
#if RC3_30_LRU
    pcieSetLanes (handle_1);
#endif

    PCIE_logPrintf ("Starting link training...\n");

    /*Enable link training*/
    if ((retVal = pcieLtssmCtrl(handle, TRUE)) != pcie_RET_OK)
    {
        PCIE_logPrintf ("Failed to Enable Link Training! (%d)\n", (INT32)retVal);
        exit(1);
    }
#if RC3_30_LRU
    /*Enable link training*/
    if ((retVal = pcieLtssmCtrl(handle_1, TRUE)) != pcie_RET_OK)
    {
        PCIE_logPrintf ("Failed to Enable Link Training! (%d)\n", (INT32)retVal);
        exit(1);
    }
#endif

    /* Wait for link to be up */
    pcieWaitLinkUp(handle);

    PCIE_logPrintf ("Link is up...................\n");

#if RC3_30_LRU
    /* Wait for link to be up */
    pcieWaitLinkUp(handle_1);

    PCIE_logPrintf ("Link is up...................\n");
#endif


    if ((retVal = pcieCheckLinkParams(handle)) != pcie_RET_OK)
    {
        PCIE_logPrintf ("Link width/speed verification FAILed: %d\n", retVal);
        /* This exit() can be removed if this example is being used as
         * template with non TI card that supports slower or narrower connections
         */
        //exit(1);
        while(1);
    }
#if RC3_30_LRU
    if ((retVal = pcieCheckLinkParams(handle_1)) != pcie_RET_OK)
    {
        PCIE_logPrintf ("Link width/speed verification FAILed: %d\n", retVal);
        /* This exit() can be removed if this example is being used as
         * template with non TI card that supports slower or narrower connections
         */
        //exit(1);
        while(1);
    }
#endif

    if ((retVal = Pcie_getMemSpaceRange (handle, &pcieBase, NULL)) != pcie_RET_OK) {
        PCIE_logPrintf ("getMemSpaceRange failed (%d)\n", (INT32)retVal);
        exit(1);
    }
#if RC3_30_LRU
    if ((retVal = Pcie_getMemSpaceRange (handle_1, &pcieBase, NULL)) != pcie_RET_OK) {
        PCIE_logPrintf ("getMemSpaceRange failed (%d)\n", (INT32)retVal);
        exit(1);
    }
#endif

#if defined(PCIE_REV1_HW)
    /* Adjust PCIE base to point at remote target buffer */
    pcieBase = (char *)pcieBase +
            PCIE_WINDOW_MEM_BASE +  /* data area doesn't start at low address */
            (((UINT32)&dstBuf) & 0xfff); /* dstBuf needs to be 4K aligned in addr tran */
#elif defined(PCIE_REV2_HW)
    /* Adjust PCIE base to point at remote target buffer */
    pcieBase = (char *) PCIE_WINDOW_MEM_BASE +  /* data area doesn't start at low address */
            (((UINT32)&dstBuf) & 0xffff); /* dstBuf needs to be 64K aligned in addr tran */
#elif defined(PCIE_REV3_HW)
    /* Adjust PCIE base to point at remote target buffer */
    pcieBase = (char *) PCIE_WINDOW_MEM_BASE +  /* data area doesn't start at low address */
            (((uintptr_t)&dstBuf) & 0x0fff); /* dstBuf needs to be 64K aligned in addr tran */
#else
    /* No adjustment needed */
#endif
    pciedstBufBase = (dstBuf_t *)pcieBase;


    if(PcieModeGbl == pcie_RC_MODE)
    {
#if defined(PCIE_REV1_HW) || (defined(PCIE_REV2_HW) && defined(BUILD_MPU)) || (defined(PCIE_REV3_HW) && defined(BUILD_MPU))
        sem = PlatformSetupMSIAndINTX (handle);
#endif
        /**********************************************************************/
        /* Push a single message to the EP then verify that it is echoed back */
        /**********************************************************************/


        /* Write from RC to EP                                                */
        for (i=0; i<PCIE_BUFSIZE_APP; i++)
        {
            pciedstBufBase->buf[i] = srcBuf[i];
        }

        /* Mark that the buffer is full, so EP can process it */
        pciedstBufBase->buf[PCIE_BUFSIZE_APP] = PCIE_EXAMPLE_BUF_FULL;


        /* Note on cache coherence: Write back is not necessary because pcieBase is in
       peripheral address space instead of physical memory*/

        /* Data sent to EP.
       RC waits for the loopback to be completed and
       receive data back from EP */

        do {
            cache_invalidate ((void *)dstBuf.buf, PCIE_EXAMPLE_DSTBUF_BYTES);
        } while(dstBuf.buf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);

        /* check all the data */
        for (i=0; i<PCIE_BUFSIZE_APP; i++)
        {
            if(dstBuf.buf[i] != srcBuf[i])
            {
                PCIE_logPrintf ("Received data = %u\nTransmited data = %u\nIndex = %u.\n\nTest failed.\n",
                                (unsigned)dstBuf.buf[i], (unsigned)srcBuf[i], (unsigned)i);
                exit(1);
            }
        }

        PCIE_logPrintf ("Root Complex received data.\n");
#if defined(PCIE_REV1_HW) || (defined(PCIE_REV2_HW) && defined(BUILD_MPU)) || (defined(PCIE_REV3_HW) && defined(BUILD_MPU))
        pcieRcWaitInts (handle, sem, pcieBase);
#endif



#ifdef PCIE_EXAMPLE_DMA_RC
        if (PcieExampleEdmaRC(pciedstBufBase, 0xbabeface, 100000,
                              (EDMA3_DRV_Handle)hEdma
        )) {
            PCIE_logPrintf ("Failed to pass token \n");
            exit(1);
        }

        PCIE_logPrintf ("Root Complex DMA received data.\n");
        PCIE_logPrintf ("EDMA Test passed.\n");
        PCIE_logPrintf ("=== this is not an optimized example ===\n");
#endif
    }
    else
    {
        /**********************************************************************/
        /* Wait for a single message from the RC then echo it back            */
        /**********************************************************************/
        PCIE_logPrintf("PCIe enumeration is done...\n");

#if 0
        /* EP waits for the data received from RC */
        do {
            cache_invalidate ((void *)dstBuf.buf, PCIE_EXAMPLE_DSTBUF_BYTES);
        } while(HW_RD_REG32(0xB0000000)!= 0xA5A5A5A5);//while(dstBuf.buf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);

        PCIE_logPrintf ("End Point received data.\n");


        /* Loopback to RC what was written in the DST buffer.
       Write from EP to RC */
        for (i=0; i<PCIE_BUFSIZE_APP; i++)
        {
            pciedstBufBase->buf[i] = dstBuf.buf[i];
        }

        /* Mark that the buffer is full, so RC can process it */
        pciedstBufBase->buf[PCIE_BUFSIZE_APP] = PCIE_EXAMPLE_BUF_FULL;

        /* Note on cache coherence: Write back is not necessary because pcieBase is in
       peripheral address space instead of physical memory*/

        PCIE_logPrintf ("End Point sent data to Root Complex, completing the loopback.\n");
#if defined(PCIE_REV1_HW) || (defined(PCIE_REV2_HW) && defined(BUILD_MPU)) || (defined(PCIE_REV3_HW) && defined(BUILD_MPU))
        /* Send msi to EP */
        pcieEpSendInts (handle);
#endif
        PCIE_logPrintf ("End of Interrupt Test.\n");




#ifdef PCIE_EXAMPLE_DMA_EP
        if (PcieExampleEdmaEP(pciedstBufBase, 0xbabeface, 100000,
                              (EDMA3_DRV_Handle)hEdma))
        {
            PCIE_logPrintf ("Failed to pass token \n");
            exit(1);
        }
        PCIE_logPrintf ("End Point sent data to Root Complex, DMA completing the loopback.\n");
        PCIE_logPrintf ("End of DMA Test.\n");
#endif
#endif
    }


#ifdef UDMA
    if (pcieUdmaTest((void *)&pciedstBufBase->edma_buf[0], sizeof(pciedstBufBase->edma_buf)))
    {
        PCIE_logPrintf("UDMA test failed\n");
        exit(1);
    }
#endif

#if 0
#ifdef EDMA
    edmaDeinit(hEdma);
#endif
#ifdef EDMAPKTBENCH
    if (PcieEdmaPktBench(&dstBuf.edmaPktBenchBuf.msiTracker[0],
                         &pciedstBufBase->edmaPktBenchBuf,
                         PcieModeGbl, sem))
    {
        PCIE_logPrintf ("EDMA packet IO benchmark failed to execute correctly\n");
        exit(1);
    }
#endif


    PCIE_logPrintf ("Test passed.\n");
#ifndef IO_CONSOLE
    Console_printf ("Test passed.\n");
#endif
#endif
    //  BIOS_exit(0);

}

  • Hi Jigyansu,

    I will need a couple of days to look through the register dumps and code.

    However, are there any logs from Intel and/or AM57x-side to see where the link stops?

    Furthermore, do you see any patterns in the randomness for when it stops (for example, does it stop after a certain time, or after a certain number of transaction, or after a certain command, etc)?

    Regards,

    Takuma

  • Hi Takuma

    We write the code to check the PCIE link status 1st ,then it goes for data transfer operation if there is a link up. If there is a link fail condition then we are doing the enumeration again and by doing this we are getting the link back. This approach gave one temporary solution to our application flow but wont help in actual testing.

    With this approach we found out that there may not be any issue from Intel side.

    The link down is happening randomly , there is no certain patterns I have observed so far. Sometimes the application runs for hours but sometimes it gives link down in 15 min.

    I am adding the code snippet for your understanding.

        while(1)
        {
            Read_Pci_Controller_Register(0);
            Read_Data_Pci_Link = HW_RD_REG32(0x5100210C);
            if(Read_Data_Pci_Link & 0x10000)
            {
                Pcie_RunTime_Test();
            }
            else
            {
                UART_printf("PCIe LINK DOWN Slot 1\n");
                Pcie_Init();
            }
        }

  • Hi Jigyansu,

    I see in your register dump that register 5100 0004 which is the STATUS_COMMAND_REGISTER that "RCVD_MASTERABORT" bit is set in "DEFAULT After Configuration" dump. This could occur when there is an "Unsupported Request" (aka, UR response).

    UR happens when there is some sort of request for an action or access to some space that is not supported by the Completer. Since you mention that the link down could take hours or as less than 15 min, perhaps there is some issue with how memory is allocated for BAR and there is a small region of memory that is just outside the supported range that gets accessed, causing random issues. Otherwise, it would be due to some rare action being sent that causes the UR response.

    Regards,

    Takuma

  • Hi Takuma

    After using the PCI driver we are using the 0x20000000 address for the communication , the code snippet I am adding below.

    With this address we are able to do the communication properly but sometimes it gets link down as I said earlier.

    Coming to the beyond supported memory region , I have used only 200 bytes of memory to do the test and did not go beyond the region.

    Please check it and let me know if there is any changes I have to do during the Initialization of PCIe.

    UINT32 Pcie_RunTime_Test()
    {
        /*Local Variables */
        static UINT32 data = 0;
        UINT32 return_val = 0;
        UINT32 Offset_Address = 0x4;
        int Read_Data_Pci_Link;
    
        PCIE_logPrintf("Checking For the Flag Slot 1\n");
        
        /*Checking for the Flag (This is to sync the transaction)*/
        while(!return_val)
        {
            /*Checking Link Status*/
            Read_Data_Pci_Link = HW_RD_REG32(0x5100210C);
            if(Read_Data_Pci_Link & 0x10000)
            {
                /*Reading The Flag*/
                return_val = HW_RD_REG32(0x20000000);
            }
            else
            {
                UART_printf("PCIe LINK DOWN Slot 1\n");
                
                /*Doing Reenumeration on failure*/
                Pcie_Init();
            }
        }
        PCIE_logPrintf("Flag Up to Read verify\n");
    
        for(int i = 0 ; i < 50 ; i++)
        {
            /*Checking Link Status*/
            Read_Data_Pci_Link = HW_RD_REG32(0x5100210C);
            if(Read_Data_Pci_Link & 0x10000)
            {
                /*Reading the data written by Root Complex*/
                return_val = HW_RD_REG32(0x20000000 + Offset_Address);
                PCIE_logPrintf("\tData received via PCIe from MPC Card is = 0x%x Expected = 0x%x\n",return_val,0xA5A5A5A5);
                
                /*Validating The data (Fixed Data)*/
                if(return_val != 0xA5A5A5A5)
                {
                    UART_printf("Data Mismatch\n");
                }
                
                /*Address Offset increment*/
                Offset_Address = Offset_Address + 0x4;
            }
            else
            {
                UART_printf("PCIe LINK DOWN Slot 1\n");
                
                /*Doing Reenumeration on failure*/
                Pcie_Init();
            }
        }
    
        /*Address offset reseting to write */
        Offset_Address = 0x4;
        for(int i = 0 ; i < 50 ; i++)
        {
            /*Checking Link Status*/
            Read_Data_Pci_Link = HW_RD_REG32(0x5100210C);
            if(Read_Data_Pci_Link & 0x10000)
            {
                /*Writing the Data to Root Complex (Fixed Data)*/
                HW_WR_REG32(0x20000000 + Offset_Address,0xB6B6B6B6);
                Offset_Address = Offset_Address + 0x4;
            }
            else
            {
                UART_printf("PCIe LINK DOWN Slot 1\n");
                
                /*Doing Reenumeration on failure*/
                Pcie_Init();
            }
        }
        
        /*Reverting the flag for Root Complex (Sync the commuincation)*/    
        HW_WR_REG32(0x20000000 , 0x0);
        return return_val;
    }
     

  • Hi Jigyansu,

    We cannot tell how the UR response is created for triggering the RCVD_MASTERABORT as that depends on your system. 

    In order to get a better idea, you will need to find patterns. For example, which line (and if it is in one of the for loops, after how many loops in the for loop) in your Pcie_RunTime_Test function do we see fails? And is it consistently in the same location? 

    Regards,

    Takuma

  • Hi Takuma 

    I have observed the fail condition multiple times and so far I did not observe any particular pattern. As I have both read and write operations inside the function ,it is happening in read and write randomly in random iteration.

  • Hi Jigyansu,

    I have found this thread from Intel about E3930 for accessing 32-bit non-prefetchable memory: https://community.intel.com/t5/Embedded-Intel-Atom-Processors/E3930-PCIe-memory-access-issues/m-p/238894?profile.language=en

    Additionally, searching through past threads for AM57x, I was not able to find any mentions about RCVD_MASTERABORT happening randomly nor a UR response happening randomly. 

    From this, I think a good experiment would be to see if you see the same random link downs when using the AM57x EP with a different RC device. If I recall, there should be an EP/RC example for AM57x, so you could try that out and see if random link downs occur with this example. Then, you could use this as a reference point to see if you can reproduce the issue by modifying the AM57x<->AM57x example.

    Regards,

    Takuma