C6678 Understanding the SRIO Multicore Loopback example and srio driver

Brandy Jabkiewicz

Hello team,

My goal is to conver the SRIO loopback example to a "normal" example that sends packets from one DSP to another DSP. I am choosing this path because I do not need the inter processor support to share memory etc. I just need to prove that the SRIO driver can aid me in sending and recieving packets becuase in my actual application the sender will not be a C6678 but another component and the DSP will just need to recieve the data stream and act on it.

I have my two C6678 EVMs attached to a srio switch in a Vadatech Chassis. The switch only allows a single device to connect to it once. So I had to change the loopback code to only enable one srio port instead of 4. The port is set to be a lane width of 1.

So, I make the above change with the ports first and run it in loopback mode. Program works.

I switch the csl call from CSL_SRIO_SetLoopBackMode to CSL_SRIO_SetNormalMode. Program fails. The SRIO port says "port ok", the drivers initialize and say ok. The first dsp attempts to send a message from Core 1 but the other DSP never receives the message.

I am fairly confident that the following is true:

- My chassis has enable the SRIO on the backplane for the two EVMs to link to the switch.

- I have configured my switch for static routes which contain the destination ids in the given program.

Another point is I am running the two DSPs in big endian. Therefore I compiled the program in big endian.

I added debug statements to the srio driver. It prints out some of the queue numbers, pointers etc so I can try to understand a bit better.

Q1: Are there ideas as to why things aren't working and what to check?

Q2: It looks like the Rx return queue and the Tx free queue are using the "stavation counter" queues (736 thru 743). Is this ok? If yes, why? If no, what should it be?

Q3: The Tx Queues are assigned to the SRIO Tx channels. However Core 1 is assigned to 672, Core 0 is assigned 673, Core 2 is 674 and Core 3 is 675. Is this order ok and why?

Q4: The high priority accumulator queues somehow attached to event 48. I don't quite understand this relationship. I also don't understand why event 48 gets connected to HWI 8? Can you explain more about this relationship and functionality?

Q5: When I send a tx message, what can I check (or where can I add a debug statement) to confirm that the message went out? I notice that the Rx side has an isr when the message comes in, but there is nothing comparable for the tx side. Is there a register I can read with a CSL command or something?

Thanks for your help,

Brandy

over 12 years ago

0 Derek Brown over 12 years ago

TI__Intellectual 1880 points

Brandy,

Here are responses to questions 2-4. I do not know the answers to questions 1 and 5, but I will keep looking for the answer for you.

Q2: It looks like the Rx return queue and the Tx free queue are using the "stavation counter" queues (736 thru 743). Is this ok? If yes, why? If no, what should it be?

It is perfectly fine to use the starvation queues. These queues are similar to general purpose queues, except that they have counters that keep track of whether or not a "pop" operation was executed when the queue no descriptors were available in the queue. There is more information about starvation queues in the Multicore Navigator User Guide SPRUG9D.

Q3: The Tx Queues are assigned to the SRIO Tx channels. However Core 1 is assigned to 672, Core 0 is assigned 673, Core 2 is 674 and Core 3 is 675. Is this order ok and why?

It is okay for any core to push a packet onto any TX queue. If that is how things get setup when you execute the example, then that is okay.

The System Event Mapping table (table 7-38) in the C6678 data manual shows the system events that can be mapped to interrupts. Event 48 is the high priority interrupt for the queue manager. The is the event that the accumulator uses for monitoring the high priority accumulation queues. This gets setup automatically during the accumulator setup. By tying the event 48 to hardware interrupt 8, this allows the interrupt from the accumulator to directly notify the core directly without going through the interrupt event combiner. It doesn't really matter which hardware interrupt is chosen, because hardware interrupts 4-15 are available for use in the system, and the programmer is free use any of those interrupts that are available. So, if as long as hardware interrupt 8 was available, then it was a valid choice.

0 Brandy Jabkiewicz over 12 years ago in reply to Derek Brown

Mastermind 6325 points

Hi Derek,

Thanks for the quick response on 2, 3, and 4. That helps with my understanding of the system design.

Looking forward to another response regarding 1 and 5!

Brandy

0 Brandy Jabkiewicz over 12 years ago in reply to Brandy Jabkiewicz

Mastermind 6325 points

Hello Derek,

I also wanted to add... I read through this post:

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/127613.aspx

At the end of it, Travis suggests that you need to enable all 4 SERDES to get the SRIO work in 4 lanes x 1 port mode.

Is this true about 1 lane x 1 port mode as well becuase I had commented out this in my change from 1 lane x 4 ports as describe originally?

For fun, I tried it real quick - I uncommented the SERDES initialization for ports 1 to 3. However, this caused the other ports to be enabled too. Interesting.

What do you think?

FYI - attached is my file with the commented out SRIO registers.

5518.multicoreLoopback - Copy.txt

/**
 *   @file  multicoreLoopback.c
 *
 *   @brief   
 *      This is an example application which showcases how the SRIO Driver
 *      API can be used in a multiple core environment. 
 *
 *      The test suite sends/receives data as follows:-
 *
 *      CORE 1 --> CORE 2 --> CORE 3 --> CORE 0 --> CORE 1
 *
 *      At each step the received data is validated (Payload and Received
 *      Identifiers).
 *
 *      The goal of the test suite is to showcase that the SRIO Driver API
 *      are multi-core safe.
 *
 *  \par
 *  NOTE:
 *      (C) Copyright 2009 Texas Instruments, Inc.
 * 
 *  Redistribution and use in source and binary forms, with or without 
 *  modification, are permitted provided that the following conditions 
 *  are met:
 *
 *    Redistributions of source code must retain the above copyright 
 *    notice, this list of conditions and the following disclaimer.
 *
 *    Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the 
 *    documentation and/or other materials provided with the   
 *    distribution.
 *
 *    Neither the name of Texas Instruments Incorporated nor the names of
 *    its contributors may be used to endorse or promote products derived
 *    from this software without specific prior written permission.
 *
 *  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
 *  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
 *  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
 *  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 
 *  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 
 *  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
 *  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
 *  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
 *  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
 *  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 
 *  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *
 *  \par
*/
#include <xdc/std.h>
#include <string.h>
#include <xdc/runtime/IHeap.h>
#include <xdc/runtime/System.h>
#include <xdc/runtime/Memory.h> 
#include <ti/sysbios/BIOS.h>
#include <ti/sysbios/knl/Task.h>
#include <ti/sysbios/heaps/HeapBuf.h>
#include <ti/sysbios/heaps/HeapMem.h>
#include <ti/sysbios/family/c64p/Hwi.h>
#include <ti/sysbios/family/c64p/EventCombiner.h> 
#include <ti/sysbios/family/c66/Cache.h>

/* IPC includes */ 
#include <ti/ipc/GateMP.h>
#include <ti/ipc/Ipc.h>
#include <ti/ipc/ListMP.h>
#include <ti/ipc/SharedRegion.h>
#include <ti/ipc/MultiProc.h>
#include <xdc/cfg/global.h>

/* SRIO Driver Include File. */
#include <ti/drv/srio/srio_drv.h>

/* CPPI/QMSS Include Files. */
#include <ti/drv/cppi/cppi_drv.h>
#include <ti/drv/qmss/qmss_drv.h>
#include <ti/drv/qmss/qmss_firmware.h>

/* CSL Include Files */
#include <ti/csl/csl_chip.h>
#include <ti/csl/csl_cacheAux.h>
#include <ti/csl/csl_psc.h>
#include <ti/csl/csl_pscAux.h>
#include <ti/csl/csl_semAux.h>

/* PDK module Headers */
#include <ti/platform/platform.h>

/**********************************************************************
 ************************** LOCAL Definitions *************************
 **********************************************************************/

#define NUM_HOST_DESC               32
#define SIZE_HOST_DESC              48
#define SRIO_MAX_MTU				256

/* Defines the core number responsible for system initialization. */
#define CORE_SYS_INIT               1

/* Number of cores for which the test is being executed. */
#define NUM_CORES                   4

/* SRIO Application Hardware Semaphore. */
#define SRIO_APP_HW_SEM             5

/**********************************************************************
 ************************** Global Variables **************************
 **********************************************************************/

/* Memory used for the descriptors. */
#pragma DATA_ALIGN (host_region, 16)
Uint8   host_region[NUM_HOST_DESC * SIZE_HOST_DESC];

Qmss_InitCfg   qmssInitConfig;

/* Global Varialble which keeps track of the core number executing the
 * application. */
UInt32          coreNum = 0xFFFF;

/* from producer_srio.c
 * #pragma DATA_SECTION (isSRIOInitialized, ".srioSharedMem");
 * volatile Uint32     isSRIOInitialized    = 0;
 */

#pragma DATA_ALIGN   (isSRIOInitialized, 128)
#pragma DATA_SECTION (isSRIOInitialized, ".srioSharedMem");
volatile Uint32     isSRIOInitialized    = 0;

#pragma DATA_ALIGN   (isSRIOSocketsCreated, 128)
#pragma DATA_SECTION (isSRIOSocketsCreated, ".srioSharedMem");
volatile Uint32     isSRIOSocketsCreated = 0;

#pragma DATA_ALIGN   (firstToRun, 128)
#pragma DATA_SECTION (firstToRun, ".srioSharedMem");
volatile Uint32     firstToRun = 0;

/* Memory used for the accumulator list. */
#pragma DATA_ALIGN (gHiPriAccumList, 16)
UInt32              gHiPriAccumList[32];

/* These are the device identifiers used used in the TEST Application */
const uint32_t DEVICE_ID1_16BIT    = 0xBEEF;
const uint32_t DEVICE_ID1_8BIT     = 0xAB;
const uint32_t DEVICE_ID2_16BIT    = 0x4560;
const uint32_t DEVICE_ID2_8BIT     = 0xCD;
const uint32_t DEVICE_ID3_16BIT    = 0x1234;
const uint32_t DEVICE_ID3_8BIT     = 0x12;
const uint32_t DEVICE_ID4_16BIT    = 0x5678;
const uint32_t DEVICE_ID4_8BIT     = 0x56;

/**********************************************************************
 ************************* Extern Definitions *************************
 **********************************************************************/

extern UInt32 malloc_counter;
extern UInt32 free_counter;

extern int32_t SrioDevice_init (void);
extern Void* Osal_srioDataBufferMalloc(UInt32 numBytes);

/* QMSS device specific configuration */
extern Qmss_GlobalConfigParams  qmssGblCfgParams;

/* CPPI device specific configuration */
extern Cppi_GlobalConfigParams  cppiGblCfgParams;

/* OSAL Data Buffer Memory Initialization. */
extern int32_t Osal_dataBufferInitMemory(uint32_t dataBufferSize);

/**********************************************************************
 ************************ SRIO TEST FUNCTIONS *************************
 **********************************************************************/

/**
 *  @b Description
 *  @n  
 *      Utility function which converts a local address to global.
 *
 *  @param[in]  addr
 *      Local address to be converted
 *
 *  @retval
 *      Global Address
 */
static UInt32 l2_global_address (Uint32 addr)
{
	UInt32 corenum;

	/* Get the core number. */
	corenum = CSL_chipReadReg(CSL_CHIP_DNUM); 

	/* Compute the global address. */
	return (addr + (0x10000000 + (corenum*0x1000000)));
}

/**
 *  @b Description
 *  @n  
 *      Utility function that is required by the IPC module to set the proc Id.
 *      The proc Id is set via this function instead of hard coding it in the .cfg file
 *
 *  @retval
 *      Not Applicable.
 */
Void myStartupFxn (Void)
{
	MultiProc_setLocalId (CSL_chipReadReg (CSL_CHIP_DNUM));
}

/**
 *  @b Description
 *  @n  
 *      This function enables the power/clock domains for SRIO. 
 *
 *  @retval
 *      Not Applicable.
 */
static Int32 enable_srio (void)
{
#ifndef SIMULATOR_SUPPORT
    /* SRIO power domain is turned OFF by default. It needs to be turned on before doing any 
     * SRIO device register access. This not required for the simulator. */

    /* Set SRIO Power domain to ON */        
    CSL_PSC_enablePowerDomain (CSL_PSC_PD_SRIO);

    /* Enable the clocks too for SRIO */
    CSL_PSC_setModuleNextState (CSL_PSC_LPSC_SRIO, PSC_MODSTATE_ENABLE);

    /* Start the state transition */
    CSL_PSC_startStateTransition (CSL_PSC_PD_SRIO);

    /* Wait until the state transition process is completed. */
    while (!CSL_PSC_isStateTransitionDone (CSL_PSC_PD_SRIO));

    /* Return SRIO PSC status */
    if ((CSL_PSC_getPowerDomainState(CSL_PSC_PD_SRIO) == PSC_PDSTATE_ON) &&
        (CSL_PSC_getModuleState (CSL_PSC_LPSC_SRIO) == PSC_MODSTATE_ENABLE))
    {
        /* SRIO ON. Ready for use */            
        return 0;
    }
    else
    {
        /* SRIO Power on failed. Return error */            
        return -1;            
    }
#else
    /* PSC is not supported on simulator. Return success always */
    return 0;
#endif
}

/**
 *  @b Description
 *  @n  
 *      System Initialization Code. This is added here only for illustrative
 *      purposes and needs to be invoked once during initialization at 
 *      system startup.
 *
 *  @retval
 *      Success     -   0
 *  @retval
 *      Error       -   <0
 */
static Int32 system_init (Void)
{
    Int32               result;
    Qmss_MemRegInfo     memRegInfo;

    /* Initialize the QMSS Configuration block. */
    memset (&qmssInitConfig, 0, sizeof (Qmss_InitCfg));

    /* Set up the linking RAM. Use the internal Linking RAM. 
     * LLD will configure the internal linking RAM address and maximum internal linking RAM size if 
     * a value of zero is specified. Linking RAM1 is not used */
    qmssInitConfig.linkingRAM0Base = 0;
    qmssInitConfig.linkingRAM0Size = 0;
    qmssInitConfig.linkingRAM1Base = 0;
    qmssInitConfig.maxDescNum      = NUM_HOST_DESC;   

#ifdef xdc_target__bigEndian
    System_printf("Using Big Endian QMSS\n");
    qmssInitConfig.pdspFirmware[0].pdspId   = Qmss_PdspId_PDSP1;
    qmssInitConfig.pdspFirmware[0].firmware = &acc48_be;
    qmssInitConfig.pdspFirmware[0].size     = sizeof (acc48_be);
#else
    System_printf("Using Little Endian QMSS\n");
    qmssInitConfig.pdspFirmware[0].pdspId   = Qmss_PdspId_PDSP1;
    qmssInitConfig.pdspFirmware[0].firmware = &acc48_le;
    qmssInitConfig.pdspFirmware[0].size     = sizeof (acc48_le);
#endif    

    /* Initialize Queue Manager Sub System */
    result = Qmss_init (&qmssInitConfig, &qmssGblCfgParams);
    if (result != QMSS_SOK)
    {
        System_printf ("Error initializing Queue Manager SubSystem error code : %d\n", result);
        return -1;
    }

    /* Start the QMSS. */
    if (Qmss_start() != QMSS_SOK)
    {
        System_printf ("Error: Unable to start the QMSS\n");
        return -1;
    }

    /* Memory Region 0 Configuration */
    memRegInfo.descBase         = (UInt32 *)l2_global_address((UInt32)host_region);
    memRegInfo.descSize         = SIZE_HOST_DESC;
    memRegInfo.descNum          = NUM_HOST_DESC;
    memRegInfo.manageDescFlag   = Qmss_ManageDesc_MANAGE_DESCRIPTOR;
    memRegInfo.memRegion        = Qmss_MemRegion_MEMORY_REGION_NOT_SPECIFIED;    

    /* Initialize and inset the memory region. */
    result = Qmss_insertMemoryRegion (&memRegInfo); 
    if (result < QMSS_SOK)
    {
        System_printf ("Error inserting memory region: %d\n", result);
        return -1;
    }

    /* Initialize CPPI CPDMA */
    result = Cppi_init (&cppiGblCfgParams);
    if (result != CPPI_SOK)
    {
        System_printf ("Error initializing Queue Manager SubSystem error code : %d\n", result);
        return -1;
    }

    /* CPPI and Queue Manager are initialized. */
    System_printf ("Debug: Queue Manager and CPPI are initialized.\n");
    System_printf ("Debug: Host Region 0x%x\n", host_region);
    return 0;
}

/**
 *  @b Description
 *  @n  
 *      The function tests multicore
 *
 *  @param[in]  hSrioDrv
 *      Handle to the SRIO driver 
 *
 *  @retval
 *      Success     - 0
 *  @retval
 *      Error       - <0
 */
static Int32 test_multicore (Srio_DrvHandle hSrioDrv)
{
    Srio_SockHandle         srioSocket;
    Srio_SockBindAddrInfo   bindInfo;
    UInt8*                  txData;
    Srio_SockAddrInfo       to;
    Srio_SockAddrInfo       from;
    UInt16                  idx;
    Int32                   num_bytes;
    UInt8*                  ptr_rxDataPayload;
    Int32                   sendToCore;
    Int32                   recvFromCore;
    Srio_DrvBuffer          hDrvBuffer;
    UInt32                  allocatedLen;    
    UInt16                  coreDeviceID[NUM_CORES];
    UInt8					errCnt =0;

    System_printf ("**********************************************\n");
    System_printf ("******** Multicore Testing (Core %d) *********\n", coreNum);
    System_printf ("**********************************************\n");
    
    /* Open SRIO Socket in Blocking Mode
     * Open a socket with the driver handle.  Get a socket handle.
     * Now you can transmit and rx*/
    srioSocket =  Srio_sockOpen (hSrioDrv, Srio_SocketType_TYPE11, TRUE);
    if (srioSocket == NULL)
    {
        System_printf ("Error: Unable to open socket1\n");
        return -1;
    }

    /* Initialize the core Device IDs: Each core has a seperate device ID. */
    coreDeviceID[0] = DEVICE_ID1_16BIT;
    coreDeviceID[1] = DEVICE_ID2_16BIT;
    coreDeviceID[2] = DEVICE_ID3_16BIT;
    coreDeviceID[3] = DEVICE_ID4_16BIT;

    /* Initialize the core bindings; we use the same mailbox & letter identifiers. */
    bindInfo.type11.tt       = TRUE;
    bindInfo.type11.id       = coreDeviceID[coreNum];
    bindInfo.type11.letter   = 2;
    bindInfo.type11.mbox     = 3;
    bindInfo.type11.segMap   = 0x0;

    /* Bind the SRIO socket
     * Bind to a queue in the srio hardware*/
    if (Srio_sockBind (srioSocket, &bindInfo) < 0)
    {
        System_printf ("Error: socket1 bind failed\n");
        return -1;
    }

    /* Get a transmit buffer from the SRIO Driver. */
    hDrvBuffer = Srio_allocTransmitBuffer(hSrioDrv, &txData, &allocatedLen);
    if (hDrvBuffer == NULL)
    {
        System_printf ("Error: Producer Memory Allocation failed.\n");
        return -1;
    }
    System_printf("Debug(Core %i): received a tx buff %p with length %i.\n", coreNum, txData, allocatedLen);

    /* Create the transmit data payload. */
    for (idx = 0; idx < 100; idx++)
        txData[idx] = 0x00 | coreNum;

    System_printf("Debug(Core %i): txData 0x%x 0x%x 0x%x...\n",coreNum, txData[0],txData[50], txData[99]);

    /* The global variable is a shared resource which is being accessed from multiple cores. 
     * So here we need to protect it and ensure that there is only 1 core which is accessing 
     * it at a time. We use a Hardware Semaphore to protect this. */
    while ((CSL_semAcquireDirect (SRIO_APP_HW_SEM)) == 0);

    /* Invalidate the cache and make sure you get the latest from the memory. */
    CACHE_invL1d ((void *) &isSRIOSocketsCreated, 128, CACHE_WAIT);

    /* The core has created the sockets:*/
    isSRIOSocketsCreated++;

    /* The SRIO Socket has been created. Writeback the contents to the cache. */
    CACHE_wbL1d ((void *) &isSRIOSocketsCreated, 128, CACHE_WAIT);

    /* Release the hardware semaphore. */
    CSL_semReleaseSemaphore (SRIO_APP_HW_SEM);

    /* We can proceed with the data transmission & reception tests only after all the
     * cores have created and bound their SRIO sockets. This is a simple counter running
     * in shared memory which allows us to SYNC up the socket creation. Wait till all the
     * cores have created the sockets */
    while (isSRIOSocketsCreated != NUM_CORES)
        CACHE_invL1d ((void *) &isSRIOSocketsCreated, 128, CACHE_WAIT);

    System_printf ("Core %i is ready to start RR-------------------------------------------\n", coreNum);

    /* Is this the starting core? */ 
//	while ((CSL_semAcquireDirect (SRIO_APP_HW_SEM)) == 0)
//	{
//		//System_printf("Debug(Core %i): Waiting for sem.\n",coreNum);
//	};
//	System_printf("Debug(Core %i): Got sem\n",coreNum);
//    CACHE_invL1d ((void *) &firstToRun, 128, CACHE_WAIT);

    if (coreNum == 1)
    {
    //	firstToRun = 2;
	 //   CACHE_wbL1d ((void *) &firstToRun, 128, CACHE_WAIT);
   //     Cache_wb ((void *) &firstToRun, 4, Cache_Type_L1D, TRUE);
	    System_printf ("Debug(Core %i) is first.\n", coreNum);

	    /* Starting Core: This is a special case because unlike the other cores; this initiates
         * the whole test cycle by sending the first packet. */
        sendToCore   = (coreNum + 1) % NUM_CORES;
        recvFromCore = coreNum - 1;
        if (recvFromCore < 0)
            recvFromCore = NUM_CORES-1;

        /* Send the data out. */
        to.type11.tt       = TRUE;
        to.type11.id       = 0xbeef;//coreDeviceID[sendToCore];
        to.type11.letter   = 2;
        to.type11.mbox     = 3;

        /* Send the data out from the producer core to the consumer core. */
        if (Srio_sockSend (srioSocket, hDrvBuffer, 100, &to) < 0)
        {
            System_printf ("Error: SRIO Socket send failed\n");
            return -1;
        }

        /* Debug Message */
        System_printf ("Debug(Core %d): Successfully sent data to ID:0x%x\n", coreNum, to.type11.id);

//        /* CORE0: Wait for data to arrive from NUM_CORES-1 */
//        System_printf ("Debug(Core %d): Waiting for data to arrive. myID: 0x%x\n", coreNum, bindInfo.type11.id);
//
//        /* Receive the data. */
//        num_bytes = Srio_sockRecv (srioSocket, (Srio_DrvBuffer*)&ptr_rxDataPayload, &from);
//        if (num_bytes > 0)
//        {
//            /* Make sure that the data was received from the producer core */
//            if (from.type11.id != coreDeviceID[recvFromCore])
//            {
//                System_printf ("Error: Invalid source id 0x%x Expected 0x%x\n", from.type11.id, coreDeviceID[recvFromCore]);
//                return -1;
//            }
//
//            /* Received and Transmitted packet length match. Payload verification */
//            for (idx = 0; idx < 100; idx++)
//            {
//                if (ptr_rxDataPayload[idx] != (0xA0 | recvFromCore))
//                {
//                	System_printf ("Error: Receive Data Payload verification failed @ index %d value = %d\n", idx, ptr_rxDataPayload[idx]);
//                   // return -1;
//                }
//            }
//            System_printf ("Debug(Core %d): Successfully received %d bytes\n", coreNum, num_bytes);
//            System_printf("Debug(Core %i): rxData 0x%x 0%x 0x%x...\n", coreNum, ptr_rxDataPayload[0],ptr_rxDataPayload[50], ptr_rxDataPayload[99]);
//
//
//            /* Cleanup the received data payload. */
//            Srio_freeRxDrvBuffer(srioSocket, (Srio_DrvBuffer)ptr_rxDataPayload);
//        }
//        else
//        {
//            /* Error: In receiving data */
//            System_printf ("Error: Unable to receive data error code: %d\n", num_bytes);
//            return -1;
//        }
    }

    CSL_semReleaseSemaphore (SRIO_APP_HW_SEM);

    Uint8 i;
    for( i = 0; i < 16; i++)
    {
        /* Any other core besides the starting core will initally wait for data to arrive */
        sendToCore   = (coreNum + 1) % NUM_CORES;
        recvFromCore = coreNum - 1;
        if (recvFromCore < 0)
            recvFromCore = NUM_CORES-1;

        System_printf ("Debug(Core %d): Waiting for data to arrive. myId: 0x%x\n", coreNum, bindInfo.type11.id);

        /* Receive the data. */
        num_bytes = Srio_sockRecv (srioSocket, (Srio_DrvBuffer*)&ptr_rxDataPayload, &from);
        if (num_bytes > 0)
        {
            /* Make sure that the data was received from the producer core */
            if (from.type11.id != coreDeviceID[recvFromCore])
            {
                System_printf ("Error: Invalid source id 0x%x Expected 0x%x\n", from.type11.id, coreDeviceID[recvFromCore]);
                errCnt++;
                //return -1;
            }

            /* Received and Transmitted packet length match. Payload verification */
            for (idx = 0; idx < 100; idx++)
            {

                {
					if (ptr_rxDataPayload[idx] != ((0x00 + ((i)<<4)) | recvFromCore))
					{
						System_printf ("Error: Receive Data Payload verification failed @ index %d value = 0x%x expected 0x%x\n", idx, ptr_rxDataPayload[idx], ((0x00 + (i<<4)) | recvFromCore));
						idx = 100;
						errCnt++;
						// return -1;
					}
                }

            }
            System_printf ("Debug(Core %d): Successfully received %d bytes. rxData 0x%x 0x%x 0x%x...\n", coreNum, num_bytes,ptr_rxDataPayload[0],ptr_rxDataPayload[50], ptr_rxDataPayload[99]);
            //System_printf("Debug(Core %i): rxData 0x%x 0%x 0x%x...\n", coreNum, ptr_rxDataPayload[0],ptr_rxDataPayload[50], ptr_rxDataPayload[99]);
            /* Cleanup the received data payload. */
            Srio_freeRxDrvBuffer(srioSocket, (Srio_DrvBuffer)ptr_rxDataPayload);
        }
        else
        {
            /* Error: In receiving data */ 
            System_printf ("Error: Unable to receive data error code: %d\n", num_bytes);
            errCnt++;
            // return -1;
        }

        /* CoreX: Sends outs the packet to CoreX+1. */
        to.type11.tt       = TRUE;
        to.type11.id       = coreDeviceID[sendToCore];
        to.type11.letter   = 2;
        to.type11.mbox     = 3;
        if(coreNum == 1) //increment i early because it initially transmitted.
        {
            for (idx = 0; idx < 100; idx++)
                txData[idx] = (0x00+((i+1)<<4)) | coreNum;
        }
        else
        {
			for (idx = 0; idx < 100; idx++)
				txData[idx] = (0x00+((i)<<4)) | coreNum;
        }
        /* Send the data out from the producer core to the consumer core. */
        if (Srio_sockSend (srioSocket, hDrvBuffer, 100, &to) < 0)
        {
            System_printf ("Error: SRIO Socket send failed\n");
            errCnt++;
            //return -1;
        }
        System_printf ("Debug(Core %d): Successfully sent data to ID:0x%x\n", coreNum, coreDeviceID[sendToCore]);
    }


    /* Close the sockets */
    Srio_sockClose (srioSocket);
 
    /* We are done with the test. */
    if(errCnt > 0)
    {
    	System_printf("Debug(Core %i): Completed with errors %i.\n",coreNum,errCnt);
    	return -1;
    }
    System_printf ("Debug(Core %d): Multicore Test Passed\n", coreNum);    
    return 0;
}

/**
 *  @b Description
 *  @n  
 *      This is the Multicore Test task
 *
 *  @retval
 *      Not Applicable.
 */
static Void multicoreTestTask(UArg arg0, UArg arg1)
{
    Srio_DrvConfig  cfg;
    Srio_DrvHandle  hSrioDriver;
    UInt8           isAllocated;
    
    /* Initialize the OSAL Data Buffer */
    if (Osal_dataBufferInitMemory(SRIO_MAX_MTU) < 0)
    	return;
 
    /* Initialize the SRIO Driver Configuration. */
    memset ((Void *)&cfg, 0, sizeof(Srio_DrvConfig));

    /* Setup the SRIO Driver Managed Configuration. */
    cfg.bAppManagedConfig                   = FALSE;

    /* Driver Managed: Receive Configuration */
    cfg.u.drvManagedCfg.bIsRxCfgValid             = 1;
    cfg.u.drvManagedCfg.rxCfg.rxMemRegion         = Qmss_MemRegion_MEMORY_REGION0;
    cfg.u.drvManagedCfg.rxCfg.numRxBuffers        = 4;
    cfg.u.drvManagedCfg.rxCfg.rxMTU               = SRIO_MAX_MTU;

    /* Accumulator Configuration. */ 
    {
	    int32_t coreToQueueSelector[4];

        /* This is the table which maps the core to a specific receive queue. */
	    coreToQueueSelector[0] = 704;
	    coreToQueueSelector[1] = 705;
	    coreToQueueSelector[2] = 706;
	    coreToQueueSelector[3] = 707;

	    /* Since we are programming the accumulator we want this queue to be a HIGH PRIORITY Queue */
	    cfg.u.drvManagedCfg.rxCfg.rxCompletionQueue = Qmss_queueOpen (Qmss_QueueType_HIGH_PRIORITY_QUEUE, 
                                                                         coreToQueueSelector[coreNum], 
                                                                         &isAllocated);
		if (cfg.u.drvManagedCfg.rxCfg.rxCompletionQueue < 0)
		{
			System_printf ("Error: Unable to open the SRIO Receive Completion Queue\n");
			return;
		}

		/* Accumulator Configuration is VALID. */
		cfg.u.drvManagedCfg.rxCfg.bIsAccumlatorCfgValid = 1;	

		/* Accumulator Configuration. */      
	    cfg.u.drvManagedCfg.rxCfg.accCfg.channel             = coreNum;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.command             = Qmss_AccCmd_ENABLE_CHANNEL;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.queueEnMask         = 0;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.queMgrIndex         = coreToQueueSelector[coreNum];
	    cfg.u.drvManagedCfg.rxCfg.accCfg.maxPageEntries      = 2;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.timerLoadCount      = 0;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.interruptPacingMode = Qmss_AccPacingMode_LAST_INTERRUPT;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.listEntrySize       = Qmss_AccEntrySize_REG_D;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.listCountMode       = Qmss_AccCountMode_ENTRY_COUNT;
	    cfg.u.drvManagedCfg.rxCfg.accCfg.multiQueueMode      = Qmss_AccQueueMode_SINGLE_QUEUE;

        /* Initialize the accumulator list memory */
        memset ((Void *)&gHiPriAccumList[0], 0, sizeof(gHiPriAccumList));
        cfg.u.drvManagedCfg.rxCfg.accCfg.listAddress = l2_global_address((UInt32)&gHiPriAccumList[0]);
        System_printf("Debug(Core %i): list address 0x%x\n", coreNum, cfg.u.drvManagedCfg.rxCfg.accCfg.listAddress);
    }
    /* Driver Managed: Transmit Configuration */
    cfg.u.drvManagedCfg.bIsTxCfgValid             = 1;
    cfg.u.drvManagedCfg.txCfg.txMemRegion         = Qmss_MemRegion_MEMORY_REGION0;
    cfg.u.drvManagedCfg.txCfg.numTxBuffers        = 4;
    cfg.u.drvManagedCfg.txCfg.txMTU               = SRIO_MAX_MTU;

    /* Start the SRIO Driver
     * Called by each core to get handle of active driver. */
    hSrioDriver = Srio_start(&cfg);
    if (hSrioDriver == NULL)
    {
        System_printf ("Error: SRIO Driver Start Failed\n");
        return;
    }

    /* SRIO Driver is operational at this time. */
    System_printf ("Debug: SRIO Driver has been started Instance Handle 0x%p\n", hSrioDriver);

    /* Hook up the SRIO interrupts with the core. */
    EventCombiner_dispatchPlug (48, (EventCombiner_FuncPtr)Srio_rxCompletionIsr, (UArg)hSrioDriver, TRUE);
	EventCombiner_enableEvent(48);

    /* Map the event id to hardware interrupt 8. */
    Hwi_eventMap(8, 48);

    /* Enable interrupt 8. */
    Hwi_enableInterrupt(8);

    /* SRIO Driver is operational at this time. */
    System_printf ("Debug: SRIO Driver ISR has been registered\n");

    /* Run the Multicore Test */
    if (test_multicore (hSrioDriver) < 0)
    {
        System_printf ("Error: Multicore Tests failed\n");
        return;
    }

    /* Print out the Malloc & Free Counter */
    System_printf ("Debug(Core %d): Allocation Counter : %d\n", coreNum, malloc_counter);
    System_printf ("Debug(Core %d): Free Counter       : %d\n", coreNum, free_counter);

    /* Check if there is a memory leak? Since we dont implement the DEINIT API
     * there are 'cfg.numRxDesc' and 'cfg.numTxDesc' buffers always allocated for 
     * the SRIO Driver. 1 allocation is there for the Driver Instance
     * Take this into account while we check for leaks? */
    if ((cfg.u.drvManagedCfg.rxCfg.numRxBuffers + cfg.u.drvManagedCfg.txCfg.numTxBuffers + free_counter + 1 ) != malloc_counter)
    {
        System_printf ("Error: Memory Leak Detected\n");
        Task_exit();
    }

    /* Control comes here implies that all testing passed. */
    System_printf ("Debug(Core %d): Multicore Example successful.\n", coreNum);    
    return;
}

/**
 *  @b Description
 *  @n  
 *      Entry point for the example
 *
 *  @retval
 *      Not Applicable.
 */
Void main(Void)
{
    Task_Params     taskParams;

    /* Get the core number. */
	coreNum = CSL_chipReadReg (CSL_CHIP_DNUM);

	while ((CSL_semAcquireDirect (SRIO_APP_HW_SEM)) == 0);
    CACHE_invL1d ((void *) &firstToRun, 128, CACHE_WAIT);
	if(firstToRun == 0)
	{
		System_printf("\n\n\n\n******** Welcome to the Multicore Loopback Left Side Test ********* \n");
		firstToRun = 1;
	    CACHE_wbL1d ((void *) &firstToRun, 128, CACHE_WAIT);
		System_printf("Debug(Core %i) firstToRun %i\n",coreNum,firstToRun);
	}
    CSL_semReleaseSemaphore (SRIO_APP_HW_SEM);

    System_printf ("****************************************************\n");
    System_printf ("******  Multicore Loopback Testing (Core %d) *******\n", coreNum);
    System_printf ("****************************************************\n");

    /* Initialize the heap in shared memory. Using IPC module to do that */ 
    Ipc_start();

    /* Initialize the system only if the core was configured to do so. */
    if (coreNum == CORE_SYS_INIT)
    {
        System_printf ("Debug(Core %d): System Initialization for CPPI & QMSS\n", coreNum);

        /* System Initialization */
        if (system_init() < 0)
            return;
        
        /* Power on SRIO peripheral before using it */
        if (enable_srio () < 0)
        {
            System_printf ("Error: SRIO PSC Initialization Failed\n");
            return;
        }
        
	    /* Device Specific SRIO Initializations: This should always be called before
         * initializing the SRIO Driver. */
    	if (SrioDevice_init() < 0)
    	{
    		System_printf ("Error: SRIO Hardware Initialization Failed\n");
    		return;
    	}
        /* Initialize the SRIO Driver
         * Called only by one core.
         */
        if (Srio_init () < 0)
        {
            System_printf ("Error: SRIO Driver Initialization Failed\n");
            return;
        }

    	/* SRIO Driver is operational at this time. */
        System_printf ("Debug(Core %d): SRIO Driver has been initialized\n", coreNum);

        /* Write to the SHARED memory location at this point in time. The other cores cannot execute
         * till the SRIO Driver is up and running. */
        isSRIOInitialized = 1;

        /* The SRIO IP block has been initialized. We need to writeback the cache here because it will
         * ensure that the rest of the cores which are waiting for SRIO to be initialized would now be
         * woken up. */
 //       CACHE_wbL1d ((void *) &isSRIOInitialized, 128, CACHE_WAIT);
        Cache_wb ((void *) &isSRIOInitialized, 4, Cache_Type_L1D, TRUE);
    }
    else
    {
        /* TODO: Testing this.... */
        System_printf ("Debug(Core %d): Waiting for SRIO to be initialized.\n", coreNum);

        /* All other cores loop around forever till the SRIO is up and running. 
         * We need to invalidate the cache so that we always read this from the memory. */

//        while (isSRIOInitialized == 0)
//            CACHE_invL1d ((void *) &isSRIOInitialized, 128, CACHE_WAIT);


        while (isSRIOInitialized == 0)
        {
          Cache_inv ((void *) &isSRIOInitialized, 4, Cache_Type_L1D, TRUE);
        }
        
        System_printf("Debug(Core %d): SRIO Initialized.\n",coreNum);

        /* Start the QMSS. */
        if (Qmss_start() != QMSS_SOK)
        {
            System_printf ("Error: Unable to start the QMSS\n");
            return;
        }

        System_printf ("Debug(Core %d): SRIO can now be used.\n", coreNum);
    }

    /* Create the Multicore Test Task. */
    Task_Params_init(&taskParams);
    Task_create(multicoreTestTask, &taskParams, NULL);

    /* Start the BIOS */
    BIOS_start();
}

0 Brandy Jabkiewicz over 12 years ago in reply to Brandy Jabkiewicz

Mastermind 6325 points

Hello Derek,

For Q1: It turns out my switch did not have the input and output port enabled. I fixed this with maintanence packets.

For Q5: you recommended via email, looking at these registers to see what kind of errors are occuring when the packet is transmitted.

Processors

Processors forum

C6678 Understanding the SRIO Multicore Loopback example and srio driver