C6678 PCIe throughput

Murad,

I am not the expert for PCIe, but the PCIe expert can be clarified for your PCIe related query.

Murad: said:

I am planning to use EDMA, but I never used DMA before and I need sometime to do that...but what am I doing wrong... can I change the outbound translation to speed up the transfer before employing EDMA?

Hi Murad,

Better to increase the payload size and use the EDMA for PCIe transfer test code.

EDMA base PCIe test code is available in MCSDK package. Please take a look at section in test code.

MCSDK Path: \ti\mcsdk_2_01_02_06\tools\boot_loader\examples\pcie\linux_host_loader\pciedemo.c

Refer readme(C:\ti\mcsdk_2_01_02_06\tools\boot_loader\examples\pcie\docs\) document for more information.

Thanks,

Can someone help please!

My code works fine when I use the CPU to read FPGA memory (3072 integers) using PCIe inside a timer interrupt as follows:

rdPtr = (int *)&algInputBuff[0];

for (k = 0; k < (3072); k++)

*rdPtr++ = *((volatile uint32_t *)pcieDataBase + PCIe_SDPRAM_block_count*(3072) + k);//3072 32-bit samples

BUT when I use the EDMA3, I get nothing!!!???...my code is shown below (not sure how to attach to this post). I have 4 blocks inside the FPGA SDPRAM, each block is 3072 integers and I am using linked EDMA PaRAM.

the folloing are defined in my edma.h

/* Define payload size and buffer size */

#define PCIe_MAX_PAYLOAD   (128)   // Assume pcie maximum payload size is 128 bytes

#define FPGA_DATA_BLOCK_SIZE_BYTES (3072) // 3072-32-bit (16 bits I and 16 bits Q)

// Define PCIe data starting address for EDMA transfer

#define PCIe_DATA_ADDR (0x60000000)

edma.c

#include "edma3.h"

extern uint32_t pcieConvert_CoreLocal2GlobalAddr (uint32_t addr);

CSL_Edma3ChannelHandle hChannel;

int cfg_edma3_INPUT(int *destAddr)

{

CSL_Edma3Handle hModule;

CSL_Edma3ParamHandle hEdma_blk0;

CSL_Edma3ParamHandle hEdma_blk1;

CSL_Edma3ParamHandle hEdma_blk2;

CSL_Edma3ParamHandle hEdma_blk3;

CSL_Edma3Obj edmaObj;

CSL_Edma3ChannelObj chObj;

//CSL_Edma3ChannelHandle hChannel;

CSL_Edma3ChannelAttr chAttr;

CSL_Edma3Context context;

CSL_Status status;

CSL_Edma3ParamSetup myParamSetup;

Uint8 instNum = 2;

Uint8 channelNum = 0;

Uint8 paRAM_number = 0;

// Initialize the EDMA3 Module.

if (CSL_edma3Init(&context) != CSL_SOK)

{

platform_write ("Error: EDMA module initialization failed\n");

return -1;

}

// Open the EDMA Module using the provided instance number

hModule = CSL_edma3Open(&edmaObj,instNum,NULL,&status);

if ( (hModule == NULL) || (status != CSL_SOK))

{

platform_write ("Error: EDMA module open failed\n");

return -1;

}

// Open DMA Channel "channelNum".

chAttr.regionNum = CSL_EDMA3_REGION_GLOBAL;

chAttr.chaNum = channelNum;

hChannel = CSL_edma3ChannelOpen(&chObj, instNum, &chAttr, &status);

if ((hChannel == NULL) || (status != CSL_SOK))

{

platform_write("Error: Unable to open EDMA Channel:%d\n", chAttr.chaNum);

return -1;

}

// /* For first EDMA instance there are only 2 TCs and 2 event queues

// * Modify the channel default queue setup from 0 to 0?

// */

// if (CSL_edma3HwChannelSetupQue(hChannel,CSL_EDMA3_QUE_0) != CSL_SOK)

// {

// platform_write ("Error: EDMA channel setup queue failed\n");

// return -1;

// }

/* Map the DMA Channel to PARAM Block 0. */

CSL_edma3MapDMAChannelToParamBlock(hModule, channelNum, 0);//MQ: ??

// CSL_edma3HwChannelSetupParam(hChannel, paRAM_number);

// Obtain a handle to PARAM Entry 0

hEdma_blk0 = CSL_edma3GetParamHandle(hChannel,0,&status);

if (hEdma_blk0 == NULL)

{

platform_write ("Error: EDMA Get Parameter Entry failed for 0.\n");

return -1;

}

// Obtain a handle to PARAM Entry 1

hEdma_blk1 = CSL_edma3GetParamHandle(hChannel,1,&status);

if (hEdma_blk1 == NULL)

{

platform_write ("Error: EDMA Get Parameter Entry failed for 1.\n");

return -1;

}

// Obtain a handle to PARAM Entry 2

hEdma_blk2 = CSL_edma3GetParamHandle(hChannel,2,&status);

if (hEdma_blk2 == NULL)

{

platform_write ("Error: EDMA Get Parameter Entry failed for 2.\n");

return -1;

}

// Obtain a handle to PARAM Entry 3

hEdma_blk3 = CSL_edma3GetParamHandle(hChannel,3,&status);

if (hEdma_blk3 == NULL)

{

platform_write ("Error: EDMA Get Parameter Entry failed for 3.\n");

return -1;

}

////////////////////// Setup the first param Entry (Ping buffer) //////////////////////

myParamSetup.option = CSL_EDMA3_OPT_MAKE(CSL_EDMA3_ITCCH_DIS, \

CSL_EDMA3_TCCH_DIS, \

CSL_EDMA3_ITCINT_DIS, \

CSL_EDMA3_TCINT_DIS,\

0,CSL_EDMA3_TCC_NORMAL,\

CSL_EDMA3_FIFOWIDTH_NONE, \

CSL_EDMA3_STATIC_DIS, \

CSL_EDMA3_SYNC_AB, \

CSL_EDMA3_ADDRMODE_INCR, \

CSL_EDMA3_ADDRMODE_INCR);

myParamSetup.srcAddr = (Uint32)PCIe_DATA_ADDR;

myParamSetup.aCntbCnt = CSL_EDMA3_CNT_MAKE(PCIe_MAX_PAYLOAD,4*FPGA_DATA_BLOCK_SIZE_BYTES/PCIe_MAX_PAYLOAD);

myParamSetup.dstAddr = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)destAddr);

myParamSetup.srcDstBidx = CSL_EDMA3_BIDX_MAKE(PCIe_MAX_PAYLOAD,PCIe_MAX_PAYLOAD);

myParamSetup.linkBcntrld = CSL_EDMA3_LINKBCNTRLD_MAKE(hEdma_blk1,0);

myParamSetup.srcDstCidx = CSL_EDMA3_CIDX_MAKE(0,0);//MQ: was (0,1)

myParamSetup.cCnt = 1;

// Configure the PARAM Entry with the setup information.

if (CSL_edma3ParamSetup(hEdma_blk0,&myParamSetup) != CSL_SOK)

{

platform_write ("Error: EDMA Parameter Entry 0 Setup failed\n");

return -1;

}

////////////////////// now set up ping block 1 //////////////////////

myParamSetup.srcAddr = (unsigned int)(PCIe_DATA_ADDR + FPGA_DATA_BLOCK_SIZE_BYTES);

myParamSetup.linkBcntrld = CSL_EDMA3_LINKBCNTRLD_MAKE(hEdma_blk2,0);

// Configure the PARAM Entry with the setup information.

if (CSL_edma3ParamSetup(hEdma_blk1,&myParamSetup) != CSL_SOK)

{

platform_write ("Error: EDMA Parameter Entry 1 Setup failed\n");

return -1;

}

////////////////////// now set up pong block 2 //////////////////////

myParamSetup.srcAddr = (unsigned int)(PCIe_DATA_ADDR + 2*FPGA_DATA_BLOCK_SIZE_BYTES);

myParamSetup.linkBcntrld = CSL_EDMA3_LINKBCNTRLD_MAKE(hEdma_blk3,0);

// Configure the PARAM Entry with the setup information.

if (CSL_edma3ParamSetup(hEdma_blk2,&myParamSetup) != CSL_SOK)

{

platform_write ("Error: EDMA Parameter Entry 2 Setup failed\n");

return -1;

}

////////////////////// now set up ping block 3 //////////////////////

myParamSetup.srcAddr = (unsigned int)(PCIe_DATA_ADDR + 3*FPGA_DATA_BLOCK_SIZE_BYTES);

myParamSetup.linkBcntrld = CSL_EDMA3_LINKBCNTRLD_MAKE(hEdma_blk0,0);

// Configure the PARAM Entry with the setup information.

if (CSL_edma3ParamSetup(hEdma_blk3,&myParamSetup) != CSL_SOK)

{

platform_write ("Error: EDMA Parameter Entry 3 Setup failed\n");

return -1;

}

// Enable Channel

if (CSL_edma3HwChannelControl(hChannel,CSL_EDMA3_CMD_CHANNEL_ENABLE,NULL) != CSL_SOK)

{

platform_write ("Error: EDMA Channel Enable failed\n");

return -1;

}

return(0);

} /* end cfg_edma */

Regards,

Murad

When you use CCS "Memory Browser" to read data from 0x60000000, the debugger reads data using DSP core on 4 bytes.

It's the same thing that your code for the DSP core :

rdPtr = (int *)&algInputBuff[0];
for (k = 0; k < (3072); k++)
*rdPtr++ = *((volatile uint32_t *)pcieDataBase + PCIe_SDPRAM_block_count*(3072) + k);//3072 32-bit samples

Dear Dmitry,
I thought the same.
- I changed the max payload size on the DSP size to 4 which affects the aCnt but nothing changed.
- The PCIe core on the FPGA allows to select the max payload size using a GUI and it is been set to 512 bytes. C6678 EDMA3 supports 128/64 bytes. I am expecting the payload size to affect speed performance, but not to get nothing from the EP!?. -Also, the FPGA PCIE for Spartan6 is v1.1 while C6678 PCIe is 2.0...do you think this can cause a problem with the EDMA3?
-CPU access to PCIe is working without any problem. My second question, Can I have the EP (FPGA) initiate outbound memory write which is Inbound memory write to the C6678 and hence no need for EDMA/CPU (SPRABK8.pdf):
"Inbound transfer means the external device initiates the transactions to write to or read
from the local device. The PCIe module has a master port to transfer the data to or from
the device memory; no CPU or EDMA is needed for inbound transfer in the local
device."

You're right Murad.
The EDMA3 can only work with data burst size (DBS) of 64 or 128 bytes. The value 4 not be able to use (I had not thought about it. Sorry.).

I do not think you have a problem versions of PCIe, as the DSP core can read data from the FPGA.

Change the logic of your test. Check the writing data using EDMA3.
Create an array of data in the local memory of DSP, it must initialize the values and move it using EDMA3 in FPGA. 
Then read the same data from the FPGA memory using the core DSP (without EDMA3).
These look like?

If you want to write data from the FPGA to the DSP memory , you must use the IP-core of CDMA in hardware platform for FPGA. The CDMA can manage from the DSP, mapped CDMA registers control with BAR-0 on the FPGA.

Hello Murad,

You are using a hardware platform for FPGA, which can only work with single DWORD payload  ( DWORD in PCI-Express - 4 bytes).

The EDMA3 at DSP will not work.
You need a different hardware platform for FPGA.

Look up page 129 in this document:

http://www.xilinx.com/support/documentation/user_guides/s6_pcie_ug654.pdf

You see:

**************

Supports single DWORD payload Read and Write PCI Express transactions to 32/64-bit address memory spaces and I/O space with support for completion TLPs

**************

Regards,

Dmitry.