Pcie Maximum Read Request Size ep

Hi,

First of all, in C66x PCIe, BAR0 is fixed to be mapped to PCIe application registers space (starting from 0x2180_0000) in both RC and EP modes.

If you want to do data transfer, you change choose to use BAR1 in RC mode (32-bit addressing). And the PCIe user guide (SPRUGS6) and PCIe use case application note (SPRABK8) should have the examples of BAR usage and inbound translation setup.

If you intend to read the values from PCIe MMR space via BAR0, the PCIe address (maybe the source address of ezdma in EP)  should match the BAR0 value in RC.

Please note that only bits [31:20] in BAR0 are configurable. SPRUGS6 Rev.C should have some update on this. 

If you still see the error, could you please share your setup of the ezdma and PCIe BAR0 (or BAR1 and inbound transaltion registers setup, if you decide to test memory region instead MMR region) ?

Hi,

as you said, the maximum read request size which the DSP can handle is 256 bytes. You can not request more than this for one TLP. I'm not sure how the ezdma splits up a transfer of 8MB. Maybe you should take a look at the Max_Read_Request_Size value in the Device Control Register of your FPGA. The PCIe default value is 512 bytes.

Ralf

Hi,

sorry steven I used BAR1 and not BAR0. I don't know why I have wrote that I use BAR0.

I post the configuration now and hope that it could help you.

RC side DSP

memset (&setRegs, 0, sizeof(setRegs));

memset (&getRegs, 0, sizeof(getRegs));

getRegs.statusCmd = &statusCmd; //status_command reg page 133

if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)

{System_printf ("Read Status Comand register failed!\n");

return retVal;

}

statusCmd.memSp = 1;

statusCmd.busMs = 1;

statusCmd.resp = 0;

statusCmd.serrEn = 1;

setRegs.statusCmd = &statusCmd;

if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)

{

System_printf ("SET Status Command register failed!\n");

return retVal;

}

memset (&setRegs, 0, sizeof(setRegs));

memset (&getRegs, 0, sizeof(getRegs));

getRegs.devStatCtrl = &devStatCtrl; //DEV_STAT_CTRL page 166

if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_LOCAL, &getRegs)) != pcie_RET_OK)

{

System_printf ("Regad Device Status Control register failed!\n");

return retVal;

}

devStatCtrl.reqRp = 1;

devStatCtrl.fatalErRp = 1;

devStatCtrl.nFatalErRp = 1;

devStatCtrl.corErRp = 1;

devStatCtrl.maxSz = 1;

setRegs.devStatCtrl = &devStatCtrl;

if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_LOCAL, &setRegs)) != pcie_RET_OK)

{

System_printf ("SET Device Status Control register failed!\n");

return retVal;

}

/* Configure Address Translation */

barCfg.location = pcie_LOCATION_LOCAL;

barCfg.mode = pcie_RC_MODE;

barCfg.base = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)PCIeBAR1);

barCfg.prefetch = pcie_BAR_NON_PREF;

barCfg.type = pcie_BAR_TYPE32;

barCfg.memSpace = pcie_BAR_MEM_MEM;

barCfg.idx = PCIE_BAR_IDX_M;

if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK)

{

System_printf ("Failed to configure BAR (%d)\n", (int)retVal);

exit(1);

}

EP side

memset (&PCIeDeviceSatCtrlReg, 0, sizeof(PCIeDeviceSatCtrlReg));

memset (&setRegs, 0, sizeof(setRegs));

PCIeDeviceSatCtrlReg.maxPayld = 1; // 000 = 128 001 = 256

PCIeDeviceSatCtrlReg.maxSz = 1;

setRegs.devStatCtrl = &PCIeDeviceSatCtrlReg;

if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs)) != pcie_RET_OK)

{

System_printf ("Read CMD STATUS register failed!\n");

}

memset (&PCIeCmdReg, 0, sizeof(PCIeCmdReg));

memset (&setRegs, 0, sizeof(setRegs));

PCIeCmdReg.memSp = 1;

PCIeCmdReg.busMs = 0;

PCIeCmdReg.serrEn = 1;

/* set everything to 1 */

PCIeCmdReg.parError = 0;

PCIeCmdReg.capList = 1;

PCIeCmdReg.resp = 1;

PCIeCmdReg.ioSp = 1;

setRegs.statusCmd= &PCIeCmdReg;

if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &setRegs)) != pcie_RET_OK)

{

System_printf ("Read CMD STATUS register failed!\n");

}

To read less than 256 datawords work fine. To start the ezdma I write in 4 datawords in pcie ep bar0 and the ezdma use then to start the work.

volatile UInt32 *bar1remote = (UInt32 *)0x60000000;

bar1remote[8] = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)PCIeBAR1); //PCIE LSB ADDRESS

bar1remote[9] = 0x0; //PCIE MSB ADDRESS

bar1remote[10] = 0x00000100; //datawords to transfer

bar1remote[11] = 0x00000014; //start ezdma

I don't know why it doesn't work with more than 256 datawords.

By the way I have I further question. First I tried to use inbound transfer. The configuration was

barCfg.location = pcie_LOCATION_LOCAL;

barCfg.mode = pcie_RC_MODE;

barCfg.base = PCIE_IB_LO_ADDR_M

barCfg.prefetch = pcie_BAR_NON_PREF;

barCfg.type = pcie_BAR_TYPE32;

barCfg.memSpace = pcie_BAR_MEM_MEM;

barCfg.idx = PCIE_BAR_IDX_M;

if ((retVal = Pcie_cfgBar(handle, &barCfg)) != pcie_RET_OK)

{

System_printf ("Failed to configure BAR (%d)\n", (int)retVal);

exit(1);

}

ibCfg.ibBar = PCIE_BAR_IDX_M; //Match BAR that was configured above//BAR1

ibCfg.ibStartAddrLo = PCIE_IB_LO_ADDR_M;//0x90000000

ibCfg.ibStartAddrHi = PCIE_IB_HI_ADDR_M;//0

ibCfg.ibOffsetAddr = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)PCIeBAR1);

ibCfg.region = PCIE_IB_REGION_M;//1

System_printf("pcie_bar1 is %08x\n", (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)PCIeBAR1));

if ((retVal = pcieIbTransCfg(handle, &ibCfg)) != pcie_RET_OK)

{

System_printf ("Failed to configure Inbound Translation (%d)\n", (int)retVal);

exit(1);

}

else

{

System_printf ("Successfully configured Inbound Translation!\n");

}

but if I use inbound transfer and try to read bar1 I get always the CPL CA error. I'm not sure if the configuration is right.

Hope the code help you further.

Regards

1. For PCIe device, "bus master" bit in cmd register should be set to 1 even in the EP mode (different from convention PCI slave device). It will enable EP to issue the memory/IO/message  transactions.

So on EP side, you could try "PCIeCmdReg.busMs = 1;" instead of "PCIeCmdReg.busMs = 0;"

2. It looks like you setup the EP (FPGA) registers from RC (DSP) side. Have you checked on the EP side after the configuration write from RC that those registers has been indeed configured correctly?

3. Have you tried to use the default setup in RC (DSP) and use 128B as max payload size (maxPayld=maxSz=0)  in EP to see if there is still the limitation of data transfer size please?

4. For the question of the inbound transfer setup, the setup on RC side seems fine. On the EP side, you should issue the PCIe packets with PCIe address matching the RC BAR1 value (barCfg.base=PCIE_IB_LO_ADDR_M). The "PCIeBAR1" should be only used on RC side as inbound address translation offset.

So are you using the following command for the ezdma setup on EP side please?

bar1remote[8] = (uint32_t)pcieConvert_CoreLocal2GlobalAddr ((uint32_t)PCIE_IB_LO_ADDR_M); //PCIE LSB ADDRESS

Hi Steven,

first i would like to thank you for you great help and fast answer.

1. I set the ep to busMs = 1 but this setting doesn't change my problem.

2. I setup the EP(FPGA) registers from RC (DSP) and checked that they has been configured correctly.

3. Yes, but the problem was the same

4. no I have used the following command and get the error. bar1remote[8] = (uint32_t)PCIE_IB_LO_ADDR_M; //PCIE LSB ADDRESS To the main problem.

After testing of you suggestions I am now sure that the problem is in the ezdma ip core.

Regards