C6670 PCIe RC does not receive 256B writes from EP

Hello,

I am making transfers with DMA engine on FPGA side. It has bus master capability as well. Right now I cannot use EDMA of DSP because there is no matching hardware on the other side. Would appreciate a hint how to debug.

And another point, could you please comment on values of DEV_STAT_CTRL register?

Hi,

the DEV_STAT_CTRL register value of 0x0000281F corresponds to a value of 0 in Bits [7:5] (128 Bytes), doesn't it?

Please note that there is a performance problem for inbound transfers in keystone devices when the max. payload value in DEV_STAT_CTRL is increased to 256 bytes:
https://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/158004

Ralf

Hi Ralf,
You're right, MAX_PAYLD in DEV_STAT_CTRL[7:5] is really zero, corresponding to 128B.
And about MAX_REQ_SZ in DEV_STAT_CTRL[14:12], which is 2=>512B. Does that mean that with default setup DSP is allowed to make requests for 512B at a time, but completions still are expected to be broken in 128B or 256B packets? Not to mention, that in SPRUGS6D in the clause 1.3 Features there is a statement:

Maximum remote read request size of 256 bytes

And thanks for pointing that performance issue topic. Its 3 years old already and we still cannot read that precaution in the manual :-( Actually, it took me efforts and fabric space to increase outbound payload in FPGA. Now I know it did not worth.

Hi,

To my memory we have to configure: 1) BAR index for IB translation; 2) IB LO/HI addresses; 3) Offset; 4) IB translation region. Here is excerpt from our code:

// Inbound Region 0
ibCfg.ibBar         = PCIE_BAR_IDX; // Match BAR that was configured above
ibCfg.ibStartAddrLo = PCIE_IB_R0_LO_ADDR;
ibCfg.ibStartAddrHi = PCIE_IB_R0_HI_ADDR;
ibCfg.ibOffsetAddr  = (u_int32) Convert_coreLocaltoGlobalAddr ( (u_int32) pcie_l2_buf.buf );
ibCfg.region        = PCIE_IB_REGION_0;

pcie_ib_trans_cfg(handle, &ibCfg);

You can easily guess values of those macro constants, and address conversion function is copied from PDK examples. It makes global address for given pointer.

What I can guess, is that your IB_OFFSET setting is not correct. You have to put true address of the receiving buffer there. You have to keep in mind three types of addresses being used with PCIe. The first one is PCIe data memory window. On C6670 which is Keystone I we have that at 0x6000_0000. It can be used directly and many people do so without translation. The second is somewhat virtual PCIe address space. You may think, that accesses to PCIe data window address get translated to PCIe address space. Many examples use 0x9000_0000 address for that. This range on my device is not used by any hardware, so its ok to use it for virtual mapping. Thus, if one define OB (here I speak about OB!), then access to 0x6000_0000 window gets translated to 0x9000_0000 window according to OB translation regions. So see the sequence. You take some buffer or variable and that implicitly defines the third address. Then copying from this DSP buffer to PCIe space is like copying from, say, your L2SRAM or DDR3 to 0x6000_0000 window. PCIe hardware translates that to 0x9000_0000 window, so final receiver gets those data targeted to 0x9000_0000. Beware, many people do not use OB translation at all, I described that just to give an idea how it works.

Transfer in the opposite direction should involve something similar by meaning, however in details. When your EP writes data towards DSP, it writes it to PCIe address space. For instance, that still could be somewhere in 0x9000_0000 window. However, at this time you need to setup DSP's BAR in way, that FPGA's writes match target BAR. If BAR accepts that address, next it goes through IB translation logic. Now we have to map PCIe address to the real receive buffer. If you have some buffer in L2SRAM, you have to define IB_OFFSET with that buffer start address, and that address to be converted to global address. If your receive buffer is in DDR3 or MCSRAM, then its address is already global.

So, once again, assuming your FPGA is writing to 0x8000_0000, and you want those writes to land to some int buf[MANY]; you have to define IB_OFFSET=&buf[0];

Hope this helps.

Hi,

Thank for the response. 

We are able to solve the issue. We were able to see the data landing into DDR area, but still there is some ambiguity wrt physical and virtual address in K2. Apart from this There was an issue with FPGA logic where in TLP packet the address was being sent incorrectly. Once this was corrected, it started working.

Will post the details once it is concluded.

Thank You

Guru