AM5K2E02: Sending data over PCI Express that spans several TLPs

Hello!
To my knowledge, both LOAD and STORE instructions to PCIe data space are mapped to single DWORD MRd/MWr transactions. Whenever you want transfer more than one DWORD of payload, you have to use DMA, EDMA3 in our case. Then if the size of the block to be transferred exceeds maximum payload size of either sender or receiver, it gets sliced in smaller pieces. Note, that among multiple transfer controllers of every channel controller only certain support large payloads, like 128 bytes.
Also be aware, that according to my observations, at least in C6670 TC0 of EDMA3CC0 does not issue 128B writes, but 64B instead. This fact is indirectly confirmed in sprabk8 PCIe Use Cases for KeyStone Devices, where read performance is better, than write.
So the answer is: larger blocks have to be sliced, but that is implemented by EDMA hardware, rather than user code.

Thank you for a quick response.
We are already using DMA .
For a write, the data are copied into the memory using a loop:
for (ui32ElemCounter = 0;
(ui32ElemCounter < ui32DataLength) && (ui32ElemCounter < C_PCIE_LOCAL_BUFFER_SIZE);
ui32ElemCounter += 1)
{
p_ui32Destination[ui32ElemCounter] = p_ui32Data[ui32ElemCounter];
returnValue += 1;
}

for a read
for (ui32ElemCounter = 0;
(ui32ElemCounter < ui32BufferLength) && (ui32ElemCounter < C_PCIE_LOCAL_BUFFER_SIZE);
ui32ElemCounter += 1)
{
p_ui32Buffer[ui32ElemCounter] = p_ui32Source[ui32ElemCounter] ;
returnValue += 1;
}

Using this mechanism, we can transfer up to 64 words (256 bytes) but it fails when 65 are sent.

dan

Hi, I was assuming that DMA had been set up as I could write data directly to the memory area. I have also tried using memcpy, which produces the same result.
The symptom is that using either method, I can send up to 64 words, but no more.
Thanks for your help.

dan

Hi,

thanks for the help. I have now added in a call to EDMA3 that uses a PaRAM register structure to set the transfer.

I am using EDMA3 CC0 which I think is correct for PCIe.

I am still limited to small transfers (less than 100 words) though.

thanks

dan

Hi again,
I have been doing more investigation.

The EP side is set up to use local buffers for the transfers.
DMA is working correctly for up to 143 words (572 bytes) for output.
The local buffers are set up 256 byte aligned.

volatile uint32_t p_ui32LocalBuffer0[C_PCIE_LOCAL_BUFFER_SIZE] __attribute__((aligned(256))) = {0};

The RC side is set up reading and writing memory at PCIE_0_DATA_BASE (0x5000 0000).

The RC side read and write can only access addresses between 0x5000 0000 and 0x5000 0040, which I have tested by inserting an offset before the first address.

The two ends are set up as follows:

RC

BAR0 is not used and is not configured.

BAR1_MASK.BAR_MASK    0x07FF FFFF
BAR1_MASK.BAR_ENABLED 1 (enabled)

The BAR1 addresses are set as follows:
BAR1.MEM_SPACE 0 (Memory BAR)
BAR1.TYPE      0 (32-bit BAR)
BAR1.PREFETCH  0 (Non-prefetchable)
BAR1.BASE_ADDR 0x0900 0000 (0x9000 0000>>4)

The OB_SIZE register is set to 0x00000003 (8MB)

The IB translation is set
IB_BAR0.IB_BAR0            0x0000 0001
IB_START0_LO.IB_START0_LO  0x0090 0000 (0x9000 0000>>8)
IB_START0_HI.IB_START0_HI  0x0000 0000
IB_OFFSET0.IB_OFFSET0      <local buffer address >> 8>
CMD_STATUS.IB_XLT_EN       1

The OB translation is set
OB_OFFSET_INDEX0.OB_OFFSET0_LO 0x00000700 (0x7000 0000>>20)
OB_OFFSET_INDEX0.OB_ENABLE0    1 (Enabled)
OB_OFFSET_INDEX0.OB_OFFSET0_HI 0x00000000
CMD_STATUS.OB_XLT_EN           1

For EP

BAR0 is not used and is not configured.

BAR1_MASK.BAR_MASK    0x07FF FFFF
BAR1_MASK.BAR_ENABLED 1 (enabled)

The BAR1 addresses are set as follows:
BAR1.MEM_SPACE 0 (Memory BAR)
BAR1.TYPE      0 (32-bit BAR)
BAR1.PREFETCH  0 (Non-prefetchable)
BAR1.BASE_ADDR 0x0700 0000 (0x7000 0000>>4)

The OB_SIZE register is set to 0x00000003 (8MB)

the IB translation is set
IB_BAR0.IB_BAR0            0x0000 0001
IB_START0_LO.IB_START0_LO  0x0070 0000 (0x7000 0000>>8)
IB_START0_HI.IB_START0_HI  0x0000 0000
IB_OFFSET0.IB_OFFSET0      <local buffer address >> 8>
CMD_STATUS.IB_XLT_EN       1

the OB translation is set
OB_OFFSET_INDEX0.OB_OFFSET0_LO 0x00000900 (0x9000 0000>>20)
OB_OFFSET_INDEX0.OB_ENABLE0    1 (Enabled)
OB_OFFSET_INDEX0.OB_OFFSET0_HI 0x00000000
CMD_STATUS.OB_XLT_EN           1

Then when a transfer is started,
For the RC
read and write using the PCIE_BASE address (0x5000 0000)

For EP read and write using the local buffer address

Is this enough information?

Hi,

As rrlagic and Yordan explained, if you use code as: p_ui32Destination[ui32ElemCounter] = p_ui32Data[ui32ElemCounter]; This is CPU style read and write, each time you have a 32-bit data payload in the TLP.

To transfer a burst of payload, you must use EDMA. The maximum payload size of K2E outbound is 128 bytes, this is limited by the EDMA transfer controller (TC). The maximum payload size of inbound is 256 bytes, some FPGA device I knew it can generate such kinds of burst. For TI K2E (RC) ----- K2E (EP) setup, there is no way you can generate burst of 256 bytes, so the maximum payload can be tested is 128 bytes.

The question, can you send a big block (much greater than 128 bytes) by EDMA over PCIE interface? Yes, you can do it without any problem. The EDMA HW breaks the data into small chuck of 128 bytes or 64 bytes (depended on which EDMA transfer controller you select) and put that into TLP. This is done by HW, we used EDMA to send a big block (probably several KB or MB) for bench-marking PCIE throughput, there is no any issue. We have such PCIE+EDMA examples already in Processor SDK RTOS PCIE driver, using PCIE LLD driver + EDMA LLD.

I am not sure the reason you like the bare metal one, to develop that, you need to familiar with PCIE and EDMA and look for the existing driver examples for reference.

Essentially, you need two parts:
1) PCIE driver, you need make sure the CPU read/write can work. E.g, on RC side you write to 0x5000_0000, RC side has an outbound translation, it becomes 0x7000_0000 over the PCIE bus. On the EP side, you have BAR setup to accept address range 0x7000_0000. Then there is also an inbound translation setup, to convert 0x7000_0000 into a memory address, e.g, Global L2, MSMC or DDR.

You can run our RTOS example first, make sure it worked. Then check the PCIE registers using CCS memory windows and compare to your bare metal driver.

2) EDMA, I am not sure how you familiar with EDMA? Are you able to write a typical EDMA bare metal code or CSL level code to do a transfer from A to B? In the OPT field, there is ACNT, BCNT, CCNT and A-sync or A-B sync to control how many bytes transmitted a time. If you have a basic EDMA working (e.g, you can transfer 1KB from MSMC to DDR within the chip), then you just need to replace either EDMA source or EDMA destination with a PCIE data space address (like 0x5000_0000) for EDMA over PCIE.

Regards, Eric

Hello!

On RC side OB translation of region 0 brings origin of data window to 0x7000_0000 address, which matches BAR1 of EP and looks like there is IB translation on EP side as well, so at the first glance that looks fine.

What we don't see is how do you configure your EDMA, how do you trigger the transfer and how do you check the result. These are still missing in your explanations. Before we see that, two things to check. First, make sure you use global address of your local buffers to both configure address translations and EDMA3 config. Second, on receiver side make sure to invalidate cache in the range of you local buffer, so you see updated data in your comparison.

Dan,

Do you have any follow-up question or I can close it?

Regards, Eric

I'm still having the same problem. I am working through the suggestions.
I have got the example working which demonstrates that it is possible.
I have been working through the DMA which uses DMA TCs where I have QDMA.
The DMA is working if I set a local address.

I have also observed that the debugger only shows the first 64 words of a global address. Is this to be expected?

thanks

dan

Dan,

I don't understand "The DMA is working if I set a local address." In the EDMA OPT field, there is a DST and SRC, assuming you used the DST and SRC with the address of the chip, it is working. Then what you need to do is replace either DST or SRC with PCIE data space address like 0x5000_0000. It also should work.

I have also observed that the debugger only shows the first 64 words of a global address. Is this to be expected?====> I also don't understand this. If you use CCS memory window, you should see a much bigger memory range as long as the address range is valid.

Regards, Eric

Hi Eric,
When I have a local SRC address and a local DST address in the EDMA OPT field then the transfer takes place as expected.
When I have a local SRC address and the DST is set to 0x5000_0000 then only the first 64 words get transferred. Similarly when the SRC address is 0x5000_0000 and the DST is local only the first 64 words are transferred.

When I open the CCS memory window, and set the address to 0x5000_0000 then the first 64 words are displayed. If I set the value to 0x5000_0100 then an error is returned.

thanks
dan

Hello!
It looks like trouble with PCIe setup. EDMA definitely places multi-dword TLPs, memory browser fetched memory content word by word, to my observations. Perhaps when error displayed for memory browser, there is some description of that error.

Hi ,

When you say that only 64 words are transferred, what is your EDMA ParamSet ACNT, BCNT, CCNT numbers and OPT field A-B sync or A-sync only? Are they big enough (to more than 256 bytes) for a transfer?

Let's say you have board A and board B connected via PCIE. On A side, you wrote data to 0x5000_0000, do you setup the the OB translation for region 0x5000_0000? If yes, what this address translated to? What is the outbound size? 1MB, 2MB ....? On B side, you have a BAR setup to match the incoming address, correct? where this address landed to through inbound translation?

Let say, using CPU write for a big loop:
for (i = 0; i < 1024; i++) {
*(unsigned int *)(0x5000_0000 + 4*i) = i;
}

Then check your CCS memory window. In B side, can you see the data pattern landed? Then in the A side, looking at 0x5000_0000 and scroll down, will you see the memory window break at 0x5000_0100? If yes, please describe what happened? Like saw all zeros, all 0xffff, lost jtag, etc? .....

Then using EDMA case, What is the error when you scroll down to 0x5000_0100?

Regards, Eric