EDMA PCIe transfer - Linux driver reference request

After studying the epdrv application, I learned that I was not doing the inbound/outbound address translation. I also found this paper: http://www.ti.com/lit/ug/sprugs6a/sprugs6a.pdf . Hopefully it will help me kick start.

Regards,

Jacek

Hi,

I am still struggling with the problem. I found this in sprugx8.pdf: 17.2.3.2.1 BAR0 Exception for In-Bound Address Translation, and changed the device endpoint and driver initialization to use BAR1. I also tried to handle the inbound translation, but it didn't work. Are the following  values acceptable?

Anyone?

Welcome to EDMA PCIe club! I'm working on this as well. My best reference so far is the Linux Device Drivers ebook. combined with TI's EDMA example which I stumbled upon in the EZSDK (it isn't listed in the software developer's guide - that would be far too easy) after a week of banging my head with the TRM.

Can you provide your code and I might be able to fix the bits that are missing/wrong. I would start by trying to read/write to a buffer created entirely in kernel space using dma_alloc_coherent using EDMA, and only once you've got that working would work on copying/mapping userspace data into kernel space.

Ralph

Dear _Ralph_,
Firstly thank you for your reply. I have stripped the driver from unnecessary stuff (like mmap, ioctl ...). The driver is a mash up of my previous drivers made from scratch (LDD + Linux/Documentation/PCI/), but it also may contain code from the Xilinx driver. For me it would be more than enough if I had the data in the driver allocated memory.

The dma part happens in fpga_char_read. My current output looks like this:

[FPGAPCIE] base_start=0x20000000;  base_len=0x00100000
[FPGAPCIE] region=0xd7a00000;
[FPGAPCIE] Device IRQ: 48
[FPGAPCIE] Read Buffer Allocation: FFC14000->89DAE000
[FPGAPCIE] Write Buffer Allocation: FFC18000->89D98000
[FPGA_PCIE] Claimed a pcie device: 0000:01:00.0
root@dm816x-evm:~# ./read.out
[FPGA_PCIE] Device opened
IB_OFFSET = 0x20000000
IB_START_HI = 0
IB_START_LO = 61616161
bar = 1
DMA Started
.............................................................................................................................................................
 From Callback 1: Channel 4 status is: 1
FPGA memory : OK
Should be (int=97) 'a', read:(int=0) '

[FPGA_PCIE] Device released
->

Attached : pcie.c
Thank you in advance for taking your time.

Regards,

Jacek

Firstly your calculation of base_len is wrong and needs brackets around the first 2 operands.

I use request_mem_region instead of pci_request_regions but perhaps I should be using the latter instead.

You set the DMA mask to 64 bits which won't help as the TI PCIe driver can only handle 32 bits of addresses, even when you are dealing with a 64 bit BAR.

Why do you have read _and_ write buffers allocated for DMA? Surely you only need 1 DMA area; the other area of memory will be the memory mapped PCIe memory.

That's my assessment of the probe function. I'll get onto the rest eventually if you can check the above.

Ralph

Hi,

I changed that line to pcie_descr.base_len = pci_resource_len(dev,1); http://lxr.linux.no/#linux+v3.1.5/include/linux/pci.h#L1319  and got the same value. I don't see any difference with or without brackets.

Changing to request_mem_region didn't help either.
The pcie_request_method makes checks pci data (start, len etc.) and then calls request_mem_region or request_region depending on pcie flags. Besides request_mem_region returns a resource structure instead of a SUCCESS/ERR integer.

Changed to  pci_set_dma_mask(dev, DMA_BIT_MASK(32));
You are absolutely right - thank you for pointing this out.

As I said, I used my previous drivers to get it working ASAP- thats why there are two buffers for ping-pong transactions. I commented all of readBuffer to avoid mistakes.

Regards,

Jacek

Before I look at the init function, can I just ask if you have successfully read and written to your PCIe device not using EDMA? Also, what BAR setup do you have with your endpoint?

Ralph

Hi,

I have successfully read and written (without EDMA) to the PCIe device. Currently the device read looks likes this ->standard write (writel)some data, EDMA, print EDMA transfer data, standard read (readl) data and check if it is the same which was fed at the beginning. 

My Bar setup is bar0 - unused, bar 1 -> 32bit 1 Mbytes of memory. I have also tried using bar0.
I am using a Xilinx ml507 board with Endoint block plus for PCIexpress v. 1.14.

The producer allows (after some changes) to scope what is happening on the device, so hopefully I will be able to say today whether the device is transmitting something during EDMA or not.

Regards
Jacek

Looking at the read function, I'll be honest and say I've never used readl/writel and I didn't use them in my EDMA clone.

For edma_set_* I used W8BIT instead of W64BIT which you used.

For edma_alloc_channel I used "EVENTQ_DEFAULT" for the 4th parameter instead of 0. From the EDMA source code: "use EVENTQ_DEFAULT unless you need a really high priority queue".

At the end of the read function you should increment f_pos by the number of bytes you read.

Maybe changing these things will make it work?

Ralph

Changed everything as you suggested - no changes.

I have scoped the fpga and I am sure that the edma reads something (and it isn't zeros) from the device. The memory line got touched 4 times during the DMA, which is what I would expect: requesting 512, and the max TLP size for the EP is 128. It looks like a memory access problem.

_Ralph_ said:

Looking at the read function, I'll be honest and say I've never used readl/writel and I didn't use them in my EDMA clone.

So how have you done the inbound address translation? Do you set the IB_offset, IB_START_HI/LO registers? Is it necessary? Do you do any extra initialization?

Currently I am planning to change the TI epdrv (rc to ep) example. Maybe having a user application(which I made myself)  with two checked and working drivers (PCIe and EDMA), will let me run the EDMA successfully and diagnose the problem.

Regards

Jacek

So it looks like I misunderstood the epdrv example and I was not accessing the device PCI registers as I should have, thus the memory translation never happened. I'm investigating this further.

Regards Jacek

I can't assist until after Christmas but maybe someone from TI could help. Also, perhaps try posting on the Linux specific forum.

Ralph

Thank You very much for your assistance. Merry Christmas

I will keep posting my progress, so sorry for spamming your inbox.

Regards Jacek

I repaired my code so it should now access the device registers - using the ioremap value.

http://processors.wiki.ti.com/index.php/DM81xx_AM38xx_PCI_Express_Endpoint_Driver_User_Guide
I have followed ti81xx_ep_app.c, lib/ti81xx_ep_lib.c , lib/ti81xx_trans.c, ti81xx_edma.c, ti81xx_edma.c  back and forth. I believe my driver is now working the same way. The only difference is the memory allocation done on system boot with the pcie_mem=8M argument, against my memory allocation (dma_alloc_coherent or kmalloc) done on driver initialization.

New code attached.

Regards

Jacek

Hi,

I am still fighting with the EDMA. I have just changed my driver to use ti81xx_ep_setup_bar from ti81xx_pcie_bootdrv.c to prepare the memory translation, but still no luck.  Is there really no straightforward way of doing this?

Dear _Ralph_ a few posts back you mentioned that you didn't use writel/readl. Which function did you use for memory translation?

Regards

Jacek

Short summary of what i have:
-the EDMA is done (completion interrupt is raised)
-the device works - writel/readl works
-the source address is right - I scoped what is going in the device, and it seams to prepare data to send back just when the EDMA starts
-the destination address is right - When copying from memory, the destination memory gets populated

I think my inbound memory translation is wrong at some point, although I have tried using code from TI examples and writing my own.

Regards

Jacek

By inbound memory translation do you mean mapping the user-space buffer to DMA memory? I haven't done this yet myself, instead for the time being I am using copy_from_user and copy_to_user.

Ralph

No, we haven't understood each other.
I meant the transfer device->davinci, not kernel->userspace. I never got any data in the driver buffer - only zeroes.

"PCIe devices need to use PCIe addresses to send/receive packets over a PCIe link. The
Address Translation Unit (ATU) within the PCIe module translates the device internal
address into a PCIe address and vice versa. The PCIe address could be 32-bit or 64-bit
(legacy EP may not support 64-bit addresses).
...
For the inbound transaction, the Base Address Register (BAR) in the PCIe module
accepts certain PCIe addresses and rejects the others. The data with an accepted PCIe
address goes through the inbound ATU and is transferred to the device internal
memory after address translation"
from http://www.ti.com/lit/an/sprabk8/sprabk8.pdf

The copy_to_user part I would like to do with mmap, but it is for the future. For now I am unable to read from the device using EDMA.

Regards
Jacek

Not sure exactly what you mean but the address of the only BAR in the DM8168 is presented to the PCIe peripheral with a start address of 0x20000000 (see arch/arm/plat-omap/include/plat/ti81xx.h) and size of 256MB (see arch/arm/mach-omap2/devices.c) if that helps...

Yes, 0x20000000 is the address of the BAR on the system (and the address which is returned by pci_resource_start(dev, 1) for BAR1 which is used by my device to avoid using BAR0 ), and it is the address I used in edma_set_src. I meant that the device and davinci might use different addresses in the PCIe bus, which are finally translated to 0x20000000, or something else...Once again:
"
For the inbound transaction, the Base Address Register (BAR) in the PCIe module
accepts certain PCIe addresses and rejects the others. The data with an accepted PCIe
address goes through the inbound ATU and is transferred to the device internal
memory after address translation
" from http://www.ti.com/lit/an/sprabk8/sprabk8.pdf

I believe this is my problem.

1.What address do you use in edma_set_src? is it 0x20000000?

2. are this the only EDMA preparation steps you make? ->

edma_alloc_channel, edma_set_src, edma_set_dest,edma_set_src_index,edma_set_dest_index, 

edma_set_transfer_param ,

edma_read_slot (dma_ch, &param_set) ,change param_set,edma_write_slot (dma_ch, &param_set);

and finally edma_start ?

Thank you for your reply

Regards

Jacek

1. What you get from pci_resource_start() assuming you are reading from the PCIe device (might be 0x20000000, who knows), otherwise for writing to PCIe you pass in the bus address (==physical address) of your Linux DMA-able memory.

2. As in TI's EDMA memory-memory copy example, the answer is yes.

Ralph

I have the same situation with reading data from FPGA through PCIe with EDMA (c6a8168 processor). When I set acnt = 4, bcnt = 1, ccnt = 1 (read 4 bytes from FPGA in one transaction) I get valid data. But if I set acnt more than 4, I get all zeros. Does somebody found the solution?