Integrating new device driver into c6x

M S1

I developed a device driver for a PCI device in Ubuntu. My customer now wants this driver ported to work on the TMDSEVM6678L, with the device itself plugged into the EVM.

My approach has been to download the c6x source, add my driver code to the arch/c6x/drivers directory, and modify the Makefile.

When I "./prj build", the compiler doesn't like my calls to pci_msi_enable and pci_msi_disable. The linux-c6x/include/linux/pci.h header file contains prototypes for these functions, so I'm a little mystified.

Is there something I'm missing here? My driver relies heavily on MSI to signal a DMA transfer is complete.

My development environment is Ubuntu 10.04 32-bit. I've got c6x 2.0.0.63, and 7.4.6 CGT.

Thanks in advance,

over 11 years ago

0 Rex Chang over 11 years ago

TI__Guru 50170 points

Hi, MS,

Could you git the latest linux-c6x from the git repositories in linux-c6x.org? PCIe support was added after 2.0.0.63 was released.

Rex

0 M S1 over 11 years ago in reply to Rex Chang

Prodigy 170 points

Thanks Rex, but I'm getting the same failure. Do you know if the standard PCI library functions are supported in c6x? Specifically, the build is unhappy about:

pci_enable_msi

pci_disable_msi

pci_iounmap

pci_iomap

The specific compile error I'm getting is: error: implicit declaration of function 'pci_enable_msi'

Thanks,

0 Rex Chang over 11 years ago in reply to M S1

TI__Guru 50170 points

Hi, MS,

It seems to me that compilation errors are caused by not having the header file for the prototype defintion. C6x also uses, for example, a grep on pci_iounmap will get

arch/c6x/include/asm/io.h:extern void pci_iounmap(struct pci_dev *dev, void __iomem *addr);

pci_enable/disable_msi is defined in include/linux/pci.h

Am I missing something?

Rex

0 M S1 over 11 years ago in reply to Rex Chang

Prodigy 170 points

Am I missing something?

That's the same question I was asking myself. In my source code, I am #including linux/pci.h. If the compiler couldn't find the header file, it would complain about that, and it isn't. In addition, my code uses a number of other pci calls such as pci_resource_start, etc. which the compiler has no problem with.

On the other hand, if the header file is present but the library itself is missing, the link will fail but the compilation will succeed. This isn't the case here either.

I don't fully understand the c6x build system, but I started looking through the .mk files for the different architectures. I noticed that for some architectures like x86, there is a USE_MSI macro defined, but not for the 6678. Does the 6678 really support MSI?

Thanks, MS

0 Rex Chang over 11 years ago in reply to M S1

TI__Guru 50170 points

Do you see pci.h file under include/linux directory and the function prototype defined in it?

From git history, there are following comments related to PCIe/MSI

0 M S1 over 11 years ago in reply to Rex Chang

Prodigy 170 points

I see the prototype, but I think it depends on another macro. Look at this search of the c6x source:

agsc6x2@klink:/home/git-linux-csx/linux-c6x$ find . -name Kconfig|xargs grep ARCH_SUPPORTS_MSI

./arch/arm/Kconfig:   select ARCH_SUPPORTS_MSI
./arch/ia64/Kconfig:   select ARCH_SUPPORTS_MSI
./arch/powerpc/Kconfig:   select ARCH_SUPPORTS_MSI
./arch/sparc/Kconfig:   select ARCH_SUPPORTS_MSI
./arch/x86/Kconfig:   select ARCH_SUPPORTS_MSI if (X86_LOCAL_APIC && X86_IO_APIC)
./arch/mips/Kconfig:   select ARCH_SUPPORTS_MSI
./drivers/pci/Kconfig:config ARCH_SUPPORTS_MSI
./drivers/pci/Kconfig:   depends on ARCH_SUPPORTS_MSI

It doesn't appear that the c6x architecture even has the option for MSI support.

0 Rex Chang over 11 years ago in reply to M S1

TI__Guru 50170 points

Hi, MS,

Taking pci_disable_msi() as an example, in pci.h, it has

#ifndef CONFIG_PCI_MSI

static inline void pci_disable_msi(struct pci_dev *dev)
{ }

#else

extern void pci_disable_msi(struct pci_dev *dev);

#endif

Whether CONFIG_PCI_MSI is defined or not, there will be a prototype for it and should not cause a compilation error. In my opinion, it has not reach where if MSI is supported or not. I think that would be linking issue. Shouldn't that be the case?

For c6x, arch/c6x/configs/ti_evmc6678_defconfig:CONFIG_PCI_MSI=y

I'll also research on the ARCH_SUPPORTS_MSI you mentined.

Rex

0 Rex Chang over 11 years ago in reply to Rex Chang

TI__Guru 50170 points

select ARCH_SUPPORTS_MSI is in arch/C6x/Kconfig.

0 M S1 over 11 years ago in reply to Rex Chang

Prodigy 170 points

Not in mine, which is the c6x source I git-cloned using your instructions.

Also, no idea why the compile fails for pci_enable_msi. I'm also calling pci_resource_start from the same pci.h header file, and it compiles fine.

0 Rex Chang over 11 years ago in reply to M S1

TI__Guru 50170 points

I just git clone the latest code yesterday and here are the history on what I did.

2349 cd linux-c6x/
2350 la
2351 git clone git://linux-c6x.org/git/linux-c6x.git

Not sure why you are not getting the code.

Rex

0 M S1 over 11 years ago in reply to Rex Chang

Prodigy 170 points

That's exactly what I did as well. I'm downloading again, and I'll report back what happens. Is it possible I need to do something with make menuconfig to bring in PCI support? Thanks, MS

0 Rex Chang over 11 years ago in reply to M S1

TI__Guru 50170 points

I think the default has both PCI_MSI and SUPPORTS_MSI =y, but you may want double check.

0 M S1 over 11 years ago in reply to Rex Chang

Prodigy 170 points

Rex,

Success this time. Not sure why it didn't work before- maybe because I used the "bootstrap" utility?

One more related question: my driver has a userland API library that I also want to include with the c6x build. Where would be the best-practice place to put this? Under tools, or maybe just lib?

Thanks for all your help,

0 Rex Chang over 11 years ago in reply to M S1

TI__Guru 50170 points

I believe bootstrap only get the GA version, not the drops after that. If the API library will be dynamically linked, it can go in lib directory. Otherwise, may be /usr/bin(?).

Rex

0 Vladimir Karakozov over 11 years ago in reply to Rex Chang

Expert 2285 points

Hello, All!

I have the same task. I have a PCI Express device implemented in FPGA and try to write device driver for it. Device have two memory regions in the BAR0 and BAR1.

I get latest linux-c6x sources, rebuild kernel, build my driver and load it into the kernel. My PCIE device was successfully found and I can read/write device register from BAR0 and BAR1 in kernel space from driver. In driver I implement a mmap syscall and want to map BARs into user space program. Than I call mmap in user space I get error: No such device. Here the log:

pexdrv # ./insert

Find pexdrv.ko module : [SUCCESS]
Loading pexdrv module : [SUCCESS]

pexdrv # lsmod

pexdrv 59958 0 - Live 0x9f060000

pexdrv # ../../exam/pex_test /dev/pexdrv0

Start testing device /dev/pexdrv0
Show board information:
DEVICE ID: 0x550E
BAR0: 0x60000000
SIZE: 0x200000
BAR1 0x60200000
SIZE: 0x200000
IRQ: 0xB3
Try to map device registers:
No such device
pexdrv #

I suppose that there are some limitation in uClinux for mmap usage. Can you give me some advice about mmap implementation in kernel driver? If you need source code or any additional information - I will provide it.

Thank you.

Vladimir

0 Rex Chang over 11 years ago in reply to Vladimir Karakozov

TI__Guru 50170 points

Hi, Vladimir,

Do you need to implement the mmap syscall in driver? I thought TI has ported mmap to c6x, but let me check internally.

Rex

0 Vladimir Karakozov over 11 years ago in reply to Rex Chang

Expert 2285 points

Hi, Rex!

Thank you for your answer! Yes, I implement mmap() in my driver. Here is code:

//-----------------------------------------------------------------------------

static inline int private_mapping_ok(struct vm_area_struct *vma)
{
    return vma->vm_flags & VM_MAYSHARE;
}

//-----------------------------------------------------------------------------

static int pex_device_mmap(struct file *file, struct vm_area_struct *vma)
{
    size_t size = vma->vm_end - vma->vm_start;

    if (!private_mapping_ok(vma))
        return -ENOSYS;

    vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);

    if (remap_pfn_range(vma,
                        vma->vm_start,
                        vma->vm_pgoff,
                        size,
                        vma->vm_page_prot)) {
        err_msg(err_trace, "%s(): error in remap_page_range.\n", __FUNCTION__ );
        return -EAGAIN;
    }
    return 0;
}

//-----------------------------------------------------------------------------

I want make two things:

1) map device BAR0 and BAR1 regions to user space. Region size is 1MB for BAR0 and 1MB for BAR1.

2) map to user space memory that was allocated by dma_alloc_coherent() for DMA channel

In two cases I have an error:

No such device.

I found that kernel call do_mmap_pgoff() from nommu.c (line 1167)

http://linux-c6x.org/git/?p=linux-c6x.git;a=blob;f=mm/nommu.c;h=49d5d7f28f0f4d615465b016f6d502b03b0f1d9b;hb=HEAD#l1167

and than validate_mmap_request() (line 817) and this function return an error after checking:

if (!(capabilities & BDI_CAP_MAP_DIRECT)) at line 918. And my pex_device_mmap not called at all.

My file operations struct:

struct file_operations pex_fops = {

    .owner = THIS_MODULE,
    .read = NULL,
    .write = NULL,

#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,37)
    .unlocked_ioctl = pex_device_ioctl,
    .compat_ioctl = pex_device_ioctl,
#else
    .ioctl = pex_device_ioctl,
#endif

    .mmap = pex_device_mmap,
    .open = pex_device_open,
    .release = pex_device_close,
    .fasync = pex_device_fasync,
    .poll = pex_device_poll,
    .aio_read = pex_device_aio_read,
    .aio_write = pex_device_aio_write,
};

User space code:

bar0 = mmap(NULL, Size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, (off_t)PhysAddress);

If you need more information - I will provide it for you.

Thank you.

Update 1:

Hello, All!

I try ti use /dev/mem for mapping DMA memory and this method is working. PCI Express device driver allocate DMA memory blocks and pass their physical addresses to user space application. In application I write a additional module that perform mapping. But mapped user space virtual addresses are the same as kernel space physical addresses. Here is the log (bold font is addresses in schema block: physical -> virtual ):

# /home/embedded/bardy/bin/mainstream 0
Stream()
Reset FPGA...
Init FPGA...
main(): Tetrade info...
main(): Start stream testing...
Allocate DMA memory...
0: 0x8f40e000 -> 0x8f40e000
1: 0x8f44e000 -> 0x8f44e000
2: 0x8f48e000 -> 0x8f48e000
3: 0x8f4ce000 -> 0x8f4ce000
Stub: 0x8f40d000 -> 0x8f40d000
Set DMA source...
Set DMA direction...
Stop DMA...
Reset DMA FIFO...
Start DMA...
BLOCKS: 33963 DATA: 0x454495b9 0x4f4e3c70 0x8c001ff8 0x72a23721
Stop DMA channel...
Free DMA memory...
Reset DMA FIFO...
~Stream()

Processors

Processors forum

Integrating new device driver into c6x