TMS320C6657: How can RC read EP Registers

Hello!
One may simply think there is configuration memory window, at 0x2180_0000 for your device, and data memory window at 0x6000_0000. So when you attempt access a word in the former window, PCIe subsystem will issue config read/write, if you access data window, then data packet will be sent, this is another type of packet. That's it, read and write to memory windows will be transformed into transaction layer packet by PCIe subsystem.

I'd like to warn you, that configuration space layout in DSP and FPGA may not be identical. I had this kind of experience with C6670 against Xilinx FPGA.

Enumeration is a process of identifying of potentially arbitrary remote device. In your design as DSP, so FPGA are fixed and known devices, so once they both up and running, there is no need to perform procedures like in regular PC. One may want make sure FPGA is okay, just try to read vendor id, device id. Once they meet your expectations, just proceed.

And just in case we are touching this topic, read and write to PCIe by processor core, so called PIO access is the easiest thing to implement, and it has extremely low speed. Once you want some PCIe like throughput, there should be DMA. One may used EDMA3 controller on DSP side, but make sure FPGA design can handle multiword TLPs. Default examples may support PIO only.

Hope this helps.

Thanks for the response. I am still having issues though, I am still unable to get the RC to gain access to EP registers. I verified that BDF address was 0:0:0 in CFG_SETUP.  I then set 

pcieConfigRootAddress = 0x21800000 + 0x2000;

and

const uint32_t vendorDeviceOffset = 0x1000;

reading from that address returns zero. 

I also cycled through all the BDF values in nested for loops, Since there is only one end point I expect to read 0xffffffff, but that is not what I'm seeing.

Simply put, the RC is not reading the EP registers. Also I verified that the link is up and running.

What am I doing wrong?

Hi Victor,

Thanks for  your insight, you make a good point. Our system is closed and simple we should not need enumeration. Actually that was my initial assumption, but my colleagues think we should go through the process, I suppose for future expandability. Secondly I think for other reasons we should have the ability to have the RC read EP registers.

I did the CFG_SETUP exercise, as Rajeet described but no luck yet. I must be doing something wrong, just can't figure what it is yet.

Hello!

Could you please disclose how you come with vendorDeviceOffset = 0x1000? I would expect them at 0x2180_2000. Googling takes me to my own old question, where 3rd picture illustrates this expectation.

support.xilinx.com/.../pcie-endpoint-config-space-layout

Hi Victor,

I got that from sprugs6d.pdf. That is the address of the Vendor_device_ID register. The more complete flow is as follows:

const uint32_t pcieConfigRootAddress = 0x21800000 + 0x2000;

const uint32_t vendorDeviceOffset = 0x1000;

uint32_t vendorDeviceID = *((uint32_t *)(pcieConfigRootAddress + vendorDeviceOffset));

That value is always zero. 

In addition to that, I also did

int bus, device, fxn;
        for(bus = 0; bus<256; ++bus)
        {
            for(device = 0; device<32; ++device)
            {
                for(fxn = 0; fxn<8; ++fxn)
                {
                    memset(&regs, 0, sizeof(regs));
                    memset(&cfgTrans, 0, sizeof(cfgTrans));
                    memset(&vndDevId, 0, sizeof(vndDevId));

                    cfgTrans.bus = bus;
                    cfgTrans.device = device;
                    cfgTrans.func = fxn;
                    cfgTrans.type = 0;

                    regs.cfgTrans = &cfgTrans;


                    if ((retVal = Pcie_writeRegs (handle, pcie_LOCATION_REMOTE, &regs)) != pcie_RET_OK)
                    {
                       PCIE_logPrintf ("write registers failed!\n");
                    }
                    else
                    {
                        PCIE_logPrintf("Written BDF = %d, %d, %d\n", bus, device, fxn);
                    }

                    // now read remote
                    uint32_t vendorDeviceId = *((uint32_t *)(pcieConfigRootAddress + vendorDeviceOffset));
                    PCIE_logPrintf("vendorDeviceId read from address = %08x\n", vendorDeviceId);

                    memset(&regs, 0, sizeof(regs));

                    regs.vndDevId = &vndDevId;
                    memset(&vndDevId, 0, sizeof(vndDevId));
                    if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &regs)) != pcie_RET_OK)
                    {
                       PCIE_logPrintf ("Read registers failed!\n");
                    }
                    else
                    {
                       PCIE_logPrintf("devID from Ti interface = %08x, vendor ID = %08x, raw = %08x\n\n", vndDevId.devId, vndDevId.vndId, vndDevId.raw );
                    }
                }
            }
        }

In my setup the endpoint BDF is 0:0:0, so my understanding is that reading any BDF other than that should return 0xffffff. That is not what happens. Here's a sample of my output:

Written BDF = 0, 1, 0
vendorDeviceId read from address = 00000000
devID from Ti interface = 00008888, vendor ID = 0000104c, raw = 8888104c

Written BDF = 0, 1, 1
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Written BDF = 0, 1, 2
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Written BDF = 0, 1, 3
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Written BDF = 0, 1, 4
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Written BDF = 0, 1, 5
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Written BDF = 0, 1, 6
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Written BDF = 0, 1, 7
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Written BDF = 0, 2, 0
vendorDeviceId read from address = 00000000
devID from Ti interface = 00008888, vendor ID = 0000104c, raw = 8888104c

Written BDF = 0, 2, 1
vendorDeviceId read from address = 00000000
devID from Ti interface = 00002180, vendor ID = 00002000, raw = 21802000

Your insight is appreciated.

Alson

That is my point: you refer TI's endpoint layout, but your FPGA may have different config space layout. I can tell you for sure that Xilinx FPGA endpoint will have different layout. So the very first thing is to see FPGA documentation. If you saw that reference to Xilinx forum, I browsed 0x2180_2000 memory range and there I saw expected VID/DID. Please try with that address, without extra 0x1000 offset.

Victor:

Sorry if I wasn't clear, this experiment is running with two tms320c6657 evaluation boards, one is Root complex and the other is EndPoint. I should also say that the link is up and running properly, both are able to write to the other, as per the PCIE_evmc6657_wSocFile_C66BiosExampleProject.

Sorry, here I'm of no help. Also, we never tried board, device, function other, that default.

Victor,

Thanks for your effort. I am currently trying your suggestion to leave out the 0x1000 offset.