This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

PCIe Remote Configuration Space Problem

Hello,

We want to connect our C6678 processor to an FPGA over PCIe link. DSP is assigned as RC and FPGA is assigned as EP.

FPGA is verified by means of a third-party tool called PCITree on a Personal Computer. Its configuration space can be retrieved normally
and PC BIOS assigns base addresses to its BARs correctly.

However, when we want to use our DSP as Root Complex, we are facing with some problems. The flow of our application goes as follows:

1. We follow the initialization procedures in the same way with PCIe sample project in the MCSDK.

2. We observe that link is up.

3. We read remote configuration registers for root complex as given the following code sniplet.(Which corresponds to the local PCIe configuration registers of FPGA)

    // READ DEVICE ID & VENDOR ID
   memset (&myRegs,   0, sizeof(myRegs));
   memset (&myRemoteCfgDevAndVendorID, 0, sizeof(myRemoteCfgDevAndVendorID));

   myRegs.vndDevId = &myRemoteCfgDevAndVendorID;

   if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
   {
        System_printf ("Read REMOTE DEVICE AND VENDOR ID register failed!\n");
   }

   // READ BAR REGISTERS
   memset (&myRegs,   0, sizeof(myRegs));
   memset (&myRemoteCfgBars, 0, sizeof(myRemoteCfgBars));

   myRegs.type0BarIdx = &myRemoteCfgBars;

   for(barNum=0; barNum < 6; barNum++)
   {
         myRemoteCfgBars.idx = barNum;
         if ((retVal = Pcie_readRegs (handle, pcie_LOCATION_REMOTE, &myRegs)) != pcie_RET_OK)
         {
              System_printf ("Read REMOTE BAR CONFIGURATION register failed!\n");
         }
   }

4. We can retrieve Device and Vendor IDs correctly. However, all BARs have the value of zero.

5. We also cannot write a desired value into any of BARs.

Why cannot we read the value of BARs although we can reach the configuration space of the EP???

Thanks in advance,
Hakan

  • Hakan,

    When you found the PCIE link is up, is it stable overtime or will drop out quickly? Do the FPGA and DSP card use the same or different reference clock? Are you able to read/write the IB_BARn, IB_STARTn_LO, IB_STARTn_HI and IB_OFFSETn (n = 0, 1, 2, 3) registers of FPGA via PCIE link?

    Regards, Eric

     

     

  • Hello Eric,

    The reference clocks of the FPGA and DSP are different but the PCIe link is stable and it doesn't drop over time.
    We didn't try to write over the registers you had mentioned. We only tried to read/write over BARs in the PCI
    configuration space of the FPGA.

    Do we need to enable something that we're missing in order to read/write those registers?

    There are some other differences which are worth mentioning between two connection types. 
    Namely, when we connect our FPGA to a PC and when we connect our FPGA to DSP
    which go as follows:

    1. The last significant three bits (BUS_MS, MEM_SP and IO_SP) of STAUS_COMMAND register of the FPGA are zero
    when we connect the FPGA to DSP, while these bits are set when we connect the FPGA to the PC.

    2. Cache Line Size Register is filled with a value rather than zero when we connect the FPGA to the PC.
    However, this value is zero when we connect the FPGA to DSP.

    Do these differences have anything to do with our remote configuration space setup problem?

    Best regards,
    Hakan

     

  • Hello Again,

    I had made a small mistake in my previous post. The clock references of the FPGA and DSP are the same.

    Additionally, we use Spartan 6 series FPGA and thus, it doesn't have built-in IB_BARn,
    IB_STARTn_LO, IB_STARTn_HI and IB_OFFSETn (n = 0, 1, 2, 3) registers.

    These registers seem to be specific to PCIESS of TI. Am I wrong?

    Best regards,
    Hakan

  • Hakan,

    When you use a PC as RC, it writes the BAR registers of the EP (FPGA) during enumeration.

    When you use the DSP as the RC, the DSP’s software DOESN’T configure the BAR of the remote-end. You need to add code to do it through PCIe Remote Configuration Space. I assume you use PDK package under MCSDK. In  packages\ti\drv\pcie\example\sample\readme.txt, it mentioned:

    At startup, each EVM configures its PCIe subsystem:
    • Serdes, clock, PLL
    • PCIe Mode and Power domain
    • Inbound/Outbound address translation and BAR registers
    • Link training is triggered

    Our example works in that each EVM configures its own BAR registers.

    Those IB_STARTn_LO, IB_STARTn_HI and IB_OFFSETn (n = 0, 1, 2, 3) registers are specified to TI PCIESS, sorry for the confusion.

    Regards, Eric

  • Hakan, how are you connecting the DSP to the FPGA?  Are you using a specific type of breakout board to connect the PCIe TX/RX lanes or are you putting the c6678 and the FPGA board in PCIe slots?

    Next, I know that the Xilinx core when used as EP requires the configuration space (memSP, busMS, serrEn) to be setup properly from the DSP (when used as RC) before you could do any memory related operations.  This RC functionality is typically done by the BIOS but the DSP does not do that.  You don't need to do this when trying to setup the configuration space.

  • @Eric:

    We already know that we are supposed to add some extra source code to configure remote configuration space of the EP.
    The problem is, as we had mentioned in my earlier posts, we don't know how to do it.

    We follow the same initialization procedure with the sample PCIe project that you mentioned.

    Is there any sample source code or a demo project of how C6678 configures a remote EP which is not a TI product?

    @Jay Shu:

    We directly connect our DSP to the FPGA over one lane. There are no PCIe slots on our board.
    But I couldn't understand what the connection method has got to do with our remote configuration
    setup problem. Could you please be more specific?

    We set the registers memSP, busMS and ioSP to the value of 1 after the link is up.

    Regards,
    Hakan

  • I actually don't have any other code than the sample pcie project also.  What I'm asking with regards to the connection is - what kind of connector are you using to breakout the PCIe lanes?  Is it a TI product or something you put together yourself.  I know that if it is a specific AMC breakout card, then the labeling is incorrect on it and will cause intermittent link issues and you definitely cannot read vendor ID.

  • Hakan, you can try my pcie init project.  I used this on the C6678 and setup the DSP as the RC and it at least let's me see the vendor ID/device ID for a Xilinx PCIe core.

    pcie_init.zip
  • Thanks Jay,

    First of all, we are working on our own custom design board that contains a C6678 DSP and Spartan 6 Series FPGA.
    This board has a very similar form factor with EVM 6678L board.

    We could also retrieve Vendor & Device IDs of the EP which can be accessed from the remote configuration space.
    This space seems to map between (0x21800000 + 0x2000) ~ (0x21800000 + 0x3FFF) in C6678 memory map. We could also
    retrieve all other information from there and the values that we retrieve are consistent with the way that we configured our EP (FPGA).

    The only problem is that we read the value of 0 for all BARs from the remote configuration space although we had assigned values
    to all BARs of EP and we had enabled them. Additionally, they are configured as 32-bit memory BARs. We also can't change the values
    of the BARs of the EP.

    Therefore, we don't seem to be able to conduct PCIe enumeration process.

    We will try to run your application on our board but I want to ask you whether it conducts this bus enumeration process. Does it write over
    the BARs of the EP which can be accessed from the remote configuration space of the RC?

    Best regards,
    Hakan

  • Hakan, with the project I have supplied, I could confirm that I could set the EP’s BAR properly.  This is using a Xilinx ML-605 board with a Virtex-6. You can see that we read that block of configuration space after we have set it.

    However with that said, I have yet to be able to read/write a block of memory in the Xilinx FPGA afterwards.  You might have better luck with that and I’d like to know if you can do that.

  • Hello Jay,

    We executed your application on our own board but after the function pcieReadLnkStatCtrlReg(handle)  is called, 
    we still read the values of the BARs within the remote configuration space (starts from 0x21800000+0x2000+0x10) as zero. 

    I think we were supposed to see these values as non-zeros since we assigned memory blocks for all BARs of EP (FPGA).

    What might have gone wrong?

    Best regards,
    Hakan.

  • Hello again,

    When we go through your sample code, we also realized that you could write over BAR0 although you don't read its value before attempting to change it.

    We also realized that you are using dstBuf for inbound translation configuration (ibOffsetAddr) and you convert its value as if resides in L2 RAM. However,
    when we checked your linker command file (pci_sample.cmd), dstBufSec (which the dstBuf is allocated in) is mapped to DDR3. This seems to mess up
    inbound translation and the readback process. Do you think that this might be the reason of why you cannot read back.???

    Regards,
    Hakan

  • Hakan,

    If you look at line 1223 in pcie_sample.c, there is a pcieEnumEpConfig(handle) call.  Inside that call - line 992, there is a Pcie_cfgBar call with setup of the EP BAR0.  Right afterwards, I did a dump of memory to checkout if the BAR registers were correct.  You can uncomment out line 1019 (pcieGetBARInfo(handle)) and it should print out the EP's BAR0 stuff also and should match what was dumped out at 0x21802000.

    With regards to the dstBuff, that could be the issue but I don't have a set of boards to test now.  This same code works with two C6678 EVMs connected together without a problem so that is where I was stumped.

    Do you have a bus analyzer to see if there are any transactions going on?  Or if one side of the link is down?  What I noticed was the FPGA (via ChipScope) thought its RX side was in electrical idle and PCIe communications completely stopped post that pcieEnumEPConfig call.

    Jay

  • Hakan, I finally got my test jig up and with what I stated in the prior reply (i.e. commenting out line 1019).  I this is what I get reading 0x21802000.

    [C66xx_0] i 0, VID 6024, DEV 10ee
    [C66xx_0] offset: 0x0 : 602410ee 00100000 05000000 00000000
    [C66xx_0] offset: 0x10 : 60000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x20 : 00000000 00000000 00000000 000710ee
    [C66xx_0] offset: 0x30 : 00000000 00000040 00000000 000001ff
    [C66xx_0] offset: 0x40 : 78034801 00000008 00806005 00000000
    [C66xx_0] offset: 0x50 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x60 : 02020010 00008e00 00012810 0003f412
    [C66xx_0] offset: 0x70 : 00110000 00000000 00000000 00000000
    [C66xx_0] offset: 0x80 : 00000000 00000002 00000000 00000000
    [C66xx_0] offset: 0x90 : 00010002 00000000 00000000 00000000
    [C66xx_0] offset: 0xa0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0xb0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0xc0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0xd0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0xe0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0xf0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x100 : 00010003 01000a35 00000001 00000000
    [C66xx_0] offset: 0x110 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x120 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x130 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x140 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x150 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x160 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x170 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x180 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x190 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x1a0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x1b0 : 00000000 00000000 00000000 00000000
    [C66xx_0] offset: 0x1c0 : 00000000 00000000 00000000 00000000

    Then using the pcieGetBARInfo(handle) function with            

       /* Configure Address Translation */
                barCfg.location = pcie_LOCATION_REMOTE;
                barCfg.mode     = pcie_EP_MODE;
                barCfg.base     = 0x60000000;
                barCfg.prefetch = pcie_BAR_NON_PREF;
                barCfg.type     = pcie_BAR_TYPE32;
                barCfg.memSpace = pcie_BAR_MEM_MEM;
                barCfg.idx      = 0;

    I get the following:

    [C66xx_0] EP: bar#0, base 0x06000000, prefetch 0x0, type 0x0, memSpace 0x0

    - At least the base address is not all zeros at that point.  Right after line 1027 (pcieReadCMDReg(handle, pcie_LOCATION_LOCAL)), then that space is all zeros.

  • I forgot to mention that yes, you are correct - that dstBuf should be in L2SRAM.  Next, my point of it being all zeros - this looking at it from the memory browser.  For some reason in CCS v5, it will be all zeros.  However, if you just read via the pcieGetBARInfo function it is correct.

  • Hello Jay,

    First of all, I would like to thank you for your fast and useful replies.

    I want to ask something different now regarding the write operation from RC to EP:

    In the line #986 of your project you seem to have assigned the base address of the remote EP the value of 0x60000000.

    barCfg.base= 0x60000000;

    However, we couldn't send data to EP using this base address. Then, we changed this line with the base
    address value of 0x90000000 (which is the same address of your outbound translation, line #1182)

    We have analyzed the bus by means of ChipScope and we observed that we could send data to EP succesfully after chaging this value.

    Is there a specific reason of your having given the value of 0x60000000 to the base address of the remote BAR?

    We are now working on the read operation from EP and I seem to have many questions to you. My first questions are:
    1. Is the read operation conducted from the same address with the write operation, namely starting from 0x60000000
    of C6678?

    2. When we want to conduct a read operation from RC to EP, is the whole process only an outbound translation or does everything
    start with an outbound translation and EP's final response is an inbound translation from our point of view?

    Please stay tuned.

    Regards,
    Hakan

  • Hakan,

    That 0x60000000 base address was used initially when we were testing and it worked (DSP to DSP) but along the way I was trying to debug why our FPGA writes are not working so you have a snapshot of where I left off.

    I'm glad you have set the base address to 0x90000000 and it works - have you verified that from reading it back via the DSP then versus seeing the TBL in Chipscope?  Did you use the same PCIe generator wizard to generate your PCIe core in the Spartan 6?  If so I'd like to know what you did on that because when I change my base address to 0x90000000, I still get junk coming back.

    I'll see what I can find out on your questions - my group does not contain the PCIe expert - but rather what you are trying to do is exactly what we started to do about 2 months ago so I could at least get you going.

    Jay

  • Hakan,

    For your questions, 0x60000000 is the PCIE data space in our C66x DSP memory map.

    When you start a read operation from the RC to EP, the PCIE address must match the BAR of the EP (0x60000000 or 0x90000000 if you changed this value), otherwise the address will be rejected. If BAR matches PCIE address, then the EP's inbound translation registers will map this to EP's internal memory address. 

    The whole process is an outbound translation from RC, and EP's final response is an inbound translation.

    Regards, Eric

     

     

  • @Jay:

    I assume that you mean TLP by TBL. Assuming that it is true, we seem to read junk data from FPGA over PCIe. However, TLPs for write and read operations
    seem to be correct although we haven't analyzed them in detail.

    We used a tool called "Spartan 6 Integrated Block for PCIe v1.4" to create our PCIe core in our FPGA. We also tried to write and read data with Virtex 6 and the result
    is the same: we can send data to FPGA but we read garbage data from FPGA. The read section of our source code goes as follows:

      volatile UInt32 *pcieDataBase = (UInt32 *)0x60000000;

      uint32_t tmp;
      for (i=0; i<PCIE_BUFSIZE_APP; i+=4)
      {
           tmp = *((volatile UInt32 *)pcieDataBase + i+0);
           tmp = *((volatile UInt32 *)pcieDataBase + i+1);
           tmp = *((volatile UInt32 *)pcieDataBase + i+2);
           tmp = *((volatile UInt32 *)pcieDataBase + i+3); 
      }

    Is there anything wrong with this source code? Is the read supposed to be done this way?

    The read section of PCIe sample apllication within the MCDSK goes as follow:

        unsigned int key;
        unsigned int loop;
        for(loop=0; loop < 5; loop++)
        {
          /* Disable Interrupts */
          key = _disable_interrupts();

          /*  Cleanup the prefetch buffer also. */
          CSL_XMC_invalidatePrefetchBuffer();

          CACHE_invL1d ((void *)dstBuf,  4, CACHE_FENCE_WAIT);
          CACHE_invL2  ((void *)dstBuf,  4, CACHE_FENCE_WAIT);

          /* Reenable Interrupts. */
          _restore_interrupts(key);

        }

    It reads the data from EP to RC onto dstBuf, which is mapped to internal memory during the inbound translation procedure of the RC?

    However, your code (and ours) conducts the read operation by accessing the memory location starting from 0x60000000 in the map. Why is there such a major difference
    between these two applications? How is the sample project in the MCSDK tested?


    @Eric,

    According to the sample project in the MCSDK, the read operation seems to be an inbound translation from the perspective of RC.

    Don't you think that there is a contradiction here between the source code of this project that comes along with the MCSDK and your last statement?

    Thanks in advance,
    Hakan

     

  • Hakan, you are now at a point where when we tried using the Virtex6, we have the same issue.  My team is not the PCIe expert but we were trying to accomplish basically what you are doing now with very little success.   The problem I had was I could not check the TLPs at all on my side so I've been trying to get usage of a PCIe bus analyzer to help check the traffic from the RC to EP and vice versa.

    That sample code I have was tested with two C6678L EVMs.  Once was EP and the other RC.  This is how the original sample code was tested and my modifications were tested on the RC side.  Both ran without issues and could read the block of memory that was written too with that piece of code.

    Is there any way you can check the block RAMs in the Spartan to see the values - just to 100% confirm that write values actually are in the FPGA?

    Jay

  • Hakan, also your point on the code in the MCSDK example - I think you mean this:

    do {

    unsigned int key;

    /* Disable Interrupts */

    key = _disable_interrupts();

    /* Cleanup the prefetch buffer also. */

    CSL_XMC_invalidatePrefetchBuffer();

    CACHE_invL1d ((

    void *)dstBuf.buf, PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

    CACHE_invL2 ((

    void *)dstBuf.buf, PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

    /* Reenable Interrupts. */

    _restore_interrupts(key);

    }

    while(dstBuf.buf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);

    /* check all the data */

    for (i=0; i<PCIE_BUFSIZE_APP; i++)

    {

    if(dstBuf.buf[i] != srcBuf[i])

     

    I think this is used in conjection with the other side of the code where it waits until the EP issues the writes to the RC.  That sample of code I gave you does not do that since I have no control of the EP doing this.  This will code basically waits until it see the completion from the data transfer to the RC and then does the check right afterwards.  I don't think you are doing this but rather just a simple RC to EP write and then RC to EP read.  If that is the case, then the code I supplied works when using two C6678s.  It just doesn't seem to work when connected to the FPGA.   

  • Hakan,

    My explanation of how the PCIE demo works for 6678 (RC) to 6678 (EP) from the MCSDK package:

    On the RC side: BAR is programed with 0x9000,0000, IB_BAR LO is 0x9000,0000 IB_BAR HI is 0x0, IB_OFSET is a global address of dest.buf from local L2. OB_LO is 0x7000,0000, OB_HI is 0x0. dstbuffer is 0, srcbuf is "i".

    On the EP side: BAR is programed with 0x7000,0000, IB_BAR LO is 0x7000,0000 IB_BAR HI is 0x0, IB_OFSET is a global address of dest.buf from local L2. OB_LO is 0x9000,0000, OB_HI is 0x0. dstbuffer is 0, srcbuf is "i".

    When RC write something to EP, it first gets pcieBase address, then this pcieBase address is written with "i" from the srcBuf. Then the OB of RC side converts the pcieBase address into PCIE address space, probably something like 0x7xxx,xxxx. In the EP side, this PCIE address matches the BAR of EP, it goes through inbound translation, IB_BAR LO and IB_BAR HI is deducted from this 0x7xxx,xxxx, add IB_OFFSET is added, so this data stored in dest.buf of EP side.

    In the EP side, when the dstbuf is full, it first gets pcieBase address, then this pcieBase address is written to "i" from the dstBuf. Then the OB of EP side converts the pcieBase address into PCIE address space, probably something like 0x9xxx,xxxx. In the RC side, this PCIE address matches the BAR of RC, it goes through inbound translation, IB_BAR LO and IB_BAR HI is deducted from this 0x9xxx,xxxx, add IB_OFFSET is added, so this data stored in dest.buf of RC side, and matched with srcbuf of RC side.

    When you configure address to readback data, you need to follow the idea how the address is translated in all BAR, IB_BAR and OB_BAR.

    Regards, Eric

     

     

  • Jay,

    For your question, the data sent from the RC can be retrieved by the EP succesfully as we observe the TLPs by means of ChipScope. However,
    FPGA does not send back the data which was sent previously from the RC when a read request arrives, but a constant data, i.e. 0x33443344.

    We also observed the completion TLP packets going out of EP in response to a read request. These TLPs do not contain any PCIe address information.
    Therefore, we assume that the read operation does not have anything to do with inbound translation from the RC side. Is this assumption correct?

    We also have an observation about outbound and inbound address translation enable/disable mechanism. When we disable inbound translation(by means of setting IB_XLT_EN = 0), nothing changes with the read operation. When we disable outbound translation(by means of setting OB_XLT_EN = 0), we read the same garbage data.
    However, there are no PCIe transactions on the EP side. How do we read the same garbage data although we disabled the outbound translation?

    Also, regarding the Address translation mechanism, when we try to read a register in PCIe addres space(i.e. 0x60000000) ; all we do is to assign a pointer to that register and read the data that pointer refers to. It seems a to take a couple of instructions from compiler side. But meanwhile we know that a read request is created by RC, sent to EP and EP responds with completion TLP.  Who handles this timing and synchronization mechanism if there is such a need? We see that we (on software side) do not do anyhing for this synchronization mechanism. 

    Regards,
    Hakan.

     

  • Hakan,

    I think you should check if the FPGA sees if its RX (TX from the DSP) is in electrical idle.  What we have noticed is basically the same think you stated.  As we dug in deeper on the FPGA side with help from the FPGA vendor, we noticed that it determined that its RX is in electrical idle even though both sides were programmed to not go into power management.

    Thus, my question to determine if you can check in Chipscope that you actually wrote into the FPGA's memory and not look at the TLP on the bus.  The TLPs should have occurred but what we kept seeing was since the FPGA determined that the RX lane was idle, it never processed any of the TLPs!

    It made no difference if we enabled or disabled the outbound translation either - we go the same result.   My team here has never made this to work properly.

  • Hakan,

    The outbound transfer refers to the transfer initialized by the local PCIe device to the external device, no matter write or read. In your case, it is outbound transfer for both write and read initialized by RC. So your assumption is correct that it is not related to inbound translation on RC side, only outbound translation may take effect.

    It is a good practice to see if the write operation from RC to EP is actually happening as Jay suggested, since both Write and Read are sharing the same configurations of PCIe. It has good chance that both of them are not working for your current setup. And it may explain why there is no difference you disable/enable the outbound translation. 

    Actually, the outbound translation is optional for the C66x PCIe transfer. But enabling it will give you more flexibility. The outbound translation is disabled by default and the PCIe address over the PCIe link from C66x will be starting from 0x60000000, which is the same as PCIe data space in C66x. It requires that on EP side, the BAR register will be setup to accept the 0x6xxxxxxx address range.

    If you enable the outbound translation on C66x (by setting OB_XLT_EN bit in CMD_STATUS register and OB_ENABLE bit in OB_OFFSET_INDEXn register), the PCIe address over the link will be replaced by other values defined in OB_OFFSETn_LO and HI registers. Then you can use any other values, like 0x90000000 as the PCIe address and EP BAR values. The range is defined in OB_SIZE register.

    Please also double check if the bits BUS_MS, MEM_SP are being set in STATUS_COMMAND register for both RC AND EP before the memory transaction, since you mentioned they are 0 on FPGA when connected to C66x in previous reply.

    Another thing to check is how the FPGA BAR register is mapped to its memory region. The PCIe in FPGA may has similar but different inbound translation mechanism as PCIe in C66x.  You have to make sure the packets accepted by BAR register will be landed/fetched on/from the destination memory eventually. Previously you mentioned that the FPGA+ host PC is working for configuration register access. I am wondering if the setup is working for memory access as well. In that way we can rule out the issues happened on FPGA side.

    Sincerely,

    Steven

  • Hello,

    When we connect the same FPGA with a different GTP to a host PC over a PCIe external cable, we do not see any problems with
    remote configuration setup, memory read and memory write operations. Additionally, we observe that correct TLPs come in and out
    to the FPGA when we connect it to the DSP. We also don't have any problems to write the contents of the arriving TLPs into the internal memory
    of the FPGA during the communication with the DSP.

    Therefore, on the contrary with the way you (Jay and Steven) think, we assume that the problem is on the DSP side, possibly PCIESS.

    Regards,
    Hakan

     

  • @Jay,

    Additionally, we observe that TLPs sent from the DSP arrive at the FPGA side correctly. Which means that RX side of the FPGA  works properly and it is not electrically idle.

    @Steven,

    We can succesfully conduct remote configuration setup after the link-up and our outbound translation (on the DSP side) before the link-up. If we hadn't accomplished it succesfully, we wouldn't have observed TLPs on the FPGA side during the read and write operations from the DSP. We also set BUS_MS and MEM_SP both on the DSP and the FPGA sides.

    Regards,
    Hakan

  • Hello,

    I think we resolved the problem. Thank you all so much for your help.

    The problem is on the FPGA side as you said. A basic description of it goes as follows:

    - There is a TD bitfield (that notifies whether to use or not to use the TLP digest) in the TLP header. If this bitfield is set,
    then there must be a valid TD data at the end of the TLP. In the read requests that goes from the DSP, this bitfield is 1 and TD data
    is at the end of the TLP as expected.

    - Although this bitfield of the completion TLP is also set, our FPGA application code does not add required TD data at the end of the TLP.
    This VHDL source code is obtained from Xilinx. We use this source code as our baseline. Our FPGA designers modify this source
    code according to our own design goals. However, the problem that I mentioned seems to come from the original source code of Xilinx.
    The cause of the problem that you are facing with the read operation might be the same!!!

    - When we change the VHDL source code of the FPGA in such a way that TD bitfield of the completion TLP header is set to zero 
       or 
      When we change your source code (that runs on the DSP side) in such
      a way that it does not apply ECRC checks on the incoming TLPs (by resetting ECRC_CHK_EN, ECRC_CHK_CAP, ECRC_GEN_EN,
      ECRC_GEN_CAP bitfields of the PCIE_ACCR register), 
      Everything works fine with the read operation.

    Best regards,
    Hakan

  • Hakan, I have verified that changing the PCIE_ACCR registers work here with our FPGA also.  Now read/writes work without an issue.  This is definitely something that is not noted in any of the FPGA's PCIe Core Gen documentation.  Thank you for testing!

  • Hello Jay,

    Hakan bey is still having reading issue from FPGA, so we need to fix the issue as soon as possible. Have you done any reading test from FPGA with C6678?

    Best Regards

    Yunus Karaborek

     

  • I'm sorry we've never tried that here where the FPGA becomes bus master - it has always been the DSP.  Have they enabled the busmaster capability in the FPGA?

  • Hello All,

    I am sorry that I did not see the last page and the problem was fixed by Hakan bey .  I want to thank you very much for him contrubition to e2e community.

    Many Thanks for your great support.

    Best Regards

    Yunus

  • Jay,

    We have a new issue. Could you please take a look at it ?

    You can find it through :

    http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/157745.aspx

    Best regards,
    Hakan

  • Hello!
    Sorry for hijacking other's thread, but your example and hint is just what needed. Lesson learnt: RC have to configure remote EP, so EP's BAR know their target address window. TI's example does not do that, because two DSPs configure their own BARs. When you have EP device like FPGA, DSP as RC should write BAR configs to EP.