Forums

Does the returned pattern of alternating data and base address indicate anything  to anyone?

Mark,

I think we could focus on the error when writing data back from EP to RC first.

Could you please confirm there is no error after RC writes data to EP?

If so, could you check if both BUS_MS bit and MEM_SP bit are set in STATUS_COMMAND register in both RC and EP devices please?

When writing data to RC, the BARn register in RC should also be set to match the PCIe address from EP side. Otherwise, the packets will be rejected and error may be reported.

So could you please double check if the PCIe address coming from EP is what we expected (if the outbound translation in EP side is being used and being set correctly)?

And what is the setup of BARn (BAR0,1,etc) on RC side  (matching the incoming PCIe address?) and if the data could be arrived on RC side internal memory correctly due to the configuration inside of RC please?

Sincerely,

Steven

Steven,

Yes both the BUS_MS and MEM_SP are set on both sides. The inbound and outbound configuration are set such that I see write transactions arriving at the RC with correct address and data. Afew packets seem to have errors in the address as observed on the bus analyzer we are using. As far as reads go on the Integra we can capture one of a series of reads with an ack but no completion packet. I am assuming the rest of the series is stalled or aborted. I am not sure where the DSP is getting data but it is moving data to the dstBuf and about half looks correct. This maybe a cache issue. I am able to write and read back data from the RC to the EP correctly. We have not captured an NAKs or Completion pakets with unsupported requests that we can see.

7245.RC_ConfigPreError.txt

I attempted to attach a dump of the configuration before the error on the RC side (Integra). may be you can find something?

Mark,

The RC register dump seems fine. Could you also share the register setup on EP side as well?

Or could you share the example code on both RC and EP sides as what modifications you have done based on the PDK loopback example please?

One thing to note is that no matter Write or Read transactions from EP to RC, it is all outbound transactions for EP. So it is using the outbound translation registers set on EP side and inbound translation registers set on RC side, if EP initializes the transaction (Read or Write).

If you could see Write transaction from EP to RC arrived on RC correctly, the Read transaction from EP to RC should be very similar as well. I am not sure how you issue the Read/Write transactions from EP to RC, maybe some incorrect addresses/memory have been used. so any code related to it will be helpful. Thanks.

Sincerely,

Steven

7317.postConfig.dat.txt

The above file is the dump of the DSP EP configuration. I created this file by usung the COde composer save memory option. I calculat the address  by subtracting 2 from the line number and multipying by 4. The start is the Application space running trhough the config space as documented in the sprugs6a.pdf

0247.PCIe.zip

The above file is the 6678 code as end point.

3465.RC_Code.zip

the Above file is the RC Integra code using vxWorks

Mark,

Thanks a lot for posting the code.

I have one question about the "PCIeLoopback()" in "pcie_support.c" file on the EP side.

You said you are testing the loopback data flow which means RC writes data to EP and then EP writes data to RC.

But on the EP side, before waiting for the completion of RC writing to EP, there are already some activities on the EP PCIe port:

/***********************start of code***************************/

void PCIeLoopback(void)
{
    int i=0;
    uint32_t             *pcieBase;
    pcieBase = (uint32_t*)  (LOCAL_OB_CONTRL_ADDR_FPGA + 0x8004);
    dstBuf[0] = pcieBase[0];

  {
    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
        dstBuf[i] = 0;
    }
    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
        dstBuf[i] = pcieBase[i];
    }
    /**********************************************************************/
    /* Wait for a single message from the RC then echo it back            */
    /**********************************************************************/

    /* EP waits for the data received from RC */
    do {
      unsigned int key;

      /* Disable Interrupts */
      key = _disable_interrupts();

      /*  Cleanup the prefetch buffer also. */
      CSL_XMC_invalidatePrefetchBuffer();

      CACHE_invL1d ((void *)dstBuf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);
      CACHE_invL2  ((void *)dstBuf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

      /* Reenable Interrupts. */
      _restore_interrupts(key);

    } while(dstBuf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);

    System_printf ("End Point received data.\n");

/***********************end of code***************************/

If the "pcieBase[]" is pointed to the PCIe data space (0x60000000~0x6FFFFFFF in C66x), "dstBuf[i] = pcieBase[i];" means you are reading something from PCIe data space (from remote device over the PCIe link) to the dstBuf.

In the header file, it looks like pcieBase[] is pointed to the PCIe data space ("#define LOCAL_OB_CONTRL_ADDR_FPGA (0x61000000)"). But I am not sure what is the role for "0x8004" here. 

Besides, after waiting for the RC writing data to EP, the EP uses the same pcieBase[] to write data to RC as

    for (i=0; i<PCIE_BUFSIZE_APP; i++)
    {
      pcieBase[i] = dstBuf[i];
    }

Then I am not sure if it is really what you intend to do in this loopback testing.

Basically, when RC initiates the transfer (no matter Write or Read), the data being transferred to the PCIe data space in RC should be translated to the correct outbound PCIe address, which should match the EP BARn register and EP inbound translation address, then the data go to (or be fetched from) the EP dstBuf over the PCIe link.

When EP initiates the transfer (no matter Write or Read), the data being transferred to PCIe data space in EP should be translated to the correct outbound PCIe address, whcih should match the RC BARn register and RC inbound translation address, then data go to (or be fetched from) the RC dstBuf.

Please double check if it is the case in your project for both RC and EP sides. And if the pcieBase is pointed to the correct PCIe data space on both sides as well. Thanks.

Sincerely,

Steven

I have since removed that code or actully moved it down below the wait for PCIE_EXAMPLE_BUF_FULL.

3073.PCIe.zip

I attempted to attach this earlier but ut seemed to not be there.

Mark,

Thanks for the updated source files.

1. In the following code, I could understand the "Read 1" section. It looks like the OB/IB on RC/EP sides are matching each other (PCIe address from EP after translation is 0x90000000, which is matching RC BAR1 register 0x90000000).

Do you see any error only after the first EP reads from RC please?

/******* Start **********/

pcieBase = (uint32_t*)  LOCAL_OB_MEMORY_ADDR_BGPP;

  // Read 1
  for (i=0; i<PCIE_BUFSIZE_APP; i++)
  {
      dstBuf[i] = 0;
  }
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
      dstBuf[i] = pcieBase[i];
  }

/******* End **********/

2. For the second part "Read 2", I am not sure which address you want EP to write to. pcieBase is pointed to other place and if I calculated correctly, the PCIe address after translation is 0x60008004, which is not matching to the RC BAR1 register (0x90000000). So you write the dstBuf data to some location over PCIe link and read the data back to EP.

I am not sure which PCIe device will accept this packet with PCIe address 0x60008004. I hope this write transaction does not change any configuration over the PCIe link (not changing the setup of PCIe Switch, RC or other system setup).

Do you see the error after Read 2 please?

/******* Start **********/

  // Read 2
  pcieBase = (uint32_t*)  (LOCAL_OB_CONTRL_ADDR_FPGA + 0x8004);
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
    pcieBase[i] = dstBuf[i];
  }

  for (i=0; i<PCIE_BUFSIZE_APP; i++)
  {
      dstBuf[i] = 0;
  }
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
      dstBuf[i] = pcieBase[i];
  }

/******* End **********/

Sincerely,

Steven

The error seem to occur on the write back of the data from the DSP EP to the RC. By error I mean that the memory targeted by the write back to the RC from the EP is not written. I reserved a portion of memory at the 0xc0000000 address space NON cached. I never see data written back arrive. It also caused the RC to become unable to arite or read past the switch. After initialializing BAR0 of the RC this particular symptom went away. When the EP attempts to read from RC I have seen a single read captured by the analyzer sometimes with an ack but we have not captured any completion packet for the read as of yet. I am going to look through the old log we have captured and send you some later. Write now we are having problem capturing anything with the protocal analyzer.

7026.PCIe.zip

I was way off on my last reply I also injected an error on my code that I now have corrected.

/*****************************************************************************
 * Function: PCIeLoopback
 */
void PCIeLoopback(void)
{
    int i=0;
    uint32_t             *pcieBase;
    /**********************************************************************/
    /* Wait for a single message from the RC then echo it back            */
    /**********************************************************************/

    /* EP waits for the data received from RC */
    do {
      unsigned int key;

      /* Disable Interrupts */
      key = _disable_interrupts();

      /*  Cleanup the prefetch buffer also. */
      CSL_XMC_invalidatePrefetchBuffer();

      CACHE_invL1d ((void *)dstBuf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);
      CACHE_invL2  ((void *)dstBuf,  PCIE_EXAMPLE_DSTBUF_BYTES, CACHE_FENCE_WAIT);

      /* Reenable Interrupts. */
      _restore_interrupts(key);

    } while(dstBuf[PCIE_BUFSIZE_APP] != PCIE_EXAMPLE_BUF_FULL);

    System_printf ("End Point received data.\n");

    /* Loopback to RC what was written in the DST buffer.
       Write from EP to RC */
    /* if ((retVal = Pcie_getMemSpaceRange (handle, &pcieBase, NULL)) != pcie_RET_OK) {
      System_printf ("getMemSpaceRange failed\n", (int)retVal);
      exit(1);
    } */

  {
    pcieBase = (uint32_t*)  LOCAL_OB_MEMORY_ADDR_BGPP;

/***********************************************************************************************************************************

HERE IS Where the ER

    for (i=0; i<PCIE_BUFSIZE_TEST; i++)
    {
     pcieBase[i] = dstBuf[i];
    }
   /* Mark that the buffer is full, so RC can process it */
    pcieBase[PCIE_BUFSIZE_APP] = PCIE_EXAMPLE_BUF_FULL;

    /* Note on cache coherence: Write back is not necessary because pcieBase is in
       peripheral address space instead of physical memory*/

    System_printf ("End Point sent data to Root Complex, completing the loopback.\nEnd of Test.\n");
  }
  // Read 1
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
   dstBuf[i] = 0;
  }
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
   dstBuf[i] = pcieBase[i];
  }
  // Write and read to/from FPGA
 

/****** The following code is a test to see if reads and writes to the FPGA are succesful **/ 
  pcieBase = (uint32_t*)  (LOCAL_OB_CONTRL_ADDR_FPGA + 0x8000);
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
    pcieBase[i] = dstBuf[i];
  }

  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
   dstBuf[i] = 0;
  }
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
   dstBuf[i] = pcieBase[i];
  }
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
   dstBuf[i] = 0;
  }
  for (i=0; i<PCIE_BUFSIZE_TEST; i++)
  {
   dstBuf[i] = pcieBase[i];
  }

}

Mark,

I am a little confused. I think you correct some error in the source code and the one attached above "7026.PCIe.zip" should be the latest code under testing.

The observations you described before is based on this current code or the previous code please?

In other words, do you still see any error happening with the current code on EP side please?

Sincerely,

Steven

The current code I just sent, still has the pronblem that read and write transaction from the EP to the RC do not complete. After writing data to the RC, I attempt to veirfy by reading from both sides of the interface. Th EP reads return results similar to the results in my first post.

Steven,

We just got our PCIe Analyzer back up. We notice that the EP device our 6678 DSP sets the requester ID to 0x0000 for memory write and read requests. These requests start at bus 2 transition to bus 1 in the switch and arrive at the bus 0 of the switch which is the port to the RC. Does this cause a problem for completion packet from the RC to the EP?

We also noticed that our command status register has not set the POSTED_WR_EN bit. Is this also  problem for writes?

Thank for your help.