TMS320C6678 and PCIe rev 2.0

I think it is correct to set up DIR_SPD bit in PL_GEN2 to switch to GEN2 in link training as you mentioned (TGT_SPEED is 0x2 by default).

 But I am not sure if you did the following in the testing as well:

" to write 0x01 to the REG_LINK_CAP register"

 Writing 0x1 to LINK_CAP register will limit maximum link speed to 2.5Gbps(GEN1). So we should leave LINK_CAP with the default reset value and set DIR_SPD bit to 0x1 in PL_GEN2 to switch to GEN2 speed for our PCIe module.

 We need to make sure the other peer (PLX3380) enables GEN2 support as well during the initialization.

 For the following statement in your original thread:

Lukasz Mitrega said:

 writing any value other than 0x01 causes the link to train

  successfully, however later reads from the data register show random
  values. The only way to get proper data after training - is to write
  0x01 to the REG_LINK_CAP register, even though the register is
  officially not R/W.

We should be able to get proper data with default setup in LINK_CAP register. If not, it might be some other issue to debug. But for this GEN2 issue, you may try to leave LINK_CAP with its reset value and see if the speed could be GEN2 between C6678 and the other peer.

 After link up, you could read LINK_SPEED in LINK_STAT_CTRL register to see if the speed after link initialization is GEN1 (LINK_SPEED=0x1) or GEN2 (LINK_SPEED=0x2).

Hello, thanks for reply, I made some investigation in the case.

I build a test program where I waited in a loop for a successful link train in gen 2. The success condition was :

- successful link training

- LINK_SPEED in LINK_STAT_CTRL equal 2

- successful read from the PLX 3380 register (using the configured BAR mechanism and pcie data memory space)

the success condition was reached only few times for hundreds loop pass.

I figured also, that setting the MAX_LINK_SPEED in LINK_CAP  to 1 or/and setting the TGT_SPEED in LINK_CTRL2 to 1, effects with training in GEN 1 only. In this situation the PCIE connection works fine (in GEN 1). While setting the MAX_LINK_SPEED and TGT_SPEED  values to 2, even if the link training ends wit success for gen 1, there is no possibility to read PLX 3380 registers.

I'm afraid that, the problems with training in GEN 2 speed is hardware related. In this case I have a question:

Is there a way to improve the physical link parameters using some register settings? Also if there was a successful link train in gen 2, together with data verification, but its so rare I can't wait for it in a target application, can I assume that the problem is not because of software configuration?

What is the reference clock frequency for DSP PCIe module please? Do you check the LOCK bit in PCIE_SERDES_STS register to make sure PLL has been locked before the link initialization please?

Are you using Spread Spectrum clock (SSC) or fixed frequency clock please? For SSC, you may need to program RX_CDR to 0x4 in SERDES_CFG0 and SERDES_CFG1 (in x2 lane case) registers to get better response from SSC.

You may also try to set SEL_DEEMPH to 0x1 (-3.5dB) in LINK_CTRL2 register to see if any change in your design for GEN2.

For the hardware SerDes requirement, please take a look at the Hardware Design Guide and SerDes Implementation Guide to see the jitter and signal integrity requirements for PCIe module. Hope they could help.

 Hello,

Thanks again for the tips. Unfortunately the tips didn't help us. By the way we use the same clock for the DSP PCIe module and the device (100MHz). Also the PLL lock is achieved. We need to focus on the Hardware Design and verify that all the requirements are met. We will ask for help again or inform about the successful results when we go further with the project.

 

Lukasz,

We are trying to get the C6678 to talk to the PLX3380 and are having quite some trouble, being new to PCIe and all.  Do you know where we could find some example code on how to configure it and transfer data?  We seem to be able to get the link to train, but haven't gotten beyond that.  Any help would be greatly appreciated.

Mike

Hello Mike,

Sorry for long delay.

For keystone pcie there is a" PCIe Use Cases for KeyStone Devices" documentation SPRABK8. As I remember we used the code example from this document to configure the pcie module.
For the plx 3380 we had an evaluation module with sample application and codes.

As I understand, You have a tms320x6678 connected with plx3380 chip using the pcie express. The first thing to do is to configure the keystone pcie express module and train the link.

Let me know what is Your current status. Perhaps I can give You some tips what to do next.

Just for information. We have a similar solution 6671 and plx3380 and it's working fine. We are reaching transfers up to 300 MB/s (it mostly depend on the host usb 3.0 controller).

Lukasz

Lukasz,

I've been working with Mike on this project, so I will fill in where we are at:

I am familiar with the SPRABK8 document and have been referencing it.  I will review the example code again.

To date, using the PCIE_sample code that is part of the PDK_C6678_1_1_2_6 we have been able to establish a link with PLX3380 board.  Curiously, using the same code, I was not able to link to a COTS USB PCIe card.

After establishing a link, using the PCIe sample, I have not determined what to use for a base address for the PLX3380.  From the SRPAK8 and SRPUGS6D docs, it seems that 2180_0000 should be the base address for the PCIe config space.  This is the handle address passed back by the Pcie_open function for device 0.  There do not appear to be any other devices on the bus, other than 0.  This doesn't seem correct.  It seems there should be at least a device 0 (Root Complex) and a device 1(End Point).  

When running the NcDev_AreYouThere function, and using 2180_0000 as the base address, I am able to r/w to the scratch area.  However, I am also able to r/w to the RO area, so that indicates to me that we are writing to something other than the target device.  Attempts to read the DevID so far have not been fruitful.  I am able to read a Dev ID at 6000_0000(local config space) of 0x104C that is the TI PCI-Bridge, but when I try to read at 6000_2000 (which should be remote configuration space) I get a number (0x4DA0) that I cannot correlate to a DevID that is documented for anything.  The PLX3380 should be 0x10B5.

At this point, I'm not sure that the PLX3380 is relevent.  I think we are still struggling to get across the bridge and communicate to whatever is on the other side.

Hello,

We check the link after training by :

DEBUG0 register in local pcie config (0x21801728) check LTSSM_STATE

LINK_STAT_CTRL register in local pcie config check LINK_SPEED

After this (using default configuration with disabled in/out address translation) we read the remote pcie config device vendor id (address 0x21802000) the read value is 0x338010B5. Also we read the address (PRODVENDID from plx 3380 reg 0x84) 0x60000084 and we got value 0x33800525.

Could You read those addresses and check the values.

Hi Lukasz,

Thanks for the pointers. Still not quite working.

Checked LTSSM_STATE and it is in L0, which I think means it is trained, true?

The link speed seems to be '2', do not know what that means, or even where to find documentation for it.

When you say "using default configuration", to you mean on the DSP side or the PLX side?  I'm using an example from TI to configure the DSP side, and it may not be doing the right things.

With IB and OB translation off, nothing seems to read correctly. With translation on, I get some values for vendor/dev ID,but not the ones you mention.

Thanks for any advice.

Mike

Hello,

The plx side configuration is done by the hardware. I mean that the dsp configuration is minimal. I'm quite sure, that after getting the link trained You should be able to read the remote (device, in this case plx) pcie configuration, the default address for pcie remote configuration is 0x21802000.

I noticed that there is some additional configuration I'm doing. Here is my code snippet

// bus mastership and response enable
  reg_value =
    SM_PCIE_CONFIG_STATUS_COMMAND_MEM_SP_BIT |
    SM_PCIE_CONFIG_STATUS_COMMAND_BUS_MS_BIT;

  SM_PCIE_CONFIG_LOCAL_STATUS_COMMAND_REG = reg_value;
  SM_PCIE_CONFIG_REMOTE_STATUS_COMMAND_REG = reg_value;

  // remote base address register 0
  SM_PCIE_CONFIG_REMOTE_BAR0_REG = SM_PCIE_DATA_BASE;

  // set request size for dma from PLX CHIP to PCIE interface to 256 bytes
  reg_value = SM_PCIE_CONFIG_REMOTE_DEVICE_CONTROL_STATUS;
  reg_value &= ~SM_PCIE_CONFIG_REMOTE_DEVICE_CONTROL_STATUS__READ_REQ_SIZE_MASK;
  reg_value |= SM_PCIE_CONFIG_REMOTE_DEVICE_CONTROL_STATUS__READ_REQ_SIZE_VAL;
  SM_PCIE_CONFIG_REMOTE_DEVICE_CONTROL_STATUS = reg_value;

  reg_value = SM_PCIE_CONFIG_REMOTE_VENDOR_DEVICE_ID_REG;
  reg_value = SM_PCIE_DATA (0x84);

  // enable pcie legacy interrupt A
  SM_PCIE_LEGACY_A_IRQ_ENABLE_SET_REG = 1;
  reg_value = SM_PCIE_CONFIG_LOCAL_MSI_CAP_REG;
  reg_value &= ~SM_PCIE_CONFIG_LOCAL_MSI_CAP__MS_EN_BIT;
  SM_PCIE_CONFIG_LOCAL_MSI_CAP_REG = reg_value;

Let me know if it helped somehow.

Lukasz.

Ok, I'm able to read the remote vendor/device id reg correctly.  Still can't read the data space register correctly.  temp = *((uint32_t *)0x60000084) returns a 0x20.

I must be using a different set of software for the driver code, I can't find SM_PCIE_* definitions anywhere in my ti directories.  But from the names I can figure out mostly what you are doing in the config.  Two that I can't guess are the SM_PCIE_DATA_BASE and *_READ_REQ_SIZE_VAL.  The local max size is set to 0x2, not sure what that means. What are you using for those?

I also moved the remote register configuration reads/writes to after link training.  The processor hangs otherwise.

Another interesting thing is that if I load code from CCS after a power cycle, the vendor/device id reg read correctly.  If I don't power cycle the EVM, I get incorrect values.  Not sure what is not being reset by the power cycle.

Thanks so much for the pointers.

Mike

Note also that I've not set any inbound or outbound translation, nor the remote BAR0.
Mike

Mike,

those defines are my custom defines (I don't use CSL or other TI codes). I sea they are understandable. The configuration should be done after link training (I forgot to mention). I remember that when I was using the EVM there also was a need to power cycle (or reset) EVM. I wrote then a routine to turn off/on the power domain for pcie (in the power sleep controller). As I remember there is a notice, that a domain should not be turned off/on but without it only the first (after power up) initialization allowed to read correct values from pcie space.

One thing is not clear for me. The correct/incorrect read are from the *((uint32_t *)0x60000084) or You mean the remote pcie config.

the SM_PCIE_DATA_BASE value is 0x60000000
the READ_REQ_SIZE_VAL is not that important As I remember that was for the plx dma.\

Lukasz,

Thanks for the clarification.

The correct read is from the remote config space, which I believe is 0x21802000.  I'm using TI's CSL and driver functions, so I don't read it directly, but I think thats the PLX Vendor/Device ID, it comes back as 3380 and 10b5 as you mentioned above.

It is the data space PRODVEND register at 0x84 that I can't yet read.

I'm going to try a couple more things. Will let you know.

Mike

Mike,

please clear one thing for me. Which is true :

1. The "The correct read is from the remote config space, which I believe is 0x21802000" works only once. Then a power cycle is needed. The "It is the data space PRODVEND register at 0x84 that I can't yet read." never read correct.

2. The "The correct read is from the remote config space, which I believe is 0x21802000" works always. The "It is the data space PRODVEND register at 0x84 that I can't yet read."works only once. Then a power cycle is needed.

Lukasz,

The first is correct.  Note that the remote config space read works fine, as long as I don't load code again.  In other words, for any given debug session, I believe the config read works multiple times.  It is just if I try and load code without powering off, it will not initialize correctly.  If I power cycle and reload code, it works again.

I have never gotten the data space PRODVEND register to read correctly yet.  I set the remote BAR to 0x60000000 per your example and now address 0x60000084 reads 0xffffffff.

Do you use outbound or inbound translation at all?

Mike

Lukasz,
Ok, I believe I have the data space read and write working. The key was to program the remote PLX BAR0 correctly. I had been programming BAR1, I think, since that is the one the controls the DSP data space and BAR0 controls the dsp config space, and is not changeable in RC mode.

Thank you for your help. We now need to configure the USB side of the PLX. Have you found any manuals that define what the PLX registers all do? When we ask PLX, they just say reverse engineer the code we give you, but that is very difficult, and it is Windows based, so a lot of translation is required. If you know of a manual or at least have sample USB 3 configuration code, that would be greatly appreciated.

Thank you again for your help.

Mike

Lukasz,

I need a little help sorting out all of the BAR and offsets and whatnot.

I am sucessfully initializing on the PCIe bus and training a link.  After linking, I am able to r/w to registers on the USB side of the PLX3380.  By setting the Product and Vendor ID to 1000 and 525 I can get the USB to enumerate on a 3.0 Bus and the transfer tool provided from PLX to connect to the device.

I am able to detect a transfer of data, and by monitoring the EP Stat register at offset 0x32C detect when data has been received and how much by looking at the counts in EP_AVAIL (330h) for GPEP0.

If I clear the flag in EP_STAT, I can continue to send packets as I please.

However, I'm not yet sure where this data is going.  I haven't sorted out where the base of the USB Ingress RAM is, or how this affects where the FIFO and/or DMA is going.

The FIFO base address for GPEP0 is default set at 200h from USB Ingress RAM per register 520h

From section 7.5.2 it appears that USB Ingress RAM is located at the address defined in BAR1 (offset 14h[31:16]).  So, if my Register base is the same as for EP Stat, etc, BAR1 should be at 0x60000014.  31:16 at this location is 0x8000.  So does this mean that my FIFO is at 0x8000_0200 or at 0x8200.  It seems that it has to be the former, as 0x8200 is not accessible memory space.  However, when I look at 0x8000_0200 I do not see anything that lines up with the data I've sent.

I have tried to set the DMA autoenable bit and configured the DMA address to 0x8120_0000 (an allocated buffer in memory), but do not see anything reflecting by input either.

So, I'm quite lost as to where the Endpoint data is going.  Is it being transferred to the FIFO, or to DMA target, or?  Is it just being flushed somewhere?

Any advice you can provide will be much appreciated.  Thanks