TMS320C6678: Unable to read C6678 PCIe endpoint PCIe Config memory (0x2180_0000)

Hi Yordan,

Thanks for the reply - looking forward to their response!
Brad

Here's the contents of our DEVSTAT register:

DEVSTAT: 0x0001_23C9

LENDIAN:         0b1             LITTLE ENDIAN

BOOTMODE[12:0]:  0x11E4          

BOOT DEVICE: 0x4             PCIe BOOT MODE

PCIe CFG:    0x3c            2048 MB, 64-BIT, BARs for BAR2/3 and BAR4/5

PCIESSMODE[1:0]: 0b00            END POINT

PCIESSEN:        0b1             PCIe module enabled

PACLKSEL:        0b0             Select PASSCLK

I also checked the CMD_STATUS and STATUS_COMMAND registers on both C6678s:

CMD_STATUS: 0x0000_0005

LTSSM_EN:          0b1 - Enable LTSSM
OB_XLT_EN:         0b1 - Enable outbound address translation
IB_XLT_EN:         0b1 - Enable inbound address translation
POSTED_WR_ENABLE:  0b0 - Disable internal bus master to use posted write commands (N/A for KeyStone)
APP_RETRY_EN:      0b0 - Disable all incoming PCIe transactions to be returned with a retry response
DBI_CS2:           0b0 - Disable writing to BAR MASK registers

STATUS_COMMAND: 0x0810_0146

IO_SP:              0b0 - N/A for KeyStone devices
MEM_SP:             0b1 - Enable memory accesses
BUS_MS:             0b1 - Enable bus mastership
PAR_ERR_RESP:       0b1 - PCIESS responds normally to parity errors
SERR_EN:            0b1 - Enable generation of error messages to RC
INTX_DIS:           0b0 - INTX generation enabled 
INT_STAT:           0b0 - No interrupts received
CAP_LIST:           0b1 - always 0b1 for PCIe
DAT_PAR_ERROR:      0b0 - No R/W parity errors from requester
SIG_TGT_ABORT:      0b1 - A function acting as completer terminates a request by issuing
                          completer abort completion status to requester

RX_TGT_ABORT:       0b0 - Requester has not received a completion with abort status
RX_MST_ABORT:       0b0 - Requester has not received a completion with unsupported request
                          status
SIG_SYS_ERROR:      0b0 - Function has not received ERR_FATAL or ERR_NONFATAL
PARITY_ERROR:       0b0 - No poisoned TLP received by function

The only troubling field is the SIG_TGT_ABORT

Hi Eric,

I am awaiting your feedback/input.

Thanks,
Brad

Hi,

CMD_STATUS: 0x0000_0005

LTSSM_EN:          0b1 - Enable LTSSM

OB_XLT_EN:         0b1 - Enable outbound address translation

IB_XLT_EN:         0b1 - Enable inbound address translation

Can you double check if this is a typo? The OB needs to be enabled.

Regards, Eric

Hi Eric,

I have adapted my driver to enable outbound translation in the C6678 endpoint:

C6678 CMD_STATUS (initial): 0x00000005
C6678 OB_SIZE (initial):    0x00000000

C6678 CMD_STATUS (final):   0x00000007
C6678 OB_SIZE (final):      0x00000000

Even with outbound translation enabled, I still am unable to access PCIe config space through BARs 2/3 - reads return 0xffff_ffff.

Thanks,
Brad

Hi Eric,

From the PCIe User Guide (sprugs6d.pdf), the lower 4 bits in BAR0, BAR2 and BAR4 are:

Bit    0: MEM_SPACE (0 = Memory BAR, 1 = I/O BAR)
Bit 2-1: TYPE (0b00 = 32-bit BAR, 0b10 = 64-bit BAR) 
Bit    3: PREFETCH

So, 

BAR0/1 = 0x0000_0000_D0C0_0004 -> 64-bit memory BAR
BAR2/3 = 0x0000_3802_0000_000C -> 64-bit PREFETCHABLE memory BAR
BAR4/5 = 0x0000_3801_8000_000C -> 64-bit PREFETCHABLE memory BAR

So, BARs 2/3 and 4/5 both have PREFETCHABLE bit enabled. Could that be the problem?

Thanks,
Brad

Hi,

My suggestion is to use BAR0/1. Why you used 0xD0C0_0004 instead of 0x0000_0004? In our PCIE driver example, the BAR is configured as no-prefetch. You can try BAR2/3 or BAR 4/5 with that as well, with inbound translation offset to 0x2180_0000.

Regards, Eric

Hi Eric,

My suggestion is to use BAR0/1.

Since BAR0/1 only provides a 4KB window, I need to use either BAR2/3 or BAR4/5 to access those config-space registers that sit above 0x1000 (e.g. STATUS_COMMAND is at offset of 0x1004 in PCIe config space and BAR0/1 doesn't provide access to that offset)

Why you used 0xD0C0_0004 instead of 0x0000_0004

The BAR of 0xD0C0_0004 is the value assigned by the operating system (Windows in my case) to the C6678 PCIe endpoint. I can't change this value. Also, I the PREFETCHABLE, TYPE, and MEM_SPACE bits are set by the OS during PCIe enumeration.

Please note that I am directly booting into PCIe boot mode.  I am not first booting into I2C and loading Boot Parameter or Boot Config tables - so I am not modifying any portion of the PCIe boot mode settings except for what is provided through the BOOTMODE pins.

You can try BAR2/3 or BAR 4/5 with that as well, with inbound translation offset to 0x2180_0000

I have tried both BAR2/3 and BAR4/5 with offsets of 0x2180_0000 and that did not make any difference.  I also tried offset of 0 (instead of 0x80_0000) and that didn't help either.

One interesting new piece of information is that (now that I am able to initialize DDR3 on my platform), I can now access (Read and Write) DDR3 memory with BAR4/5 offset set to 0x8000_0000. So I am not sure it has to do with the inbound offset.

Regards,
Brad

Hi Eric,

I tested reading the following areas through BAR2/3:

DDR3 EMIF Configuratiom:
0x2100_0000:  Reads returned 0xCCCC_CCCC
0x2100_0004:  Reads returned 0xCCCC_CCCC

HyperLink Configuration:
0x2140_0000:  Reads returned 0xCCCC_CCCC
0x2140_0004:  Reads returned 0xCCCC_CCCC

PCIe Configuration:
0x2180_0000: Reads returned 0xFFFF_FFFF
0x2180_0004: Reads returned 0xFFFF_FFFF

Queue Manager Configuration:
0x3400_0000: Reads returned 0xCCCC_CCCC
0x3400_0004: Reads returned 0xCCCC_CCCC

HyperLink Data:
0x4000_0000: Reads returned 0xCCCC_CCCC
0x4000_0004: Reads returned 0xCCCC_CCCC

PCIe Data:
0x6000_0000: Reads returned 0xCCCC_CCCC
0x6000_0000: Reads returned 0xCCCC_CCCC

Should I be able to access any of these areas over the PCIe link?  

Hi Eric,

If you can use BAR4/5 to access DDR, then there should be no issue to use BAR2/3 to access DDR as well by changing the inbound offset (please move BAR4/5 to pointing someplace else). 

Yes, I can use both BAR2/3 and BAR4/5 to access DDR!

You used BAR2/3 to access a lots of peripheral registers, I thought your trial is arbitrary and the returned data is garbage, the access doesn't work. But be aware that even your PCIE setting is correct, you are not necessarily able to access those. EMIF is 36-bit, need 36 to 32 bit mapping; Hyperlink register needs power up, Hyperlink data needs link established, etc....

OK. Understood that some of the peripheral register sets can only be accessed once enabled.  

Please make sure that either use BAR2/3 or BAR 4/5 you can access MSMC, or DDR or global L2. Then for the BAR0/1, where did you setup the BAR mask to 4KB, can you assign a 16KB mask?

Yes, currently BAR2/3 can access MSMC and L2SRAM while BAR4/5 can access DDR3.  

But, how can I modify the window size of the BAR0/1 beyond 4kB? I believe that is done by the ROM Boot Loader. Currently I am booting directly into PCIe Boot Mode.  My understanding is that to modify the PCIe BAR window sizes further than what is available from the RBL, I would need to boot into, for example, I2C boot mode and provide Boot Parameter and Boot Config Tables that would set the BAR0/1 MASK for 32KB window size.

Even the "PCIe Boot Mode Parameter Table" (in sprs691e.pdf) does not provide for setting Window 0 size - only Windows 1 (BAR2/3) and 2 (BAR4/5) are configurable in this table.  

So, how should I configure BAR0/1 for 32KB window size?

Thanks,
Brad