Another Keystone II PCIe link up issue

Don Clementi

Other Parts Discussed in Thread: CDCM6208V1, TCI6636K2H, TCI6638K2K

We have 2 Keystone II (EVMK2H) connected together using a Silicon Turnkey Express AMC to SMA Ultra 9000 board to interconnect Lanes 4 and 5, and provide 100 MHz (FCLK+) and 25 MHz (TCLKB+) reference clocks. We are using a TI CDCM6208V1 as the clock module for the 25 and 100 MHz clocks.

Note that one EVMK2H is at Rev. 4 and the other is at Rev. 3.

We have used CCS V5.5 to build the MCSDK (V3.1.3.6) PDK libraries with the PCIe LLD.

We have also used CCS to build the PCIe example code, with one built as an RC and the other as an EP.

Both EVMK2Hs are powered up in DSP No-boot mode, and are connected to 2 different Host PCs using XDS200 USB emulator.

The PCIe example code are downloaded to C66xx_0 on both boards in debug mode and then started up.

The PCIe sample code successfully initializes everything for both RC and EP, but link training fails. In fact, the port link DEBUG0 register (offset 0x1728) indicates LTSSM state is 0 (DETECT_QUIET state) and never transitions. We have tried to manually start RC and EP simultaneously and asynchronously but link state never changes.

Does anyone have an idea of what issue there could be with this configuration or setup?

over 10 years ago

0 Raja over 10 years ago

TI__Guru* 81335 points

Welcome to the TI E2E forum. I hope you will find many good answers here and in the TI.com documents and in the TI Wiki Pages (for processor issues). Be sure to search those for helpful information and to browse for the questions others may have asked on similar topics (e2e.ti.com). Please read all the links below my signature.

Please go through below wiki for PCIe related FAQs and Resources for PCIe link issue,

http://processors.wiki.ti.com/index.php/PCI_Express_(PCIe)_Resource_Wiki_for_Keystone_Devices

Thank you.

0 Ganapathi Dhandapani95 over 10 years ago

TI__Mastermind 28085 points

Hi,

I have connected 2 Keystone II (EVMK2H) together using Dual EVM Break out Card(CI2EVMBOC) to test the PCIe example between two devices. It is working fine. I use internal clock for PCIe_CLK.

The MCU is control the MUX selection for internal and external PCIe reference clock to SOC. I think by-default MCU is configured as internal PCIe reference clock, If you using external clock means you need to change the MCU MUX section.

Thanks,

0 Don Clementi over 10 years ago in reply to Ganapathi Dhandapani95

Prodigy 60 points

Thanks for the quick reply.

Is the MCU MUX selection software controlled through a register? I cannot find any documentation that indicates how to configure it.

I also cannot find the dual EVM card that you referenced to see how it is different from our configuration.

Thanks again,
Don

0 Ganapathi Dhandapani95 over 10 years ago in reply to Don Clementi

TI__Mastermind 28085 points

Hi,

You able to control the MUX selection via BMC console. Operating the PCIe from the local clock is the standard use case. All EVMs are tested for this fuctionality prior to shipment from the factory. See below BMC console log for enable/disable PCIe CLK MUX.

[00:00:52]  BMC>pcie
[00:00:52]  Executing command "pcie"
[00:00:52]  The PCIE CLK Mux select is currently 0
[00:01:22]  BMC>pcie enable
[00:01:22]  Executing command "pcie"
[00:01:22]  The PCIE CLK Mux select is currently 1
[00:01:22]  Setting select to 0
[00:01:30]  BMC>pcie disable
[00:01:30]  Executing command "pcie"
[00:01:30]  The PCIE CLK Mux select is currently 0
[00:01:30]  Setting select to 0
BMC>

Note: Better to use latest BMC version. If it is not the latest, v1.0.2.6, please upgrade it by following the steps in "UCD Power Management Modules In-Field Update" at the EVMK2H Hardware Setup link in Getting Started Chapter of the Keystone-2 User's Guide.

Thanks,

0 Don Clementi over 10 years ago in reply to Ganapathi Dhandapani95

Prodigy 60 points

Hi,

I had to update the Rev 3 EVMK2H to BMC v1.0.2.6. The Rev 4 EVMK2h already had the latest BMC version.

Once I did I ran test again with PCIE external clock mux enabled for both the RC and EP configured EVMs. However, both sides failed to complete the PCIE power domain configuration operation. It seems that pciePowerCfg() function is waiting for PCIe power domain state transition to occur on PTSTAT [bit 3] .

I also setup breakpoints after the above function completed and the enabled the PCIE external clock mux. However, PCIe still failed to link. In one test, the RC LTSSM state did transition to DETECT_ACT, but never transitioned to POLL state. The EP LTSSM state always remained in DETECT_QUIET.

Is there a different power domain / module to configure for external clock?

Since the PCIE clock is external, is PCIE power domain operation still needed (i..e can I remove pciePowerCfg() function from the example?

Regards,
Don

0 Ganapathi Dhandapani95 over 10 years ago in reply to Don Clementi

TI__Mastermind 28085 points

Hi,

I do not have resource to reproduce your issue. I will check with my team and get back to you. Take a look at below thread and add "pciePowerDownCfg()" before "pciePowerCfg()". If you got same issue, please share the debug console log message.

CI2EVMBOC can be ordered through the distributors such as Avnet.

Thanks,

0 Don Clementi over 10 years ago in reply to Ganapathi Dhandapani95

Prodigy 60 points

I added pciePowerDownCfg() function and it assures the power is off prior to power domain configuration, but it did not seem to change PCIe link state.

For your working configuration, can you provide the MC SDK version, PDK version, and CCS version you used?

Did you rebuild the PDK platform libraries prior to building your working example?

What GEL file(s) did you use to initialize your target configuration? (Note, we used "TCI6636K2H" type with the XDS2xx USB Emulator.)

Can you provide binaries that we can try on our EVM platform? (This might help us assure the problem is not our example software build but HW or tool configuration issue.)

Thanks,
Don

0 Ganapathi Dhandapani95 over 10 years ago in reply to Don Clementi

TI__Mastermind 28085 points

Hi,

My working configuration: MCSDK(mcsdk_bios_3_01_03_06), PDK(pdk_keystone2_3_01_03_06) and CCS 6.1

My test of K2H EVM PCIE uses two EVM connected via TI breakout card(CI2EVMBOC).
· The 100MHz reference clock is generated on board, that is each EVM uses its own internal clock.
· There is no need to re-build the PCIE driver library.
· only needs to build the test application, using CCS.
· whether or not having GEL file to initialize the board shouldn’t matter, the main PLL is slower if no GEL. I use "xtcievmk2x.gel" file for my testing.(MCSDK Path: C:\ti\mcsdk_bios_3_01_03_06\tools\program_evm\gel\xtcievmk2x.gel).

Please try to test the PCIe example with internal reference clock mode.

Thanks,

0 Don Clementi over 10 years ago in reply to Ganapathi Dhandapani95

Prodigy 60 points

Hi,

I tried PCIe DSP test example using internal reference clock but still no PCIe link occurs.

Using BMC v1.0.2.6, I assured both EVMs were using internal clock :

BMC>pcie
[01:50:56] Executing command "pcie"
[01:50:56] The PCIE CLK Mux select is currently 0

I am still using AMC to SMA to connect PCIe Tx/RX lanes (port 4 & 5) since we cannot purchase the CI2EVMBOK.

Below is the Console outputs for both the EVMs, which contains configuration and initialization after loading and executing test example. I also included the Port Link Control registers' values while attempting to link.

===============================================================================

Keystone II PCIe Example Test (EVMK2H) Rev. 3.0
-----------------------------------------------

PCIE_K2HC66BiosExampleProject built with CCS V5.4 and loaded through XDS2xx USB Emulator on C66xx_0

GEL file (xtcievmk2x.gel from D:\TI\mcsdk_bios_3_01_03_06\tools\program_evm\gel) initialization:
------------------------------------------------------------------------------------------------
C66xx_0: GEL Output:
Connecting Target...
C66xx_0: GEL Output: TCI6638K2K GEL file Ver is 1.3
C66xx_0: GEL Output: Detected PLL bypass enabled: SECCTL[BYPASS] = 0x00800000
C66xx_0: GEL Output: (2a) MAINPLLCTL1 = 0x00000040
C66xx_0: GEL Output: (2b) PLLCTL = 0x00000048
C66xx_0: GEL Output: (2c) PLLCTL = 0x00000048
C66xx_0: GEL Output: (2d) Delay...
C66xx_0: GEL Output: (2e) SECCTL = 0x00810000
C66xx_0: GEL Output: (2f) PLLCTL = 0x0000004A
C66xx_0: GEL Output: (2g) Delay...
C66xx_0: GEL Output: (2h) PLLCTL = 0x00000048
C66xx_0: GEL Output: (4)PLLM[PLLM] = 0x0000000F
C66xx_0: GEL Output: MAINPLLCTL0 = 0x05000000
C66xx_0: GEL Output: (5) MAINPLLCTL0 = 0x07000000
C66xx_0: GEL Output: (5) MAINPLLCTL1 = 0x00000040
C66xx_0: GEL Output: (6) MAINPLLCTL0 = 0x07000000
C66xx_0: GEL Output: (7) SECCTL = 0x00890000
C66xx_0: GEL Output: (8a) Delay...
C66xx_0: GEL Output: PLL1_DIV3 = 0x00008002
C66xx_0: GEL Output: PLL1_DIV4 = 0x00008004
C66xx_0: GEL Output: PLL1_DIV7 = 0x00000000
C66xx_0: GEL Output: (8d/e) Delay...
C66xx_0: GEL Output: (10) Delay...
C66xx_0: GEL Output: (12) Delay...
C66xx_0: GEL Output: (13) SECCTL = 0x00090000
C66xx_0: GEL Output: (Delay...
C66xx_0: GEL Output: (Delay...
C66xx_0: GEL Output: (14) PLLCTL = 0x00000041
C66xx_0: GEL Output: PLL has been configured (CLKIN * PLLM / PLLD / PLLOD = PLLOUT):
C66xx_0: GEL Output: PLL has been configured (122.88 MHz * 16 / 1 / 2 = 983.04 MHz)
C66xx_0: GEL Output: Power on all PSC modules and DSP domains...
C66xx_0: GEL Output: Power on all PSC modules and DSP domains... Done.
C66xx_0: GEL Output: WARNING: SYSCLK is the input to the PA PLL.
C66xx_0: GEL Output: Completed PA PLL Setup
C66xx_0: GEL Output: PAPLLCTL0 - before: 0x0x098804C0 after: 0x0x07080400
C66xx_0: GEL Output: PAPLLCTL1 - before: 0x0x00000040 after: 0x0x00002040
C66xx_0: GEL Output: DDR begin
C66xx_0: GEL Output: XMC setup complete.
C66xx_0: GEL Output: DDR3 PLL (PLL2) Setup ...
C66xx_0: GEL Output: DDR3 PLL Setup complete, DDR3A clock now running at 666 MHz.
C66xx_0: GEL Output: DDR3A initialization complete
C66xx_0: GEL Output: DDR3 PLL Setup ...
C66xx_0: GEL Output: DDR3 PLL Setup complete, DDR3B clock now running at 800MHz.
C66xx_0: GEL Output: DDR3B initialization complete
C66xx_0: GEL Output: DDR done

PCIE EP Example Test Output:
----------------------------
[C66xx_0]
**********************************************
* PCIe Test Start *
* EP mode *
**********************************************

Version #: 0x02010001; string PCIE LLD Revision: 02.01.00.01:Sep 3 2015:18:59:57

Debug: Serdes Setup Successfully
PCIe Power Up.
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Starting link training...
[hangs, no further output]

Port Link Control Registers:
---------------------------
0x21801700 00C00040 FFFFFFFF 07000004 1B0F6400

0x21801710 00030120 00000000 000103AA 00000500

0x21801720 00000000 00000003 0000AC00 08200000

0x21801730 00000000 00000000 00000000 00000000

0x21801740 0000000F 00000000 4021702F 00217005

0x21801750 00800000 00000000 00000000 00000000

0x21801760 00000000 00000000 00000000 00000000

0x21801770 00000000 00000000 00000000 00000000

0x21801780 00000000 00000000 00000000 00000000

0x21801700 00000000 00000000 00000000 00000000

0x21801790 00000000 00000000 00000000 00000000

0x218017a0 00000000 00000000 00000000 00000000

0x218017c0 00000000 00000000 00000000 00000000

0x218017d0 00000000 00000000 00000000 00000000

0x218017e0 00000000 00000000 00000000 00000000

0x218017f0 00000000 00000000 00000000 00000000

0x21801800 00000000 00000000 00000000 0000010F

===============================================================================

Keystone II PCIe Example Test (EVMK2H) Rev 4.0
----------------------------------------------

PCIE_K2HC66BiosExampleProject built with CCS V5.5 and loaded through XDS2xx USB Emulator on C66xx_0

GEL file (xtcievmk2x.gel from C:\TI\mcsdk_bios_3_01_03_06\tools\program_evm\gel) initialization:
------------------------------------------------------------------------------------------------
C66xx_0: GEL Output:
Connecting Target...
C66xx_0: GEL Output: TCI6638K2K GEL file Ver is 1.3
C66xx_0: GEL Output: Detected PLL bypass disabled: SECCTL[BYPASS] = 0x00000000
C66xx_0: GEL Output: (3a) PLLCTL = 0x00000040
C66xx_0: GEL Output: (3b) PLLCTL = 0x00000040
C66xx_0: GEL Output: (3c) Delay...
C66xx_0: GEL Output: (4)PLLM[PLLM] = 0x0000000F
C66xx_0: GEL Output: MAINPLLCTL0 = 0x07000000
C66xx_0: GEL Output: (5) MAINPLLCTL0 = 0x07000000
C66xx_0: GEL Output: (5) MAINPLLCTL1 = 0x00000040
C66xx_0: GEL Output: (6) MAINPLLCTL0 = 0x07000000
C66xx_0: GEL Output: (7) SECCTL = 0x00090000
C66xx_0: GEL Output: (8a) Delay...
C66xx_0: GEL Output: PLL1_DIV3 = 0x00008002
C66xx_0: GEL Output: PLL1_DIV4 = 0x00008004
C66xx_0: GEL Output: PLL1_DIV7 = 0x00000000
C66xx_0: GEL Output: (8d/e) Delay...
C66xx_0: GEL Output: (10) Delay...
C66xx_0: GEL Output: (12) Delay...
C66xx_0: GEL Output: (13) SECCTL = 0x00090000
C66xx_0: GEL Output: (Delay...
C66xx_0: GEL Output: (Delay...
C66xx_0: GEL Output: (14) PLLCTL = 0x00000041
C66xx_0: GEL Output: PLL has been configured (CLKIN * PLLM / PLLD / PLLOD = PLLOUT):
C66xx_0: GEL Output: PLL has been configured (122.88 MHz * 16 / 1 / 2 = 983.04 MHz)
C66xx_0: GEL Output: Power on all PSC modules and DSP domains...
C66xx_0: GEL Output: Power on all PSC modules and DSP domains... Done.
C66xx_0: GEL Output: Completed PA PLL Setup
C66xx_0: GEL Output: PAPLLCTL0 - before: 0x0x07080400 after: 0x0x07080400
C66xx_0: GEL Output: PAPLLCTL1 - before: 0x0x00002040 after: 0x0x00002040
C66xx_0: GEL Output: DDR begin
C66xx_0: GEL Output: XMC setup complete.
C66xx_0: GEL Output: DDR3 PLL (PLL2) Setup ...
C66xx_0: GEL Output: DDR3 PLL Setup complete, DDR3A clock now running at 666 MHz.
C66xx_0: GEL Output: DDR3A initialization complete
C66xx_0: GEL Output: DDR3 PLL Setup ...
C66xx_0: GEL Output: DDR3 PLL Setup complete, DDR3B clock now running at 800MHz.
C66xx_0: GEL Output: DDR3B initialization complete
C66xx_0: GEL Output: DDR done

PCIE RC Example Test Output:
----------------------------------

[C66xx_0]
**********************************************
* PCIe Test Start *
* RC mode *
**********************************************

Version #: 0x02010001; string PCIE LLD Revision: 02.01.00.01:Aug 24 2015:13:21:33

Debug: Serdes Setup Successfully
Power domain is already enabled. You probably re-ran without device reset (which is OK)
PCIe Power Up.
PLL configured.
Successfully configured Inbound Translation!
Successfully configured Outbound Translation!
Starting link training...
[hangs, no further output]

Port Link Control registers:
---------------------------
0x21801700 00C00040 FFFFFFFF 07000004 1B0F6400

0x21801710 00030120 00000000 000103AA 00000500

0x21801720 00000000 00000003 00000F00 08200000

0x21801730 00000000 00000000 00000000 00000000

0x21801740 0000000F 00000000 4021702F 00217005

0x21801750 00800000 00000000 00000000 00000000

0x21801760 00000000 00000000 00000000 00000000

0x21801770 00000000 00000000 00000000 00000000

0x21801780 00000000 00000000 00000000 00000000

0x21801790 00000000 00000000 00000000 00000000

0x218017a0 00000000 00000000 00000000 00000000

0x218017b0 00000000 00000000 00000000 00000000

0x218017c0 00000000 00000000 00000000 00000000

0x218017d0 00000000 00000000 00000000 00000000

0x218017e0 00000000 00000000 00000000 00000000

0x218017f0 00000000 00000000 00000000 00000000

0x21801800 00000000 00000000 00000000 0000010F

===============================================================================

Let me know if the attached output is helpful or if you need more information.

I think there must still be some interconnect configuration issue that is not correct.

Thanks for the help,

Don

0 Ganapathi Dhandapani95 over 10 years ago in reply to Don Clementi

TI__Mastermind 28085 points

Hi,

Based on your debug log and PCie control register, both EVM's are stuck in link training and the LTSSM State is stuck in "DETECT_QUIET" state.

The LTSSM States table in Appendix. A of C66x PCIe user guide, the POLL_ACTIVE state is coded as 0x02. If your current step is stuck in "DETECT_QUIET" and "DETECT_ACT" stages, you can set the "LINK_STATE" field to be 0x02 and set 1 in "FORCE_LINK" field to see if PCIe module could be forced to Polling Active status and move forward.

The LTSSM_STATE field (bits [4:0]) in DEBUG0 register (0x21801728) to show the status of the PCIe link. It will not be 0x11 if the link is disconnected.

Could you please share your Hardware setup connection image?

Thanks,

0 Don Clementi over 10 years ago in reply to Ganapathi Dhandapani95

Prodigy 60 points

I built and loaded the PCIe test example to run on each EVMK2H using the internal PCIE reference clock, one is configured as an EP and the other as as RC.

Using the XDS debugger, I start execution of each and then set the Port Force Link register's LINK_STATE to 2 for both the EP and RC, then set the FORCE_LINK bit to 1.

However the Port Link DEBUG0 register for both still indicated no link activity and did not transition to new states.

We currently have two EVMK2H, each connected to an SMA to AMC Ultra 9000 board, in attached document, with Lanes 4 & 5 TX / RX differential signals connected. The length of the SMA connector cables vary from 6 to 12 inches.

/cfs-file/__key/communityserver-discussions-components-files/791/7701.Ultra_2D00_9000.doc

The clock module board, CDCM6208V1 , mentioned previously was not used for this test.

I also assured that each EVMK2H was also setup to use the internal clock by setting pcie 0 through the BMC serial port interface.

Are there any test points on the EVMK2H which we can attach a scope to verify the PCIE reference clocks, power domain, settings are OK?

Thanks,

Don

Processors

Processors forum

Another Keystone II PCIe link up issue