Linux/66AK2E02: PCIe end point not detecting in the PDK 4.1.0.6

sathish v

Part Number: 66AK2E02

Tool/software: Linux

Hi,

We have our K2E based custom board which contains Xilinx Spartan-6 FPGA connected to K2E via PCIe interface. We configured K2E ARM core to be root complex and FPGA is configured as end point.

Previously, we were using the K2E with MCSDK version 3.1 with linux 3.10. In this, K2E was detecting the FPGA end point (verified in boot log as well as by lspci command).

Now we need to port our existing work to the newer version of PDK 4.1.0.6. In this PDK version, K2E is not able to detect the FPGA end point. By analysing the boot log we see PHY link never came up error. Below is the boot log for the same. We dont know what is going wrong in the PDK. Any help is appreciated.

[    3.776986] OF: PCI: host bridge /soc/pcie@21800000 ranges:
[    3.777017] OF: PCI:   MEM 0x50000000..0x5fffffff -> 0x50000000
[    4.775261] keystone-pcie 21801000.pcie: phy link never came up
[    5.775260] keystone-pcie 21801000.pcie: phy link never came up
[    6.775260] keystone-pcie 21801000.pcie: phy link never came up
[    7.775260] keystone-pcie 21801000.pcie: phy link never came up
[    8.775260] keystone-pcie 21801000.pcie: phy link never came up
[    8.775273] keystone-pcie 21801000.pcie: phy link never came up
[    8.775670] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    8.775687] pci_bus 0000:00: root bus resource [bus 00-ff]
[    8.775701] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    8.775750] pci 0000:00:00.0: [104c:b009] type 01 class 0x060400
[    8.776336] PCI: bus0: Fast back to back transfers disabled
[    8.776605] PCI: bus1: Fast back to back transfers enabled
[    8.776765] pci 0000:00:00.0: PCI bridge to [bus 01]
[    8.777212] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    8.777231] pcie_pme 0000:00:00.0:pcie001: service driver pcie_pme loaded
[    8.777533] aer 0000:00:00.0:pcie002: service driver aer loaded
[    8.779163] OF: PCI: host bridge /soc/pcie@21020000 ranges:
[    8.779191] OF: PCI:   MEM 0x60000000..0x6fffffff -> 0x60000000
[    9.775261] keystone-pcie 21021000.pcie: phy link never came up
[   10.775260] keystone-pcie 21021000.pcie: phy link never came up
[   11.775260] keystone-pcie 21021000.pcie: phy link never came up
[   12.775260] keystone-pcie 21021000.pcie: phy link never came up
[   13.775260] keystone-pcie 21021000.pcie: phy link never came up
[   13.775273] keystone-pcie 21021000.pcie: phy link never came up
[   13.775670] keystone-pcie 21021000.pcie: PCI host bridge to bus 0001:00
[   13.775686] pci_bus 0001:00: root bus resource [bus 00-ff]
[   13.775700] pci_bus 0001:00: root bus resource [mem 0x60000000-0x6fffffff]
[   13.775748] pci 0001:00:00.0: [104c:b009] type 01 class 0x060400
[   13.776320] PCI: bus0: Fast back to back transfers disabled
[   13.776592] PCI: bus1: Fast back to back transfers enabled
[   13.776768] pci 0001:00:00.0: PCI bridge to [bus 01]
[   13.777210] pcieport 0001:00:00.0: Signaling PME through PCIe PME interrupt
[   13.777229] pcie_pme 0001:00:00.0:pcie001: service driver pcie_pme loaded
[   13.777522] aer 0001:00:00.0:pcie002: service driver aer loaded

over 6 years ago

0 Rex Chang over 6 years ago

TI__Guru 50170 points

Hi, Satish,

The RC mode was set in kernel in MCSDK, but moved to U-boot in ProcSDK for upstreaming reason. Please refer to this thread, e2e.ti.com/.../567282 , and check if that is your case?

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Thank you for the response rex and sry for the late reply. I have checked that thread already. The thing in that case is, latest linux was booted with 2013 uboot version. But in my case, both uboot and linux are from same PDK release. Corect me if I am wrong, ideally without any modifications uboot should initialize the PCIe rc node right, or do I have enable some config. How to make sure that uboot is initializing the PCIe to rc mode?

regards,

Sathish V

0 sathish v over 6 years ago in reply to sathish v

Intellectual 520 points

Hi,

On further debugging, following observations were made :
1 pcie lssm state is changing between 6 (detect wait) and 2(poll active).
2 On debugfs, clock enable count for pcie is 0 which was accquired by executing following command
cat /sys/kernel/debug/clk/pcie/clk_enable_count
Output :0
so to cross verify whether clock is enabled during enumeration, __clk_is_enabled() is used before and after pci scaning in ks_pcie_probe()
[drivers/pci/dwc/pci-keystone.c]. this shows clock is getting enabled during enumeration.

Based on the above observation, we don't know why enumeration is getting stuck at "poll active" state. Please help us for further debugging.

Kernel log:

3.777071] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[ 3.777201] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[ 3.781366] OF: PCI: host bridge /soc/pcie@21020000 ranges:
[ 3.781397] OF: PCI: MEM 0x60000000..0x6fffffff -> 0x60000000
[ 3.783326] ks_dw_pcie_link_up lssm = |00000000|
[ 3.783341] ks_dw_pcie_link_up lssm = |00000006|
[ 3.875284] ks_dw_pcie_link_up lssm = |00000002|
[ 3.975284] ks_dw_pcie_link_up lssm = |00000006|
[ 4.075284] ks_dw_pcie_link_up lssm = |00000002|
[ 4.175283] ks_dw_pcie_link_up lssm = |00000002|
[ 4.275284] ks_dw_pcie_link_up lssm = |00000006|
[ 4.375283] ks_dw_pcie_link_up lssm = |00000002|
[ 4.475283] ks_dw_pcie_link_up lssm = |00000006|
[ 4.575283] ks_dw_pcie_link_up lssm = |00000002|
[ 4.675283] ks_dw_pcie_link_up lssm = |00000002|
[ 4.775286] keystone-pcie 21021000.pcie: phy link never came up
[ 4.775302] ks_dw_pcie_link_up lssm = |00000006|
[ 4.875284] ks_dw_pcie_link_up lssm = |00000002|
[ 4.975284] ks_dw_pcie_link_up lssm = |00000002|
[ 5.075283] ks_dw_pcie_link_up lssm = |00000006|
[ 5.175284] ks_dw_pcie_link_up lssm = |00000002|
[ 5.275285] ks_dw_pcie_link_up lssm = |00000002|
[ 5.375285] ks_dw_pcie_link_up lssm = |00000006|
[ 5.475285] ks_dw_pcie_link_up lssm = |00000002|
[ 5.575283] ks_dw_pcie_link_up lssm = |00000002|
[ 5.675283] ks_dw_pcie_link_up lssm = |00000006|
[ 5.775285] keystone-pcie 21021000.pcie: phy link never came up

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Hi, Sathish,

Could you dump the following serdes registers:

0x02321FE0 0x02327FE0
0x02321FE4 0x02327FE4
0x02321FF4 0x02327FF4

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Hi Rex,
These are the serdes register dumps which you have asked for.

serdes register 0x02321FE0 |00000001|
serdes register 0x02321FE4 |00000001|
serdes register 0x02321FF4 |00000000|
serdes register 0x02327FE0 |00000000|
serdes register 0x02327FE4 |00000000|
serdes register 0x02327FF4 |10000003|

Now we are trying to dump same serdes register in the 3.10 kernel (in which pcie is working )
When I get the result I will post the result .

Regards,
Sathish

0 sathish v over 6 years ago in reply to sathish v

Intellectual 520 points

Hi Rex,
Serdes register dump for 3.10 kernel.

serdes register 0x02321FE0 |00000001|
serdes register 0x02321FE4 |00000000|
serdes register 0x02321FF4 |10000002|
serdes register 0x02327FE0 |00000000|
serdes register 0x02327FE4 |00000000|
serdes register 0x02327FF4 |10000003|

Regards,
Sathish

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Sathish,

You use both of the PCIe0 and PCIe1 interfaces, right? The serdes register dump sort of answered my suspicion on PCIe LTSSM not up. However, I can't explain with the same hardware why serdes PLL doesn't lock. I'll need to research a little bit more.

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Rex,

Thank you for helping us out. Yes we am using both interfaces. So what is the cause for the LTSSM not up? Where can I find the SERDES register information. I am not able to find it.

Sathish

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Sathish,

All documents are under 66ak2e device page, www.ti.com/.../66AK2E05. In User's Guide section under Technical Documents tab, The Serdes for Keystone-II User's Guide is the one.

Rex

0 Rex Chang over 6 years ago in reply to Rex Chang

TI__Guru 50170 points

Hi, Sathish,

Just checking if you make any progress. I compared the serdes configuration between MCSDK and ProcSDK and noticed that it was done in PCIe node of the dts file in MCSDK, but separate node in ProcSDK. In ProcSDk, only pcie1_phy is defined. If you didn't have the other phy defined, then the serdes won't get configured for the other pcie port.

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Hi Rex

We have added pcie0_phy node entry in keystone.dtsi file. Please find the attachement for the same.

Initially pcie 0 was disabled, we enabled it by modifying status = "okay" in keystone.dtsi file, so both pcie 0 and pcie 1 was enabled, but still enumeration was failing. Then we have disabled pcie 1 (In keystone-k2e-evm.dts file, pcie1_phy, pcie1 status changed to disabled) and tried only with pcie 0. Still enumeration was failing.

These are the Updated sardes register for kernel 4.9 with pcie 0 enabled and pcie 1 disabled.

serdes register 0x02321FE0 |00000000|
serdes register 0x02321FE4 |00000000|
serdes register 0x02321FF4 |10000003|
serdes register 0x02327FE0 |00000001|
serdes register 0x02327FE4 |00000001|
serdes register 0x02327FF4 |00000000|

After this we tried enabling phy loop-back by writing 0x40000000 in the 0x02320200 and 0x02320400 serdes register. In this case, enumeration was completing without any failure. But , we don't know how to test the data transfer in loop back mode.

Regards,
Sathish

keystone_dtsi.zip

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

HI, Sathish,

Serdes is now up, what's the LTSSM_EN state now? Could you see if it keeps trasitioning between idle and training?

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Hi Rex,

LTSSM_EN state is still transition between (0x02 POLL_ACTIVE) and (0x06 DETECT_WAIT).
Any suggestion??

[ 3.793634] ks_dw_pcie_link_up val = |00000006|
[ 3.885288] ks_dw_pcie_link_up val = |00000002|
[ 3.985288] ks_dw_pcie_link_up val = |00000006|
[ 4.085288] ks_dw_pcie_link_up val = |00000002|
[ 4.185288] ks_dw_pcie_link_up val = |00000002|
[ 4.285287] ks_dw_pcie_link_up val = |00000006|
[ 4.385287] ks_dw_pcie_link_up val = |00000002|
[ 4.485288] ks_dw_pcie_link_up val = |00000006|
[ 4.585287] ks_dw_pcie_link_up val = |00000002|
[ 4.685288] ks_dw_pcie_link_up val = |00000002|
[ 4.785290] keystone-pcie 21801000.pcie: phy link never came up
[ 4.785317] ks_dw_pcie_link_up val = |00000006|
[ 4.885288] ks_dw_pcie_link_up val = |00000002|
[ 4.985287] ks_dw_pcie_link_up val = |00000002|
[ 5.085287] ks_dw_pcie_link_up val = |00000006|
[ 5.185288] ks_dw_pcie_link_up val = |00000002|
[ 5.285289] ks_dw_pcie_link_up val = |00000002|
[ 5.385288] ks_dw_pcie_link_up val = |00000006|
[ 5.485289] ks_dw_pcie_link_up val = |00000002|
[ 5.585288] ks_dw_pcie_link_up val = |00000002|
[ 5.685287] ks_dw_pcie_link_up val = |00000006|

Regards
Sathish

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Hi, Sathish,

I just realized that you were dumping the ltssm_state bit in DEBUG0 register. We actually want to monitor LTSSM_EN bit in CMD_STATUS register. It should be toggling between 0 and 1. I want to be sure the training get initiated.

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

oh... sry for the misunderstanding rex. I will check that field and post the same once I get back to my office.

0 sathish v over 6 years ago in reply to sathish v

Intellectual 520 points

Hi Rex,

CMD_STATUS registre value is |00100107|, and LTSSM_EN bit is always 1.
I am reading CMD_STATUS in ks_dw_pcie_link_up function.

Regards,
Sathish

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Hi, Sathish,

All configuration look good to me. My understanding is that when the LTSSM_EN bit is set to 1, the training is not software related, but I can't explain why it shows differently between different kernel versions. I am checking internally for an explanation. Is your setup allowed to connect other EP device to see if it works? With TI EVM, I see no differences using either Kernel version. I have a SATA drive connected, and lspci can see the SATA drive.

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Hi rex,
Thank you for the your help. The thing is the pcie line is hard wired. There is no facility to connect other EP. One think I wanted to ask is the EP(FPGA) is running PCIe gen 1.1 core. will this have any impact on this?

sathish

0 sathish v over 6 years ago in reply to sathish v

Intellectual 520 points

Hi Rex,

We also have following observation.

In kernel 4.9 vendor id of root complex is appear as 0xb009 but in kernel 3.10 it is 0x8888

Kernel 4.9 log.

# lspci
00:00.0 PCI bridge: Texas Instruments Device b009 (rev 01)

Kernel 3.10 log

# lspci
00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01)
01:00.0 RAM memory: Xilinx Corporation Default PCIe endpoint ID

Sathish.

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Hi, Sathish,

I had discussion internally and one of the suspicions is the serdes change between 2 releases. We moved the serdes configuration out of c file and is one of the binary files in k2-fw-initrd.cpio.gz. This binary file for PCIe should be compared with the binary in c file for 3.10 kernel. You can compare what values were written to the PCIe IP.

The PCIe IP should work with Gen 1 mode, but you can try to force it in Gen 1 mode to exclude if it is the issue.

One question was brought up during the discussion is that why TI sees no issue on our setup. A suggestion isto check if it requires calibration. I am not sure if this applies, but it's one of suggestions to check.

I'll also do the serdes binary comparison. If you see something different, you may want to give it a try to see if it helps.

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Hi Rex,

I compared the sardes configuration present in c file of kernel 3.10 with ks2_gbe_serdes.bin file of kernl 4.9.
In ks2_gbe_serdes.bin some of the configurations are extra and some configurations are modified.

Extar added configurations in kernel 4.9 are
02080000 00170000 00100000 00010000
04080000 00170000 00100000 00010000
06080000 00170000 00100000 00010000
08080000 00170000 00100000 00010000
0aa40000 00070000 00000000 00350000
0aa40000 000f0000 00080000 00a80000

Modified configurations in kernel 4.9
{0x022c, 23, 16, 0x20}, -kernel 3.10
022c0000 00170000 00100000 00300000 -kernel 4.9

{0x042c , 23, 16, 0x20}, -kernel 3.10
042c0000 00170000 00100000 00300000 -kernel 4.9

{0x082c, 23, 16, 0x20}, -kernel 3.10
082c0000 00170000 00100000 00300000 -kernerl 4.9

I had copy serdes_config structure from kernel 3.10 to drivers/phy/phy-keystone-serdes.c file(kernel 4.9) and load configuration from structure instead of ks2_gbe_serdes.bin, but still enumeration was failing.

Sathish.

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Hi, Sathish,

Is it a typo that you meant ks2_pcie_serdes.bin, instead of ks2_gbe_serdes.bin?

When you mentioned Kernel 3.10, which exact release were you using? I believe there were Serdes changes between release 3.1.3.6, and 3.1.4.7. They are kernel 3.10.61 and 3.10.72 repectively. PSDK 4.1 should be closer to kernel 3.10.72, but I want to be sure.

I'll be out of office for a week, but will continue to work with you after I am back in office.

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Hi Rex,

Sorry its typo, i mean ks2_pcie_serdes.bin only.
We are using kernel version 3.10.61.
Thanks again for helping us out.

Sathish.

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Hi, Satish,

We are not sure what else is involved. We don't have the setup to investigate further.

Rex

0 sathish v over 6 years ago in reply to Rex Chang

Intellectual 520 points

Hi Rex,

For time being we are moved to older version of kernel 3.10.

Sathish

0 Rex Chang over 6 years ago in reply to sathish v

TI__Guru 50170 points

Satish,

Thanks. Since I can't reproduce the issue and don't have the setup to investigate further. I'll close this thread.

Rex

Processors

Processors forum

Linux/66AK2E02: PCIe end point not detecting in the PDK 4.1.0.6