PCIe device not detected with PCIe driven by external clock

Pervez, 

I had to re-context myself with the Thread 90537. But I did not see replies from your team about the two register values:

• CM_CLKMODE_APLL_PCIE
• CTRL_CORE_SMA_SW_6

Also could you clarify the "working" and "non-working" setup, are they just different devices on the other side of the switch, or the driver software /clock configuration changed?

The dw_pcie_link_up() function simply polls the status register during link training. Timing out can be potentially caused by clock configurations. so please help me to understand your system - there is an external clock source that supplies refclk to both the AM57 and the switch (common clock topology). so we need to ensure the AM57:

1. ACSPCIe control MMR is configured in RX mode

2. clock mux is set to use the external refclk

3. sequence-wise, the refclk is enabled 100us before the reset pin to the switch is pulled high (reset pin is active low)

thanks

Jian

Hi Jian,

Exact register values are not given explicitly.

• CM_CLKMODE_APLL_PCIEis defined as,
  • #define DRA7XX_CM_CORE_CKGEN_INST 0x0104
  • cm2_7xx.h 126 #define DRA7XX_CM_CLKMODE_APLL_PCIE_OFFSET 0x0118
  • cm2_7xx.h 127 #define DRA7XX_CM_CLKMODE_APLL_PCIE DRA7XX_CM_CORE_REGADDR(DRA7XX_CM_CORE_CKGEN_INST, 0x0118)
• CTRL_CORE_SMA_SW_6 is NOT defined anywhere in code.

Also could you clarify the "working" and "non-working" setup, are they just different devices on the other side of the switch, or the driver software /clock configuration changed?

- Device 1 (“working”): External clock source, Am57 PCIe1 and AM57 PCIe2 each connected to a PCIe peripheral
- Device 2 (“not working”): Internal clock source, Am57 PCIe1 connected to a PCIe peripheral (as on Device 1)
- HW Problem can be excluded, Device 1 and Device 2 are ok under VxWorks.

The dw_pcie_link_up() function simply polls the status register during link training. Timing out can be potentially caused by clock configurations. so please help me to understand your system - there is an external clock source that supplies refclk to both the AM57 and the switch (common clock topology). so we need to ensure the AM57:
1. ACSPCIe control MMR is configured in RX mode
2. clock mux is set to use the external refclk
3. sequence-wise, the refclk is enabled 100us before the reset pin to the switch is pulled high (reset pin is active low)

  >>  We can't accurately find the data for above queries. We assume the HW setup would be correct, as they works under VxWorks, not in Linux.

Regards,
Perumal

Hello Jian,

Is it possible to have a short call to expedite this request? We can explain the problem better and get an idea of what information needs to be collected to RCA issue.

Thanks.

Hi Jain
I have attached kernel dmesg logs for your reference. We are using the dra7.dtsi file pcie related configuration.

[ 0.645973] pinctrl-single 4a003400.pinmux: 282 pins at pa fc003400 size 1128
[ 0.647230] dra7-pcie 51000000.pcie: Linked as a consumer to phy-4a094000.pciephy.3
[ 0.647309] dra7-pcie 51000000.pcie: GPIO lookup for consumer (null)
[ 0.647316] dra7-pcie 51000000.pcie: using device tree for GPIO lookup
[ 0.647329] of_get_named_gpiod_flags: can't parse 'gpios' property of node '/ocp/axi@0/pcie@51000000[0]'
[ 0.647339] of_get_named_gpiod_flags: can't parse 'gpio' property of node '/ocp/axi@0/pcie@51000000[0]'
[ 0.647346] dra7-pcie 51000000.pcie: using lookup tables for GPIO lookup
[ 0.647354] dra7-pcie 51000000.pcie: lookup for GPIO (null) failed
[ 0.647379] dra7-pcie 51000000.pcie: WA for Errata i870 not applied
[ 0.647476] OF: PCI: host bridge /ocp/axi@0/pcie@51000000 ranges:
[ 0.647509] OF: PCI: IO 0x20003000..0x20012fff -> 0x00000000
[ 0.647527] OF: PCI: MEM 0x20013000..0x2fffffff -> 0x20013000
[ 1.641774] dra7-pcie 51000000.pcie: phy link never came up
[ 1.641916] dra7-pcie 51000000.pcie: PCI host bridge to bus 0000:00
[ 1.641927] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 1.641936] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 1.641944] pci_bus 0000:00: root bus resource [mem 0x20013000-0x2fffffff]
[ 1.641952] pci_bus 0000:00: scanning bus
[ 1.641986] pci 0000:00:00.0: [104c:8888] type 01 class 0x060400
[ 1.642017] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit]
[ 1.642048] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x68
[ 1.642090] pci 0000:00:00.0: supports D1
[ 1.642097] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[ 1.642106] pci 0000:00:00.0: PME# disabled
[ 1.642288] pci_bus 0000:00: fixups for bus
[ 1.642299] PCI: bus0: Fast back to back transfers disabled
[ 1.642309] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 0
[ 1.642396] pci_bus 0000:01: scanning bus
[ 1.642406] pci_bus 0000:01: fixups for bus
[ 1.642412] PCI: bus1: Fast back to back transfers enabled
[ 1.642419] pci_bus 0000:01: bus scan returning with max=01
[ 1.642431] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 1
[ 1.642441] pci_bus 0000:00: bus scan returning with max=ff
[ 1.642468] pci 0000:00:00.0: BAR 0: assigned [mem 0x20100000-0x201fffff 64bit]
[ 1.642483] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[ 1.642632] pcieport 0000:00:00.0: assign IRQ: got 178
[ 1.642792] pcieport 0000:00:00.0: Signaling PME with IRQ 146
[ 1.642913] pcieport 0000:00:00.0: AER enabled with IRQ 146
[ 1.686816] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[ 1.689522] 48020000.serial: ttyS2 at MMIO 0x48020000 (irq = 45, base_baud = 3000000) is a 8250
[ 1.689595] console [ttyS2] enabled
[ 1.690476] I2C device entries driver



=============================

DTS File Configuration from dra7.dtsi

=============================

pcie1_phy: pciephy@4a094000 {
compatible = "ti,phy-pipe3-pcie";
reg = <0x4a094000 0x80>, /* phy_rx */
<0x4a094400 0x64>; /* phy_tx */
reg-names = "phy_rx", "phy_tx";
syscon-phy-power = <&scm_conf_pcie 0x1c>;
syscon-pcs = <&scm_conf_pcie 0x10>;
clocks = <&dpll_pcie_ref_ck>,
<&dpll_pcie_ref_m2ldo_ck>,
<&optfclk_pciephy1_32khz>,
<&optfclk_pciephy1_clk>,
<&optfclk_pciephy1_div_clk>,
<&optfclk_pciephy_div>,
<&sys_clkin1>;
clock-names = "dpll_ref", "dpll_ref_m2",
"wkupclk", "refclk",
"div-clk", "phy-div", "sysclk";
#phy-cells = <0>;
};

pcie1_rc: pcie@51000000 {
compatible = "ti,dra7-pcie";
reg = <0x51000000 0x2000>, <0x51002000 0x14c>, <0x1000 0x2000>;
reg-names = "rc_dbics", "ti_conf", "config";
interrupts = <0 232 0x4>, <0 233 0x4>;
#address-cells = <3>;
#size-cells = <2>;
device_type = "pci";
ranges = <0x81000000 0 0 0x03000 0 0x00010000
0x82000000 0 0x20013000 0x13000 0 0xffed000>;
bus-range = <0x00 0xff>;
#interrupt-cells = <1>;
num-lanes = <1>;
linux,pci-domain = <0>;
ti,hwmods = "pcie1";
phys = <&pcie1_phy>;
phy-names = "pcie-phy0";
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 &pcie1_intc 1>,
<0 0 0 2 &pcie1_intc 2>,
<0 0 0 3 &pcie1_intc 3>,
<0 0 0 4 &pcie1_intc 4>;
status = "disabled";
pcie1_intc: interrupt-controller {
interrupt-controller;
#address-cells = <0>;
#interrupt-cells = <1>;
};
};

Hari checked on the board the register CTRL_CORE_SMA_SW_6  is reading 0x0, that means the clk buffer is not enabled. 

we tried to write bit[17:16] to 0b01 but writing with devmem utility was not successful. 

Hari, regarding to the issue of writing to CTRL_CORE_SMA_SW_6  register, my colleague tried the following command:

//unlock the MMR block

devmem2 0x4A002550 w 0x6F361E05

devmem2 0x4A002550

//enable acspcie output buffers

devmem2 0x4A003C14 w 0x00010000

devmem2 0x4A003C14

on a similar board (but newer sdk), and we he was able to write to the register. if you are able to find enable the devmem2 unity, can you give it another try?

thanks
Jian 

Additional notes for actual code implement after manual testing:

Writing to this register shall be inserted in the driver, before the PERST is deasserted. pls add 100us delay between clock enablement and reset. 

Jian

Hi Jian,

I tried following approach to read the register first but none of them worked and it resulted in a hung state of kernel.

diff --git a/kernel/linux-4.14/drivers/pci/dwc/pci-dra7xx.c b/kernel/linux-4.14/drivers/pci/dwc/pci-dra7xx.c
index 06eae13..3b21eed 100644
--- a/kernel/linux-4.14/drivers/pci/dwc/pci-dra7xx.c
+++ b/kernel/linux-4.14/drivers/pci/dwc/pci-dra7xx.c
@@ -140,6 +140,12 @@ static int dra7xx_pcie_establish_link(struct dw_pcie *pci)
u32 reg;
u32 exp_cap_off = EXP_CAP_ID_OFFSET;

+ volatile uint32_t* const reg_temp = (uint32_t*) 0x4A003C14;
+ pr_info("Debug::: reg_temp = %x",*reg_temp);

+ //u32 val;
+ //val = dra7xx_pcie_readl(dra7xx, 0x1c14);
+ //pr_info("Debug::: val = %x",val);
+

From the PCI configuration space, do we need to modify anything to access ? I am doing something wrong here to update it to the register. Can you help me on this ? Do we need to use virt_to_phys or phys_to_virt functions to get the value ?

Thanks,
Harivignesh GS