This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

J784S4XEVM: Enable J784S4 PCIe1 port in 4x

Part Number: J784S4XEVM

Tool/software:

Hello, 

We are trying to have both PCIe ports of the J784S4 EVM working in 4x, but unfortunately we can't atm.

Here the setup :

  • 2 J784S4XEVM :
    • First one (EVM #A) has the two PCIe ports configured in RC.
    • Second one (EVM #B) has the two PCIe ports configured in EP.
  • One wiring harness between EVM #A Port 1 connected to EVM #B Port 0.
  • SW11 set to 10000010.
  • SW7 set to 00000000.

  • SW2 set to 100001101. (Both PCIe ports in RC).

  • Linux SDK version : 10.01.00.05.

Hardware modifications on EVM #A to avoid using USB but rather have the 2x PCIe lanes :

  • Populated components :
    • C696, C697.
    • C683, C684.
    • R891, R892.
    • R876, R877.
    • R168, R169.
    • C125, C129.

  • Unpopulated components :
    • C694, C695.
    • C679, C680.
    • R885, R886.
    • R867, R868.
    • R170, R171.
    • R152, R155.

Software modifications :

  • k3-j784s4-evm.dts :

diff --git a/arch/arm64/boot/dts/ti/k3-j784s4-evm.dts b/arch/arm64/boot/dts/ti/k3-j784s4-evm.dts
index 56a037532..dd92863be 100644
--- a/arch/arm64/boot/dts/ti/k3-j784s4-evm.dts
+++ b/arch/arm64/boot/dts/ti/k3-j784s4-evm.dts
@@ -13,7 +13,7 @@
 
 / {
 	compatible = "ti,j784s4-evm", "ti,j784s4";
-	model = "Texas Instruments J784S4 EVM";
+	model = "Texas Instruments J784S4 EVM (PCIe1 4X)";
 
 	chosen {
 		stdout-path = "serial2:115200n8";
@@ -458,13 +458,6 @@ J784S4_IOPAD(0x010, PIN_INPUT_PULLUP, 8) /* (AH33) MCAN13_RX.I2C4_SDA */
 		>;
 	};
 
-	main_usbss0_pins_default: main-usbss0-default-pins {
-		bootph-all;
-		pinctrl-single,pins = <
-			J784S4_IOPAD(0x0ec, PIN_OUTPUT, 6) /* (AN37) TIMER_IO1.USB0_DRVVBUS */
-		>;
-	};
-
 	main_mcan4_pins_default: main-mcan4-default-pins {
 		pinctrl-single,pins = <
 			J784S4_IOPAD(0x088, PIN_INPUT, 0) /* (AF36) MCAN4_RX */
@@ -1372,40 +1365,6 @@ &dss {
 				 <&k3_clks 218 22>;
 };
 
-&serdes0 {
-	status = "okay";
-
-	serdes0_usb_link: phy@3 {
-		reg = <3>;
-		cdns,num-lanes = <1>;
-		#phy-cells = <0>;
-		cdns,phy-type = <PHY_TYPE_USB3>;
-		resets = <&serdes_wiz0 4>;
-	};
-};
-
-&serdes_wiz0 {
-	status = "okay";
-};
-
-&usb_serdes_mux {
-	idle-states = <0>; /* USB0 to SERDES lane 3 */
-};
-
-&usbss0 {
-	status = "okay";
-	pinctrl-0 = <&main_usbss0_pins_default>;
-	pinctrl-names = "default";
-	ti,vbus-divider;
-};
-
-&usb0 {
-	dr_mode = "otg";
-	maximum-speed = "super-speed";
-	phys = <&serdes0_usb_link>;
-	phy-names = "cdns3,usb3-phy";
-};
-
 &serdes_wiz4 {
 	status = "okay";
 };

  • k3-j784s4-main.dtsi :

diff --git a/arch/arm64/boot/dts/ti/k3-j784s4-main.dtsi b/arch/arm64/boot/dts/ti/k3-j784s4-main.dtsi
index 105a8fd39..69e7e5a6d 100644
--- a/arch/arm64/boot/dts/ti/k3-j784s4-main.dtsi
+++ b/arch/arm64/boot/dts/ti/k3-j784s4-main.dtsi
@@ -159,8 +159,8 @@ serdes_ln_ctrl: mux-controller@4080 {
 					<0x28 0x3>, <0x2c 0x3>; /* SERDES2 lane2/3 select */
 			idle-states = <J784S4_SERDES0_LANE0_PCIE1_LANE0>,
 				      <J784S4_SERDES0_LANE1_PCIE1_LANE1>,
-				      <J784S4_SERDES0_LANE2_IP3_UNUSED>,
-				      <J784S4_SERDES0_LANE3_USB>,
+				      <J784S4_SERDES0_LANE2_PCIE1_LANE2>,
+				      <J784S4_SERDES0_LANE3_PCIE1_LANE3>,
 				      <J784S4_SERDES1_LANE0_PCIE0_LANE0>,
 				      <J784S4_SERDES1_LANE1_PCIE0_LANE1>,
 				      <J784S4_SERDES1_LANE2_PCIE0_LANE2>,

Here is what we got after we enumareted the EVM #B:

And the link caps & status :

What are we missing?

    • Populated components :
      • C696, C697.
      • C683, C684.
      • R891, R892.
      • R876, R877.
      • R168, R169.
      • C125, C129.

    • Unpopulated components :
      • C694, C695.
      • C679, C680.
      • R885, R886.
      • R867, R868.
      • R170, R171.
      • R152, R155.

    This part was wrong, I swapped unpopulated / populated components. here is th correct version :

    • Unpopulated components :
      • C696, C697.
      • C683, C684.
      • R891, R892.
      • R876, R877.
      • R168, R169.
      • C125, C129.

    • Ppopulated components :
      • C694, C695.
      • C679, C680.
      • R885, R886.
      • R867, R868.
      • R170, R171.
      • R152, R155.
  • Hi Cedric,

    Could you send the whole logs from "dmesg"? A bit long, but I can parse through it to see which PCIe port is coming up as x2 lanes.

    However, in terms of what I see so far:

    1. serdes_ln_ctrl looks correct
    2. removal of USB nodes look correct
    3. serdes0 node will need some changes so that the lanes previously used for USB is now used for PCIe
    4. pcie1_rc node will need to change num-lanes for 4 lanes

    Something like below patch should fix points 3 and 4:

    https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/0001_2D00_PCIe_2D00_x4_2D00_lane_2D00_changes_2D00_for_2D00_PCIe1.patch

    I haven't reviewed the hardware changes yet, but try the changes above in software and let me know if that fixes it. If it does not fix, then I will also take a look at the schematics in case issue is hardware related.

    Regards,

    Takuma

  • Hi Takuma,

    I'll try to give you the kernel logs as soon as I can. Anyways, I actually did try the patch you showed before writing this post:

    Also put the serdes0 lanes to 4, but unfortunately it didn't change.

    I'll come back with the logs asap.

    Cedric

  • Just tried your patch, and it's even worse on our side:

    Here is the dmesg log (pcie part only):

    root@j784s4-evm:~# dmesg | grep pci
    [    2.140232] j721e-pcie 2900000.pcie: host bridge /bus@100000/pcie@2900000 ranges:
    [    2.147717] j721e-pcie 2900000.pcie: Parsing ranges property...
    [    2.153635] j721e-pcie 2900000.pcie:       IO 0x0010001000..0x0010010fff -> 0x0010001000
    [    2.161716] j721e-pcie 2900000.pcie:      MEM 0x0010011000..0x0017ffffff -> 0x0010011000
    [    2.169793] j721e-pcie 2900000.pcie: Parsing dma-ranges property...
    [    2.176049] j721e-pcie 2900000.pcie:   IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000
    [    2.394526] j721e-pcie 2900000.pcie: Link up
    [    2.399129] j721e-pcie 2900000.pcie: PCI host bridge to bus 0000:00
    [    2.405390] pci_bus 0000:00: root bus resource [bus 00-ff]
    [    2.410863] pci_bus 0000:00: root bus resource [io  0x0000-0xffff] (bus address [0x10001000-0x10010fff])
    [    2.420323] pci_bus 0000:00: root bus resource [mem 0x10011000-0x17ffffff]
    [    2.427183] pci_bus 0000:00: scanning bus
    [    2.431204] pci 0000:00:00.0: [104c:b012] type 01 class 0x060400
    [    2.437200] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C bits
    [    2.446941] pci 0000:00:00.0: supports D1
    [    2.450940] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
    [    2.456675] pci 0000:00:00.0: PME# disabled
    [    2.461011] pci 0000:00:00.0: vgaarb: pci_notify
    [    2.467395] pci_bus 0000:00: fixups for bus
    [    2.471572] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
    [    2.478258] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [    2.486249] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
    [    2.492983] pci_bus 0000:01: scanning bus
    [    2.498016] pci 0000:01:00.0: [10ee:9034] type 00 class 0x058000
    [    2.504606] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
    [    2.511702] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x0000ffff]
    [    2.518943] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x001fffff]
    [    2.532764] pci 0000:01:00.0: vgaarb: pci_notify
    [    2.538057] pci_bus 0000:01: fixups for bus
    [    2.542229] pci_bus 0000:01: bus scan returning with max=01
    [    2.547794] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
    [    2.554399] pci_bus 0000:00: bus scan returning with max=01
    [    2.559969] pci 0000:00:00.0: BAR 14: assigned [mem 0x10100000-0x104fffff]
    [    2.566830] pci 0000:01:00.0: BAR 2: assigned [mem 0x10200000-0x103fffff]
    [    2.575567] pci 0000:01:00.0: BAR 0: assigned [mem 0x10100000-0x101fffff]
    [    2.582808] pci 0000:01:00.0: BAR 1: assigned [mem 0x10400000-0x1040ffff]
    [    2.589661] pci 0000:00:00.0: PCI bridge to [bus 01]
    [    2.594615] pci 0000:00:00.0:   bridge window [mem 0x10100000-0x104fffff]
    [    2.601441] pcieport 0000:00:00.0: vgaarb: pci_notify
    [    2.606493] pcieport 0000:00:00.0: of_irq_parse_pci: failed with rc=-22
    [    2.613093] pcieport 0000:00:00.0: assign IRQ: got 0
    [    2.618051] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
    [    2.624144] pcieport 0000:00:00.0: enabling bus mastering
    [    2.629698] pcieport 0000:00:00.0: PME: Signaling with IRQ 490
    [    2.635722] pcieport 0000:00:00.0: AER: enabled with IRQ 490
    [    2.641411] pcieport 0000:00:00.0: save config 0x00: 0xb012104c
    [    2.647319] pcieport 0000:00:00.0: save config 0x04: 0x00100406
    [    2.653224] pcieport 0000:00:00.0: save config 0x08: 0x06040000
    [    2.659129] pcieport 0000:00:00.0: save config 0x0c: 0x00010000
    [    2.665034] pcieport 0000:00:00.0: save config 0x10: 0x00000000
    [    2.670939] pcieport 0000:00:00.0: save config 0x14: 0x00000000
    [    2.676844] pcieport 0000:00:00.0: save config 0x18: 0x00010100
    [    2.682749] pcieport 0000:00:00.0: save config 0x1c: 0x000001f1
    [    2.688654] pcieport 0000:00:00.0: save config 0x20: 0x10401010
    [    2.694559] pcieport 0000:00:00.0: save config 0x24: 0x0001fff1
    [    2.700463] pcieport 0000:00:00.0: save config 0x28: 0x00000000
    [    2.706368] pcieport 0000:00:00.0: save config 0x2c: 0x00000000
    [    2.712273] pcieport 0000:00:00.0: save config 0x30: 0x00000000
    [    2.718177] pcieport 0000:00:00.0: save config 0x34: 0x00000080
    [    2.724082] pcieport 0000:00:00.0: save config 0x38: 0x00000000
    [    2.729987] pcieport 0000:00:00.0: save config 0x3c: 0x00020100
    [    2.735939] pcieport 0000:00:00.0: vgaarb: pci_notify
    [    2.741182] j721e-pcie 2910000.pcie: host bridge /bus@100000/pcie@2910000 ranges:
    [    2.748658] j721e-pcie 2910000.pcie: Parsing ranges property...
    [    2.754571] j721e-pcie 2910000.pcie:       IO 0x0018001000..0x0018010fff -> 0x0018001000
    [    2.762653] j721e-pcie 2910000.pcie:      MEM 0x0018011000..0x001fffffff -> 0x0018011000
    [    2.770727] j721e-pcie 2910000.pcie: Parsing dma-ranges property...
    [    2.776984] j721e-pcie 2910000.pcie:   IB MEM 0x0000000000..0xffffffffffff -> 0x0000000000
    [    2.994520] j721e-pcie 2910000.pcie: Link up
    [    2.998853] j721e-pcie 2910000.pcie: PCI host bridge to bus 0001:00
    [    3.005108] pci_bus 0001:00: root bus resource [bus 00-ff]
    [    3.010581] pci_bus 0001:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x18001000-0x18010fff])
    [    3.020213] pci_bus 0001:00: root bus resource [mem 0x18011000-0x1fffffff]
    [    3.027072] pci_bus 0001:00: scanning bus
    [    3.031089] pci 0001:00:00.0: [104c:b012] type 01 class 0x060400
    [    3.037087] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x4 may corrupt adjacent RW1C bits
    [    3.046826] pci 0001:00:00.0: supports D1
    [    3.050824] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
    [    3.056559] pci 0001:00:00.0: PME# disabled
    [    3.060875] pci 0001:00:00.0: vgaarb: pci_notify
    [    3.067282] pci_bus 0001:00: fixups for bus
    [    3.071456] pci 0001:00:00.0: scanning [bus 00-00] behind bridge, pass 0
    [    3.078141] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
    [    3.086133] pci 0001:00:00.0: scanning [bus 00-00] behind bridge, pass 1
    [    3.092870] pci_bus 0001:01: scanning bus
    [    3.096924] pci 0001:01:00.0: [10ee:9034] type 00 class 0x058000
    [    3.102956] pci 0001:01:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
    [    3.109227] pci 0001:01:00.0: reg 0x14: [mem 0x00000000-0x0000ffff]
    [    3.115496] pci 0001:01:00.0: reg 0x18: [mem 0x00000000-0x001fffff]
    [    3.122140] pci 0001:01:00.0: 7.876 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x1 link at 0001:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
    [    3.137177] pci 0001:01:00.0: vgaarb: pci_notify
    [    3.141809] pci_bus 0001:01: fixups for bus
    [    3.145982] pci_bus 0001:01: bus scan returning with max=01
    [    3.151543] pci_bus 0001:01: busn_res: [bus 01-ff] end is updated to 01
    [    3.158147] pci_bus 0001:00: bus scan returning with max=01
    [    3.163710] pci 0001:00:00.0: BAR 14: assigned [mem 0x18100000-0x184fffff]
    [    3.170572] pci 0001:01:00.0: BAR 2: assigned [mem 0x18200000-0x183fffff]
    [    3.177352] pci 0001:01:00.0: BAR 0: assigned [mem 0x18100000-0x181fffff]
    [    3.184131] pci 0001:01:00.0: BAR 1: assigned [mem 0x18400000-0x1840ffff]
    [    3.190910] pci 0001:00:00.0: PCI bridge to [bus 01]
    [    3.195865] pci 0001:00:00.0:   bridge window [mem 0x18100000-0x184fffff]
    [    3.202676] pcieport 0001:00:00.0: vgaarb: pci_notify
    [    3.207726] pcieport 0001:00:00.0: of_irq_parse_pci: failed with rc=-22
    [    3.214325] pcieport 0001:00:00.0: assign IRQ: got 0
    [    3.219280] pcieport 0001:00:00.0: enabling device (0000 -> 0002)
    [    3.225372] pcieport 0001:00:00.0: enabling bus mastering
    [    3.230911] pcieport 0001:00:00.0: PME: Signaling with IRQ 492
    [    3.236890] pcieport 0001:00:00.0: AER: enabled with IRQ 492
    [    3.242575] pcieport 0001:00:00.0: save config 0x00: 0xb012104c
    [    3.248483] pcieport 0001:00:00.0: save config 0x04: 0x00100406
    [    3.254387] pcieport 0001:00:00.0: save config 0x08: 0x06040000
    [    3.260292] pcieport 0001:00:00.0: save config 0x0c: 0x00010000
    [    3.266197] pcieport 0001:00:00.0: save config 0x10: 0x00000000
    [    3.272102] pcieport 0001:00:00.0: save config 0x14: 0x00000000
    [    3.278006] pcieport 0001:00:00.0: save config 0x18: 0x00010100
    [    3.283911] pcieport 0001:00:00.0: save config 0x1c: 0x000001f1
    [    3.289817] pcieport 0001:00:00.0: save config 0x20: 0x18401810
    [    3.295722] pcieport 0001:00:00.0: save config 0x24: 0x0001fff1
    [    3.301626] pcieport 0001:00:00.0: save config 0x28: 0x00000000
    [    3.307530] pcieport 0001:00:00.0: save config 0x2c: 0x00000000
    [    3.313435] pcieport 0001:00:00.0: save config 0x30: 0x00000000
    [    3.319340] pcieport 0001:00:00.0: save config 0x34: 0x00000080
    [    3.325244] pcieport 0001:00:00.0: save config 0x38: 0x00000000
    [    3.331149] pcieport 0001:00:00.0: save config 0x3c: 0x00020100
    [    3.337101] pcieport 0001:00:00.0: vgaarb: pci_notify
    
    

    Tell me if it's enough or if you want the complete dmesg log.

    Best regards,

    Cedric

  • Hi Cedric,

    Could you send the complete dmesg logs? 

    So far, the behavior being observed is very strange. For example:

    • I see in your original logs, it is x2 lanes, but the speed looks to be downgraded to PCIe gen 1 speeds of 2.5GT/s per lane. This could happen if there is a software limit you have imposed, or signal integrity, but both I doubt since gen 3 speeds are achievable in latest logs.
    • Original logs also mention the link is capable of 63Gb/s with 8.0 GT/s PCIe x8 link. This is also strange if J784S4 is the EP and RC, since both only support up to x4 lanes
    • The newest dmesg logs negotiate to x1 lane at PCIe gen 3 speeds of 8.0GT/s per lane. This looks the most normal logs, just not the desired outcome since only 1 lane is up

    There is a check in the pcie kernel driver that when triggered will fall back to 1 lane. It could be that this is somehow hit in the latest logs: https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/tree/drivers/pci/controller/cadence/pci-j721e.c?h=ti-linux-6.6.y#n514. Or, it could be something in serdes kernel driver, since I think there were similar checks. But either way, full dmesg logs might give me more of an idea for what is happening.

    Regards,

    Takuma

  • Hello Takuma,

    I did some testings back again this morning, and it's now working fine.

    A Hardware issue was causing the Link to be downgraded to 1x.

    Now we are getting the 2 PCie RC links working fine in 4x:

    Best regards,

    Cedric

  • Hi Cedric,

    Thanks for the update. Good to hear that it is working!

    Regards,

    Takuma