PROCESSOR-SDK-J784S4: [NETWORK] Migrating Kernel device-tree from SDK 9.2 to 10.1

Daniel Braunwarth

Tool/software:

We are currently migrating from SDK 9.2 to 10.1. First problem we are currently seeing is that the SERDES, CPSW9G and PCIe configuration is somehow broken.

If we connect the interfaces we see for some of them that unexpected interfaces receives link up/down.

Also the interfaces are not really sending any data to the line.

We see the following errors / warnings:

$ dmesg --level err
[    0.216096] mux muxchip0: unable to set idle state
[    0.301939] ti-sci-clk 44083000.system-controller:clock-controller: get-parent failed for dev=157, clk=34, ret=-19
[    0.302040] ti-sci-clk 44083000.system-controller:clock-controller: get-parent failed for dev=157, clk=34, ret=-19
[    0.302181] ti-sci-clk 44083000.system-controller:clock-controller: get-parent failed for dev=157, clk=34, ret=-19
[    1.426716] pcieport 0000:00:00.0: of_irq_parse_pci: failed with rc=-22
[    1.657589] pcieport 0001:00:00.0: of_irq_parse_pci: failed with rc=-22
[    2.662996] pcieport 0002:00:00.0: of_irq_parse_pci: failed with rc=-22
[    2.892869] pcieport 0003:00:00.0: of_irq_parse_pci: failed with rc=-22

$ dmesg --level warn
[    0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[    0.192978] SPI driver spidev has no spi_device_id for rohm,bh2228fv
[    0.216100] mmio-mux: probe of 104080.mux-controller failed with error -5
[    0.419937] cdns-torrent-phy: probe of 5020000.serdes failed with error -22
[    1.424255] pci_bus 0000:00: 2-byte config write to 0000:00:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    1.654006] pci_bus 0001:00: 2-byte config write to 0001:00:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    2.660583] pci_bus 0002:00: 2-byte config write to 0002:00:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    2.889686] pci_bus 0003:00: 2-byte config write to 0003:00:00.0 offset 0x4 may corrupt adjacent RW1C bits
[    3.246298] systemd[1]: MachineID 'uninitialized' is not valid, ignoring: Invalid argument

The relevant parts of our device tree are:

&serdes0 {
	status = "okay";
	serdes0_pcie1_link: phy@0 {
		reg = <0>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz0 1>;
	};

	serdes0_pcie3_link: phy@2 {
		reg = <2>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz0 3>;
	};
};

&serdes_wiz0 {
	status = "okay";
};

&serdes1 {
	status = "okay";
	serdes1_pcie0_link: phy@0 {
		reg = <0>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz1 1>;
	};

	serdes1_pcie2_link: phy@2 {
		reg = <2>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_PCIE>;
		resets = <&serdes_wiz1 3>;
	};
};

&serdes_wiz1 {
	status = "okay";
};

&serdes2 {
	status = "okay";
	#address-cells = <1>;
	#size-cells = <0>;

	serdes2_sgmii_link1: phy@0 {
		reg = <0>;
		cdns,num-lanes = <2>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_SGMII>;
		resets = <&serdes_wiz2 1>, <&serdes_wiz2 2>;
	};

	serdes2_qsgmii_link: phy@2 {
		reg = <2>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_QSGMII>;
		resets = <&serdes_wiz2 3>;
	};

	serdes2_sgmii_link2: phy@3 {
		reg = <3>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_SGMII>;
		resets = <&serdes_wiz2 4>;
	};
};

&serdes_wiz2 {
	status = "okay";
};

&serdes4 {
	status = "okay";

	serdes4_sgmii_link: phy@2 {
		reg = <2>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_SGMII>;
		resets = <&serdes_wiz4 3>;
	};

	serdes4_usb_link: phy@3 {
		reg = <3>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_USB3>;
		resets = <&serdes_wiz4 4>;
	};
};

&serdes_wiz4 {
	status = "okay";
};

/* refer to include/dt-bindings/mux/ti-serdes.h */
&serdes_ln_ctrl {
	mux-reg-masks = <0x00 0x3>, <0x04 0x3>, /* SERDES0 lane0/1 select */
			<0x08 0x3>, <0x0c 0x3>, /* SERDES0 lane2/3 select */
			<0x10 0x3>, <0x14 0x3>, /* SERDES1 lane0/1 select */
			<0x18 0x3>, <0x1c 0x3>, /* SERDES1 lane2/3 select */
			<0x20 0x3>, <0x24 0x3>, /* SERDES2 lane0/1 select */
			<0x28 0x3>, <0x2c 0x3>, /* SERDES2 lane2/3 select */
			<0x40 0x3>, <0x44 0x3>, /* SERDES4 lane0/1 select */
			<0x48 0x3>, <0x4c 0x3>; /* SERDES4 lane2/3 select */

	idle-states = <J784S4_SERDES0_LANE0_PCIE1_LANE0>,   /* PCIe1 */
		      <J784S4_SERDES0_LANE1_IP4_UNUSED>,
		      <J784S4_SERDES0_LANE2_PCIE3_LANE0>,   /* PCIe3 */
		      <J784S4_SERDES0_LANE3_IP4_UNUSED>,
		      <J784S4_SERDES1_LANE0_PCIE0_LANE0>,   /* PCIe0 */
		      <J784S4_SERDES1_LANE1_IP4_UNUSED>,
		      <J784S4_SERDES1_LANE2_PCIE2_LANE0>,   /* PCIe2 */
		      <J784S4_SERDES1_LANE3_IP4_UNUSED>,
		      <J784S4_SERDES2_LANE0_QSGMII_LANE5>,  /* SGMII: RTL8211 */
		      <J784S4_SERDES2_LANE1_QSGMII_LANE6>,  /* SGMII: RTL8211 */
		      <J784S4_SERDES2_LANE2_QSGMII_LANE1>,  /* QSGMII: YT8614 */
		      <J784S4_SERDES2_LANE3_QSGMII_LANE8>,  /* SGMII: RTL8211 */
		      <J784S4_SERDES4_LANE0_IP3_UNUSED>,
		      <J784S4_SERDES4_LANE1_IP3_UNUSED>,
		      <J784S4_SERDES4_LANE2_QSGMII_LANE7>,  /* SGMII: RTL8211 */
		      <J784S4_SERDES4_LANE3_USB>;           /* USB */
};

&serdes_refclk {
	clock-frequency = <100000000>;
};

&main_gpio0 {
	status = "okay";
	pinctrl-names = "default";
	pinctrl-0 = <&pcie_reset_pins_default>,
		<&network_pins_default>,
		<&board_id_eeprom_wp_pins_default>,
		<&pwr_mgmt_pins_main_default>,
		<&sleep_led_pins_default>;
	gpio-line-names = MAIN_GPIO0_LINE_NAMES;
};

&wkup_gpio0 {
	status = "okay";
	pinctrl-names = "default";
	pinctrl-0 = <&mcu_adc0_sel_on_pins_default>,
		    <&wkup_gpio_0_vddshv2_mcu_pins_default>,
		    <&pwr_mgmt_pins_wkup_default>;
	gpio-line-names = WKUP_GPIO0_LINE_NAMES;
};

&pcie0_rc {
	status = "okay";
	num-lanes = <1>;
	phys = <&serdes1_pcie0_link>;
	phy-names = "pcie-phy";
};

&pcie1_rc {
	status = "okay";
	num-lanes = <1>;
	phys = <&serdes0_pcie1_link>;
	phy-names = "pcie-phy";
};

&pcie2_rc {
	status = "okay";
	num-lanes = <1>;
	phys = <&serdes1_pcie2_link>;
	phy-names = "pcie-phy";
};

&pcie3_rc {
	status = "okay";
	num-lanes = <1>;
	phys = <&serdes0_pcie3_link>;
	phy-names = "pcie-phy";
};

&cpsw0_phy_gmii_sel {
	ti,qsgmii-main-ports = <1>, <1>;
};

&main_cpsw0 {
	status = "okay";
	pinctrl-names = "default";
};

&main_cpsw0_port1 {
	status = "okay";
	phy-handle = <&cpsw9g_eexb_tp>;
	phy-mode = "qsgmii";
	local-mac-address = [02 60 C8 00 00 00];
	phys = <&cpsw0_phy_gmii_sel 1>;
	phy-names = "mac";
};

&main_cpsw0_port2 {
	status = "okay";
	phy-handle = <&cpsw9g_ksi>;
	phy-mode = "qsgmii";
	local-mac-address = [02 60 C8 00 00 01];
	phys = <&cpsw0_phy_gmii_sel 2>;
	phy-names = "mac";
};

&main_cpsw0_port3 {
	status = "okay";
	phy-handle = <&cpsw9g_tp>;
	phy-mode = "qsgmii";
	local-mac-address = [02 60 C8 00 00 02];
	phys = <&cpsw0_phy_gmii_sel 3>;
	phy-names = "mac";
};

&main_cpsw0_port4 {
	status = "okay";
	phy-handle = <&cpsw9g_diob>;
	phy-mode = "qsgmii";
	local-mac-address = [02 60 C8 00 00 03];
	phys = <&cpsw0_phy_gmii_sel 4>;
	phy-names = "mac";
};

&main_cpsw0_port5 {
	status = "okay";
	phy-handle = <&cpsw9g_koni>;
	phy-mode = "sgmii";
	local-mac-address = [02 60 C8 00 00 04];
	phys = <&cpsw0_phy_gmii_sel 5>;
	phy-names = "mac";
};

&main_cpsw0_port6 {
	status = "okay";
	phy-handle = <&cpsw9g_kli_ot>;
	phy-mode = "sgmii";
	local-mac-address = [02 60 C8 00 00 05];
	phys = <&cpsw0_phy_gmii_sel 6>;
	phy-names = "mac";
};

&main_cpsw0_port7 {
	status = "okay";
	phy-handle = <&cpsw9g_profinet4>;
	phy-mode = "sgmii";
	local-mac-address = [02 60 C8 00 00 06];
	phys = <&cpsw0_phy_gmii_sel 7>;
	phy-names = "mac";
};

&main_cpsw0_port8 {
	status = "okay";
	phy-handle = <&cpsw9g_profinet3>;
	phy-mode = "sgmii";
	local-mac-address = [02 60 C8 00 00 07];
	phys = <&cpsw0_phy_gmii_sel 8>;
	phy-names = "mac";
};

&main_cpsw0_mdio {
	status = "okay";
	pinctrl-names = "default";
	pinctrl-0 = <&main_cpsw9g_mdio_pins_default>;
	bus_freq = <1000000>;
	reset-gpios = <&main_gpio0 10 GPIO_ACTIVE_LOW>;
	reset-post-delay-us = <120000>;
	#address-cells = <1>;
	#size-cells = <0>;

	cpsw9g_koni: ethernet-phy@1 {
		reg = <1>;
	};

	cpsw9g_kli_ot: ethernet-phy@2 {
		reg = <2>;
	};

	cpsw9g_profinet4: ethernet-phy@3 {
		reg = <3>;
	};

	cpsw9g_profinet3: ethernet-phy@4 {
		reg = <4>;
	};

	cpsw9g_eexb_tp: ethernet-phy@8 {
		reg = <8>;
	};

	cpsw9g_ksi: ethernet-phy@9 {
		reg = <9>;
	};

	cpsw9g_tp: ethernet-phy@10 {
		reg = <10>;
	};

	cpsw9g_diob: ethernet-phy@11 {
		reg = <11>;
	};
};

&usb_serdes_mux {
	idle-states = <1>; /* USB0 to SERDES4 lane 3 */
};

&usbss0 {
	status = "okay";
	pinctrl-0 = <&main_usbss0_pins_default>;
	pinctrl-names = "default";
	ti,vbus-divider;
};

&usb0 {
	status = "okay";
	dr_mode = "host";
	maximum-speed = "super-speed";
	phys = <&serdes4_usb_link>;
	phy-names = "cdns3,usb3-phy";
};

The SERDES configuration has been ported like this:

Adding SERDES4 here seems to cause the error "mux muxchip0: unable to set idle state".

Is there anything else we need to adjust?

Just let me know if you need any further information.

Regards

Daniel

8 months ago

0 Chris Tsongas 8 months ago

TI__Genius 15440 points

Your question has been initially assigned to Tanmay

0 Takuma Fujiwara 8 months ago in reply to Chris Tsongas

TI__Guru 53403 points

Hi Daniel,

Three things:

Can you also send the full logs from "dmesg", so that we can get the bigger picture?
Can you share the output from "devmem2 0x00105008" and "devmem2 0x0010500C", which are the CTRL_MMR_CFG0_LOCK1_KICK0 and KICK1 registers?
1. Values should be 0x68EF3490 and 0xD172BC5A respectively.
Can you share the output from "devmem2 0x00104080", which is the CTRL_MMR_CFG0_SERDES0_LN0_CTRL?
1. Value should be 0x1.

My current suspicion is that the CTRLMMR registers are not being unlocked and preventing writes to the registers controlled by the serdes_ln_ctrl node.

Regards,

Takuma

0 Daniel Braunwarth 8 months ago in reply to Takuma Fujiwara

Intellectual 360 points

Hi Takuma

Here is the output of the requested registers:

./devmem2 0x00105008
/dev/mem opened.
Memory mapped at address 0xffffa4373000.
Value at address 0x105008 (0xffffa4373008): 0x68EF3491

./devmem2 0x0010500C
/dev/mem opened.
Memory mapped at address 0xffff8dc74000.
[1]    1009 bus error  ./devmem2 0x0010500C

./devmem2 0x00104080
/dev/mem opened.
Memory mapped at address 0xffff92830000.
Value at address 0x104080 (0xffff92830080): 0x1

I'm going to send you the dmesg output as a private message.

Regards

Daniel

0 Takuma Fujiwara 8 months ago in reply to Daniel Braunwarth

TI__Guru 53403 points

Hi Daniel,

It looks like the CTRLMMR is locked. However, the SERDES LN CTRL register has been written with non-default value. From this, I suspect there are some changes in U-Boot between 9.2 and 10.0.

Could you remind me if your project is using U-Boot for booting? Or is a different bootflow being used?
If U-Boot is used could you also send U-Boot logs as well?

Ideally, U-Boot should simply unlock the CTRLMMR registers by writing the unlock values to the lock kick registers, then leave the lane control CTRLMMR registers as well as the SERDES registers untouched and allow Linux to do the initialization.

Regards,

Takuma

0 Daniel Braunwarth 8 months ago in reply to Takuma Fujiwara

Intellectual 360 points

Hi Takuma

Yes, we are booting with u-boot (default SPL method).

I've checked to u-boot code and it is doing an unlock during SPL initialization. Call stack is board_init_f() -> k3_spl_init() -> ctrl_mmr_unlock():

static void ctrl_mmr_unlock(void)
{
        /* Unlock all WKUP_CTRL_MMR0 module registers */
        mmr_unlock(WKUP_CTRL_MMR0_BASE, 0);
        mmr_unlock(WKUP_CTRL_MMR0_BASE, 1);
        mmr_unlock(WKUP_CTRL_MMR0_BASE, 2);
        mmr_unlock(WKUP_CTRL_MMR0_BASE, 3);
        mmr_unlock(WKUP_CTRL_MMR0_BASE, 4);
        mmr_unlock(WKUP_CTRL_MMR0_BASE, 6);
        mmr_unlock(WKUP_CTRL_MMR0_BASE, 7);

        /* Unlock all MCU_CTRL_MMR0 module registers */
        mmr_unlock(MCU_CTRL_MMR0_BASE, 0);
        mmr_unlock(MCU_CTRL_MMR0_BASE, 1);
        mmr_unlock(MCU_CTRL_MMR0_BASE, 2);
        mmr_unlock(MCU_CTRL_MMR0_BASE, 3);
        mmr_unlock(MCU_CTRL_MMR0_BASE, 4);

        /* Unlock all CTRL_MMR0 module registers */
        mmr_unlock(CTRL_MMR0_BASE, 0);
        mmr_unlock(CTRL_MMR0_BASE, 1);
        mmr_unlock(CTRL_MMR0_BASE, 2);
        mmr_unlock(CTRL_MMR0_BASE, 3);
        mmr_unlock(CTRL_MMR0_BASE, 5);
        mmr_unlock(CTRL_MMR0_BASE, 7);
}

Unlock values are correct: 0x68ef3490 / 0xd172bc5a.

For testing I have NOPed mmr_lock() in mcu1_0 firmware.

But in Linux I still see the values and the bus error from above.

I'm wondering where is wrong value for CTRL_MMR_CFG0_LOCK1_KICK0 is coming from.

I've sent you the u-boot logs as a private message.

Regards

Daniel

0 Takuma Fujiwara 8 months ago in reply to Daniel Braunwarth

TI__Guru 53403 points

Hi Daniel,

Thank you for the logs. I see DFU boot is used, so USB driver is in use at U-Boot. And thank you for confirming the mmr_unlocks are ran. It looks like at least in U-Boot, the registers are all unlocked.

Now, I re-reviewed the Linux kernel logs shared via private message once more, and noticed that all 4 PCIe interfaces look to be up, and the serdes reporting an error is related to CPSW. I also do not see any errors with USB, except for the mux chip error for serdes4.

Can you confirm whether there are any issues with PCIe and USB? Or are all the interfaces including ethernet, PCIe, and USB are not sending data?

Regards,

Takuma

0 Daniel Braunwarth 8 months ago in reply to Takuma Fujiwara

Intellectual 360 points

Hi Takuma

while testing the different SERDES lines I found that one patch from Siddharth somehow broke. We had this one already in our tree on SDK version 9.2, but 10.1 is missing it.

All SERDES lines seems to be operational now.

We are still seeing the following errors during the boot:

[    0.222586] mux muxchip0: unable to set idle state
[    0.658894] pcieport 0000:00:00.0: of_irq_parse_pci: failed with rc=-22
[    0.890035] pcieport 0001:00:00.0: of_irq_parse_pci: failed with rc=-22
[    1.121153] pcieport 0002:00:00.0: of_irq_parse_pci: failed with rc=-22
[    1.352472] pcieport 0003:00:00.0: of_irq_parse_pci: failed with rc=-22

Can you help to identify the root cause, please?

Regards

Daniel

0 Takuma Fujiwara 8 months ago in reply to Daniel Braunwarth

TI__Guru 53403 points

Hi Daniel,

Ah, understood. That is a patch for multi-link/multi-protocol. Glad you were able to figure out the issue.

As for PCIe port errors, you may ignore these. These errors come from removing support of the legacy interrupts for PCIe.

As for the mux error, I assume this is from serdes4. In our default devicetree, I do not see these defined for some reason.

Could you dump the content of 0x001040c0, 0x001040c4, 0x001040c8, and 0x001040cc?
Can you confirm if PCIe, USB, and/or CPSW are functional?

Regards,

Takuma

0 Daniel Braunwarth 8 months ago in reply to Takuma Fujiwara

Intellectual 360 points

HI Takuma

Takuma Fujiwara said:
As for PCIe port errors, you may ignore these. These errors come from removing support of the legacy interrupts for PCIe.

Do you see any possibility to get rid of them? We try to have no errors during the boot to prevent our customer support to hunt bugs which are actually non.

Takuma Fujiwara said:
As for the mux error, I assume this is from serdes4. In our default devicetree, I do not see these defined for some reason.

Can you align this internally and add the Serdes4 configuration to SDK 11.0?

Takuma Fujiwara said:
Could you dump the content of 0x001040c0, 0x001040c4, 0x001040c8, and 0x001040cc?

Here are the register values:

# devmem2 0x001040c0
/dev/mem opened.
Memory mapped at address 0xffffb4640000.
Value at address 0x1040C0 (0xffffb46400c0): 0x2

# devmem2 0x001040c4
/dev/mem opened.
Memory mapped at address 0xffff99bc4000.
[1]    1186 bus error  devmem2 0x001040c4

# devmem2 0x001040c8
/dev/mem opened.
Memory mapped at address 0xffff910fd000.
Value at address 0x1040C8 (0xffff910fd0c8): 0x1

# devmem2 0x001040cc
/dev/mem opened.
Memory mapped at address 0xffffb4db9000.
[1]    1190 bus error  devmem2 0x001040cc

Takuma Fujiwara said:
Can you confirm if PCIe, USB, and/or CPSW are functional?

As far as I can see for now PCIe, USB and the CPSW are fully functional. But we just started with the migration and don't have any long-running machines so far.

Regards

Daniel

0 Takuma Fujiwara 8 months ago in reply to Daniel Braunwarth

TI__Guru 53403 points

Hi Daniel,

For the PCIe errors, this commit should fix it: https://patchwork.kernel.org/project/linux-pci/patch/20240726135903.1255825-1-s-vadapalli@ti.com/

SERDES4, I will check.

A bit concerning that there is a bus error for 1040c4 and 1040cc, but the two other registers seem to show the correct value is programmed, so we can be certain some non-default value is programmed. If all interfaces are fully functional, then should be fine.

Regards,

Takuma

+1 Takuma Fujiwara 8 months ago in reply to Takuma Fujiwara

TI__Guru 53403 points

Hi Daniel,

Checked with development team and we will have the serdes4 mux defined in future releases (hopefully 11.0, unless upstreaming does not happen in time).

I think this concludes this forum thread. Please feel free to start a new thread if a new question arises.

Regards,

Takuma

Processors

Processors forum

PROCESSOR-SDK-J784S4: [NETWORK] Migrating Kernel device-tree from SDK 9.2 to 10.1