SK-AM69: J721EXENETXPANEVM on-board clock generator problem

Ok, it was a struggle, but finally the ENET board works. As a final solution, we use REFCLK directly from SERDES2. Here are the modifications that need to be made:

1. Replace resistors R26 and R27 with 100 nF capacitors on the ENET board.

In the debugging process we thought there was a problem with REFCLK. We have seen capacitors on the CDCI line, but not on this direct line from SERDES2. When we removed resistors R26 and R27, the voltage on the PHY REFCLK_N/P pads was around 1V, so the PHY itself boosts the voltage. So we replaced these two resistors with 100 nF capacitors. I'm not 100% sure if this modification is necessary, this may be a question for your hardware team.

2. Modify am65-cpsw-nuss.c Linux driver to enable REFCLK output.

The phy-cadence-torrent.c driver creates a clock provider with the ability to enable REFCLK as an output clock, but to actually be enabled, it must be used as a clock source in another driver. Add this code to the am65-cpsw-nuss.c probe function:

	clk = devm_clk_get(dev, "fck");
	if (IS_ERR(clk))
		return dev_err_probe(dev, PTR_ERR(clk), "getting fck clock\n");
	common->bus_freq = clk_get_rate(clk);
	
+	clk = devm_clk_get_optional(dev, "serdes_refclk");
+	if (IS_ERR(clk)) {
+		ret = PTR_ERR(clk);
+		dev_err(dev, "failed to get serdes_refclk\n");
+	} else if (clk) {
+	    ret = clk_prepare_enable(clk);
+	    if (ret) {
+		    dev_err(dev, "failed to enable serdes_refclk\n");
+	    }
+	}

	pm_runtime_enable(dev);
	ret = pm_runtime_resume_and_get(dev);
	if (ret < 0) {
		pm_runtime_disable(dev);
		return ret;
	}

Of course there should be error checks, but for our debugging purposes it was OK for now.

Build the kernel and move it to your board as usual.

3. Create k3-am69-sk-quad-port-eth-exp.dtso device-tree overlay.

This overlay was created as a modification of the k3-j784s4-evm-quad-port-eth-exp1.dtso. It is located in the <SDK_INSTALL_PATH>board-support/ti-linux-kernel-6.1.80+gitAUTOINC+2e423244f8-ti/arch/arm64/boot/dts/ti.

Full code:

// SPDX-License-Identifier: GPL-2.0
/**
 * DT Overlay for CPSW9G in QSGMII mode using J7 Quad Port ETH EXP Add-On Ethernet Card with
 * J7AHP board. The Add-On Ethernet Card has to be connected to ENET Expansion 1 slot on the
 * board.
 *
 * Product Datasheet: https://www.ti.com/lit/ug/spruj74/spruj74.pdf
 * Product Link: https://www.ti.com/tool/J721EXENETXPANEVM
 *
 * Copyright (C) 2023 Texas Instruments Incorporated - https://www.ti.com/
 */

/dts-v1/;
/plugin/;

#include <dt-bindings/gpio/gpio.h>
#include <dt-bindings/mux/ti-serdes.h>
#include <dt-bindings/phy/phy-cadence.h>
#include <dt-bindings/phy/phy-ti.h>
#include <dt-bindings/phy/phy.h>

#include "k3-pinctrl.h"

&{/} {
	aliases {
		ethernet1 = "/bus@100000/ethernet@c000000/ethernet-ports/port@5";
		ethernet2 = "/bus@100000/ethernet@c000000/ethernet-ports/port@6";
		ethernet3 = "/bus@100000/ethernet@c000000/ethernet-ports/port@7";
		ethernet4 = "/bus@100000/ethernet@c000000/ethernet-ports/port@8";
	};
};

&main_cpsw0 {
	status = "okay";
	clock-names = "fck", "serdes_refclk";
	clocks = <&k3_clks 64 0>, <&serdes2 CDNS_TORRENT_REFCLK_DRIVER>;
	assigned-clock-parents = <&serdes2 CDNS_TORRENT_RECEIVED_REFCLK>;
	assigned-clocks = <&serdes2 CDNS_TORRENT_REFCLK_DRIVER>;
};

&main_cpsw0_port5 {
	status = "okay";
	phy-handle = <&cpsw9g_phy1>;
	phy-mode = "qsgmii";
	mac-address = [00 00 00 00 00 00];
	phys = <&cpsw0_phy_gmii_sel 5>, <&serdes2_qsgmii_link>;
	phy-names = "mac", "serdes";
};

&main_cpsw0_port6 {
	status = "okay";
	phy-handle = <&cpsw9g_phy2>;
	phy-mode = "qsgmii";
	mac-address = [00 00 00 00 00 00];
	phys = <&cpsw0_phy_gmii_sel 6>, <&serdes2_qsgmii_link>;
	phy-names = "mac", "serdes";
};

&main_cpsw0_port7 {
	status = "okay";
	phy-handle = <&cpsw9g_phy3>;
	phy-mode = "qsgmii";
	mac-address = [00 00 00 00 00 00];
	phys = <&cpsw0_phy_gmii_sel 7>, <&serdes2_qsgmii_link>;
	phy-names = "mac", "serdes";
};

&main_cpsw0_port8 {
	status = "okay";
	phy-handle = <&cpsw9g_phy0>;
	phy-mode = "qsgmii";
	mac-address = [00 00 00 00 00 00];
	phys = <&cpsw0_phy_gmii_sel 8>, <&serdes2_qsgmii_link>;
	phy-names = "mac", "serdes";
};

&cpsw0_phy_gmii_sel {
	ti,qsgmii-main-ports = <8>, <8>;
};

&davinci_mdio {
	reset-gpios = <&exp1 9 GPIO_ACTIVE_LOW>;
	reset-post-delay-us = <120000>;
	#address-cells = <1>;
	#size-cells = <0>;

	cpsw9g_phy0: ethernet-phy@16 {
		reg = <16>;
	};
	cpsw9g_phy1: ethernet-phy@17 {
		reg = <17>;
	};
	cpsw9g_phy2: ethernet-phy@18 {
		reg = <18>;
	};
	cpsw9g_phy3: ethernet-phy@19 {
		reg = <19>;
	};
};

&exp1 {
	/* Power-down to high */
	qsgmii-pwdn-hog {
		gpio-hog;
		gpios = <8 GPIO_ACTIVE_HIGH>;
		output-low;
	};
	/* Clock generator I2C enable */
	cg-i2c-hog {
		gpio-hog;
		gpios = <10 GPIO_ACTIVE_HIGH>;
		output-high;
	};
};

&serdes_ln_ctrl {
	idle-states = 
        <J784S4_SERDES0_LANE0_PCIE1_LANE0>, <J784S4_SERDES0_LANE1_PCIE1_LANE1>,
        <J784S4_SERDES0_LANE2_PCIE3_LANE0>, <J784S4_SERDES0_LANE3_USB>,
		
        <J784S4_SERDES1_LANE0_PCIE0_LANE0>, <J784S4_SERDES1_LANE1_PCIE0_LANE1>,
		<J784S4_SERDES1_LANE2_PCIE0_LANE2>, <J784S4_SERDES1_LANE3_PCIE0_LANE3>,
      
        <J784S4_SERDES2_LANE0_IP2_UNUSED>, <J784S4_SERDES2_LANE1_IP2_UNUSED>,
        <J784S4_SERDES2_LANE2_IP3_UNUSED>, <J784S4_SERDES2_LANE3_QSGMII_LANE8>;

};

&serdes_wiz2 {
	status = "okay";
};

&serdes2 {
	status = "okay";
	#address-cells = <1>;
	#size-cells = <0>;
	
	assigned-clock-parents = <&k3_clks 406 6>, <&k3_clks 406 6>, <&k3_clks 406 5>;
	assigned-clocks = <&serdes_wiz2 TI_WIZ_PLL0_REFCLK>, <&serdes_wiz2 TI_WIZ_PLL1_REFCLK>, <&serdes_wiz2 TI_WIZ_REFCLK_DIG>;
	clocks = <&serdes_wiz2 TI_WIZ_PLL0_REFCLK>, <&serdes_wiz2 TI_WIZ_PHY_EN_REFCLK>;

	serdes2_qsgmii_link: phy@0 {
		reg = <3>;
		cdns,num-lanes = <1>;
		#phy-cells = <0>;
		cdns,phy-type = <PHY_TYPE_QSGMII>;
		resets = <&serdes_wiz2 4>;
	};
};

Build the DT overlay and move it to your board as usual.

4. Add DT overlay to uEnv.txt to load at boot.

The SD card is expected to be used as the boot source. On your board, change directory to /run/media/BOOT-mmcblk1p1 and add the name of the overlay to the name_overlays variables. For me it is:

name_overlays=k3-j784s4-edgeai-apps.dtbo k3-am69-sk-quad-port-eth-exp.dtbo
name_overlays_fit=ti/k3-j784s4-edgeai-apps.dtbo ti/k3-am69-sk-quad-port-eth-ext.dtboCode

5. Reboot the board.

After these steps you should see eth1 through eth4 in ifconfig and should be able to communicate. For me, the IP address was obtained from a DHCP server and the communication speed measured with iperf3 was ~950 Mbps, as expected.


These steps may not be 100% correct, they should probably be confirmed and tested by the TI team.

Thank you and the EVM team for your time.