DP83869HM: Link not working with RGMII -> SGMII bridge -> 1000BASE-T using Linux

Hi Tgbe,

It appears auto-neg is disabled via straps on the daughter board, which is causing communication issues when the carrier board attempts the auto-neg process.

Please verify auto-neg is enabled on both boards via straps (pins 45/46/47 strapped low).

Hopefully this solves the initial link issue.

In terms of the error packets you are seeing while link is up, please try programming the daughter board in RGMII shift mode via 0x32[1:0] = '00' and noting any difference in performance.

Thank you,

Evan

Hi Evan,

thanks for your reply. That already cleared up most of our problems.

0x32[1:0] was indeed set to '11', which caused errors on the RGMII side (not SGMII, as I had suspected). After setting it to '00' and also setting both delay values to 7 in 0x86, the links work.

For future reference, there were two causes:

1. I had used phy-mode = "rgmii"; insted of phy-mode = "rgmii-id"; in the device tree, which (correctly) deactivates the internal delays.
2. The Linux driver had a bug that leads to incorrect default values for the delays (0xd0 instead of 0x77) if no delays are specified in the device tree. This is fixed by a rather new patch in the mainline kernel: https://lore.kernel.org/all/20230403140406.289977821@linuxfoundation.org

As for the auto-neg problem, it still persists. The DP83869 on the daughter card is indeed strapped high on pins 45/46/47 (I'm not sure why). However, I thought  that register settings would override that. Also, the data sheet gives the impression that these straps are not relevant for RGMII-SGMII-bridge mode, only pin 38.

Anyway, I will try and change the straps on one of our modules for testing and see if it makes a difference.

I also have the same issue with a second DP83869 that is configured as RGMII-1000BASE-X connected to an SFP transceiver. Here I need to set

ethtool -s eth1 autoneg off speed 1000 duplex full

and replug the cable to get the link to work. A possible issue is that LOS/SD# is not connected to the PHY. That one is strapped with OP_MODE 001 and pin 45 high, pin 46 low, pin 47 low (Fiber Auto-negotiation ON and Signal Detect disable on Pin 24). If the problem persists, I will post some more info on that.

Thank you,

Thomas

Hi Thomas,

Glad to hear the link issues are mostly resolved.

Straps 45/46/47 are not required if both link partners are set to force the same speed, however we generally recommend enabling auto-negotiation to simplify the link-up process.

For the RGMII-1000BASE-X PHY, I believe the issue is that 1000M speed is not being advertised. 45 high, 46 low, 47 low sets the device in Auto-neg with only 10/100 advertised.

Curious to hear if changing the straps resolves the problem entirely.

Thank you,

Evan

Hi Evan,

we performed the strapping changes for both the RGMII-SGMII bridge and the 1000BASE-X phy (pins 45, 46 and 47 to GND). Unfortunately, that didn't solve the autoneg problem. The behavior is as before.

For the RGMII-to-SGMII-to-1000BASE-T link, after plugging the cable auto-negotiation must be disabled in Linux with ethtool to make the link work.

For the RGMII-to-1000BASE-X (SFP) link, even more steps are necessary. Either auto-negotiation must be disabled and then the cable must be un-/regplugged; or auto-negotiation must be disabled, then enabled, then disabled.

For the both the bridge/PHY combo and the SFP link I provide register dumps below for the steps.

It looks like the 1000BASE-T link comes up fine including auto-negotiation, but is not forwarded correctly over the SGMII bridge (even though the whole connection works without errors after disabling autoneg). In FX_STS the link up doesn't come up on the PHY side after plugging the cable, and the bridge side seems to indicate a remote failure (?).

The SFP link also looks strange. Here auto-negotiation appears to succeed at first, but the link doesn't come up. After dis-/reenabling the autoneg, the link comes up but auto-negotiation doesn't succeed anymore. After disabling it again, the link can be used...

Data transfer after link up works fine without errors for both links.

All DP83869 used are rev. 3, if that makes any difference.

I'm unsure what to try next and would be grateful for any suggestions.

Thanks,

Thomas

Hi Thomas,

Thank you for sharing the detailed updates.

What SFP modules are you using? From my past experience with Fiber link setups, some SFP modules only support 1G speeds and require both link partners to have auto-negotiation turned off in order to link up properly. That, or leaving auto-negotiation enabled with 100/10M advertisements disabled may help link-up as well in this case.

I need time to review the register logs and create a test plan to help isolate the root cause, please expect a follow-up by Wednesday 2/7 for this.

Thank you,

Evan

Hi Evan,

I am using HP fiber SFP modules. I cannot tell you the exact type right now because I don't have the hardware on hand. Anyway, I don' think for fiber SFP modules the transceiver model matters (as opposed to RJ45 SFP modules). The fiber link connects to a QNAP switch with auto-neg capable SFP ports and a web interface for configuration and monitoring.

For 1000BASE-X the 10/100M strap options should not really matter, since 1000BASE-X can only run at 1000M. Auto-negotiation should only determine duplex and flow-control capabilities.

Thanks again for your support.

Thomas

Hi Thomas,

Thank you for clarifying.

For fiber applications with DP83869, the link speed should be manually set with auto-neg disabled at each endpoint.

Regards,

Evan

Hi Evan,

the link speed for 1000BASE-X is always 1000 Mbps, but there is still the 1000BASE-X auto-negotiation for the duplex and PAUSE settings, which the DP83868 supports (there are registers for it and even an explicit strap setting "Fiber Auto-negotiation ON" as per table 9-12). I rather suspect that this is an issue with the Linux driver, and the bits relevant for auto-negotiaion are somehow not written/read correctly.

The same is true for the RGMII-SGMII bridge. Here was at first suspicious of the link-fault bit, but another post seems to indicate that this bit can be ignored for this mode.

In fact, it looks like the link comes up fine on the PHY side, including auto-negotiation, but this info is somehow not correctly forwarded over the SGMII link; or else the Linux driver doesn't pick up the information from the relevant bits.

Can I conclude that these are all driver issues? Do you know if these modes have been tested with the Linux driver before? Would it be possible for a driver expert to have a look at this issue?

Thanks,

Thomas

Hi Thomas,

Which version of Linux are you working with? I will take a look at the driver and discuss with team to confirm if this is a driver related issue.

In the past I have seen auto-negotiation issues for fiber, but was unclear if this was device or driver-level (using auto-neg disable as a temporary workaround).

Hoping to uncover the specific driver issue and upstream a patch to fix.

Thank you,

Evan

Hi Evan,

we are using Linux 5.15 from linux-imx. However, I have also updated the DP83869 driver to the newest version from the mainline (https://github.com/torvalds/linux/blob/master/drivers/net/phy/dp83869.c) as there have been some changes. I tried both versions. (I had to update the PHY ID in both cases because the driver doesn't have the new ID of the DP83869 Rev. 3 0x2000a1a4).

I would really like to get this to work. The CPU board is meant as a general-purpose daughter-board for customer-specific applications, so we want to be able to integrate it into most systems without issues like manually disabling auto-negotiation.

Many thanks,

Thomas

Hi Thomas,

Could you share the CPU ethtool log in the case with auto-neg enabled on both PHYs?

There is another test case I would also like to confirm:

- Daughter board auto-neg enabled, carrier board auto-neg disabled; probe and note RX_CLK frequency on RGMII of daughter board

Referring to the other E2E post's RGMII-SGMII autoneg issue after reviewing the driver, it seems the fix they implemented was on the system level rather than the driver level. The status bits referenced for the link-up logic (0x1[2], 0x11[15:14], ..,) are not referenced in the PHY driver. I would like to get further clarity on the exact problem statement before suggesting possible driver changes.

Thank you,

Evan

Hi Evan,

yes, I can perform the tests you asked tomorrow when I have the hardware on hand again.

I have updated the graphic in my original post above to make the whole setup a bit clearer.

I can access the RGMII-SGMII bridge and the RGMII-1000BASE-X PHY on the daughter board directly with ethtool.

The SGMII-1000BASE-T PHY on the carrier is supposed to be strap-only. I can now access its registers with mdio-tool because I made a hardware patch and included it in the device tree using a GPIO MDIO bus, but it is not associated with a network interface, so I cannot access it with ethtool.

Below is an excerpt from the device tree for completeness.

Thanks,

Thomas

&fec1 {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_enet1>;
    assigned-clocks = <&clks IMX7D_ENET_PHY_REF_ROOT_SRC>,
              <&clks IMX7D_ENET_AXI_ROOT_SRC>,
              <&clks IMX7D_ENET1_TIME_ROOT_SRC>,
              <&clks IMX7D_ENET1_TIME_ROOT_CLK>,
              <&clks IMX7D_ENET_AXI_ROOT_CLK>;
    assigned-clock-parents = <&clks IMX7D_PLL_ENET_MAIN_25M_CLK>,
                 <&clks IMX7D_PLL_ENET_MAIN_250M_CLK>,
                 <&clks IMX7D_PLL_ENET_MAIN_100M_CLK>;
    assigned-clock-rates = <0>, <0>, <0>, <100000000>, <250000000>;
    phy-mode = "rgmii-id";
    phy-handle = <&dp83869_fec1>;
    fsl,magic-packet;
    fsl,mii-exclusive;
    status = "okay";

    mdio {
        #address-cells = <1>;
        #size-cells = <0>;

        dp83869_fec1: ethernet-phy@0 {
            reg = <0>;
            compatible = "ethernet-phy-id2000.a0f3";
            ti,op-mode = <DP83869_RGMII_SGMII_BRIDGE>;
        };
    };
};

&fec2 {
    pinctrl-names = "default";
    pinctrl-0 = <&pinctrl_enet2>;
    assigned-clocks = <&clks IMX7D_ENET_PHY_REF_ROOT_SRC>,
              <&clks IMX7D_ENET_AXI_ROOT_SRC>,
              <&clks IMX7D_ENET2_TIME_ROOT_SRC>,
              <&clks IMX7D_ENET2_TIME_ROOT_CLK>,
              <&clks IMX7D_ENET_AXI_ROOT_CLK>;
    assigned-clock-parents = <&clks IMX7D_PLL_ENET_MAIN_25M_CLK>,
                 <&clks IMX7D_PLL_ENET_MAIN_250M_CLK>,
                 <&clks IMX7D_PLL_ENET_MAIN_100M_CLK>;
    assigned-clock-rates = <0>, <0>, <0>, <100000000>, <250000000>;
    phy-mode = "rgmii-id";
    phy-handle = <&dp83869_fec2>;
    fsl,magic-packet;
    fsl,mii-exclusive;
    status = "okay";

    mdio {
        #address-cells = <1>;
        #size-cells = <0>;

        dp83869_fec2: ethernet-phy@0 {
            reg = <0>;
            compatible = "ethernet-phy-id2000.a0f3";
            ti,op-mode = <DP83869_RGMII_1000_BASE>;
        };
    };
};

/ {
    mdio0: mdio {
        pinctrl-names = "default";
        pinctrl-0 = <&pinctrl_gpio_mdio>;

        compatible = "virtual,mdio-gpio";

        #address-cells = <1>;
        #size-cells = <0>;

        gpios = <&gpio4 8 GPIO_ACTIVE_HIGH>, <&gpio4 9 GPIO_ACTIVE_HIGH>;

        dp83869_sk: ethernet-phy@0 {
            reg = <0>;
            compatible = "ethernet-phy-id2000.a0f3";
            ti,op-mode = <DP83869_SGMII_COPPER_ETHERNET>;
        };
    };

    aliases {
        mdio-gpio0 = &mdio0;
    };
};

Thank you for clarifying the DTS and setup.

I will review the DTS settings, otherwise waiting on ethtool log and RX_CLK probe when possible.

Best regards,

Evan 

Hi Evan,

here are the ethtool outputs for the two DP83869 on the daughter board.

eth0 is the RGMII-SGMII bridge. eth1 is the RGMII-1000BASE-X PHY.

I cannot access the SGMII-1000BASE-T PHY on the carrier board with ethtool, since that one doesn't have a native MDIO connection.

# ethtool eth0
Settings for eth0:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: no


# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half
                                         100baseT/Half
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no

I also did the RX_CLK measurements on the RGMII-SGMII bridge on the daughter board.

At boot time, auto-neg is enabled on both the RGMII-SGMII bridge on the daughter board and the SGMII-1000BASE-T PHY on the carrier board. With no cable plugged in, the RX_CLK frequency on the daughter board is 2.5 MHz.

I disable auto-neg on the carrier board PHY by setting 0x0[12] to 0:

mdio gpio-0 mmd-c22 0:31 0x00 0x0140

At the same moment, the RX_CLK frequency on the daughter board changes to 125 MHz.

If I leave auto-neg on on the carrier board and plug in a TP cable to a GbE-capable switch, the RX_CLK frequency also changes to 125 MHz.

Best,

Thomas

Hi Thomas,

Please allow me some time to review with team and get back to you.

I expect to have feedback for next debug steps by Tuesday 2/20 due to U.S. holiday.

Thank you,

Evan

Hi Thomas,

RX_CLK frequency is behaving as expected - it should switch to 125MHz either when auto-negotiating or forcing 1000M speed.

For next steps:

- Please share eth0 and eth1 ethtool log for passing case with auto-negotiation disabled

- Test with link loss pass through disabled on eth1 and eth0 PHYs (reg 0x1ECh = 0x1FFC)

- Test with 10/100M advertisements disabled. I understand 1000BASE-X won't negotiate to lower speeds, but I would like to sanity check that negotiation isn't failing due to speed miscommunication

Thank you,

Evan

Hi Evan,

thanks for your help. here are the output for the tests you asked for. I marked the parts of the ethtool outputs that changed from the previous state for each interface.

- Please share eth0 and eth1 ethtool log for passing case with auto-negotiation disabled

# # AFTER REBOOT
#
# ethtool eth0
Settings for eth0:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
#
# ethtool -s eth0 autoneg off speed 1000 duplex full
# ethtool -s eth1 autoneg off speed 1000 duplex full
#
# ethtool eth0
Settings for eth0:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: off
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
#
# # CABLES PLUGGED IN
#
[ 9672.242135] fec 30be0000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[ 9672.249816] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 9672.281252] 8021q: 802.1Q VLAN Support v1.8
[ 9673.452126] fec 30bf0000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
[ 9673.459760] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[ 9674.481742] fec 30bf0000.ethernet eth1: Link is Down
[ 9675.522138] fec 30bf0000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
[ 9675.529774] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
# ethtool eth0
Settings for eth0:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: off
    master-slave cfg: forced slave
    master-slave status: slave
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: yes
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: off
    master-slave cfg: forced slave
    master-slave status: slave
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: yes

- Test with link loss pass through disabled on eth1 and eth0 PHYs (reg 0x1ECh = 0x1FFC)

# # AFTER REBOOT
#
# ethtool eth0
Settings for eth0:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
#
# mdio 30be0000.ethernet-1 mmd-c22 0:31 0x1EC 0x1FFC
# mdio 30bf0000.ethernet-2 mmd-c22 0:31 0x1EC 0x1FFC
# mdio gpio-0 mmd-c22 0:31 0x1EC 0x1FFC
#
# # -> NO ethtool CHANGES on eth0 or eth1
#
# # CABLES PLUGGED IN
#
# ethtool eth0
Settings for eth0:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half
                                         100baseT/Half
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no

- Test with 10/100M advertisements disabled.

# # AFTER REBOOT
#
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
#
# ethtool -s eth1 advertise 100baseT/Half off
# ethtool -s eth1 advertise 100baseT/Full off
# ethtool -s eth1 advertise 1000baseT/Full off
#
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
#
# # CABLE PLUGGED IN
#
# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half
                                         100baseT/Half
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
 

Disabling link loss pass through or 10M/100M advertised modes don't seem to help fixing auto-negotiation. However, the link-partner advertised modes for 1000BASE-X seem suspicious. I'm wondering if the driver is checking the wrong registers or bits here.

Best,

Thomas

Hi Thomas,

Thank you for the prompt response.

There is one driver issue I've noticed for setting RGMII to 1000BASE-X mode - some register writes are missing from the case statement (line 780). Please try writing this configuration on the eth1 PHY in the auto-neg on case:

If this doesn't yield any difference, I'd like to focus on debugging eth1 for now - do you have two of the daughter/carrier board setups for a back-to-back setup on eth1 through the SFP?

E.g. :

CPU | - RGMII - | DP83869 | --- 1000BASE-X --- |  SFP | --- 1000BASE-X --- | DP83869 | - RGMII - | CPU 

If so, please try connecting with auto-negotiation enabled, and writing 0x1F = 0x8000 to both PHYs to reset the straps to original values (in-case driver is incorrectly overriding strap settings).

For this test case, please unplug/replug the SFP module.

Best regards,

Evan

Hi Evan,

thank you, I'll be able to perform your suggested tests this Friday.

Best,

Thomas

Sounds good, looking forward to results.

Thank you,

Evan

I repeated the 1000BASE-X connection test after setting the register values as per your post. This didn't appear to make any difference.

As for the the test setup with two boards, I will need some more time to prepare this. I'm hoping to be able to prepare the hardware next week. I'll post again when I have performed the tests.

Hi Thomas,

Looking forward to hear more results.

I will share any suggestions that come up while discussing the query with the team.

Thank you,

Evan

Hi Evan,

I performed your suggested test with two of our cards connected, so two DP83869 in 1000BASE-X mode back to back. I post the output from one the cards below. (The other one is idendical except for using a different IP address).

The result is a bit strange. I now get a "link up" status in Linux, but the link still doesn't work (no ping or data going through). It seems to me that the output from ethtool, especially for Link partner advertised link modes is nonsensical (only 10baseT/Half), perhaps because the driver doesn't access the correct registers, and therefore Linux cannot determine the correct settings (speed and duplex remain "Unknown").

Once again, if I disable auto-negotiation and force 1000 Mbps/full-duplex, the link starts working.

~# # After reboot
~# # SFP not connected
~#
~# mdio 30bf0000.ethernet-2 mmd-c22 0:31 0x1F 0x8000
~#
~# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
~#
~# # Connect SFP
~#
~# [  133.442355] fec 30bf0000.ethernet eth1: Link is Up - Unknown/Unknown - flow control off
[  133.450418] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  133.477711] 8021q: 802.1Q VLAN Support v1.8
~#
~# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII     FIBRE ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: slave
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: yes
~#
~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 86:2d:84:c0:34:5d brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 82:6f:ac:e2:89:96 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::806f:acff:fee2:8996/64 scope link
       valid_lft forever preferred_lft forever
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
~#
~# ip addr add dev eth1 192.168.0.1/24
~#
~# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
^C
--- 192.168.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2105ms

~# # Ping fails, all packets lost
~#
~# ethtool -s eth1 speed 1000 duplex full autoneg off
[  246.007540] fec 30bf0000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
~#
~# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=4.58 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=64 time=2.25 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=64 time=2.23 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=64 time=2.27 ms
^C
--- 192.168.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 2.229/2.833/4.583/1.010 ms
~#
~# # Link works

Thanks,

Thomas

Hi Thomas,

Interesting, so ethtool for both DP83869's shows all advertised link modes, but the link partner advertised modes is read incorrectly.

I agree this seems to be on the software-layer with the driver... I will spend time more time reviewing the driver with team to see if we can identify the source of the bug.

Thank you,

Evan 

Hi Thomas,

Apologies for the delay here, I wanted to update you that we are still actively looking into this.

Thank you,

Evan

Hi Thomas,

Is it possible to test this patch in your system?

https://lore.kernel.org/lkml/20200903114259.14013-2-dmurphy@ti.com/

Thank you,

Evan

Hi Evan,

sorry for the late reply, I was busy with another project.

You linked a proposed patch from the LKML that was revised a few days later:

https://lore.kernel.org/lkml/20200918191453.13914-3-dmurphy@ti.com/

The revised version was actually merged into the kernel and is present in the driver version I am using.

Would you suggest that I try to replace the patch that is in the kernel with the originally proposed patch that you linked?

Thank you,

Thomas

Hi Thomas, 

Yes, if you can share the ethtool log after replacing the patch this would help us further understand how the advertisements are being communicated on the software side. I appreciate your patience through this debug, there are related DP83869 driver patches that we are actively working on and looking to upload.

Thank you,

Evan

Hi Evan,

I have repeated the test with two boards back-to-back as you requested.

I reverted

https://github.com/torvalds/linux/commit/a29de52ba2a156873505d8b8cef44e69925b8114

and applied the patch you linked

https://lore.kernel.org/lkml/20200903114259.14013-2-dmurphy@ti.com/

instead. I attach the results below.

The output looks and behaves exactly as before, with the exception of the Supported ports item in ethtool. Before, this included FIBER, wich is now gone.

~# # After reboot
~# # SFP not connected
~#
~# mdio 30bf0000.ethernet-2 mmd-c22 0:31 0x1F 0x8000
[  218.770168] mdio_netlink: loading out-of-tree module taints kernel.
~# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: unknown
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: no
~#
~# # Connect SFP
~#
~# [  292.492049] fec 30bf0000.ethernet eth1: Link is Up - Unknown/Unknown - flow control off
[  292.500118] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  292.526614] 8021q: 802.1Q VLAN Support v1.8
~#
~# ethtool eth1
Settings for eth1:
    Supported ports: [ TP     MII ]
    Supported link modes:   100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Half 100baseT/Full
                            1000baseT/Full
                            1000baseX/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    master-slave cfg: forced slave
    master-slave status: slave
    Port: FIBRE
    PHYAD: 0
    Transceiver: external
    Supports Wake-on: g
    Wake-on: d
    Link detected: yes
~#
~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 2a:07:c1:2a:03:6e brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 02:2a:95:ac:53:5f brd ff:ff:ff:ff:ff:ff
    inet 169.254.204.181/16 brd 169.254.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::2a:95ff:feac:535f/64 scope link
       valid_lft forever preferred_lft forever
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
~#
~# ip addr add dev eth1 192.168.0.1/24
~#
~# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
From 192.168.0.1 icmp_seq=1 Destination Host Unreachable
From 192.168.0.1 icmp_seq=2 Destination Host Unreachable
From 192.168.0.1 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.0.2 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4165ms
pipe 3
~# ethtool -s eth1 speed 1000 duplex full autoneg off
[  440.588888] fec 30bf0000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
~#
~# ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=3.41 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=64 time=2.20 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=64 time=2.21 ms
64 bytes from 192.168.0.2: icmp_seq=4 ttl=64 time=2.23 ms
^C
--- 192.168.0.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 2.199/2.511/3.405/0.515 ms
~#
~# # Link works

Thanks,

Thomas

Hi Thomas,

Thanks for sharing this log.

Please allow me until early next week to review and get back to you with suggestions.

Best regards,

Evan

Hi Thomas,

Please test again after writing 0x1DF = 0x3.

I believe the pin mapping may have been inadvertently swapped between RGMII-to-SGMII vs. SGMII-to-RGMII mode.

Thank you,

Evan

Hi Thomas,

May I ask the status of this debug on your side?

My colleague Alvaro recently worked on a similar issue, there is more clarifying information on the correct bridge mode setting here:

https://e2e.ti.com/support/interface-group/interface/f/interface-forum/1350481/dp83869hm-rgmii-to-sgmii-bridge-auto-negotiation-not-succeeding

Thank you,

Evan