DP83869HM: Issue Setting up PHY Level (MII) Loopback

Hi all, 

I am attempting to setup MII level loopback within our PHY to confirm communications between MAC<-->PHY. I have been able to successfully perform loopback testing with passing results at the MAC & PCS level (both internal to the MAC), but not at the MII level (external to the MAC). I'm looking for some advice or clues as to what might be missing / going wrong.

Our Hardware:

• Intel Cyclone 10 GX FPGA
• NIOS II process (soft-core, instantiated in FPGA fabric)
• TI DP83869HM PHY

Our Software / Tools:

• Quartus Prime Pro 19.4
• NIOS II Software Build Tools for Eclipse 19.4
• Intel Ethernet Packet Generator / Ethernet Packet Monitor 19.4 IP
• Intel Triple Speed Ethernet 19.4 IP

Our Settings / Setup:

• SGMII-->Copper mode (OP_MODE_DECODE = 0x0046)
• Gigabit speed
• Auto-Negotiation disabled (docs suggest this should be disabled for loopback)
• Full Duplex

Our Documentation:

• Triple-Speed Ethernet Intel FPGA IP User Guide (UG-01008 - 2021.10.04)
• DP83869HM Datasheet - Revised December 2018p
• Intel Cyclone 10 LP FPGA Triple-Speed Ethernet and Intel On-Board PHY Chip Reference Design
  • pg. 4, figure 1 provides a basic diagram of our system. We however have an additional muxer to allow flow of 'normal' traffic when connected to a link partner
• How to Configure DP838xx for Ethernet Compliance Testing

Things we've tried to get it it working

• We've confirmed ability to R/W all registers in MAC / PCS / PHY (including extended registers) via reading data back from registers / observing data on a scope
• We've confirmed all MAC / PCS / PHY settings match each other in terms of speed, duplex and auto-negotiation
• We've confirmed MAC and PCS level loopback are working. Can see statistics in both the ethernet packet monitor registers and MAC registers
• We've confirmed we are accurately following PHY loopback setup steps found in "Intel Cyclone 10 LP FPGA Triple-Speed Ethernet and Intel On-Board PHY Chip Reference Design" package
  • Detailed steps themselves are found in .tcl scripts included in the .zip
• Attempted setting the proprietary 0xC6 register to 0x10.
  • This register has no description about what it does. But is called out as a necessary step in the "How to Configure DP838xx for Ethernet Compliance Testing document", pg. 13. Unclear if this needs to be set for loopback
• Attempted setting the LOOPCR (0xFE) to 0xE720.
  • This register has no description about what it does, and is only briefly mentioned in "DP83869HM Datasheet", pg 27. It is unclear if this needs to be set for MII loopback.
• Attempting forcing our LINK_STS1 in the BMSR (0x01) register to up by setting 'FORCE_LINK_GOOD' in the PHY_CONTROL (0x10) register.
  • NOTE: When we attempted digital loopback (which did not work), LINK_STS1 was high. Should this bit be high for MII loopback?

In this post the user states that "we just set the bit 14 of register 0x00  to 1 ,and  write 0x0004 to register 0x16. The test results were successful". In the next response, TI employee states "The configuration for MII and Analog/Digital loopback is correct." We have attempted these settings and confirmed they do not work in our case. Is there more to these steps that was left out?

Here is our current implementation:

#include <stdio.h>
#include <system.h>
#include <io.h>
#include <altera_eth_tse_regs.h>

typedef volatile struct tse_pcs_struct
{
    unsigned int control;
    unsigned int status;
    unsigned int phy_id1;
    unsigned int phy_id2;
    unsigned int dev_ability;
    unsigned int partner_ability;
    unsigned int an_expansion;
    unsigned int device_next_page;
    unsigned int partner_next_page;
    unsigned int master_slave_cntl;
    unsigned int master_slave_stat;
    unsigned int reserved1;
    unsigned int reserved2;
    unsigned int reserved3;
    unsigned int reserved4;
    unsigned int extended_status;
    unsigned int scratch;
    unsigned int rev;
    unsigned int link_timer1;
    unsigned int link_timer2;
    unsigned int if_mode;
} tse_pcs;

// Offsets from MAC Base
#define MAC_REV 0x00
#define MAC_SCRATCH 0x01
#define MAC_CMD_CFG 0x02
#define MAC_MAC_0 0x03
#define MAC_MAC_1 0x04
#define MAC_FRM_LEN 0x05
#define MAC_PAUSE_QUANT 0x06
#define MAC_RX_SEC_EMPTY 0x07
#define MAC_RX_SEC_FULL 0x08
#define MAC_TX_SEC_EMPTY 0x09
#define MAC_TX_SEC_FULL 0x0A
#define MAC_RX_ALMOST_EMPTY 0x0B
#define MAC_RX_ALMOST_FULL 0x0C
#define MAC_TX_ALMOST_EMPTY 0x0D
#define MAC_TX_ALMOST_FULL 0x0E
#define MAC_MDIO_ADDR0 0x0F
#define MAC_MDIO_ADDR1 0x10
#define MAC_HOLDOFF_QUANT 0x11
#define MAC_TX_IPG_LENGTH 0x17
#define MAC_TX_CMD_STAT 0x3A
#define MAC_RX_CMD_STAT 0x3B

// Offsets from Packet Gen Base
#define PKT_GEN_NUM_PKT 0x00
#define PKT_GEN_CONFIG_REG 0x01
#define PKT_GEN_RAND_SEED0 0x02
#define PKT_GEN_RAND_SEED1 0x03
#define PKT_GEN_SRC_ADDR0 0x04
#define PKT_GEN_SRC_ADDR1 0x05
#define PKT_GEN_DEST_ADDR0 0x06
#define PKT_GEN_DEST_ADDR1 0x07
#define PKT_GEN_OPERATION 0x08
#define PKT_GEN_PKT_TX_CNT 0x09

// Offsets from Packet Monitor Base
#define PKT_MON_NUM_PKT 0x00
#define PKT_MON_RX_OK 0x01
#define PKT_MON_RX_ERROR 0x02
#define PKT_MON_BYTE_RX_CNT0 0x03
#define PKT_MON_BYTE_RX_CNT1 0x04
#define PKT_MON_CYCLE_RX_CNT0 0x05
#define PKT_MON_CYCLE_RX_CNT1 0x06
#define RX_CTRL_STATUS 0x07

// Offsets for PCS Registers
#define ETH_PCS_BASE 0x10002200
#define PCS_CTRL_REG 0x00
#define PCS_IF_MODE_REG 0x14

// Offsets for PHY Registers
#define ETH_PHY_BASE 0x10002280
#define PHY_BMCR 0x00
#define PHY_BMSR 0x01
#define PHY_REGCR 0x0D
#define PHY_ADDAR 0x0E
#define PHY_CONTROL 0x10
#define PHY_STATUS 0x11
#define PHY_BIST 0x16
#define PHY_GEN_CTRL 0x1F
#define PHY_LOOPCR 0xFE
#define OP_MODE_DECODE 0x1DF

#define NUM_PKTS_TX_RX 1000

np_tse_mac *mac_registers = ((void*) ETH_TSE_0_BASE);
tse_pcs *pcs_registers = ((void*) ETH_PCS_BASE);

int main()
{
    alt_u32 selector = 0;

    // Step 1a - Configure MAC
    IOWR(ETH_TSE_0_BASE, MAC_CMD_CFG, 0x003B); // tx/rx enable, gigE, promiscuous mode
    IOWR(ETH_TSE_0_BASE, MAC_MAC_0, 0x17231C00); // Set upper portion of MAC address
    IOWR(ETH_TSE_0_BASE, MAC_MAC_1, 0x0000CB4A); // Set lower portion of MAC address
    IOWR(ETH_TSE_0_BASE, MAC_FRM_LEN, 1518); // Set dframe length
    IOWR(ETH_TSE_0_BASE, MAC_PAUSE_QUANT, 0xFFFF); // Set pause quant to max
    IOWR(ETH_TSE_0_BASE, MAC_RX_SEC_EMPTY, 4080);
    IOWR(ETH_TSE_0_BASE, MAC_RX_SEC_FULL, 16);
    IOWR(ETH_TSE_0_BASE, MAC_TX_SEC_EMPTY, 4080);
    IOWR(ETH_TSE_0_BASE, MAC_TX_SEC_FULL, 16);
    IOWR(ETH_TSE_0_BASE, MAC_RX_ALMOST_EMPTY, 8);
    IOWR(ETH_TSE_0_BASE, MAC_RX_ALMOST_FULL, 8);
    IOWR(ETH_TSE_0_BASE, MAC_TX_ALMOST_EMPTY, 8);
    IOWR(ETH_TSE_0_BASE, MAC_TX_ALMOST_FULL, 3);
    IOWR(ETH_TSE_0_BASE, MAC_MDIO_ADDR0, 0);
    IOWR(ETH_TSE_0_BASE, MAC_MDIO_ADDR1, 0);
    IOWR(ETH_TSE_0_BASE, MAC_TX_IPG_LENGTH, 12);
    IOWR(ETH_TSE_0_BASE, MAC_TX_CMD_STAT, 0); // OMIT_CRC = 0, TX_SHIFT16 = 0
    IOWR(ETH_TSE_0_BASE, MAC_RX_CMD_STAT, 0); // RX_SHIFT16 = 0

    IOWR(ETH_TSE_0_BASE, MAC_CMD_CFG, 0x203B); // Reset MAC using same values as above
    while ( (IORD(ETH_TSE_0_BASE, MAC_CMD_CFG) & 0x2000) == 0x2000)
    {
        // do nothing. Wait for MAC reset to complete
    }
    IOWR(ETH_TSE_0_BASE, MAC_CMD_CFG, 0x003B); // re-enable Tx/Rx lines (reset disables these)

    // Step 1b - Configure PCS ------------------------------------------------
    IOWR(ETH_PCS_BASE, PCS_CTRL_REG, 0x0140); // disable AN, gigE
    IOWR(ETH_PCS_BASE, PCS_IF_MODE_REG, 0x0009); // disable SGMII AN, gigE, full duplex

    // Step 1c - Configure PHY ------------------------------------------------
    IOWR(ETH_TSE_0_BASE, MAC_MDIO_ADDR1, 0); // set mdio address 1 to PHY 0

    IOWR(ETH_PHY_BASE, PHY_REGCR, 0x001F); // Set OP_MODE_DECODE
    IOWR(ETH_PHY_BASE, PHY_ADDAR, OP_MODE_DECODE);
    IOWR(ETH_PHY_BASE, PHY_REGCR, 0x401F);
    IOWR(ETH_PHY_BASE, PHY_ADDAR, 0x0046);

    IOWR(ETH_PHY_BASE, PHY_REGCR, 0x001F); // set LOOPCR - Data sheet says this needs to be done. No description on what it does
    IOWR(ETH_PHY_BASE, PHY_ADDAR, PHY_LOOPCR);
    IOWR(ETH_PHY_BASE, PHY_REGCR, 0x401F);
    IOWR(ETH_PHY_BASE, PHY_ADDAR, 0xE720);

    IOWR(ETH_PHY_BASE, PHY_BMCR, 0x0140); // Set BMCR - AN disabled, gigE (need to turn off AN before enabling MII Loopback mode)
    IOWR(ETH_PHY_BASE, PHY_CONTROL, 0x5008); // Set MDI Mode - 0x5028 = MDI-X, 0x5008 = MDI (need to turn off Auto MDI-x before enabling MII Loopback mode)
    IOWR(ETH_PHY_BASE, PHY_BMCR, 0x4140); // Enable MII Loopback
    IOWR(ETH_PHY_BASE, PHY_GEN_CTRL, 0x4000); // SW Reset - preserves register values

    // Step 2 - Configure packet generator ------------------------------------
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_NUM_PKT, NUM_PKTS_TX_RX); // set number of pkts to be sent
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_CONFIG_REG, 0x0BB8); // Set # pkts, fixed pkt length of 1500, incremental pattern
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_RAND_SEED0, 0x12345678); // Set random seed lower (can be anything, chose value from .tcl script)
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_RAND_SEED1, 0x00009ABC); // Set random seed upper (can be anything, chose value from .tcl script)
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_SRC_ADDR0, 0x17231C00); // Set src mac address lower (pulled MAC address from TI docs)
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_SRC_ADDR1, 0x0000CB4A); // Set src mac address upper (pulled MAC address from TI docs)
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_DEST_ADDR0, 0x17231C00); // Set dest mac address lower (pulled MAC address from TI docs)
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_DEST_ADDR1, 0x0000CB4A); // Set dest mac address upper (pulled MAC address from TI docs)

    // Step 3 - Configure packet monitor --------------------------------------
    IOWR(ETH_MON_DW_0_BASE, PKT_MON_NUM_PKT, NUM_PKTS_TX_RX); // set number of pkts to be received

    // Step 4 - Configure muxers ----------------------------------------------
    IOWR(ST_MUX_2_TO_1_DW_0_BASE, 0, 0x01); // Set mux 0 to a 1 (data coming in from mux 1)
    IOWR(ST_MUX_2_TO_1_DW_1_BASE, 0, 0x00); // Set mux 1 to a 0 (data coming in from pkt generator)

    // Step 5 - Turn on packet monitor ----------------------------------------
    IOWR(ETH_MON_DW_0_BASE, RX_CTRL_STATUS, 0x01);

    // Step 6 - Turn on packet generator --------------------------------------
    IOWR(ETH_GEN_DW_0_BASE, PKT_GEN_OPERATION, 0x01);

    while(!selector)
    {
        // do nothing until we pause and change selector to 1
    }

    alt_u32 rx_OK = IORD(ETH_MON_DW_0_BASE, PKT_MON_RX_OK);
    alt_u32 tx_ERROR = IORD(ETH_MON_DW_0_BASE, PKT_MON_RX_ERROR);
    alt_u32 last_bmcr = IORD(ETH_PHY_BASE, PHY_BMCR);
    alt_u32 last_bmsr = IORD(ETH_PHY_BASE, PHY_BMSR);
    last_bmsr = IORD(ETH_PHY_BASE, PHY_BMSR); // need to be read 2x for valid link up read
    alt_u32 last_phy_ctrl = IORD(ETH_PHY_BASE, PHY_CONTROL);
    alt_u32 last_phy_sts = IORD(ETH_PHY_BASE, PHY_STATUS);

    while(1)
    {
        // do nothing
    }

    return 0;
}

See above code for exact steps we're taking to enable MII level loopback. By examining the MAC statistics registers, we are able to see that all of the packets get sent out OK, but the MAC is not receiving any of those frames which should have been looped back. This suggests to us that the data is not being looped back in the PHY as we expect.

Overall it feels like there is some step(s) / knowledge we're missing to get this working properly. Is there something missing from our setup? Any help or advice would be welcomed and appreciated!