DP83869HM: RGMII to 1000-base-X SFP connection not working

Just one more information, the link comes up only after we disable autonegotiation and we force speed and duplex:
ethtool -s eth1 autoneg off speed 1000 duplex full

Please advice. 

Hello,

For best practices the PHY should be bootstrapped to be in RGMII to 1000Base-X mode. This should take care of most if not all configurations. 

If you are sending TX packets out, is the opposite MAC acknowledging receiving packets? If using same boards with each other, it may be important to switch to a known good board to isolate where the issue lies.

Once we have this datapoint, we can do more pinpoint debugging to isolate where the issue lies.

Sincerely,

Gerome

Hi Gerome Cacho, Yes, we have the pins configured to boot in RGMII to 1000Base-X since the beginning. We are testing the board against a known working on-the-shelf ethernet switch.

On the MAC on the opposite side, we don't detect received packages.
here you can find the register dump with "ethtool -d eth1":
# ethtool -d eth1
0x004: 0x00008000
0x008: 0x0a0000aa
0x010: 0x01000000
0x014: 0x00000000
0x024: 0x70000132
0x040: 0x6f8e0000
0x044: 0x0000037c
0x064: 0x40000000
0x084: RCR (Receive Control Register) 0x47c00044
MAX_FL (Maximum frame length) 1984
FCE (Flow control enable) 0
BC_REJ (Broadcast frame reject) 0
PROM (Promiscuous mode) 0
DRT (Disable receive on transmit) 0
LOOP (Internal loopback) 0
0x0c4: TCR (Transmit Control Register) 0x00000004
RFC_PAUSE (Receive frame control pause) 0
TFC_PAUSE (Transmit frame control pause) 0
FDEN (Full duplex enable) 1
HBC (Heartbeat control) 0
GTS (Graceful transmit stop) 0
0x0e4: 0xf8dc7ad7
0x0e8: 0x14378808
0x0ec: 0x00010000
0x0f0: 0xcc801312
0x0f4: 0xcc801312
0x0f8: 0xcc801312
0x100: 0xcc801312
0x104: 0xcc801312
0x108: 0xcc801312
0x118: IAUR (Individual Address Upper Register) 0x00000000
IADDR1 0x0000000000000000
0x11c: IALR (Individual Address Lower Register) 0x00000000
IADDR2 0x0000000000000000
0x120: GAUR (Group Address Upper Register) 0x00401000
GADDR1 0x0040100000000000
0x124: GALR (Group Address Lower Register) 0x00800001
GADDR2 0x0000000000800001
0x144: TFWR (Transmit FIFO Watermark Register) 0x00000100
X_WMRK 64 bytes
0x14c: FRBR (FIFO Receive Bound Register) 0x00000000
R_BOUND (Highest valid FIFO RAM address) 0x00
0x150: 0x00000000
0x160: 0xf5b22000
0x164: 0xf5b2a000
0x168: 0x000007c0
0x16c: 0xf5b24000
0x170: 0xf5b2e000
0x174: 0x000007c0
0x180: 0xf5b20000
0x184: 0xf5b26000
0x188: EMRBR (Maximum Receive Buffer Size) 0x000007c0
R_BUF_SIZE (Receive buffer size) 124
0x190: 0x00000000
0x194: 0x00000000
0x198: 0x00000004
0x19c: 0x00000004
0x1c4: 0x00000086
0x1c8: 0x00013210
0x1cc: 0x00017654
0x1d8: 0x00010200
0x1dc: 0x00010200
0x1e0: 0x01000000
0x1e4: 0x00000000
0x1e8: 0x01000000
0x1ec: 0x00000000
0x1f0: 0x00000000
0x200: 0x00000000
0x204: 0x000002a7
0x208: 0x00000078
0x20c: 0x0000022f
0x210: 0x00000000
0x214: 0x00000000
0x218: 0x00000000
0x21c: 0x00000000
0x220: 0x00000000
0x224: 0x00000000
0x228: 0x00000000
0x22c: 0x00000175
0x230: 0x000000ba
0x234: 0x00000078
0x238: 0x00000000
0x23c: 0x00000000
0x240: 0x00000000
0x244: 0x00019b59
0x248: 0x00000000
0x24c: 0x000002a7
0x250: 0x00000000
0x254: 0x00000000
0x258: 0x00000000
0x25c: 0x00000000
0x260: 0x00000000
0x264: 0x00000000
0x268: 0x00000000
0x26c: 0x00000000
0x270: 0x00000000
0x274: 0x00019b59
0x284: 0x00000000
0x288: 0x00000000
0x28c: 0x00000000
0x290: 0x00000000
0x294: 0x00000000
0x298: 0x00000000
0x29c: 0x00000000
0x2a0: 0x00000000
0x2a4: 0x00000000
0x2a8: 0x00000000
0x2ac: 0x00000000
0x2b0: 0x00000000
0x2b4: 0x00000000
0x2b8: 0x00000000
0x2bc: 0x00000000
0x2c0: 0x00000000
0x2c4: 0x00000000
0x2c8: 0x00000000
0x2cc: 0x00000000
0x2d0: 0x00000000
0x2d4: 0x00000000
0x2d8: 0x00000000
0x2dc: 0x00000000
0x2e0: 0x00000000

Hi, we found that the TX-Disable pin on the SFP cage was reversed compared to what we expected. setting this pin high enabled the module.
Now we have transmission from the SFP module but yet we are having several RX errors, seems that the CRC is compromised. 

# ethtool -S eth1
NIC statistics:
tx_dropped: 0
tx_packets: 416
tx_broadcast: 414
tx_multicast: 2
tx_crc_errors: 0
tx_undersize: 0
tx_oversize: 0
tx_fragment: 0
tx_jabber: 0
tx_collision: 0
tx_64byte: 414
tx_65to127byte: 2
tx_128to255byte: 0
tx_256to511byte: 0
tx_512to1023byte: 0
tx_1024to2047byte: 0
tx_GTE2048byte: 0
tx_octets: 26644
IEEE_tx_drop: 0
IEEE_tx_frame_ok: 416
IEEE_tx_1col: 0
IEEE_tx_mcol: 0
IEEE_tx_def: 0
IEEE_tx_lcol: 0
IEEE_tx_excol: 0
IEEE_tx_macerr: 0
IEEE_tx_cserr: 0
IEEE_tx_sqe: 0
IEEE_tx_fdxfc: 0
IEEE_tx_octets_ok: 26644
rx_packets: 155
rx_broadcast: 0
rx_multicast: 0
rx_crc_errors: 42
rx_undersize: 0
rx_oversize: 0
rx_fragment: 113
rx_jabber: 0
rx_64byte: 0
rx_65to127byte: 1


While the phy reports: 
ethtool -d eth1
0x004: 0x00008000
0x008: 0x0a0000aa
0x010: 0x01000000
0x014: 0x00000000
0x024: 0x70000132
0x040: 0x6f8e0000
0x044: 0x0000037c
0x064: 0x40000000
0x084: RCR (Receive Control Register) 0x47c00044
MAX_FL (Maximum frame length) 1984
FCE (Flow control enable) 0
BC_REJ (Broadcast frame reject) 0
PROM (Promiscuous mode) 0
DRT (Disable receive on transmit) 0
LOOP (Internal loopback) 0
0x0c4: TCR (Transmit Control Register) 0x00000004
RFC_PAUSE (Receive frame control pause) 0
TFC_PAUSE (Transmit frame control pause) 0
FDEN (Full duplex enable) 1
HBC (Heartbeat control) 0
GTS (Graceful transmit stop) 0
0x0e4: 0xf8dc7ad7
0x0e8: 0x14378808
0x0ec: 0x00010000
0x0f0: 0xcc801312
0x0f4: 0xcc801312
0x0f8: 0xcc801312
0x100: 0xcc801312
0x104: 0xcc801312
0x108: 0xcc801312
0x118: IAUR (Individual Address Upper Register) 0x00000000
IADDR1 0x0000000000000000
0x11c: IALR (Individual Address Lower Register) 0x00000000
IADDR2 0x0000000000000000
0x120: GAUR (Group Address Upper Register) 0x04401002
GADDR1 0x0440100200000000
0x124: GALR (Group Address Lower Register) 0x00800041
GADDR2 0x0000000000800041
0x144: TFWR (Transmit FIFO Watermark Register) 0x00000100
X_WMRK 64 bytes
0x14c: FRBR (FIFO Receive Bound Register) 0x00000000
R_BOUND (Highest valid FIFO RAM address) 0x00
0x150: 0x00000000
0x160: 0xf5b22000
0x164: 0xf5b2a000
0x168: 0x000007c0
0x16c: 0xf5b24000
0x170: 0xf5b2e000
0x174: 0x000007c0
0x180: 0xf5b20000
0x184: 0xf5b26000
0x188: EMRBR (Maximum Receive Buffer Size) 0x000007c0
R_BUF_SIZE (Receive buffer size) 124
0x190: 0x00000000
0x194: 0x00000000
0x198: 0x00000004
0x19c: 0x00000004
0x1c4: 0x00000086
0x1c8: 0x00013210
0x1cc: 0x00017654
0x1d8: 0x00010200
0x1dc: 0x00010200
0x1e0: 0x01000000
0x1e4: 0x00000000
0x1e8: 0x01000000
0x1ec: 0x00000000
0x1f0: 0x00000000
0x200: 0x00000000
0x204: 0x00000200
0x208: 0x000001fe
0x20c: 0x00000002
0x210: 0x00000000
0x214: 0x00000000
0x218: 0x00000000
0x21c: 0x00000000
0x220: 0x00000000
0x224: 0x00000000
0x228: 0x000001fe
0x22c: 0x00000002
0x230: 0x00000000
0x234: 0x00000000
0x238: 0x00000000
0x23c: 0x00000000
0x240: 0x00000000
0x244: 0x00008014
0x248: 0x00000000
0x24c: 0x00000200
0x250: 0x00000000
0x254: 0x00000000
0x258: 0x00000000
0x25c: 0x00000000
0x260: 0x00000000
0x264: 0x00000000
0x268: 0x00000000
0x26c: 0x00000000
0x270: 0x00000000
0x274: 0x00008014
0x284: 0x000000c4
0x288: 0x00000000
0x28c: 0x00000000
0x290: 0x00000037
0x294: 0x00000000
0x298: 0x00000000
0x29c: 0x0000008d
0x2a0: 0x00000000
0x2a4: 0x00000000
0x2a8: 0x00000000
0x2ac: 0x00000001
0x2b0: 0x00000029
0x2b4: 0x00000006
0x2b8: 0x00000007
0x2bc: 0x00000000
0x2c0: 0x00000000
0x2c4: 0x00004288
0x2c8: 0x00003852
0x2cc: 0x00000000
0x2d0: 0x00000037
0x2d4: 0x00000000
0x2d8: 0x00000000
0x2dc: 0x00000000
0x2e0: 0x00000000


rx_128to255byte: 31
rx_256to511byte: 3
rx_512to1023byte: 7
rx_1024to2047byte: 0
rx_GTE2048byte: 0
rx_octets: 13881
IEEE_rx_drop: 10964
IEEE_rx_frame_ok: 0
IEEE_rx_crc: 42
IEEE_rx_align: 0
IEEE_rx_macerr: 0
IEEE_rx_fdxfc: 0
IEEE_rx_octets_ok: 0
rx_xdp_redirect: 0
rx_xdp_pass: 0
rx_xdp_drop: 0
rx_xdp_tx: 0
rx_xdp_tx_errors: 0
tx_xdp_xmit: 0
tx_xdp_xmit_errors: 0
rx_pp_alloc_fast: 756
rx_pp_alloc_slow: 12
rx_pp_alloc_slow_ho: 0
rx_pp_alloc_empty: 12
rx_pp_alloc_refill: 0
rx_pp_alloc_waive: 0
rx_pp_recycle_cached: 0
rx_pp_recycle_cache_full: 0
rx_pp_recycle_ring: 0
rx_pp_recycle_ring_full: 0
rx_pp_recycle_released_ref: 0

Hello,

Seeing how switching up the SD pin was able to enable packet transfer (even corrupted), I would like to switch up approaches to isolating where the issue lies.

First I would like to test MDI; can we place DUT in reverse loopback (Reg 0x16) and have LP send packets? This would have PHY send packets back and ensure that the MDI is good.

Next, I would like to test MII loopback. Place DUT into MII loopback (Reg 0x0) and have DUT's MAC send packets. This would have PHY send packets back to verify the MII. 

My most likely hunch is that the RGMII skew is affecting the RX transmission and may need to be adjusted. These test benches would help in achieving that goal.

Sincerely,

Gerome

Hi Gerome,

I set the loopback bit for mii loopback:

# mdio stmmac-0 0x05
BMCR(0x00): 0x4000
flags: -reset +loopback -aneg-enable -power-down -isolate -aneg-restart
-collision-test
speed: 10-half

BMSR(0x01): 0x614d
capabilities: -100-t4 +100-tx-f +100-tx-h -10-t-f -10-t-h -100-t2-f -100-t2-h
flags: +ext-status -aneg-complete -remote-fault +aneg-capable +link
-jabber +ext-register

ID(0x02/0x03): 0x2000a0f3

ESTATUS(0x0F): 0xf000
capabilities: +1000-x-f +1000-x-h +1000-t-f +1000-t-h

running 100 iterations of ping I get:

ethtool -S eth1 | grep -v " 0"
NIC statistics:
tx_packets: 974
tx_broadcast: 255
tx_multicast: 704
tx_64byte: 113
tx_65to127byte: 477
tx_128to255byte: 232
tx_256to511byte: 147
tx_1024to2047byte: 5
tx_octets: 146184
IEEE_tx_frame_ok: 974
IEEE_tx_octets_ok: 146184
rx_packets: 654
rx_crc_errors: 616
rx_fragment: 38
rx_65to127byte: 384
rx_128to255byte: 140
rx_256to511byte: 87
rx_1024to2047byte: 5
rx_octets: 93292
IEEE_rx_drop: 9403
IEEE_rx_crc: 616
rx_pp_alloc_fast: 756
rx_pp_alloc_slow: 12
rx_pp_alloc_empty: 12

# ping -I eth1 -i 0.01 -c 100 192.168.6.205
ping: Warning: source address might be selected on device other than: eth1
PING 192.168.6.205 (192.168.6.205) from 192.168.6.134 eth1: 56(84) bytes of data.

--- 192.168.6.205 ping statistics ---
100 packets transmitted, 0 received, 100% packet loss, time 49204ms

# ethtool -S eth1 | grep -v " 0"
NIC statistics:
tx_packets: 1087
tx_broadcast: 317
tx_multicast: 755
tx_64byte: 164
tx_65to127byte: 509
tx_128to255byte: 251
tx_256to511byte: 158
tx_1024to2047byte: 5
tx_octets: 159345
IEEE_tx_frame_ok: 1087
IEEE_tx_octets_ok: 159345
rx_packets: 767
rx_crc_errors: 729
rx_fragment: 38
rx_65to127byte: 467
rx_128to255byte: 159
rx_256to511byte: 98
rx_1024to2047byte: 5
rx_octets: 106566
IEEE_rx_drop: 9403
IEEE_rx_crc: 729
rx_pp_alloc_fast: 756
rx_pp_alloc_slow: 12
rx_pp_alloc_empty: 12

Not sure if we are able to perform the reverse loopback test right now, we're working on this .

Does this point to a RGMII skew?

Thanks,

Filippo

One more information that we think might be useful.

We are using the phy in rgmii-to-1000base-x and we needed to segment the SGMII lines using a TMUX1511PWR 4 channel analog switch. 

In order to reduce the power consumption we are powering down the SFP module and the SGMII lines with a GPIO when not used. 

Do you think this might introduce problems on the SGMII connection?

In all of these test both the module and the analog switch are obviously always enabled. 

Hello,

If off of MII loopback issues are being seen, then the issue most likely is the RGMII interface. Skew is a great way to correct for errors since most are likely setup/hold time issues that are seen due to the strict speed of 1G (125MHz DDR). Please consult Reg 0x86 and 0x32 for more information of what skew is required. It may also be prudent to scope out the clock and data signals at the receiver (MAC input) to get a better analysis of required skew adjustment.

Sincerely,

Gerome

Hi Gerome,

Your tests were perfectly targetting the point. We found that we have a bad clock @125 Mhz targetting 1000BASE-X, we downgraded manually the speed to 100BASE-X and now we finally have a working link!

We see the same lines working correctly on the evaluation board, the same settings are applied on the final system.

I tried acting on both 0x86 and 0x32 registers, but in both cases we don't see improvements in the signal, since these only acts in shifting the clock.

We deeply inspected the clock line, it's correctly equalized, not very long, does't have interference with other lines. The TXC line is very similarly routed but is behaving much better.

Do you know if there is a way to optimize this behavior? Increase the drive strenght of this line? 
Thanks,

Filippo

Some additional scope shots at 25 MHz. The signal is already degraded. the RXC @125MHz has a pk-pk voltage of 48mV with a baseline of 65mV. 

Hi Filippo,

This is unexpected. I would imagine parasitic are affecting the signal as with increase of speed, I think evaluating the MAC traces against our recommendations in our layout check would be a prudent move. The MAC section starts with row 26.

Sincerely,

Gerome