• State
  • Locked Locked
  • Replies 3 replies
  • Subscribers 33 subscribers
  • Views 1107 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DP83867ERGZ-S-EVM: RFSoC (Ultrascale+) DP83867ERGZ-S-EVM Link Up but No Traffic (Ping)

Alejandro Martin
Prodigy 20 points
Part Number: DP83867ERGZ-S-EVM
Other Parts Discussed in Thread: DP83867E, , DP83869EVM

Tool/software:

Hello dear TI Team,

In our company, I am developing a network interface for the RFSoC ZU27DR based on the PHY DP83867E, that is using SGMII interface. I am using the original evaluation board from TI DP83867ERGZ-S-EVM, where I removed the R72 Resistor to apply VDDIO_EXT 3v3 from external power supply (about 35-38mA consume @ 3v3)

The FPGA is a module from KR (Knowledge Resources) with a KRC4700 carrier board. This carrier has no GT connectors (SMA or similar) and all PS-GTR are pined over a M.2 connector. I used and adapter M.2 to PCIe x4 to get access to the GT Lane 3, because the RefClk3 is the only one I can get access from a Si5332C clock generator.

Well I configured a Petalinux Project 2023.2 for this FPGA with the following device-tree:

&psgtr {
/* GEM3, USB3 */
clocks = <&pcie_gtr_clk>, <&eth_gtr_clk>, <&usb3_gtr_clk>;
clock-names = "ref0", "ref1", "ref2";
};
&gem3 {
status = "okay";
phy-mode = "sgmii";
phys = <&psgtr 3 PHY_TYPE_SGMII 3 1>;
phy-handle = <&phy0>;
is-internal-pcspma;
// Fix MAC Address
//local-mac-address = [00 0a 35 00 22 03];

// Only to avoid the message ...
// macb ff0e0000.ethernet eth0: unable to generate target frequency: 125000000 Hz
assigned-clocks = <&zynqmp_clk GEM3_REF>;
assigned-clock-parents = <&zynqmp_clk GEM3_REF_UNG>;
 
assigned-clock-rates = <250000000>;

mdio: mdio {
#address-cells = <1>;
#size-cells = <0>;
phy0: ethernet-phy@0 {
#phy-cells = <1>;
compatible = "ethernet-phy-id2000.a231";
reg = <0>;
ti,fifo-depth = <DP83867_PHYCR_FIFO_DEPTH_4_B_NIB>;
ti,dp83867-rxctrl-strap-quirk; // Only if RX_DV/RX_CTRL in mode 1 or 2

// Reset EMIO GPIO 410 (EMIO 0 BANK 3)#
reset-names = "phy";
reset-assert-us = <5000>; // Default 300 @ zynqmp-sck-kr-g-revB.dts
reset-deassert-us = <5000>; // Default 280 @ zynqmp-sck-kr-g-revB.dts
reset-gpios = <&gpio 78 GPIO_ACTIVE_LOW>; // MDIO Reset in EMIO_0 (GPIO_78)
};

};

};

The registers of the DP83867 seem to be ok, in the U-BOOT,

ZynqMP> mii read 0x00 0x00-0x1F
addr=00 reg=00 data=1140
addr=00 reg=01 data=796D
addr=00 reg=02 data=2000
addr=00 reg=03 data=A231
addr=00 reg=04 data=01E1
addr=00 reg=05 data=CDE1
addr=00 reg=06 data=006F
addr=00 reg=07 data=2001
addr=00 reg=08 data=4006
addr=00 reg=09 data=0300
addr=00 reg=0a data=3C00
addr=00 reg=0b data=0000
addr=00 reg=0c data=0000
addr=00 reg=0d data=401F
addr=00 reg=0e data=0000
addr=00 reg=0f data=3000
addr=00 reg=10 data=5840
addr=00 reg=11 data=BF02
addr=00 reg=12 data=0000
addr=00 reg=13 data=1C42
addr=00 reg=14 data=29C7
addr=00 reg=15 data=0000
addr=00 reg=16 data=0000
addr=00 reg=17 data=0040
addr=00 reg=18 data=6150
addr=00 reg=19 data=4440
addr=00 reg=1a data=0002
addr=00 reg=1b data=0000
addr=00 reg=1c data=0000
addr=00 reg=1d data=0000
addr=00 reg=1e data=0002
addr=00 reg=1f data=0000

ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x0031
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E
1031
ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x0032
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E        
0000
ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x0033
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E        
0000
ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x0037
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E        
0000
ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x006E
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E        
8820
ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x006F
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E        
0110
ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x00D3
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E        
0000
ZynqMP> mii write 0x00 0x0D 0x001F
ZynqMP> mii write 0x00 0x0E 0x0135
ZynqMP> mii write 0x00 0x0D 0x401F
ZynqMP> mii read 0x00 0x0E        
0000

The reset is performed through an EMIO pin (pin EMIO0, or GPIO 78) and I observed it ocurrs once while FSBL and again after loading the kernel. The second reset (kernel) lasts double time than the first one (FSBL).

I fixed the ipaddr in the u-boot environment to try to ping from U-BOOT, but without success.

The GEM_CLK_CTL 0x00FF180308 register in the U-Boot shows the active mode for SGMII, but the GEM_CTRL 0x00FF180360 register indicates activity in GEM1, not GEM3 ????

ZynqMP> md 0x00FF180308 0x01
ff180308: 00030180                             ....
ZynqMP> md 0x00FF180360 0x01
ff180360: 00000040                             @...

After loading the Petalinux, the status from PHY registers is a bit different:

Reg: 0x00    Value: 0x1140
Reg: 0x01    Value: 0x796D
Reg: 0x02    Value: 0x2000
Reg: 0x03    Value: 0xA231
Reg: 0x04    Value: 0x09E1
Reg: 0x05    Value: 0xCDE1
Reg: 0x06    Value: 0x006F
Reg: 0x07    Value: 0x2001
Reg: 0x08    Value: 0x4006
Reg: 0x09    Value: 0x0300
Reg: 0x0A    Value: 0x3C00
Reg: 0x0D    Value: 0x401F
Reg: 0x0E    Value: 0x1111
Reg: 0x0F    Value: 0x3000
Reg: 0x10    Value: 0x5848
Reg: 0x11    Value: 0xAF02
Reg: 0x12    Value: 0x0000
Reg: 0x13    Value: 0x1C02
Reg: 0x14    Value: 0x2BC7
Reg: 0x15    Value: 0x0000
Reg: 0x16    Value: 0x0000
Reg: 0x17    Value: 0x0040
Reg: 0x18    Value: 0x6150
Reg: 0x19    Value: 0x4440
Reg: 0x1A    Value: 0x0002
Reg: 0x1E    Value: 0x0202
Reg: 0x1F    Value: 0x0000
Reg: 0x31    Value: 0x1111
Reg: 0x32    Value: 0x00D3
Reg: 0x33    Value: 0x0000
Reg: 0x37    Value: 0x0000
Reg: 0x43    Value: 0x07A0
Reg: 0x6E    Value: 0x8820
Reg: 0x6F    Value: 0x0110
Reg: 0x86    Value: 0x0057
Reg: 0xC6    Value: 0x0000
Reg: 0xD3    Value: 0x0000
Reg: 0xFE    Value: 0xE721
Reg: 0x134    Value: 0x1000
Reg: 0x135    Value: 0x0000
Reg: 0x16F    Value: 0x0015
Reg: 0x170    Value: 0x0C12
Reg: 0x172    Value: 0x0000
Reg: 0x1D5    Value: 0xF500

Now are both modes SGMII (Reg: 0x10/Value: 0x5848) and RGMII (Reg: 0x32/Value: 0x00D3) enabled but no Next-Page Autonegotiation as received (Reg: 0x11/Value: 0xAF02).

Regarding the strapping, I observed that depends on reset time: for short (< 300µs) resets, bootstrapping is a bit random. PHY address is sometimes 0x4, other 0x8 or even 0xC. This is due to the presence of signal in SO pins, while strapping. Pins are in mode 0 (floating) in the evaluation board, so only the 2 LSB from PHY address are right. Just extending the reset time (about 2000µs) is the problem corrected.

The link is up and no problem reported by macb,

[    2.129416] macb ff0e0000.ethernet: Not enabling partial store and forward
[    2.149860] macb ff0e0000.ethernet eth0: Cadence GEM rev 0x50070106 at 0xff0e0000 irq 45 (00:0a:35:00:22:03)
[    7.627405] macb ff0e0000.ethernet eth0: PHY [ff0e0000.ethernet-ffffffff:00] driver [TI DP83867] (irq=POLL)
[    7.637161] macb ff0e0000.ethernet eth0: configuring for phy/sgmii link mode
[    7.650759] macb ff0e0000.ethernet: gem-ptp-timer ptp clock registered.
[   11.743339] macb ff0e0000.ethernet eth0: Link is Up - 1Gbps/Full - flow control tx
[   11.751079] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Even I have IPv6, but impossible to obtain IPv4 (traffic?), even when connected to a DHCP server, nothing.

When I take a look to the network interfaces with ifconfig ...

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::20a:35ff:fe00:2203  prefixlen 64  scopeid 0x20<link>
        ether 00:0a:35:00:22:03  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 77  bytes 22051 (21.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 45  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And the status of the eth0 with ethtools ...

root@sgmii:~# ethtool eth0
Settings for eth0:
    Supported ports: [ TP     MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Half 1000baseT/Full
    Supported pause frame use: Transmit-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Half 1000baseT/Full
    Advertised pause frame use: Transmit-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Auto-negotiation: on
    master-slave cfg: preferred slave
    master-slave status: slave
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: external
    MDI-X: Unknown
    Supports Wake-on: a
    Wake-on: d
    Link detected: yes

root@sgmii:~# ethtool -S eth0
NIC statistics:
     tx_octets: 50685
     tx_frames: 198
     tx_broadcast_frames: 175
     tx_multicast_frames: 23
     tx_pause_frames: 0
     tx_64_byte_frames: 38
     tx_65_127_byte_frames: 21
     tx_128_255_byte_frames: 0
     tx_256_511_byte_frames: 139
     tx_512_1023_byte_frames: 0
     tx_1024_1518_byte_frames: 0
     tx_greater_than_1518_byte_frames: 0
     tx_underrun: 0
     tx_single_collision_frames: 0
     tx_multiple_collision_frames: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     tx_deferred_frames: 0
     tx_carrier_sense_errors: 0
     rx_octets: 0
     rx_frames: 0
     rx_broadcast_frames: 0
     rx_multicast_frames: 0
     rx_pause_frames: 0
     rx_64_byte_frames: 0
     rx_65_127_byte_frames: 0
     rx_128_255_byte_frames: 0
     rx_256_511_byte_frames: 0
     rx_512_1023_byte_frames: 0
     rx_1024_1518_byte_frames: 0
     rx_greater_than_1518_byte_frames: 0
     rx_undersized_frames: 0
     rx_oversize_frames: 0
     rx_jabbers: 0
     rx_frame_check_sequence_errors: 0
     rx_length_field_frame_errors: 0
     rx_symbol_errors: 0
     rx_alignment_errors: 0
     rx_resource_errors: 0
     rx_overruns: 0
     rx_ip_header_checksum_errors: 0
     rx_tcp_checksum_errors: 0
     rx_udp_checksum_errors: 0
     q0_rx_packets: 0
     q0_rx_bytes: 0
     q0_rx_dropped: 0
     q0_tx_packets: 154
     q0_tx_bytes: 47664
     q0_tx_dropped: 0
     q1_rx_packets: 0
     q1_rx_bytes: 0
     q1_rx_dropped: 0
     q1_tx_packets: 44
     q1_tx_bytes: 2905
     q1_tx_dropped: 0

The GEM_CLK_CTRL is a bit different after loading the Petalinux, ...

GEM_CLK_CTRL (IOU_SLCR) 0XFF180308 : 0x00020180

It seems to be not associated, the PHY GT with the GEM3.

I would be very gratefull if you could help me with this issue.

  • 0
    TI__Mastermind 19621 points

    Hi Alejandro!

    Thank you for submitting your query, I'll gladly assist in resolving this issue. You have provided a wealth of information, and I want to clarify a few items to make sure I am understanding the problem correctly.

    You are connecting your FPGA board to our 867-S-EVM and hoping to evaluate the SGMII connection. In both register logs, U-boot & PentaLinux, I can see that Reg 0x37 = 0, meaning that SGMII auto-negotiation never completed. You are however, receiving link status (Register 0x1 = 0x796D confirms link and Reg 0x11 = AF02 confirms 1000Mbps link speed). I want to confirm that SGMII link and MDI link (connector side) are independent, in your case we are linked up over the ethernet cable, but the SGMII link between our PHY and your FPGA is not established. This is the root cause of your communication issues.

    Before focusing on the SGMII connection, you mentioned that the strap values were changing after reset, and correctly identified the problem. Is this reset-strapping issue resolved? This needs to be stable before debugging further. 

    Regards,

    Alvaro

  • 0 in reply to Alvaro (Al-vuh-roe) Reyes
    Prodigy 20 points

    Hola Alvaro! Wink

    Thanks a lot for your quick response and your guidance. Sorry for delay, but I am currently living in Germany (I come from Spain) and there is a lag time.

    I have a Raspberry Pi 4 as DHCP server to see when the DHCP Client is properly working.

    Here you have a Photo with the Testbench I am working with:

    Testbench: KRC4700 and DP83867ERGZ-S-EVM SGMII

    I removed the R72 and sourced the VDDIO_EXT with 3v3 to make it compatible with my B87 Bank of the RFSoC

    R72 Remove and VDDIO_EXT=3v3

    The PSGTR on the KRC4700 (RFSoC Evaluation Board) are all in the M.2 Connector, so I used an adapter M2 -> PCIe x4,

    M.2 to PCie x4 Adapter

    There are 4x GT Lanes on the PCIe x4, but I can get the PS_MGTREFCLK3 only from the Si5332C clock generator in the carrier board. So I decided to use a self-made cable that get the GT Lane 3 from the PCI3 to 4x SMA connectors, and be able to connect to the PHY evaluation board,

    PCIe to 4x SMA (Tx_p, Tx_n, Rx_p and Rx_n)

    By this way, and configuraing the Si5332C to get the 125MHz clock for the PS_MGTREFCLK3, I configured the device tree with thephy parameter in the gem3

      phys = <&psgtr 3 PHY_TYPE_SGMII 3 1>;

    The first 3 is related to GT Lane 3 to use in the gem3 with SGMII.

    The second 3 is related to the CONTROLLER_INSTANCE. Teoreticall, I could use any number between 0 and 3, but only seems to "work" with value 3. With 0, 1 or 2 it is not possible to probe the eth0, no eth0 appears in the list of ethernet interfaces (ifconfig) and get an error getting the PHY (-22).

    [ 2.130032] macb ff0e0000.ethernet: Not enabling partial store and forward
    [ 2.136944] macb ff0e0000.ethernet: error -EINVAL: failed to get SGMII PHY
    [ 2.143982] macb: probe of ff0e0000.ethernet failed with error -22

    The number 1 makes reference to the clock 1 (ref1 clock defined in the &psgtr in the device-tree), which is a fixed clock of 125MHz.

    In reference to the reset, yes, I found, after some tests, that one value around 2ms (reset-assert-us = <2000>;) is stable, so I obtain the parameters as stated in the strap with PHY address 0x00. I tested the MDIO bus to make some measurements and here you have some screenshots:

     NOTE:     D7 MDC     D6 RESET     D5 MDIO 

    U-BOOT Reset

    This is the first reset in the U-BOOT. The timing is 2ms, as specified in the device tree, and the clock MDC comes 3ms later.

    KERNEL Reset

    When the kernel loads, makes another reset, but now twice as specified in the device-tree (4ms) and the clock MDC starts about 8.25ms later, also much bigger time from reset deassertion to start of MDC.

    Messages after reset (MDC, MDIO)

    MDC timing

    There are 2 messages in the MDIO bus, I did not analize, but timing seems to be ok. The period of MDC is 480ns, that is a frequency of 2.08MHz.

    32 Clock Cycles MDC to MDIO

    Reset Timing

    There are 32 MDC clock cycles to wait for MDIO stabilization, as specified in the Figure 7-2 of the DP83867E datasheet (page 14)

    Regarding the registers, I confirmed your information from the register mapping in the datasheet, as you said, but I also observed a couple of things:

    The register 0x0032 shows that RGMII mode is enable, but also is enabled the SGMII mode, is this normal? Teoretically, the mode is SGMI if both modes are active as stated in the Table 8-1 of this datasheet,

    This happens after second reset, but after first one in u-boot, this register 0x0032 is 0x0000 (no RGMII).

    In both cases, in U-BOOt and in the kernel, the strap configuration register 0x006E takes value 0x8820, and it enables RGMII mode,

    RGMII and SGMII enabled

    Maybe this has any effect in the association of the GT Lane 3 with the GEM3? But in U-BOOT is the register 0x0032 null and even so no ping.

    Well, to answer your question, yes with 2ms it is the bootstrap properly configured and seems to be stable. Is there any rule to calculate or estimate this reset assertion and deassertion time? I got just testing, but ... too much time )

    Best Regards

  • 0 in reply to Alejandro Martin
    TI__Mastermind 19621 points

    Buenas Alejandro!

    Glad to hear back from you. Regarding Reg 0x10 & Reg 0x32, you are correct, the data sheet mentions that SGMII has higher priority, as long as SGMII is enabled we are in the correct mode. Side comment, I believe you are using an older version of our data sheet, please find the most up to date one online.

    Thank you for showing me the set up! Focusing on the SMA to PCIE adaptor you made, would it be possible to use a scope to confirm the SGMII output from the FPGA? The SGMII output from the PHY can also be confirmed by plugging the EVM into a scope. Below is a scope shot of the SGMII signals from the DP83869EVM, a similar part to the DP83867.

    Channel 1: SO_P

    Channel 2: SO_N

    Channel M1: CH1 - CH 2

    Regards,

    Alvaro