This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Linux/66AK2G12: U-Boot: Ethernet works 100Mb, does not work 1Gb

Part Number: 66AK2G12

Tool/software: Linux

Hello,

I have a custom design using K2G running Linux.  We have a gigabit phy (DP83867) connected to the K2G using RGMII.  When the PHY autodetects the link as gigabit, no data seems to get transferred.  If I connect the PHY to a 10/100 port, the link gets established as 100Mb and everything works normally.  For the majority of my testing, I have been using the same switch, but with one port configured as 10/100 only and one as 10/100/1000.  I have tried the system with a 10/100 only switch and everything works properly.  For the purposes of this post, when I refer to a 1Gb port and a 10/100 port, I am talking about ports on the same switch that are configured for different speeds.

Here's a summary of what I'm running:

Processor SDK K2G Linux 4.2.0.9
U-Boot 2017.01
Linux 4.9.59
66AK2G12 ARM + DSP
DP83867 Gigabit PHY
Netgear GS108T Managed Gigabit Switch

Here is what I have added to the U-Boot DTS (keystone-k2g-evm.dts):

&mdio {
	status = "okay";

	ethphy0: ethernet-phy@0 {
		reg = <0>;
		ti,rx-internal-delay = <DP83867_RGMIIDCTL_2_25_NS>;
		ti,tx-internal-delay = <DP83867_RGMIIDCTL_2_75_NS>;
		ti,fifo-depth = <DP83867_PHYCR_FIFO_DEPTH_8_B_NIB>;
		ti,min-output-impedance;
		//ti,dp83867-rxctrl-strap-quirk;
	};
};

&gbe0 {
	phy-handle = <&ethphy0>;
};

And the Linux DTS (keystone-k2g-evm.dts):

&mdio {
	status = "okay";
	ethphy0: ethernet-phy@0 {
		reg = <0>;
		ti,rx-internal-delay = <DP83867_RGMIIDCTL_2_25_NS>;
		ti,tx-internal-delay = <DP83867_RGMIIDCTL_2_75_NS>;
		ti,fifo-depth = <DP83867_PHYCR_FIFO_DEPTH_4_B_NIB>;
		//phy-mode = "rgmii-id";
		ti,min-output-impedance;
		/* ti,dp83867-rxctrl-strap-quirk; */
		//compatible="ethernet-phy-ieee802.3-c22";
	};
};
&gbe0 {
	phy-handle = <&ethphy0>;
};

It is difficult to do much diagnostic work from the Linux side of things because I am running everything through NFS.  If I break the network connection, the system goes unresponsive until I reconnect to a 100Mb port.  It MUST be NFS because I do not have a viable local filesystem for this system becuase the NAND driver for this part is unusable (an unfortunate fact we learned after ordering prototypes).  While connected to a 1Gb port, the console shows that NFS is attempting to reconnect to the server.  I tried a wireshark capture and saw no traffic that seemed to be from my unit.  This seems to indicate that the ethernet traffic is not making it out of the box when in 1Gb mode. 

On the U-Boot side of things, I can do register dumps through "mii dump".  Another post on a similar topic asked for mii dump 0 0 and mii dump 0 1, so those are below:

mii dump 0 0                                                                 
0.     (1140)                 -- PHY control register --                        
  (8000:0000) 0.15    =     0    reset                                          
  (4000:0000) 0.14    =     0    loopback                                       
  (2040:0040) 0. 6,13 =   b10    speed selection = 1000 Mbps                    
  (1000:1000) 0.12    =     1    A/N enable                                     
  (0800:0000) 0.11    =     0    power-down                                     
  (0400:0000) 0.10    =     0    isolate                                        
  (0200:0000) 0. 9    =     0    restart A/N                                    
  (0100:0100) 0. 8    =     1    duplex = full                                  
  (0080:0000) 0. 7    =     0    collision test enable                          
  (003f:0000) 0. 5- 0 =     0    (reserved)                                     
                                                                                
                                                                                
=> mii dump 0 1                                                                 
1.     (796d)                 -- PHY status register --                         
  (8000:0000) 1.15    =     0    100BASE-T4 able                                
  (4000:4000) 1.14    =     1    100BASE-X  full duplex able                    
  (2000:2000) 1.13    =     1    100BASE-X  half duplex able                    
  (1000:1000) 1.12    =     1    10 Mbps    full duplex able                    
  (0800:0800) 1.11    =     1    10 Mbps    half duplex able                    
  (0400:0000) 1.10    =     0    100BASE-T2 full duplex able                    
  (0200:0000) 1. 9    =     0    100BASE-T2 half duplex able                    
  (0100:0100) 1. 8    =     1    extended status                                
  (0080:0000) 1. 7    =     0    (reserved)                                     
  (0040:0040) 1. 6    =     1    MF preamble suppression                        
  (0020:0020) 1. 5    =     1    A/N complete                                   
  (0010:0000) 1. 4    =     0    remote fault                                   
  (0008:0008) 1. 3    =     1    A/N able                                       
  (0004:0004) 1. 2    =     1    link status                                    
  (0002:0000) 1. 1    =     0    jabber detect                                  
  (0001:0001) 1. 0    =     1    extended capabilities 

and when connected to a 10/100 port on the switch, I get the same output.  It's worth noting that autonegotiation completes in both cases and the device (and drivers) indicate that the link is up.  That said, when I try to do DHCP from U-Boot when connected to a 1Gb port, the request just times out repeatedly.

On the hardware side of things, a data rate change can be seen on the RMGII lines when switching from 10/100 to 1Gb, but it is difficult to judge the signal integrity with our 300MHz oscilloscopes.  The 10/100 RGMII waveforms look pretty clean, and we tried to stick to the TI recommended PCB layout for the PHY design.  When running at 100Mb, the RGMII clocks are 25MHz, and at 1Gb the clocks are 125MHz.

In summary, on the Linux side, the driver reports that the link is up when connected to a 1Gb port, but the NFS driver cannot find the server.  I do not see any clear network traffic from the unit on a wireshark capture.  In U-Boot, the MII registers seem to report that the link is up, but DHCP does not work.  I have confirmed that the PHY is operating at a different speed in both cases.

Thank you for any help you can offer.  I'm happy to provide any information that may help solve the problem.

Jeff

  • Hi Jeff,

    Do you have the following in u-boot/board/ti/ks2/baord_k2g.c:
    .sgmii_link_type = SGMII_LINK_MAC_PHY,
    .phy_if = PHY_INTERFACE_MODE_RGMII,
    and in u-boot/board/ti/ks2/board.c you should have:
    if (cpu_is_k2g())
    writel(KS2_ETHERNET_RGMII, KS2_ETHERNET_CFG);
    Have you done any modifications here?

    Best Regards,
    Yordan
  • Yordan,

    I have all of the source you mentioned. I had at one time added code to board_eth_init to enable RMII_CLKOUT, but that has since been removed.

    Thank you,

    Jeff
  • Hi Jeff,

    Can you stop at u-boot and share the value of BOOTCFG_ETHERNET_CFG register, located at address 0x02620E20?
    Maybe u-boot does not set the CPSW correctly.

    Best Regards,
    Yordan

  • BOOTCFG_ETHERNET_CFG = 0x00000002, which seems like what I want it to be.
  • Yes, the setting is correct.

    I've notified the design team to help here.. I am out of ideas. The only thing that comes to my mind is check the Router/Switch, but I'm sure you've done that :).

    Best Regards,
    Yordan
  • Indeed!  A different switch is how I found out it works at 10/100.  I'm worried it's a hardware issue, though I'm not sure what I would be.  Hopefully it's just software...

    Thanks for your help,

    Jeff

  • HI, Jeff,

    For debugging, you can use ramfs boot which will download the filesystem to the RAM to get around the NFS issue. It can be done by setting boot=ramfs. You may need to prepare the cpio.gz file as shown in default name_fs which is arago-console-image-[platform].cpio.gz. I can't remember if tisdk-server-rootfs is too big to use. In any case, you need to extract the file system from tar.xz which you have been using for NFS and issue the following commands from the root directory of the filesystem to create the .cpio.gz file

    find . | sudo cpio -H newc -o -O ../myfile.cpio
    cd ..
    gzip myfile.cpio

    Have myfile.cpio.gz copied to tftp server and in u-boot

    setenv boot ramfs
    setenv name_fs myfile.cpio.gz
    saveenv

    There was a similar issue before on e2e which DP83867 only got 100Mb, not 1Gb rate. But that was on different KS2 platform.
    In that case, it was DP83867SC which supports and was configured to use SGMII. I am not sure if it applies to you.
    What we found was that in the new DP83867 datasheet (revised Feb 2017) had a change in strap resistor setting
    RX_CTRL:Modes 1 & 2 should not be used and mode 3 & 4 should be used only. So, after changing to mode 3 and 1GbE works fine.

    However, in that earlier case, when the strap mode register showed the autoneg was enabled in mode 1 but SGMII autoneg failed.
    After configuring it as mode 3, SGMII autoneg worked. I am not sure if RGMII is the same case. Please also check that you have
    correct pin straps on the device.

    Rex
  • Rex,

    Thanks for the tip on the ramfs, I had the same idea.

    As far as the strapping goes, my strap status register reads: 0x1030, showing auto-negotiation enabled. We have the datasheet recommended strapping resistors located on pin 38.
    Interestingly, it's showing RGMII disabled. I'm using the RGZ package, which only supports RGMII. The full part number is DP83867CRRGZ.

    It's also worth mentioning that the phy indicates the link is up and that autoneg completes in all cases. In order to make it work, I have to use a 100Mb port so that the autoneg results in 100Mb operation.

    Jeff
  • Hi Jeff,

    Just to be sure, can you share the relevant ethernet parts of the schematics, so that we can check it?

    Best Regards,
    Yordan
  • Yordan,

    Here's the schematic. Note that there is some rework.

    C154 has been removed.
    C67, X6, and R63 have been installed.

    /cfs-file/__key/communityserver-discussions-components-files/791/8311.dsp_2D00_gige.pdf

    Some relevant part numbers:

    X6: ABLS-25.000MHZ-B4-F-T
    J25: XFGIB100JM-CLGY1-4MS

    Thank you for taking a look at this.  I can also provide details relating to the copper.

    Jeff

  • Hi, Jeff,

    Is the strap status register you mentioned the STRAP_STS1 register at 0x006E? and the RGMII showing disabled indicated by bit 12? What is the RGMII_EN bit set to in RGMIICTL register at 0x0032?

    In this thread, e2e.ti.com/.../2070564, the other user was able to configure and run it at 1Gb with tweaks of modifying compatible and changing RX_CTRL to mode 3. I am not sure if the dts nodes make difference, but may be worth comparing against it.

    I'll be out for the rest of the week. My response will be slow.

    Rex
  • Rex,

    That is correct. The strap status register I mentioned is STRAP_STS1 at 0x6E, and the bit is bit 12. Interestingly, RGMII_EN is set to 0. In fact, that whole register == 0x00. So, using the U-Boot command line mdio tool, I set the register manually to it's defaults, and was able to get a DHCP lease! I can then use the interface until Linux takes control and (I assume) un-sets that bit. Once Linux takes control, the interface stops working. So it seems that I just need to ensure that it gets set.

    Enjoy your time off!

    Jeff
  • Hi Jeff,

    Sorry for the delay on the hw schematics. I was OoO a couple of days last week.

    I checked your design and it is ok. The difference between your design and K2G ICE (which also uses DP83867 phy) is that you don't use clock generator to produce the GIGETH_25MHzCLK clock. You also don't have a filter (LC) on the 3.3V power supply. If you can verify that you have stable +3.3V and no jitter on the 25MHz clock connected to the PHY XI, then your schematic is correct.

    Best Regards,
    Yordan
  • Yordan,

    Thanks for checking that. I'll take a look at our supplies/clock. I'm pretty confident that they are good, but it never hurts to check.

    Good to know that the ICE uses the same PHY. Would you suggest looking at the ICE device tree/U-Boot/Linux code to see how I should set it up? I can get it to work if I manually set the RGMIICTL register in the PHY using MDIO in the U-Boot command line, but Linux seems to be resetting the value, so it won't finish booting.

    Jeff
  • I take that back. Manually changing the register does not seem to have an effect anymore. I'm not sure why it seemed to work before...

    I've been digging through the source and cannot seem to find any obvious differences between the ICE setup and the setup I am using.

    Jeff
  • It appears that the problem was ultimately a timing issue. I worked through the Timing Budget sheet for RGMII and figured out that I needed to disable the internal delay on the processor and set the internal delay on the PHY. I modified the device tree to include my calculated delay values, then modified board.c in U-Boot to set BOOTGFC_ETHERNET_CFG = 0x12 (disable internal delay, set mode to rgmii).

    Further testing will be necessary to consider this the actual fix, and the post will be updated once the fix is verified.
  • Thanks for sharing your solution.

    Best Regards,
    Yordan