DP83867CS: Help with using three PHY components connected to the same I2C MDIO lines

Hi, 

Does the MDIO access come back if the PHY is reseted and is the PHY still detectable on Linux after this error happens?
Also, how often does this happen?
Could you give us more detail about the PHY's strap configuration? 

Best,
J

Reset solves the problem,
The problem recurs frequently (once a day).

Hi Shira, 
Which PHY is dying? Also, is the problematic PHY detectable on Linux after the error statement?
It seems like all the PHYs are in the same strap setting with only difference being the PHY address. 

Is there any additional log on the Linux side? Could you also share the schematic so we can better take a look at this problem? 
Currently, it seems like it could be power dip on one of the voltage rails causing the PHY to get stuck in a bad state. 

When the PHY gets into the bad state, could you probe RBIAS pin and see if it is outputting 1V and could you check if there is clock going into XI pin when the PHY is in the bad state?

Best,
J

Hi,

Which phy dying is not permanent, when the phy is dying linux can't communicate with the phy but with the other phys it does. 

linux log -

dmesg | grep eth0-
[ 2.791714] macb ff0b0000.ethernet eth0: Cadence GEM rev 0x50070106 at 0xff0b0000 irq 60 (00:0a:35:16:c7:be)
[ 10.100274] macb ff0b0000.ethernet eth0: PHY [ff0b0000.ethernet-ffffffff:0c] driver [TI DP83867] (irq=POLL)
[ 10.100311] macb ff0b0000.ethernet eth0: configuring for phy/sgmii link mode
[ 14.182864] macb ff0b0000.ethernet eth0: unable to generate target frequency: 125000000 Hz
[ 14.184424] macb ff0b0000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[ 14.184466] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

dmesg | grep eth1-
[ 2.818679] macb ff0d0000.ethernet eth1: Cadence GEM rev 0x50070106 at 0xff0d0000 irq 61 (00:c1:be:33:00:17)
[ 10.016515] macb ff0d0000.ethernet eth1: PHY [ff0b0000.ethernet-ffffffff:08] driver [TI DP83867] (irq=POLL)
[ 10.016546] macb ff0d0000.ethernet eth1: configuring for phy/sgmii link mode
[ 14.118845] macb ff0d0000.ethernet eth1: unable to generate target frequency: 125000000 Hz
[ 14.120428] macb ff0d0000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
[ 14.120471] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

dmesg | grep eth2-
[ 2.852286] macb ff0e0000.ethernet eth2: Cadence GEM rev 0x50070106 at 0xff0e0000 irq 62 (9e:3e:21:13:ea:5c)
[ 9.992175] macb ff0e0000.ethernet eth2: Could not attach PHY (-19)

schematic -

all power rails are common to all phys so if there is problem in power source it will affect all the phys.
I checked when the phy dying RBIAS= 1V, and there is clk in XI pin.

Hi Shira,

To confirm, any of the three PHYs may die while they are in its operation?
Also, could you send a better quality picture or file of the schematic? I cannot read the letters with the current quality. 

I have some hypothesis as to why this could be happening:
1. MDIO bus load is too high with three devices that setup/hold time specs are not met causing the PHY to be dropped from the Linux. For this, I would suggest switching MDIO pull up resistor to a lower resistance and see if this is causing it. You can possibly check this by using MSP430 Launchpad to communicate to the problematic PHY when the PHY is dropped from Linux and see if the PHY responds to the MSP430's MDIO read/writes. Please make sure that the Linux is not driving the MDIO bus when the MSP430 Launchpad is driving the bus. 
2. Possible power dip. As you said they all share the same power rail, power dip can cause the PHY to be stuck in a bad state and not recover. This would be possible for any of the three PHYs to go into a bad state. Could you possibly try out putting the board on the oscilloscope and see if there ever is a power dip?

Please let me know. 

Best,
J

Hi,
We added a pullup resistor to the mdio line
I'll update if that helps,thanks!

Hi Shira,

Please keep me updated. Please note that MDIO line requires 1.5k to 2.2k pullup resistor by MDIO bus protocol.

Best,
J

Hi,
Adding pullup resistors didn't help...
Do you have any other ideas for a solution?
link for schematic-
There are 3 pictures of each ethernet connection with a different address
https://imgur.com/a/XHzolxd

Thanks

Hi Shira, 

Have you tried driving the problematic PHY with different MDIO driver like MSP430 Launchpad?

J said:

You can possibly check this by using MSP430 Launchpad to communicate to the problematic PHY when the PHY is dropped from Linux and see if the PHY responds to the MSP430's MDIO read/writes. Please make sure that the Linux is not driving the MDIO bus when the MSP430 Launchpad is driving the bus. 

Also, have you checked if there are any power dips that can cause this?

Best,
J

Hi,
I can't connect MSP430, because we used custom board and mdio lines connect to zynqMP (FPGA Xilinx) so i cant disconnect the lines and connect to another driver.
We checked and there are not power dips.

Hi Shira,

US office is closed for holiday 9/1. I will get back to you on 9/2 US time.

Best,

J

Hi Shira,
Could you use phytool to read/write to the problematic PHY?

I discussed this internally and it is very unfortunate that you cannot access MDIO bus. If you could make this happen somehow that would be very helpful in debugging. 
If this is a loading issue on the bus, using 1.5kohm resistor on MDIO may solve the issue. You can also try to add 1.5kohm resistor on the MDC to ensure that there is a proper pull up. Could you also try that?

Best,
J