This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM625: DP83867 Ethernet PHY Unable to Recover Post Deepsleep Wake-up

Part Number: AM625
Other Parts Discussed in Thread: SK-AM62B-P1,

Hi TI Experts,

We have AM62 base custom HW modules which uses on module DP83867 Ethernet PHY. This PHY uses 25MHz EXT_REFCLK1 from SOC.

I found that EXT_REFCLK1 25MHz clock goes off in deepsleep mode and comes back with a wrong frequency of 50 MHz. (Already have this thread in e2e for the same). However with the below workaround in ti-sci firmware driver, I could recover the 25 MHz clock back after waking up from deepsleep.

diff --git a/drivers/firmware/ti_sci.c b/drivers/firmware/ti_sci.c
index a58d25724365..9b211ce386d7 100644
--- a/drivers/firmware/ti_sci.c
+++ b/drivers/firmware/ti_sci.c
@@ -3619,6 +3620,20 @@ static int ti_sci_resume(struct device *dev)
                return ret;
        dev_dbg(dev, "%s: disable isolation: %d\n", __func__, ret);
 
+       /* ti_sci_cmd_clk_set_freq(const struct ti_sci_handle *handle,
+                                   u32 dev_id, u32 clk_id, u64 min_freq,
+                                   u64 target_freq, u64 max_freq); */
+
+       ret = ti_sci_cmd_clk_set_freq(&info->handle, 157, 20, 0, 25000000, 50000000);
+
+       if (ret) {
+               dev_info(dev, "%s: setting EXT_REFCLK1 at 25MHz failed with error : %d\n", __func__, ret);
+               return ret;
+       }
+
+       dev_info(dev, "%s: setting EXT_REFCLK1 at 25MHz success : %d\n", __func__, ret);

        ti_sci_msg_cmd_lpm_wake_reason(&info->handle, &source, &time);
        dev_info(dev, "%s: wakeup source: 0x%X\n", __func__, source);

By doing above workaround, I am still unable to make Ethernet interface (DP83867 Ethernet PHY) working. Eth0 link status is not resuming back online. Unable to get IP for eth0 interface after wake up from deepsleep.

Please see below status logs of eth0 upon resuming from deepsleep.

root@verdin-am62-15133510:~# ifconfig
eth0: flags=-28605<UP,BROADCAST,RUNNING,MULTICAST,DYNAMIC>  mtu 1500
        inet6 fe80::214:2dff:fee6:eb46  prefixlen 64  scopeid 0x20<link>
        ether 00:14:2d:e6:eb:46  txqueuelen 1000  (Ethernet)
        RX packets 77  bytes 6838 (6.6 KiB)
        RX errors 0  dropped 43  overruns 0  frame 0
        TX packets 115  bytes 20379 (19.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 213  bytes 15432 (15.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 213  bytes 15432 (15.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.11.1  netmask 255.255.255.0  broadcast 192.168.11.255
        inet6 fe80::a413:aff:feda:7f7d  prefixlen 64  scopeid 0x20<link>
        ether a6:13:0a:da:7f:7d  txqueuelen 1000  (Ethernet)
        RX packets 112  bytes 16551 (16.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 44  bytes 7996 (7.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Settings for eth0:
        Supported ports: [ TP    MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 10Mb/s
        Duplex: Half
        Auto-negotiation: on
        master-slave cfg: preferred slave
        master-slave status: unknown
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        MDI-X: Unknown
        Supports Wake-on: ubgs
        Wake-on: ubgs
        SecureOn password: ff:ff:ff:ff:ff:ff
        Current message level: 0x000020f7 (8439)
                               drv probe link ifdown ifup rx_err tx_err hw
        Link detected: no

root@verdin-am62-15133510:~# ip addr 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,DYNAMIC,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:14:2d:e6:eb:46 brd ff:ff:ff:ff:ff:ff
3: can0: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
    link/can 
4: can1: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
    link/can 
5: mlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6c:1d:eb:9d:38:4d brd ff:ff:ff:ff:ff:ff
6: uap0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 6c:1d:eb:9d:3a:4d brd ff:ff:ff:ff:ff:ff
7: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether a6:13:0a:da:7f:7d brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.1/24 brd 192.168.11.255 scope global usb0
       valid_lft forever preferred_lft forever
    inet6 fe80::a413:aff:feda:7f7d/64 scope link 
       valid_lft forever preferred_lft forever

Questions :

  1. Is DP83867 Ethernet PHY supports XI clock breakage (going off and getting on again) during it's operation in current SW ? (As per the datasheet https://www.ti.com/lit/gpn/dp83867ir, PHY supports deepsleep mode in which XI can be turned off, however present dp83867 Linux driver is not supporting this feature as far as I can understand)
  2. If answer to Q#1 is no, what could be the solution for this issue ?
  3. How to make eth0 (DP83867) functionally working after deepsleep wakeup while using an internal EXT_REFCLK1 25MHz clock ?

SW details:

Looking forward for support.

Thank you.

Regards,

Parth P

  • Hi Parth,

    I am not a network expert, and unable to comment on why the PHY or network doesn't not work as expected.

    But let's first check if the ti_sci_cmd_clk_set_freq() hack is correct or not. (I actually think you should use ti_sci_cmd_clk_set_parent() instead, since the bug is not keeping the parent 22 and switched back to 21).

    Can you please use k3conf to dump clock for device 157 and check clock ID 20, 21, 22 to see all the values are the same before and after suspend/resume, with your patch?

    Also use command 'devmem2 0x108010 w' before and after suspend/resume to check its value.

  • Hi Bin,

    Thank you for response.

    I tried with ti_sci_cmd_clk_set_parent() API instead of ti_sci_cmd_clk_set_freq() where I could see no difference in k3conf to dump clock for device 157, clock IDs 20, 21, 22. Please check below snippet.

    Changes in ti-sci firmware driver:

    diff --git a/drivers/firmware/ti_sci.c b/drivers/firmware/ti_sci.c
    index a58d25724365..c0dc40f07b01 100644
    --- a/drivers/firmware/ti_sci.c
    +++ b/drivers/firmware/ti_sci.c
    @@ -11,6 +11,7 @@
     #include <linux/bitmap.h>
     #include <linux/debugfs.h>
     #include <linux/dma-mapping.h>
    +//#include <linux/delay.h>
     #include <linux/export.h>
     #include <linux/firmware.h>
     #include <linux/io.h>
    @@ -3619,6 +3620,28 @@ static int ti_sci_resume(struct device *dev)
                    return ret;
            dev_dbg(dev, "%s: disable isolation: %d\n", __func__, ret);
     
    +#if 0
    +       /* ti_sci_cmd_clk_set_freq(const struct ti_sci_handle *handle,
    +                                   u32 dev_id, u32 clk_id, u64 min_freq,
    +                                   u64 target_freq, u64 max_freq); */
    +
    +       ret = ti_sci_cmd_clk_set_freq(&info->handle, 157, 20, 0, 25000000, 50000000);
    +#else
    +       /*static int ti_sci_cmd_clk_set_parent(const struct ti_sci_handle *handle,
    +                                     u32 dev_id, u32 clk_id, u32 parent_id)*/
    +
    +       ret = ti_sci_cmd_clk_set_parent(&info->handle, 157, 20, 22);
    +
    +#endif
    +
    +       if (ret) {
    +               dev_info(dev, "%s: setting EXT_REFCLK1 at 25MHz failed with error : %d\n", __func__, ret);
    +               return ret;
    +       }
    +
    +       dev_info(dev, "%s: setting EXT_REFCLK1 at 25MHz success : %d\n", __func__, ret);
    +
            ti_sci_msg_cmd_lpm_wake_reason(&info->handle, &source, &time);
            dev_info(dev, "%s: wakeup source: 0x%X\n", __func__, source);

    'devmem2 devmem2 0x108010 w' with above change for before and after deepsleep shows the same value as below.

    root@verdin-am62-15133431:~# devmem2 0x108010 w
    /dev/mem opened.
    Memory mapped at address 0xffffadfa5000.
    Read at address  0x00108010 (0xffffadfa5010): 0x00000001

    With this change, Ethernet PHY/interface is still not recovering and the same issue persist as mentioned in original post.

    It would be a really great if you could please help involve Ethernet/network experts to have a look into this issue.

    Thank you.

    Regards,

    Parth P

  • Hi Parth,

    Could you run the following after waking up from deepsleep? These messages would help us see if there were any indication that phy was detected and if the MAC could communicate with the phy via mdio. 

    dmesg | grep phy
    dmesg | grep mdio
    dmesg | grep ethernet

    However, typically a link should be established via auto-negotiation without needing MAC to phy communication established. Another check is to see if the green LED is on, on the RJ45 connector when the ethernet cable is connected. If the green LED is off, you may not have a firm connection and/or you might want to switch the ethernet cable with a known working one.

    You could also try verifying that the clock frequency of the RGMII interface is the 25 MHz that you expect with an O-scope/logic analyzer.

    Could you also share your dts file?

    Best regards,

    Daolin

  • Hi Daolin,

    Thank you for quick response. Please find below logs as requested.

    Before Sleep Entry : 
    -------------------------------------------------------------------------
    root@verdin-am62-15133431:~# dmesg | grep phy
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
    [    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
    [    1.678299] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
    [    1.686514] davinci_mdio 8000f00.mdio: phy[7]: device 8000f00.mdio:07, driver Microchip KSZ9131 Gigabit PHY
    [   13.853902] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    
    root@verdin-am62-15133431:~# dmesg | grep mdio
    [    1.293355] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.336310] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.582931] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.628327] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.678299] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
    [    1.686514] davinci_mdio 8000f00.mdio: phy[7]: device 8000f00.mdio:07, driver Microchip KSZ9131 Gigabit PHY
    [   13.810958] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    
    root@verdin-am62-15133431:~# dmesg | grep ethernet
    [    1.345196] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006
    [    1.358086] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
    [    1.365229] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
    [    1.376164] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:1
    [    1.386867] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [   12.524848] using random self ethernet address
    [   12.529660] using random host ethernet address
    [   13.810958] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [   13.853902] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [   17.434473] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
    
    root@verdin-am62-15133431:~# rtcwake -s 10 -m mem -d rtc1
    rtcwake: assuming RTC uses UTC ...
    rtcwake: wakeup from "mem" using rtc1 at Thu Jan  1 00:01:49 1970
    [   92.181202] PM: suspend entry (deep)
    [   92.185841] Filesystems sync: 0.000 seconds
    [   92.193881] Freezing user space processes
    [   92.200945] Freezing user space processes completed (elapsed 0.002 seconds)
    [   92.208066] OOM killer disabled.
    [   92.211293] Freezing remaining freezable tasks
    [   92.217237] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
    [   92.224713] printk: Suspending console(s) (use no_console_suspend to debug)
    [   92.325948] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
    [   92.347079] Disabling non-boot CPUs ...
    [   92.349251] psci: CPU1 killed (polled 0 ms)
    [   92.350348] Enabling non-boot CPUs ...
    [   92.350779] Detected VIPT I-cache on CPU1
    [   92.350984] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
    [   92.351054] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [   92.352152] CPU1 is up
    [   92.353759] ti-sci 44043000.system-controller: ti_sci_resume: setting EXT_REFCLK1 at 25MHz success : 0
    [   92.353928] ti-sci 44043000.system-controller: ti_sci_resume: wakeup source: 0x50
    [   92.367488] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [   92.412084] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [   92.412757] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [   92.439642] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (1) failed: -110
    [   92.439768] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [   92.439843] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (2) failed: -110
    [   92.439914] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [   92.439985] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (3) failed: -110
    [   92.440054] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [   92.440062] mwifiex_sdio mmc2:0001:1: DNLD_CMD: host to card failed
    [   92.830201] OOM killer enabled.
    [   92.833356] Restarting tasks ... done.
    [   92.887296] random: crng reseeded on system resumption
    [   92.893706] PM: suspend exit
    
    After wakeup from deepsleep : 
    -------------------------------------------------------------------------
    root@verdin-am62-15133431:~# dmesg | grep phy
    [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
    [    0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
    [    1.678299] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
    [    1.686514] davinci_mdio 8000f00.mdio: phy[7]: device 8000f00.mdio:07, driver Microchip KSZ9131 Gigabit PHY
    [   13.853902] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [   92.412757] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    
    root@verdin-am62-15133431:~# dmesg | grep mdio
    [    1.293355] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.336310] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.582931] davinci_mdio 8000f00.mdio: Configuring MDIO in manual mode
    [    1.628327] davinci_mdio 8000f00.mdio: davinci mdio revision 9.7, bus freq 1000000
    [    1.678299] davinci_mdio 8000f00.mdio: phy[0]: device 8000f00.mdio:00, driver TI DP83867
    [    1.686514] davinci_mdio 8000f00.mdio: phy[7]: device 8000f00.mdio:07, driver Microchip KSZ9131 Gigabit PHY
    [   13.810958] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [   92.412084] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    
    root@verdin-am62-15133431:~# dmesg | grep ethernet
    [    1.345196] am65-cpsw-nuss 8000000.ethernet: initializing am65 cpsw nuss version 0x6BA01103, cpsw version 0x6BA81103 Ports: 3 quirks:00000006
    [    1.358086] am65-cpsw-nuss 8000000.ethernet: initialized cpsw ale version 1.5
    [    1.365229] am65-cpsw-nuss 8000000.ethernet: ALE Table size 512
    [    1.376164] am65-cpsw-nuss 8000000.ethernet: CPTS ver 0x4e8a010c, freq:500000000, add_val:1 pps:1
    [    1.386867] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [   12.524848] using random self ethernet address
    [   12.529660] using random host ethernet address
    [   13.810958] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [   13.853902] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [   17.434473] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
    [   92.325948] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
    [   92.367488] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [   92.412084] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [   92.412757] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    

    Regarding LEDs I am not sure if we have those in our HW design or not, however, I could not see ethernet link coming up post wake-up in the dmesg logs.

    I really did measure a physical 25 MHz clock with an O-scope along with Ethernet PHY RESET, both the signals looks fine too in my opinion.  Please check below snippets of the same measurement for entry and exit from deepsleep.

    DTS file and SW details are already mentioned in the original post.

    SW details:

    Please let me know in case you need further information on this issue.

    Thank you for support.

    Regards,

    Parth P

  • Hi Daolin,

    Adding one observation which might be helpful for you to know. After waking up from the deepsleep, when I enter 2-3 ethtool commands of Ethernet link related to MID/MDI-X negotiations and configs (ethtool -s eth0 mdix <on/off/auto>), I see Ethernet link coming back to life. Please see full logs below.

    root@verdin-am62-15133431:~# rtcwake -s 10 -m mem -d rtc1                                                                                                                                                                                                                                                                                                                              
    rtcwake: assuming RTC uses UTC ...
    rtcwake: wakeup from "mem" using rtc1 at Thu Jan  1 00:07:32 1970
    [  435.413704] PM: suspend entry (deep)
    [  435.418652] Filesystems sync: 0.001 seconds
    [  435.427074] Freezing user space processes
    [  435.433907] Freezing user space processes completed (elapsed 0.002 seconds)
    [  435.440989] OOM killer disabled.
    [  435.444221] Freezing remaining freezable tasks
    [  435.450163] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
    [  435.457636] printk: Suspending console(s) (use no_console_suspend to debug)
    [  435.553562] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
    [  435.573508] Disabling non-boot CPUs ...
    [  435.575678] psci: CPU1 killed (polled 0 ms)
    [  435.577042] Enabling non-boot CPUs ...
    [  435.577489] Detected VIPT I-cache on CPU1
    [  435.577690] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
    [  435.577762] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [  435.578807] CPU1 is up
    [  435.579734] ti-sci 44043000.system-controller: ti_sci_resume: setting EXT_REFCLK1 at 25MHz success : 0
    [  435.579797] ti-sci 44043000.system-controller: ti_sci_resume: wakeup source: 0x50
    [  435.593378] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [  435.636846] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [  435.637534] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [  435.664169] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (1) failed: -110
    [  435.664256] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [  435.664328] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (2) failed: -110
    [  435.664437] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [  435.664511] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (3) failed: -110
    [  435.664583] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [  435.664591] mwifiex_sdio mmc2:0001:1: DNLD_CMD: host to card failed
    [  436.056599] OOM killer enabled.
    [  436.059766] Restarting tasks ... done.
    [  436.078020] random: crng reseeded on system resumption
    [  436.086390] PM: suspend exit
    root@verdin-am62-15133431:~# 
    root@verdin-am62-15133431:~# 
    root@verdin-am62-15133431:~# ethtool -s eth0 mdix on  
    root@verdin-am62-15133431:~# ethtool -s eth0 mdix off
    root@verdin-am62-15133431:~# ethtool -s eth0 mdix auto
    [  461.474490] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
    root@verdin-am62-15133431:~# [  461.483226] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    
    root@verdin-am62-15133431:~# 
    root@verdin-am62-15133431:~# 
    root@verdin-am62-15133431:~# 
    root@verdin-am62-15133431:~# ifconfig
    eth0: flags=-28605<UP,BROADCAST,RUNNING,MULTICAST,DYNAMIC>  mtu 1500
            inet 10.0.214.175  netmask 255.255.255.0  broadcast 10.0.214.255
            inet6 fe80::214:2dff:fee6:eaf7  prefixlen 64  scopeid 0x20<link>
            ether 00:14:2d:e6:ea:f7  txqueuelen 1000  (Ethernet)
            RX packets 1746  bytes 111236 (108.6 KiB)
            RX errors 0  dropped 210  overruns 0  frame 0
            TX packets 138  bytes 18388 (17.9 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10<host>
            loop  txqueuelen 1000  (Local Loopback)
            RX packets 124  bytes 10082 (9.8 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 124  bytes 10082 (9.8 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.11.1  netmask 255.255.255.0  broadcast 192.168.11.255
            inet6 fe80::f09b:64ff:feed:4dc0  prefixlen 64  scopeid 0x20<link>
            ether f2:9b:64:ed:4d:c0  txqueuelen 1000  (Ethernet)
            RX packets 117  bytes 16653 (16.2 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 51  bytes 9372 (9.1 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    

    Thanks.

    Regards,

    Parth P

  • Hi Parth,

    Thanks for sharing your findings on MDI-X, this is very helpful. Based on your findings, there doesn't seem to be a problem with the phy state in establishing a link. The issue appears to be more related to eth0 configuration in Linux. Could you try the below commands to force eth0 link up and post the resulting kernel messages?

    root@am62xx-evm:~# ip link set dev eth0 down
    [ 3234.055210] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
    root@am62xx-evm:~# ip link set dev eth0 up  
    [ 3238.057720] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=POLL)
    [ 3238.057751] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    root@am62xx-evm:~# [ 3240.098640] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
    [ 3240.098701] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    

    The result of this ip link command probably performs similar to the result of the ethtool mdix commands you ran. Based on the dmesg output you shared, it seems like the process of entering deepsleep forces eth0 link down. After wakeup, eth0 is still down and requires action to force it back up. Since you probably aren't physically reconnecting the ethernet cable, there probably is no auto-negotiation happening through the event of connecting to a link partner. I need to check on this but I think that the cpsw driver would not detect that a link is already established if eth0 was forced down in software.

    Best regards,

    Daolin

  • Hi Daolin,

    I tried these ip link commands for putting down eth0 and making it up after wake up from deep sleep and it did not help any. Only after 'ethtool -s eth0 mdix auto' command, eth0 is able to establish a link.

    Please have a look at complete logs as below.

    root@verdin-am62-15133431:~# ifconfig
    eth0: flags=-28605<UP,BROADCAST,RUNNING,MULTICAST,DYNAMIC>  mtu 1500
            inet 10.0.214.175  netmask 255.255.255.0  broadcast 10.0.214.255
            inet6 fe80::214:2dff:fee6:eaf7  prefixlen 64  scopeid 0x20<link>
            ether 00:14:2d:e6:ea:f7  txqueuelen 1000  (Ethernet)
            RX packets 43  bytes 4272 (4.1 KiB)
            RX errors 0  dropped 18  overruns 0  frame 0
            TX packets 56  bytes 7871 (7.6 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10<host>
            loop  txqueuelen 1000  (Local Loopback)
            RX packets 104  bytes 8734 (8.5 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 104  bytes 8734 (8.5 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.11.1  netmask 255.255.255.0  broadcast 192.168.11.255
            inet6 fe80::f09b:64ff:feed:4dc0  prefixlen 64  scopeid 0x20<link>
            ether f2:9b:64:ed:4d:c0  txqueuelen 1000  (Ethernet)
            RX packets 48  bytes 6602 (6.4 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 32  bytes 6064 (5.9 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    root@verdin-am62-15133431:~# ip link set dev eth0 down
    [   73.314029] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
    
    root@verdin-am62-15133431:~# ip link set dev eth0 up  
    [   80.983340] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [   80.993807] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [   81.007426] 8021q: adding VLAN 0 to HW filter on device eth0
    
    root@verdin-am62-15133431:~# [   83.799950] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
    [   83.808688] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    
    root@verdin-am62-15133431:~# rtcwake -s 10 -m mem -d rtc1                                                                                                                                                                                                                                                                                                                              
    rtcwake: assuming RTC uses UTC ...
    rtcwake: wakeup from "mem" using rtc1 at Thu Jan  1 00:01:49 1970
    [   92.672065] PM: suspend entry (deep)
    [   92.677038] Filesystems sync: 0.001 seconds
    [   92.684665] Freezing user space processes
    [   92.691653] Freezing user space processes completed (elapsed 0.002 seconds)
    [   92.698722] OOM killer disabled.
    [   92.701968] Freezing remaining freezable tasks
    [   92.707931] Freezing remaining freezable tasks completed (elapsed 0.001 seconds)
    [   92.715384] printk: Suspending console(s) (use no_console_suspend to debug)
    [   92.820561] am65-cpsw-nuss 8000000.ethernet eth0: Link is Down
    [   92.835455] Disabling non-boot CPUs ...
    [   92.836901] psci: CPU1 killed (polled 4 ms)
    [   92.838552] Enabling non-boot CPUs ...
    [   92.838989] Detected VIPT I-cache on CPU1
    [   92.839193] GICv3: CPU1: found redistributor 1 region 0:0x00000000018a0000
    [   92.839267] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
    [   92.840338] CPU1 is up
    [   92.841529] ti-sci 44043000.system-controller: ti_sci_resume: setting EXT_REFCLK1 at 25MHz success : 0
    [   92.841593] ti-sci 44043000.system-controller: ti_sci_resume: wakeup source: 0x50
    [   92.854739] am65-cpsw-nuss 8000000.ethernet: set new flow-id-base 19
    [   92.886253] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [   92.886944] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [   92.906160] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (1) failed: -110
    [   92.906251] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [   92.906323] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (2) failed: -110
    [   92.906393] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [   92.906467] mwifiex_sdio mmc2:0001:1: host_to_card, write iomem      (3) failed: -110
    [   92.906537] mwifiex_sdio mmc2:0001:1: write CFG reg failed
    [   92.906544] mwifiex_sdio mmc2:0001:1: DNLD_CMD: host to card failed
    [   93.301973] OOM killer enabled.
    [   93.305125] Restarting tasks ... done.
    [   93.356083] random: crng reseeded on system resumption
    [   93.362362] PM: suspend exit
    
    root@verdin-am62-15133431:~# ip link set dev eth0 down
    
    root@verdin-am62-15133431:~# ip link set dev eth0 up  
    [  113.537545] am65-cpsw-nuss 8000000.ethernet eth0: PHY [8000f00.mdio:00] driver [TI DP83867] (irq=359)
    [  113.547494] am65-cpsw-nuss 8000000.ethernet eth0: configuring for phy/rgmii-rxid link mode
    [  113.564803] 8021q: adding VLAN 0 to HW filter on device eth0
    root@verdin-am62-15133431:~# 
    root@verdin-am62-15133431:~# ethtool -s eth0 mdix auto                                                                                                                                                                                                                                                                                                                                 
    root@verdin-am62-15133431:~# [  134.824388] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
    [  134.827857] am65-cpsw-nuss 8000000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
    

    Regarding your below comment, I believe that this is what exactly implemented and working on SK-AM62B-P1 with latest SDK release (December 2023) so I assume it should also work for me. (Except for the internal EXT_REFCLK1 of 25MHz for DP83867 PHY, we are more or less on par with Ethernet HW design and SW part with SK boards)

    Based on the dmesg output you shared, it seems like the process of entering deepsleep forces eth0 link down. After wakeup, eth0 is still down and requires action to force it back up. Since you probably aren't physically reconnecting the ethernet cable, there probably is no auto-negotiation happening through the event of connecting to a link partner. 

    It would be really helpful if you could look into DTS and related linux configs from our public repo and give some hints to further debug this issue.

    Please find below DTS/Kernel configs.

    Thank you for your support.

    Regards,

    Parth P

  • Hi Parth,

    After consulting with a team member I may have made the wrong conclusion with this earlier statement. The fact that forcing "mdix auto" is needed to bring up the link indicates that there might be an issue with the phy not waking up properly.

    Daolin Qiu said:
    there doesn't seem to be a problem with the phy state in establishing a link.

    We will need to consult with the phy team and get back to you in the next day or so.

    Best regards,

    Daolin

  • Hi Daolin,

    I have a couple of updates/findings to share from my side on this issue.

    1. Ethernet PHY link is able to resume if I configure PHY in a polling mode(The way it is done on AM625 SK/SDK).
      1. By enabling the polling mode from DTSI, phy link resumes however the complete context of PHY/PHY registers are lost (PHY RESET and CLK are going off in deepsleep) due to which PHY could not resume at lastly done/modified configurations.
      2. We do not plan to use polling mode and would like to use PHY in interrupt mode.
      3. Full Logs: Link
    2. I tried testing Ethernet with a reworked HW where Ethernet PHY RESET is not triggered upon deepsleep entry so that PHY can retain it's configurations and context. However, with this HW change too, in an interrupt mode, PHY is unable to establish the link and resume fully.
      1. Full Logs: Link

    With the above findings, I have following questions.

    1. How to make DP83867 PHY correctly resume and functional in an interrupt mode after waking up from deeplseep while using an internal EXT_REFCLK1 25MHz clock ?
    2. Even with the PHY registers and context retention (point#2), why interrupt is not properly reconfigured upon resume ? (I tried setting eth0 down/up, using PHY mainline driver - https://github.com/torvalds/linux/blob/master/drivers/net/phy/dp83867.c which kind of has interrupt reconfiguration upon resume though, nothing really helped to make PHY working properly)

    Looking forward to your support.

    Thank you.

    Regards,

    Parth P

  • Hi Parth,

    We received some feedback from the PHY team.

    The question is if prior to power down is the PHY being told of the shutdown?

    In the datasheet the section 8.5.6.2 there is a description that says that setting bit 7 for Reg 0x10 is required before asserting the PWRDN pin.

    Best Regards,

    Schuyler 

  • Hi Schuyler,

    Thank you for your response. Yes, I tired to make changes in DP83867 driver according to the datasheet as a first thing and this did not help to make PHY resume properly. Thus, there was an ask of below questions in my original post.

    Questions :

    1. Is DP83867 Ethernet PHY supports XI clock breakage (going off and getting on again) during it's operation in current SW ? (As per the datasheet https://www.ti.com/lit/gpn/dp83867ir, PHY supports deepsleep mode in which XI can be turned off, however present dp83867 Linux driver is not supporting this feature as far as I can understand)
    2. If answer to Q#1 is no, what could be the solution for this issue ?

    Nevertheless, please review below changes which I did in DP83867 PHY driver related to your proposal. I am not a PHY expert here hence there is a high chance of myself missing out on something important here.

    $git diff drivers/net/phy/dp83867.c
    diff --git a/drivers/net/phy/dp83867.c b/drivers/net/phy/dp83867.c
    index f7436191fa80..023e1941c069 100644
    --- a/drivers/net/phy/dp83867.c
    +++ b/drivers/net/phy/dp83867.c
    @@ -151,6 +151,9 @@
     /* FLD_THR_CFG */
     #define DP83867_FLD_THR_CFG_ENERGY_LOST_THR_MASK       0x7
     
    +/* DP83867 Low Power Mode Bits */
    +#define DP83867_PHYCTRL_DS              BIT(7)
    +
     enum {
            DP83867_PORT_MIRROING_KEEP,
            DP83867_PORT_MIRROING_EN,
    @@ -960,6 +963,32 @@ static void dp83867_link_change_notify(struct phy_device *phydev)
            }
     }
     
    +static int dp83867_suspend(struct phy_device *phydev)
    +{
    +
    +       phy_clear_bits_mmd(phydev, DP83867_DEVADDR, MII_DP83867_PHYCTRL,
    +                                   DP83867_PHYCTRL_DS);
    +
    +       return genphy_suspend(phydev);
    +}
    +
    +static int dp83867_resume(struct phy_device *phydev)
    +{
    +       int ret = 0;
    +
    +       genphy_resume(phydev);
    +
    +        ret = dp83867_config_init(phydev);
    +        if (ret < 0)
    +                return ret;
    +
    +        ret = dp83867_config_intr(phydev);
    +        if (ret < 0)
    +                return ret;
    +
    +       return 0;
    +}
    +
     static struct phy_driver dp83867_driver[] = {
            {
                    .phy_id         = DP83867_PHY_ID,
    @@ -982,9 +1011,13 @@ static struct phy_driver dp83867_driver[] = {
                    .config_intr    = dp83867_config_intr,
                    .handle_interrupt = dp83867_handle_interrupt,
     
    -               .suspend        = genphy_suspend,
    -               .resume         = genphy_resume,
    -
    +#if 1
    +                .suspend        = dp83867_suspend,
    +                .resume         = dp83867_resume,
    +#else
    +                .suspend        = genphy_suspend,
    +                .resume         = genphy_resume,
    +#endif
                    .link_change_notify = dp83867_link_change_notify,
            },
     };
    

    Are DP83867 PHY HW and associated driver ever tested to support this PHY deepsleep mode as mentioned in a datasheet in which XI CLK pad can go off completely ? If we have some references for the same, it would be really helpful to have a look into it.

    Thank you for support.

    Regards,

    Parth P

  • My bad, I sent the wrong changes in the above response. Apologies.

    Please have a look at below changes. phy_clear_bits_mmd is replaced with phy_set_bits_mmd and still the same issue persist.

    diff --git a/drivers/net/phy/dp83867.c b/drivers/net/phy/dp83867.c
    index f7436191fa80..ea5ff8cedf44 100644
    --- a/drivers/net/phy/dp83867.c
    +++ b/drivers/net/phy/dp83867.c
    @@ -151,6 +151,9 @@
     /* FLD_THR_CFG */
     #define DP83867_FLD_THR_CFG_ENERGY_LOST_THR_MASK       0x7
     
    +/* DP83867 Low Power Mode Bits */
    +#define DP83867_PHYCTRL_DS              BIT(7)
    +
     enum {
            DP83867_PORT_MIRROING_KEEP,
            DP83867_PORT_MIRROING_EN,
    @@ -960,6 +963,32 @@ static void dp83867_link_change_notify(struct phy_device *phydev)
            }
     }
     
    +static int dp83867_suspend(struct phy_device *phydev)
    +{
    +
    +       phy_set_bits_mmd(phydev, DP83867_DEVADDR, MII_DP83867_PHYCTRL,
    +                                   DP83867_PHYCTRL_DS);
    +
    +       return genphy_suspend(phydev);
    +}
    +
    +static int dp83867_resume(struct phy_device *phydev)
    +{
    +       int ret = 0;
    +
    +       genphy_resume(phydev);
    +
    +        ret = dp83867_config_init(phydev);
    +        if (ret < 0)
    +                return ret;
    +
    +        ret = dp83867_config_intr(phydev);
    +        if (ret < 0)
    +                return ret;
    +
    +       return 0;
    +}
    +
     static struct phy_driver dp83867_driver[] = {
            {
                    .phy_id         = DP83867_PHY_ID,
    @@ -982,9 +1011,13 @@ static struct phy_driver dp83867_driver[] = {
                    .config_intr    = dp83867_config_intr,
                    .handle_interrupt = dp83867_handle_interrupt,
     
    -               .suspend        = genphy_suspend,
    -               .resume         = genphy_resume,
    -
    +#if 1
    +                .suspend        = dp83867_suspend,
    +                .resume         = dp83867_resume,
    +#else
    +                .suspend        = genphy_suspend,
    +                .resume         = genphy_resume,
    +#endif
                    .link_change_notify = dp83867_link_change_notify,
            },
     };
    

    Thank you for understanding.

    Regards,

    Parth P

  • Hi Parth,

    Let us pull in the TI PHY team directly for further assistance on debugging this issue. 

    Best regards,

    Daolin

  • Hi Daolin,

    Could you please update on this issue ? I hope TI PHY team is looking into this.

    It would be a great help if you could kindly share an intermediate updates/findings to debug DP83867 PHY suspend/resume issue while using PHY in an interrupt mode.

    Thank you.

    Regards,

    Parth P

  • Hi Parth,

    I apologize for the delay in response. We have informed the PHY team to directly respond to what driver changes are needed based on your changes to the PHY driver. I will prompt them again today for a response.

    We also obtained some feedback from our internal team that the DP83867 uses Level Interrupts to signal events such as Link Down/Link Up while the AM62x SoC supports Edge interrupts. Due to this, it could be possible that interrupts are missed if the PHY is in an interrupt mode.

    Based on their comments, it appears that testing of Ethernet resume after deep sleep was done with the PHY operating in Polled mode rather than Interrupt mode.

    I will try to get more information from the PHY team as well and provide an update later today or tomorrow.

    Best regards,

    Daolin

  • Hi Parth,

    As an update, we are still communicating with the PHY team and our internal development team about how to get the link up after deep sleep.

    Reading the https://git.toradex.com/cgit/linux-toradex.git/tree/arch/arm64/boot/dts/ti/k3-am62-verdin-dev.dtsi?h=toradex_ti-linux-6.1.y that you shared, it appears that you have set the interrupt type to be falling edge type. Just to clarify, you shared several dts files but is this the most relevant dtsi file to look at? The first dts you shared in the first post had the &cpsw3g_mdio node disabled.

    We are currently trying to understand if the internal team has information on how to configure for edge interrupt mode in order to compare with what you have defined for interrupt mode. At the same time we are trying to get information from the phy team on how they distinguish between the interrupt types for the DP83867 phy. 

    ~Daolin

  • Hi Daolin,

    Thank you for the updates.

    Regarding your below comment, this case of missing an interrupt might be just applicable for resume cycle/transition. The real problem is (once the system is completely resumed), after reading/clearing the ISR PHY reg, PHY interrupt line goes high and when I disconnect the Ethernet cable, interrupt line goes low again. That means there is an falling edge interrupt from PHY hardware and that too is missed by PHY driver/AM62. All in all, any PHY interrupts never work after resume and that is not the case of just one interrupt being missed in a resume transition cycle.

    We also obtained some feedback from our internal team that the DP83867 uses Level Interrupts to signal events such as Link Down/Link Up while the AM62x SoC supports Edge interrupts. Due to this, it could be possible that interrupts are missed if the PHY is in an interrupt mode.

    Please see below explanation about DTS:

    Appreciate your support. Thank you.

    Regards,

    Parth P

  • Hi Daolin,

    I am writing this to provide an update on this issue. I believe that complete GPIO interrupt functionality is not working upon deepsleep resume. This might not have anything to do with PHY driver or related interrupts as such.

    I tested GPIO interrupt functionality with one of the GPIOs which we have on our boards and results surprised me with unexpected findings. GPIO interrupt functionality is broken after wake up from deepsleep. Please have a look at this e2e thread for more details for the same.

    You might not be the right person to help debug this GPIO interrupt issue, I would really appreciate if you could kindly loop relevant TI team for the same.

    Thank you.

    Regards,

    Parth P

  • Hi Parth,

    I appreciate the update on your findings regarding the GPIO interrupt functionality and for opening up a new thread on that topic as that is best way to get the people with the right knowledge on this issue. 

    I've went ahead to inform the relevant team members about this GPIO interrupt issue and the thread you've opened up and they will hopefully address it soon.

    Since the issue may be related to GPIO interrupts not working after deepsleep and not specifically related to the PHY functionality, I will leave this thread on waiting for customer for now. If you find later that the issue still may be related to the PHY, feel free to respond to this thread to open it back up.

    Best regards,

    Daolin

  • Hi Daolin,

    Just to keep you posted about the updates on this topic.

    I am able to resume DP83867 Ethernet PHY properly in an interrupt mode with the workaround for mentioned GPIO interrupt issue discussed here.
    PHY is able to resume properly for the case when it is not getting reset from HW during deepsleep suspend/resume cycle. However, when PHY resets due to RESTATSTATz going low for deepsleep mode on our modules, It can not resume it's function and link is not up.

    For now, we are not planning to reset Ethernet PHY in deep sleep which is one way to solve the issue.

    In case of any further assistance related to this topic, I will reopen this thread.

    Appreciate your support. Thank you.

    Regards,

    Parth P