AM3352: Kernel panic in booting up the custom board

Hello,

We have a custom board based on AM3352. We are in process of migrating kernel from 4.9 to 5.4 and we are observing kernel panic at boot up and it hangs. The problem seem to be related cpsw/phy/mdio. Below is the panic message, please find full logs here /cfs-file/__key/communityserver-discussions-components-files/791/5.4_2D00_kernel_2D00_crash.txt

[    2.220984] cpsw 4a100000.ethernet: cpts: overflow check period 500 (jiffies)
[    2.235639] cpsw 4a100000.ethernet: Detected MACID = 38:0b:3c:95:ed:04
[    2.246117] cpsw 4a100000.ethernet: RAX: Exiting cpsw_probe
[    2.254361] rtc-pcf8523 0-0068: setting system clock to 2022-08-11T12:56:37 UTC (1660222597)
[    2.266245] cpsw 4a100000.ethernet: initializing cpsw version 1.12 (0)
[    2.282120] 8<--- cut here ---
[    2.285348] Unable to handle kernel NULL pointer dereference at virtual address 00000004
[    2.294082] pgd = 487c313e
[    2.296932] [00000004] *pgd=00000000
[    2.300774] Internal error: Oops: 5 [#1] PREEMPT ARM
[    2.305999] Modules linked in:
[    2.309223] CPU: 0 PID: 1 Comm: swapper Not tainted 5.4.161-alc-g999 #1
[    2.316183] Hardware name: Generic AM33XX (Flattened Device Tree)
[    2.322617] PC is at dp83822_config_intr+0x30/0x124
[    2.327751] LR is at mdiobus_read+0x58/0x68
[    2.332152] pc : [<c04f40d8>]    lr : [<c04f3180>]    psr: 60000113
[    2.338746] sp : cd03bce0  ip : cd039640  fp : c0c4e2e0
[    2.344243] r10: c0b4078c  r9 : cd59a8e0  r8 : 00000001
[    2.349739] r7 : 00000000  r6 : c050e884  r5 : 00000000  r4 : cd500000
[    2.356604] r3 : cd039640  r2 : 00000000  r1 : 00000000  r0 : 00000000
[    2.363475] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[    2.370977] Control: 10c5387d  Table: 80004019  DAC: 00000051
[    2.377021] Process swapper (pid: 1, stack limit = 0x7b048a44)
[    2.383151] Stack: (0xcd03bce0 to 0xcd03c000)
[    2.387743] bce0: cd500000 00000000 c050e884 c04ee07c cd543cc0 cd500000 cd500000 c04f1874
[    2.396352] bd00: cd500000 cd619000 c050e884 c05ee358 cd59a8c0 cd6194a0 cd500840 cd6194a0
[    2.404960] bd20: 00000001 c07b39c8 00000007 00000003 00000000 cd619000 cd500840 cd6194a0
[    2.413570] bd40: 00000002 cd59a8e0 c0b4078c c0510088 00000000 00000003 cd161b40 00000000
[    2.422182] bd60: 0000000e c06ac63c 00000001 c06f4d38 00000000 00000a20 cd584000 cd115000
[    2.430786] bd80: cd584000 00000000 fffffff5 00000000 c0736c60 fffffff3 00000000 fffffff2
[    2.439396] bda0: 00000000 cd03bdec fffffff0 00000000 cd03bdec c0c04248 cd619000 cd619000
[    2.448007] bdc0: c0851748 00000000 00000000 00001002 cd619000 c0b4078c c0c4e2e0 c062005c
[    2.456617] bde0: fffffff0 cd619000 ffffe000 cd619000 00000000 c0c04248 cd619000 00000001
[    2.465229] be00: 00001003 c0620430 00000000 cd115000 00000001 c061fd0c cd115000 c0c04248
[    2.473839] be20: cd619000 00000001 00000000 cd619138 00001002 c06204b8 c0c4e334 00000001
[    2.482446] be40: 00001002 c0b4079c c09f02dc c0b307c8 00000001 00000000 00000000 c0c56000
[    2.491057] be60: 000000e1 c02b7088 cd02aec8 00000002 00000001 00000000 00000002 cd68b688
[    2.499669] be80: c0812e20 cd02aec8 c0c56000 c02b8a34 00000000 00000002 62f4fc85 c0c4db70
[    2.508283] bea0: 00000000 c02b9470 00000000 c0b27c2c 00010000 00000000 c0812e20 c0c4db70
[    2.516891] bec0: 00000000 00000000 cd042d00 c0c4db70 00000000 c02b9804 00000000 c0c04248
[    2.525499] bee0: 00000000 c0b30558 ffffe000 00000000 c0c56000 000000e1 c0b38834 c0b004a4
[    2.534105] bf00: c0b4c478 c0102dc8 00000000 00000000 cfbc6436 c0a49600 000000e1 c0154758
[    2.542715] bf20: c0a484ac c097a60c c0b004a4 00000000 00000007 00000007 00000000 cfbc6426
[    2.551329] bf40: cfbc6431 c0c04248 00000000 00000008 c0b38850 c0c04248 c0c56000 00000008
[    2.559938] bf60: c0b38854 c0c56000 c0c56000 c0b00e94 00000007 00000007 00000000 c0b004a4
[    2.568555] bf80: c07b667c 00000000 c07b667c 00000000 00000000 00000000 00000000 00000000
[    2.577163] bfa0: 00000000 c07b6684 00000000 c01010e8 00000000 00000000 00000000 00000000
[    2.585771] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.594379] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[    2.603021] [<c04f40d8>] (dp83822_config_intr) from [<c04ee07c>] (phy_request_interrupt+0x7c/0xe4)
[    2.612453] [<c04ee07c>] (phy_request_interrupt) from [<c04f1874>] (phy_connect_direct+0x48/0x58)
[    2.621793] [<c04f1874>] (phy_connect_direct) from [<c05ee358>] (of_phy_connect+0x38/0x60)
[    2.630496] [<c05ee358>] (of_phy_connect) from [<c07b39c8>] (cpsw_slave_open+0x1d4/0x2a0)
[    2.639108] [<c07b39c8>] (cpsw_slave_open) from [<c0510088>] (cpsw_ndo_open+0x4b8/0x68c)
[    2.647638] [<c0510088>] (cpsw_ndo_open) from [<c062005c>] (__dev_open+0xe0/0x16c)
[    2.655613] [<c062005c>] (__dev_open) from [<c0620430>] (__dev_change_flags+0x164/0x1d4)
[    2.664132] [<c0620430>] (__dev_change_flags) from [<c06204b8>] (dev_change_flags+0x18/0x48)
[    2.673027] [<c06204b8>] (dev_change_flags) from [<c0b307c8>] (ip_auto_config+0x270/0x1110)
[    2.681826] [<c0b307c8>] (ip_auto_config) from [<c0102dc8>] (do_one_initcall+0x4c/0x1c0)
[    2.690361] [<c0102dc8>] (do_one_initcall) from [<c0b00e94>] (kernel_init_freeable+0x10c/0x1c8)
[    2.699523] [<c0b00e94>] (kernel_init_freeable) from [<c07b6684>] (kernel_init+0x8/0x114)
[    2.708135] [<c07b6684>] (kernel_init) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
[    2.716094] Exception stack(0xcd03bfb0 to 0xcd03bff8)
[    2.721411] bfa0:                                     00000000 00000000 00000000 00000000
[    2.730021] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.738628] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    2.745592] Code: e5945274 ebfffc15 e3500000 b8bd8070 (e5953004)
[    2.752916] ---[ end trace b75dcf016a97b2d4 ]---
[    2.757796] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.765859] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

Here is how relevant nodes are extended in case of 4.9 kernel for custom board. Same set of nodes are used in 5.4 kernel except "phy_sel".

&mac {
    pinctrl-names = "default", "sleep";
    pinctrl-0 = <&cpsw_default>;
    pinctrl-1 = <&cpsw_sleep>;
    status = "okay";
};

&davinci_mdio {
    pinctrl-names = "default", "sleep";
    pinctrl-0 = <&davinci_mdio_default>;
    pinctrl-1 = <&davinci_mdio_sleep>;
    status = "okay";

    reset-gpios = <
        &gpio2 2 GPIO_ACTIVE_LOW        /* GPIO2_2 (ETH0_RESETn) */
        &gpio2 3 GPIO_ACTIVE_LOW        /* GPIO2_3 (ETH1_RESETn) */
    >;
    reset-delay-us = <30>;              /* PHY datasheet states 25uS min */
    ethernetphy0: ethernet-phy@0 {
//        compatible = "ethernet-phy-id2000.a140", "ethernet-phy-ieee802.3-c22", "ethernet-phy-ieee802.3-c45";
        reg = <0>;
        interrupt-parent = <&gpio2>;
        interrupts = <6 IRQ_TYPE_EDGE_FALLING>;
    };
    ethernetphy1: ethernet-phy@1 {
//        compatible = "ethernet-phy-id2000.a140", "ethernet-phy-ieee802.3-c22", "ethernet-phy-ieee802.3-c45";
        reg = <1>;
        interrupt-parent = <&gpio2>;
        interrupts = <7 IRQ_TYPE_EDGE_FALLING>;
    };
};

/* ETH0 */
&cpsw_emac0 {
    phy-mode = "rmii";
    phy_id = <&davinci_mdio>, <0>;
    phy-handle = <&ethernetphy0>;
    status = "okay";
};

/* ETH1 */
&cpsw_emac1 {
    phy-mode = "rmii";
    phy_id = <&davinci_mdio>, <1>;
    phy-handle = <&ethernetphy1>;
    status = "okay";
};

/* Deprecrated in 5.4 kernel */
&phy_sel {
    rmii-clock-ext;
};

IMP: one thing to notice here is our custom board is using external clock for phy.

So, "rmii-clock-ext" is extended for "phy_sel" node. Now, when it comes to 5.4 kernel I observed that "phy_sel" is deprecated and documentation recommends to use phy_gmii_sel but I do not see any property like "rmii-clock-ext". Is this causing the kernel panic?

Edit 1: Further debugging, I found that dp83822_config_intr is being called and in that function when it is trying to deference phydev->priv, panic gets triggered and this is because priv is NULL. Then I put debug logs in dp83822_probe function and I do not see probe is being called which means driver's private structure doesn't get initialized. I am still not clear why it is happening and how to resolve this. 

Regards,

Raxesh