PROCESSOR-SDK-AM335X: kernel hangs at boot up after migrating from 5.4 to 5.10 kernel.

Hi Siva,

I think the first thing you can try is to use the previous U-Boot binary for kernel 5.4 to boot the new kernel 5.10 to see if the kernel can boot further.

u-boot is not changed.

we are using u-boot 2020 before for 5.4 kernel and it works fine.

Now we just migrated to 5.10 kernel with same u-boot 2020 and i can see the hang.  

U-Boot SPL 2020.01-g2781231a33 (Mar 26 2021 - 17:56:23 +0000)
Trying to boot from MMC2

U-Boot 2020.01-g2781231a33 (Mar 26 2021 - 17:56:23 +0000)

CPU : AM335X-GP rev 2.1
Model: Platforms Edwards 4-CPU
DRAM: 256 MiB
WDT: Started with servicing (60s timeout)
NAND: 0 MiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

Net:
Warning: ethernet@4a100000 MAC addresses don't match:
Address in ROM is 34:84:e4:18:01:90
Address in environment is 00:b0:19:74:c9:68
eth0: ethernet@4a100000
Warning: usb_ether MAC addresses don't match:
Address in ROM is de:ad:be:ef:00:01
Address in environment is 00:b0:19:74:c9:69
, eth1: usb_ether
switch to partitions #0, OK
mmc1(part 0) is current device
SD/MMC found on device 1
4097 bytes read in 4 ms (1000 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
5579264 bytes read in 949 ms (5.6 MiB/s)
93779 bytes read in 21 ms (4.3 MiB/s)
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Device Tree to 8df1b000, end 8df34e52 ... OK

Starting kernel ...

I removed the below Nodes that was modified while Migrating from 5.4 to 5.10 kernel. After this kernel boots & hangs further while initializing lcd.

---------------------------------------------

  ocp {
      target-module@49000000 {
       clocks = <&l3_gclk>;
        /*clock-names="fck";*/
       };
  };

------------------------------------------------                     

ocp {                                              
edma: edma@49000000 {            
clocks = <&l3_gclk>;                     
clock-names="fck";                       
};                                                    
};                                                    

---------------------------------------------

/*
** Backlight Support
** -----------------
** Creating the Pulse Width Module Sub-System
*/
&epwmss0 {
status = "okay";
ecap0: ecap@100 {
status = "okay";
pinctrl-names = "default", "sleep";
pinctrl-0 = <&ecap0_pins_default>;
pinctrl-1 = <&ecap0_pins_sleep>;
};
};         

-----------------------------------------------------

&epwmss0 {
status = "okay";
ecap0: pwm@100 {
status = "okay";
pinctrl-names = "default", "sleep";
pinctrl-0 = <&ecap0_pins_default>;
pinctrl-1 = <&ecap0_pins_sleep>;
};
};

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0 MiB
MMC: OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from FAT... *** Warning - bad CRC, using default environment

Net:
Warning: ethernet@4a100000 MAC addresses don't match:
Address in ROM is 34:84:e4:18:09:28
Address in environment is 00:b0:19:74:c9:4c
eth0: ethernet@4a100000
Warning: usb_ether MAC addresses don't match:
Address in ROM is de:ad:be:ef:00:01
Address in environment is 00:b0:19:74:c9:4d
, eth1: usb_ether
switch to partitions #0, OK
mmc1(part 0) is current device
SD/MMC found on device 1
4097 bytes read in 6 ms (666 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
5579264 bytes read in 948 ms (5.6 MiB/s)
93735 bytes read in 21 ms (4.3 MiB/s)
## Flattened Device Tree blob at 88000000
Booting using the fdt blob at 0x88000000
Loading Device Tree to 8df1b000, end 8df34e26 ... OK

Starting kernel ...

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 5.10.162-rt79-g1f0dabb52e (oe-user@oe-host) (arm-poky-linux-gnueabi-gcc (GCC) 9.5.0, GNU ld (GNU Binutils) 2.34.0.20200910) #1 PREEMPT_RT Fri Jan 27 07:49:52 UTC 2023
[ 0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Platforms Edwards 4-CPU v. 00.01.00
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] efi: UEFI not found.
[ 0.000000] cma: Failed to reserve 512 MiB
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000080000000-0x000000008fdfffff]
[ 0.000000] HighMem empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x000000008fdfffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000008fdfffff]
[ 0.000000] CPU: All CPU(s) started in SVC mode.
[ 0.000000] AM335X ES2.1 (sgx neon)
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 64452
[ 0.000000] Kernel command line: console=ttyS0,115200n8 root=/dev/mmcblk0p2 rootfstype=ext3 rw rootwait ip=off lpj=3590144 video=/dev/fb0:480x640@60 fbcon=rotate:1 threadirqs
[ 0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 241608K/260096K available (10023K kernel code, 811K rwdata, 3876K rodata, 364K init, 291K bss, 18488K reserved, 0K cma-reserved, 0K highmem)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU priority boosting: priority 1 delay 500 ms.
[ 0.000000] rcu: RCU_SOFTIRQ processing moved to rcuc kthreads.
[ 0.000000] No expedited grace period (rcu_normal_after_boot).
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] IRQ: Found an INTC at 0x(ptrval) (revision 5.0) with 128 interrupts
[ 0.000000] TI gptimer clocksource: always-on /ocp/interconnect@44c00000/segment@200000/target-module@31000
[ 0.000002] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[ 0.000022] clocksource: dmtimer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[ 0.000441] TI gptimer clockevent: 24000000 Hz at /ocp/interconnect@48000000/segment@0/target-module@40000
[ 0.001843] Console: colour dummy device 80x30
[ 0.001917] Calibrating delay loop (skipped) preset value.. 718.02 BogoMIPS (lpj=3590144)
[ 0.001936] pid_max: default: 32768 minimum: 301
[ 0.002082] LSM: Security Framework initializing
[ 0.002256] AppArmor: AppArmor initialized
[ 0.002321] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.002337] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.003359] CPU: Testing write buffer coherency: ok
[ 0.003435] CPU0: Spectre v2: using BPIALL workaround
[ 0.004738] Setting up static identity map for 0x80008200 - 0x80008278
[ 0.004906] rcu: Hierarchical SRCU implementation.
[ 0.005361] EFI services will not be available.
[ 0.005846] devtmpfs: initialized
[ 0.021096] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[ 0.021332] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.021367] futex hash table entries: 256 (order: 1, 8192 bytes, linear)
[ 0.021658] pinctrl core: initialized pinctrl subsystem
[ 0.022406] DMI not present or invalid.
[ 0.023168] NET: Registered protocol family 16
[ 0.023908] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.039444] l3-aon-clkctrl:0000:0: failed to disable
[ 0.040071] audit: initializing netlink subsys (disabled)
[ 0.041459] thermal_sys: Registered thermal governor 'fair_share'
[ 0.041475] thermal_sys: Registered thermal governor 'bang_bang'
[ 0.041479] thermal_sys: Registered thermal governor 'step_wise'
[ 0.041483] thermal_sys: Registered thermal governor 'user_space'
[ 0.050622] audit: type=2000 audit(0.030:1): state=initialized audit_enabled=0 res=1
[ 1.145366] No ATAGs?
[ 1.145388] hw-breakpoint: debug architecture 0x4 unsupported.
[ 1.607860] iommu: Default domain type: Translated
[ 1.610168] SCSI subsystem initialized
[ 1.611113] mc: Linux media interface: v0.10
[ 1.611168] videodev: Linux video capture interface: v2.00
[ 1.611320] pps_core: LinuxPPS API ver. 1 registered
[ 1.611327] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 1.611348] PTP clock support registered
[ 1.612489] Advanced Linux Sound Architecture Driver Initialized.
[ 1.614163] clocksource: Switched to clocksource dmtimer
[ 1.615114] AppArmor: AppArmor Filesystem Enabled
[ 1.623888] NET: Registered protocol family 2
[ 1.624395] IP idents hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 1.625735] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 7168 bytes, linear)
[ 1.625812] TCP established hash table entries: 2048 (order: 1, 8192 bytes, linear)
[ 1.625850] TCP bind hash table entries: 2048 (order: 3, 49152 bytes, linear)
[ 1.625912] TCP: Hash tables configured (established 2048 bind 2048)
[ 1.626053] UDP hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 1.626087] UDP-Lite hash table entries: 256 (order: 2, 16384 bytes, linear)
[ 1.626292] NET: Registered protocol family 1
[ 1.627066] RPC: Registered named UNIX socket transport module.
[ 1.627080] RPC: Registered udp transport module.
[ 1.627084] RPC: Registered tcp transport module.
[ 1.627088] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1.627098] NET: Registered protocol family 44
[ 1.627119] PCI: CLS 0 bytes, default 64
[ 1.628489] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[ 1.629854] Initialise system trusted keyrings
[ 1.630299] workingset: timestamp_bits=14 max_order=16 bucket_order=2
[ 1.635513] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 1.636499] NFS: Registering the id_resolver key type
[ 1.636572] Key type id_resolver registered
[ 1.636580] Key type id_legacy registered
[ 1.636690] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 1.636701] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[ 1.636744] ntfs: driver 2.1.32 [Flags: R/O].
[ 1.637566] NET: Registered protocol family 38
[ 1.637594] Key type asymmetric registered
[ 1.637602] Asymmetric key parser 'x509' registered
[ 1.637660] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 244)
[ 1.637669] io scheduler mq-deadline registered
[ 1.637676] io scheduler kyber registered
[ 1.894239] ti-sysc: probe of 44e31000.target-module failed with error -16
[ 1.962032] ti-sysc: probe of 48040000.target-module failed with error -16
[ 2.037530] omap-mailbox 480c8000.mailbox: omap mailbox rev 0x400
[ 2.148902] debugfs: Directory '49000000.dma' with parent 'dmaengine' already present!
[ 2.148942] edma 49000000.dma: TI EDMA DMA engine driver
[ 2.169510] pinctrl-single 44e10800.pinmux: 142 pins, size 568
[ 2.173888] pwm-backlight backlight: supply power not found, using dummy regulator
[ 2.233378] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[ 2.238112] printk: console [ttyS0] disabled
[ 2.238338] 44e09000.serial: ttyS0 at MMIO 0x44e09000 (irq = 20, base_baud = 3000000) is a 8250
[ 2.238510] printk: console [ttyS0] enabled
[ 2.242593] omap_rng 48310000.rng: Random Number Generator ver. 20
[ 2.245628] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.246020] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.246280] tilcdc 4830e000.lcdc: no encoders/connectors found
[ 2.246771] random: crng init done
[ 2.266492] brd: module loaded
[ 2.276399] loop: module loaded
[ 2.283404] CAN device driver interface
[ 2.285182] c_can_platform 481d0000.can: c_can_platform device registered (regs=439fb2ef, irq=43)
[ 2.344197] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6, bus freq 1000000
[ 2.349494] davinci_mdio 4a101000.mdio: phy[1]: device 4a101000.mdio:01, driver TI DP83620 10/100 Mbps PHY
[ 2.349508] davinci_mdio 4a101000.mdio: phy[3]: device 4a101000.mdio:03, driver TI DP83620 10/100 Mbps PHY
[ 2.350297] cpsw 4a100000.ethernet: initialized cpsw ale version 1.4
[ 2.350313] cpsw 4a100000.ethernet: ALE Table size 1024
[ 2.350430] cpsw 4a100000.ethernet: cpts: overflow check period 500 (jiffies)
[ 2.350505] cpsw 4a100000.ethernet: Detected MACID = 00:b0:19:74:c9:4c
[ 2.351429] cpsw 4a100000.ethernet: cpsw: Detected MACID = 00:b0:19:74:c9:4d
[ 2.355175] i2c /dev entries driver
[ 2.357469] sdhci: Secure Digital Host Controller Interface driver
[ 2.357479] sdhci: Copyright(c) Pierre Ossman
[ 2.360368] sdhci-pltfm: SDHCI platform and OF driver helper
[ 2.367632] davinci-mcasp 48038000.mcasp: IRQ common not found
[ 2.368953] davinci-mcasp 4803c000.mcasp: IRQ common not found
[ 2.370969] NET: Registered protocol family 10
[ 2.372780] Segment Routing with IPv6
[ 2.372891] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[ 2.373739] NET: Registered protocol family 17
[ 2.373771] can: controller area network core
[ 2.373871] NET: Registered protocol family 29
[ 2.374625] Key type dns_resolver registered
[ 2.375403] omap_voltage_late_init: Voltage driver support not added
[ 2.376124] printk: console [ttyS0]: printing thread started
[ 2.376182] Loading compiled-in X.509 certificates
[ 2.385618] alg: No test for pkcs1pad(rsa,sha1) (pkcs1pad(rsa-generic,sha1))
[ 2.393690] Loaded X.509 cert 'Build time autogenerated kernel key: 4085288fd5a5b48a43f5e7db26fefd43ddd74bd1'
[ 2.410249] AppArmor: AppArmor sha1 policy hashing enabled
[ 2.436048] OMAP GPIO hardware version 0.1
[ 2.438777] at24 0-0050: supply vcc not found, using dummy regulator
[ 2.440077] at24 0-0050: 2048 byte 24c16 EEPROM, writable, 16 bytes/write
[ 2.440151] i2c i2c-0: Failed to register i2c client 24c16 at 0x51 (-16)
[ 2.440163] i2c i2c-0: of_i2c: Failure registering /ocp/interconnect@44c00000/segment@200000/target-module@b000/i2c@0/sfp0@51
[ 2.440188] i2c i2c-0: Failed to create I2C device for /ocp/interconnect@44c00000/segment@200000/target-module@b000/i2c@0/sfp0@51
[ 2.441125] at24 0-005f: supply vcc not found, using dummy regulator
[ 2.442342] at24 0-005f: 2048 byte 24c16 EEPROM, writable, 16 bytes/write
[ 2.442429] omap_i2c 44e0b000.i2c: bus 0 rev0.11 at 100 kHz
[ 2.474463] tps65910 1-002d: No interrupt support, no core IRQ
[ 2.476564] vrtc: supplied by vbat
[ 2.480669] vio: supplied by vbat
[ 2.484093] vdd_mpu: supplied by vbat
[ 2.488685] vdd_core: supplied by vbat
[ 2.498723] vdig1: supplied by vbat
[ 2.506106] vdig2: supplied by vbat
[ 2.511026] vpll: supplied by vbat
[ 2.514821] vdac: supplied by vbat
[ 2.521751] vaux1: supplied by vbat
[ 2.537650] vaux2: supplied by vbat
[ 2.545570] vaux33: supplied by vbat
[ 2.553482] vmmc: supplied by vbat
[ 2.560179] vbb: supplied by vbat
[ 2.561323] at24 1-0050: supply vcc not found, using dummy regulator
[ 2.562626] at24 1-0050: 2048 byte 24c16 EEPROM, writable, 16 bytes/write
[ 2.562725] omap_i2c 4802a000.i2c: bus 1 rev0.11 at 400 kHz
[ 2.580148] at24 2-0050: supply vcc not found, using dummy regulator
[ 2.581453] at24 2-0050: 2048 byte 24c16 EEPROM, writable, 16 bytes/write
[ 2.581525] i2c i2c-2: Failed to register i2c client 24c16 at 0x51 (-16)
[ 2.581535] i2c i2c-2: of_i2c: Failure registering /ocp/interconnect@48000000/segment@100000/target-module@9c000/i2c@0/sfp1@51
[ 2.581559] i2c i2c-2: Failed to create I2C device for /ocp/interconnect@48000000/segment@100000/target-module@9c000/i2c@0/sfp1@51
[ 2.581905] at24 2-005f: supply vcc not found, using dummy regulator
[ 2.583093] at24 2-005f: 2048 byte 24c16 EEPROM, writable, 16 bytes/write
[ 2.583180] omap_i2c 4819c000.i2c: bus 2 rev0.11 at 100 kHz
[ 2.588497] pwm-backlight backlight: supply power not found, using dummy regulator
[ 2.590181] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.590541] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.590801] tilcdc 4830e000.lcdc: no encoders/connectors found
[ 2.604372] pwm-backlight backlight: supply power not found, using dummy regulator
[ 2.606121] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.606859] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.607151] tilcdc 4830e000.lcdc: no encoders/connectors found
[ 2.609712] ALSA device list:
[ 2.609735] No soundcards found.
[ 2.625028] Waiting for root device /dev/mmcblk0p2...
[ 2.625801] pwm-backlight backlight: supply power not found, using dummy regulator
[ 2.628026] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.628379] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.628660] tilcdc 4830e000.lcdc: no encoders/connectors found
[ 2.702501] mmc1: new high speed MMC card at address 0001
[ 2.714698] mmcblk1: mmc1:0001 M32508 7.13 GiB
[ 2.714989] mmcblk1boot0: mmc1:0001 M32508 partition 1 4.00 MiB
[ 2.715258] mmcblk1boot1: mmc1:0001 M32508 partition 2 4.00 MiB
[ 2.715566] mmcblk1rpmb: mmc1:0001 M32508 partition 3 4.00 MiB, chardev (243:0)
[ 2.717605] mmcblk1: p1 p2 p3 p4
[ 2.725499] pwm-backlight backlight: supply power not found, using dummy regulator
[ 2.727365] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.727757] OF: graph: no port node found in /ocp/interconnect@48000000/segment@300000/target-module@e000/lcdc@0
[ 2.728042] tilcdc 4830e000.lcdc: no encoders/connectors found

Hi BinLu,

My issue looks similar to this below e2e ticket.

e2e.ti.com/.../4211722

Here is emmc node in my dts.

&mmc2 {
compatible = "ti,omap4-hsmmc";
status = "okay";
bus-width = <8>;
pinctrl-names = "default", "sleep";
pinctrl-0 = <&emmc_pins_default>;
pinctrl-1 = <&emmc_pins_sleep>;
ti,non-removable;
vmmc-supply = <&vmmc_reg>;
};

Is there any changes needed to this node with 5.10 kernel to resolve hang with mounting rootfs. 

Regards,

Siva.