Because of the holidays, TI E2E™ design support forum responses will be delayed from Dec. 25 through Jan. 2. Thank you for your patience.

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM3354: failure to resume from suspend (standby/mem)

Part Number: AM3354

Hi,

We have a custom board with AM3354 SoC running our Yocto based Linux distro with upstream Linux kernel v5.10.166. We are using the Yocto layer meta-ti tagged: 07.03.00.005

We have run into suspend/resume issue on this SoC.

We are referring to the document "AM335x Low Power Design Guide Rev. A" (SPRAC74A – February 2017 – Revised March 2017) to enable power management on this SoC.

This is the state of kernel configs:

CONFIG_MAILBOX=y
CONFIG_OMAP2PLUS_MBOX=m
CONFIG_WKUP_M3_RPROC=m
CONFIG_SOC_TI=y
CONFIG_AMX3_PM=m
CONFIG_WKUP_M3_IPC=m
CONFIG_TI_EMIF_SRAM=m

This shows that the firmware was loaded into CM3 successfully:

root@lp-am3354:~# dmesg | grep -iE "remote|ipc|wkup|mailbox"
[   12.766082] remoteproc remoteproc0: wkup_m3 is available
[   12.785892] wkup_m3_ipc 44e11324.wkup_m3_ipc: IPC Request for A8->M3 Channel failed! -517
[   13.105999] wkup_m3_ipc 44e11324.wkup_m3_ipc: IPC Request for A8->M3 Channel failed! -517
[   13.231233] wkup_m3_ipc 44e11324.wkup_m3_ipc: IPC Request for A8->M3 Channel failed! -517
[   13.478769] wkup_m3_ipc 44e11324.wkup_m3_ipc: IPC Request for A8->M3 Channel failed! -517
[   13.522549] omap-mailbox 480c8000.mailbox: omap mailbox rev 0x400
[   13.661037] remoteproc remoteproc0: powering up wkup_m3
[   13.920713] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 244024
[   13.929014] remoteproc remoteproc0: remote processor wkup_m3 is now up
[   13.929036] wkup_m3_ipc 44e11324.wkup_m3_ipc: CM3 Firmware Version = 0x192

We are using v1.9.2 of the CM3 PM firmware built from git.ti.com/processor-firmware/ti-amx3-cm3-pm-firmware.git branch: ti-v4.1.y, commit hash: fb484c5e54f2e31cf0a338d2927a06a2870bcc2c.

We have also tried the CM3 firmware binary downloaded from the above mentioned git repo (git.ti.com/.../ against the release tag: 08.02.00.004. Which essentially points to the same tag as branch: ti-v4.1.y, commit hash: fb484c5e54f2e31cf0a338d2927a06a2870bcc2c.

There is no VTT termination on the DRAM: MT41K256M16, and there is a 10K ohm pull-down resistor on the DDR_CKE. The DDR_CKE pin goes LOW when we put the SoC to suspend mode using either of the following commands:

$ echo mem > /sys/power/state
$ echo standby > /sys/power/state

The SoC does not wake up when we type anything on the UART0 console.

root@lp-am3354:~# echo mem > /sys/power/state
root@lp-am3354:~# [  110.842841] PM: suspend entry (deep)
[  110.853425] Filesystems sync: 0.006 seconds
[  110.874009] Freezing user space processes ... (elapsed 0.001 seconds) done.
[  110.882481] OOM killer disabled.
[  110.885838] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[  110.894957] printk: Suspending console(s) (use no_console_suspend to debug)

However, when we add the "no_console_suspend" in the kernel commandline args, we are able to wake up the SoC using UART0 only from standby mode:

root@lp-am3354:~# echo standby > /sys/power/state
root@lp-am3354:~# [  986.212016] PM: suspend entry (shallow)
[  986.222320] Filesystems sync: 0.006 seconds
[  986.255885] Freezing user space processes ... (elapsed 0.001 seconds) done.
[  986.264063] OOM killer disabled.
[  986.267286] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[  986.428728] Disabling non-boot CPUs ...
[  986.432517] pm33xx pm33xx: PM: Successfully put all powerdomains to target state
[  986.432517] PM: Wakeup source MPU_WAKE
��[  986.443311] spi-nor spi0.0: trying to lock already unlocked area
[  986.449524] cpsw 4a100000.ethernet: initializing cpsw version 1.12 (0)
[  986.541566] Micrel KSZ8081 or KSZ8091 4a101000.mdio:05: attached PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=4a101000.mdio:05, irq=POLL)
[  986.561872] Micrel KSZ8081 or KSZ8091 4a101000.mdio:05: Micrel KSZ8081 or KSZ8091: resetting PHY on link down
[  986.768201] OOM killer enabled.
[  986.771343] Restarting tasks ... done.
[  986.885854] PM: suspend exit
root@lp-am3354:~# [  989.776988] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off

... but not from mem/deepsleep mode:

root@lp-am3354:~# echo mem > /sys/power/state
root@lp-am3354:~# [  281.843135] PM: suspend entry (deep)
[  281.857175] Filesystems sync: 0.010 seconds
[  281.880809] Freezing user space processes ... (elapsed 0.001 seconds) done.
[  281.889450] OOM killer disabled.
[  281.892697] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[  281.922471] Disabling non-boot CPUs ...

The "AM335x Low Power Design Guide Rev. A" (SPRAC74A – February 2017 – Revised March 2017) document states at page 23, section 3.3.4.2 "Serial Console Output During Suspend/Resume" that, "Note that if this (i.e. no_console_suspend) is used then the serial port cannot be used as a wakeup source for the processor.". This is totally opposite to our case.

Adding wakeup-source to the uart0 node in the dts also did not help:

&uart0 {
        pinctrl-names = "default";
        pinctrl-0 = <&uart0_pins>;

        status = "okay";
        wakeup-source;                                                                                                                                                                                                                                                          
};

We have also tried plugging in/out a Flash drive into USB1 to wake up the SoC, but it did not work. Is this a correct test?

The internal RTC is not in use and linux is not able to enable/disable its clock probably because of:

 TPS65910

root@lp-am3354:~# dmesg | grep -iE "rtc"
[    1.478468] l4-rtc-clkctrl:0000:0: failed to enable
[    2.936790] rtc-ds1307 0-0068: registered as rtc0
[    2.941742] rtc-ds1307 0-0068: setting system clock to 2000-01-01T01:23:14 UTC (946689794)
[    3.054168] vrtc: supplied by vcc5v
[    6.654989] l4-rtc-clkctrl:0000:0: failed to disable
[   16.656550] PM: no-rtc available, rtc-only mode disabled.

Therefore we have patched the pm33xx.c driver to not go for rtc-only mode suspend:

diff --git a/drivers/soc/ti/pm33xx.c b/drivers/soc/ti/pm33xx.c
index dc21aa855..16455539b 100644
--- a/drivers/soc/ti/pm33xx.c
+++ b/drivers/soc/ti/pm33xx.c
@@ -306,12 +306,12 @@ static void am33xx_pm_end(void)
 	u32 val = 0;
 	struct nvmem_device *nvmem;
 
-	nvmem = devm_nvmem_device_get(&omap_rtc->dev, "omap_rtc_scratch0");
-	if (IS_ERR(nvmem))
-		return;
-
 	m3_ipc->ops->finish_low_power(m3_ipc);
 	if (rtc_only_idle) {
+		nvmem = devm_nvmem_device_get(&omap_rtc->dev, "omap_rtc_scratch0");
+		if (IS_ERR(nvmem))
+			return;
+
 		if (retrigger_irq) {
 			/*
 			 * 32 bits of Interrupt Set-Pending correspond to 32
@@ -443,7 +443,7 @@ static int am33xx_pm_rtc_setup(void)
 
 	np = of_find_node_by_name(NULL, "rtc");
 
-	if (of_device_is_available(np)) {
+	if (of_device_is_available(np) && 0) {
 		/* RTC interconnect target module clock */
 		rtc_fck = of_clk_get_by_name(np->parent, "fck");
 		if (IS_ERR(rtc_fck))

We never got to wake up the SoC from deepsleep mode. However, only once, we saw this when we kept pressing the Enter key on the UART0 console:

root@lp-am3354:~# echo mem > /sys/power/state
[   40.918408] PM: suspend entry (deep)
[   40.935296] Filesystems sync: 0.013 seconds
[   40.954440] Freezing user space processes ... (elapsed 0.003 seconds) done.
[   40.965119] OOM killer disabled.
[   40.968240] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[   41.030803] Disabling non-boot CPUs ...
[   41.034843] pm33xx pm33xx: PM: Kernel suspend failure
[   41.046149] spi-nor spi0.0: trying to lock already unlocked area
[   41.052138] cpsw 4a100000.ethernet: initializing cpsw version 1.12 (0)
[   41.140257] Micrel KSZ8081 or KSZ8091 4a101000.mdio:05: attached PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=4a101000.mdio:05, irq=POLL)
[   41.160627] Micrel KSZ8081 or KSZ8091 4a101000.mdio:05: Micrel KSZ8081 or KSZ8091: resetting PHY on link down
[   41.624064] usb 2-1: reset high-speed USB device number 2 using musb-hdrc

We need your help to get the suspend/resume to work on this SoC.

Best regards,
Arsalan H. Awan

  • Hi Arsalan,

    We have a custom board with AM3354 SoC running our Yocto based Linux distro with upstream Linux kernel v5.10.166. We are using the Yocto layer meta-ti tagged: 07.03.00.005

    This doesn't seem to match - kernel v5.10.y is used in Processor SDK v8.x, while tag 07.03.xx.xxx should be on kernel v5.4.y.

    Instead of using the Yocto build, but just for testing, can you please take the u-boot, kernel, and rootfs from the Processor SDK v8.2 to see if you can get suspend/resume working on your platform?

  • Hi Bin,

    I downloaded the AM335x Linux SDK prebuilt BSP binaries and root filesystem:

    am335x-evm-linux-sdk-bin-08.02.00.24.tar.xz  — 2229010 K

    from here: https://www.ti.com/tool/PROCESSOR-SDK-AM335X

    I used the rootfs and kernel from this, but used our DTS which is built on top of am33xx.dtsi from upstream kernel v5.10.166.

    The suspend failed probably because of the RTC clock issue I shared earlier which causes the drivers/soc/ti/pm33xx.c probe failure.

     _____                    _____           _         _   
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  
                  |___|                    |___|            
    
    Arago Project http://arago-project.org am335x-evm ttyS0
    
    Arago 2021.09 am335x-evm ttyS0
    
    am335x-evm login: root
    
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~# uname -a
    Linux am335x-evm 5.10.100-g7a7a3af903 #1 PREEMPT Sat May 14 22:36:58 UTC 2022 armv7l GNU/Linux
    root@am335x-evm:~#
    root@am335x-evm:~# lsmod
    Module                  Size  Used by
    omap_aes_driver        24576  0
    omap_crypto            16384  1 omap_aes_driver
    omap_sham              28672  0
    libaes                 16384  1 omap_aes_driver
    crypto_engine          16384  2 omap_aes_driver,omap_sham
    pm33xx                 16384  0
    rtc_omap               20480  1 pm33xx
    ti_emif_sram           16384  1 pm33xx
    musb_dsps              20480  0
    musb_hdrc             110592  1 musb_dsps
    udc_core               28672  1 musb_hdrc
    usbcore               241664  1 musb_hdrc
    phy_am335x             16384  2
    phy_generic            16384  1 phy_am335x
    usb_common             16384  5 phy_am335x,udc_core,musb_hdrc,musb_dsps,usbcore
    rtc_tps65910           16384  0
    at24                   20480  0
    rtc_ds1307             32768  0
    wkup_m3_ipc            16384  1 pm33xx
    phy_am335x_control     16384  1 phy_am335x
    omap_wdt               16384  0
    wkup_m3_rproc          16384  1
    sch_fq_codel           20480  1
    cryptodev              53248  0
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~# dmesg | grep -iE "remote|ipc|wkup|pm33xx"
    [    9.480554] systemd[1]: Reached target Remote File Systems.
    [   15.201011] remoteproc remoteproc0: wkup_m3 is available
    [   16.983341] remoteproc remoteproc0: powering up wkup_m3
    [   17.197838] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 243276
    [   17.377879] remoteproc remoteproc0: remote processor wkup_m3 is now up
    [   17.377901] wkup_m3_ipc 44e11324.wkup_m3_ipc: CM3 Firmware Version = 0x192
    [   23.577425] pm33xx: probe of pm33xx failed with error -22
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~# echo standby > /sys/power/state                                                                                                              
    [  279.001303] PM: suspend entry (shallow)
    [  279.005382] Filesystems sync: 0.000 seconds
    [  279.009604] PM: suspend exit
    -sh: echo: write error: Operation not permitted
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~# echo mem > /sys/power/state
    [  283.941003] PM: suspend entry (s2idle)
    [  283.944993] Filesystems sync: 0.000 seconds
    [  283.972688] Freezing user space processes ... (elapsed 0.002 seconds) done.
    [  283.982069] OOM killer disabled.
    [  283.985319] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [  283.994116] PM not initialized for pm33xx, wkup_m3_ipc, or am335x-pm-firmware.elf
    [  284.001672] OOM killer enabled.
    [  284.004825] Restarting tasks ... done.
    [  284.040735] PM: suspend exit
    -sh: echo: write error: Invalid argument
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~#
    root@am335x-evm:~# systemctl suspend
    root@am335x-evm:~# [  289.307578] PM: suspend entry (s2idle)
    [  289.314380] Filesystems sync: 0.000 seconds
    [  289.319753] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [  289.328709] OOM killer disabled.
    [  289.332166] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [  289.340900] PM not initialized for pm33xx, wkup_m3_ipc, or am335x-pm-firmware.elf
    [  289.348417] OOM killer enabled.
    [  289.351595] Restarting tasks ... done.
    [  289.358344] PM: suspend exit
    [  289.362337] PM: suspend entry (shallow)
    [  289.366304] Filesystems sync: 0.000 seconds
    [  289.371197] PM: suspend exit
    [  289.374264] PM: suspend entry (s2idle)
    [  289.378092] Filesystems sync: 0.000 seconds
    [  289.384156] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [  289.393053] OOM killer disabled.
    [  289.396297] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [  289.404996] PM not initialized for pm33xx, wkup_m3_ipc, or am335x-pm-firmware.elf
    [  289.412561] OOM killer enabled.
    [  289.415715] Restarting tasks ... done.
    [  289.422494] PM: suspend exit
    
    root@am335x-evm:~# 
    root@am335x-evm:~# dmesg | grep -i "rtc"
    [    1.413972] l4-rtc-clkctrl:0000:0: failed to enable
    [    2.973658] vrtc: supplied by vcc5v
    [    7.469891] l4-rtc-clkctrl:0000:0: failed to disable
    [   16.969033] rtc-ds1307 0-0068: read: 32 22 02 07 01 01 00
    [   16.969060] rtc-ds1307 0-0068: read secs=32, mins=22, hours=2, mday=1, mon=0, year=100, wday=6
    [   16.969456] rtc-ds1307 0-0068: char device (252:0)
    [   16.969488] rtc-ds1307 0-0068: registered as rtc0
    [   17.078827] rtc-ds1307 0-0068: read: 32 22 02 07 01 01 00
    [   17.078865] rtc-ds1307 0-0068: read secs=32, mins=22, hours=2, mday=1, mon=0, year=100, wday=6
    [   17.079072] rtc-ds1307 0-0068: setting system clock to 2000-01-01T02:22:32 UTC (946693352)
    [   17.743375] tps65910-rtc tps65910-rtc: Enabling rtc-tps65910.
    [   17.761098] tps65910-rtc tps65910-rtc: char device (252:1)
    [   17.761126] tps65910-rtc tps65910-rtc: registered as rtc1
    [   33.362503] rtc-ds1307 0-0068: write secs=48, mins=22, hours=2, mday=1, mon=0, year=100, wday=6
    [   33.362534] rtc-ds1307 0-0068: write: 48 22 02 07 01 01 00
    [   33.363312] rtc-ds1307 0-0068: read: 48 22 02 07 01 01 00
    [   33.363337] rtc-ds1307 0-0068: read secs=48, mins=22, hours=2, mday=1, mon=0, year=100, wday=6
    root@am335x-evm:~# 
    

    I fixed this in the past with this change:

    diff --git a/drivers/soc/ti/pm33xx.c b/drivers/soc/ti/pm33xx.c
    index dc21aa855..16455539b 100644
    --- a/drivers/soc/ti/pm33xx.c
    +++ b/drivers/soc/ti/pm33xx.c
    @@ -443,7 +443,7 @@ static int am33xx_pm_rtc_setup(void)
     
     	np = of_find_node_by_name(NULL, "rtc");
     
    -	if (of_device_is_available(np)) {
    +	if (of_device_is_available(np) && 0) {
     		/* RTC interconnect target module clock */
     		rtc_fck = of_clk_get_by_name(np->parent, "fck");
     		if (IS_ERR(rtc_fck))

    I tried removing the rtc node from the dts using the following but the pm33xx.c probe still failed with err -22:

    /delete-node/ &rtc;

    I noticed that the wkup_m3_ipc driver in this 08.02.00.24 release where we have 5.10.100 kernel looks for the wkup_m3_scale_data binary for voltage scaling at suspend/resume. This code does not exist in the 5.10.7 kernel that came in meta-ti 07.03.00.005. This also does not exist in the upstream kernel v5.10.166 that we are using. I wonder there might be other differences as well in the 5.10.100 kernel...

  • Hi Arsalan,

    The TI SDK releases were only validated with the kernel from git.ti.com. It is possible the mainline kernel missing some patches.

  • Hi Bin,

    I was able to test the suspend/resume with the prebuilt binaries (08.02.00.24) from TI on our custom platform. I removed the rtc node from the dts and also renamed the external rtc node from rtc to rtc-ext. Now the pm33xx driver is able to probe/load properly with this msg:

    [   23.720639] PM: no-rtc available, rtc-only mode disabled.

     _____                    _____           _         _   
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_ 
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|  
                  |___|                    |___|            
    
    Arago Project http://arago-project.org am335x-evm ttyS0
    
    Arago 2021.09 am335x-evm ttyS0
    
    am335x-evm login: root
    root@am335x-evm:~# 
    root@am335x-evm:~# 
    root@am335x-evm:~# echo standby > /sys/power/state 
    [ 1581.270070] PM: suspend entry (shallow)
    [ 1581.274147] Filesystems sync: 0.000 seconds
    [ 1581.293654] Freezing user space processes ... (elapsed 0.002 seconds) done.
    [ 1581.303412] OOM killer disabled.
    [ 1581.306656] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [ 1581.444096] cpsw 4a100000.ethernet eth0: Link is Down
    [ 1581.464392] pm33xx pm33xx: PM: Successfully put all powerdomains to target state
    [ 1581.464392] PM: Wakeup source MPU_WAKE
    �[ 1581.477079] cpsw 4a100000.ethernet: initializing cpsw version 1.12 (0)
    
    [ 1581.564337] Micrel KSZ8081 or KSZ8091 4a101000.mdio:05: attached PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=4a101000.mdio:05, irq=POLL)
    
    [ 1581.782144] 8<--- cut here ---
    [ 1581.785287] Unable to handle kernel NULL pointer dereference at virtual address 00000194
    [ 1581.794160] pgd = 605b4467
    [ 1581.796887] [00000194] *pgd=82e5b831, *pte=00000000, *ppte=00000000
    [ 1581.803256] Internal error: Oops: 17 [#1] PREEMPT ARM
    [ 1581.808332] Modules linked in: omap_aes_driver omap_crypto omap_sham libaes crypto_engine pm33xx rtc_omap ti_emif_sram musb_dsps musb_hdrc udc_core phy_am33)
    [ 1581.834885] CPU: 0 PID: 773 Comm: sh Tainted: G           O      5.10.100-g7a7a3af903 #1
    [ 1581.843010] Hardware name: Generic AM33XX (Flattened Device Tree)
    [ 1581.849156] PC is at nvmem_device_get+0x8/0x40
    [ 1581.853621] LR is at devm_nvmem_device_get+0x38/0x70
    [ 1581.858606] pc : [<c08468d4>]    lr : [<c0846944>]    psr: a0000113
    [ 1581.864897] sp : c28dfe28  ip : c2fb8740  fp : c1003040
    [ 1581.870142] r10: c10b10b8  r9 : 00000001  r8 : c28de000
    [ 1581.875387] r7 : 00000000  r6 : 00000000  r5 : bf0a608c  r4 : bf0a608c
    [ 1581.881940] r3 : 00000000  r2 : 00000000  r1 : bf0a608c  r0 : 00000000
    [ 1581.888498] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
    [ 1581.895664] Control: 10c5387d  Table: 82e88019  DAC: 00000051
    [ 1581.901435] Process sh (pid: 773, stack limit = 0x64f049cf)
    [ 1581.907030] Stack: (0xc28dfe28 to 0xc28e0000)
    [ 1581.911409] fe20:                   c2fb8740 c0846944 bf0a7280 00000002 00000000 bf0a50dc
    [ 1581.919626] fe40: 00000000 c1003048 c28de000 c10b10cc 00000002 c01652f4 c28dfe7c 00000001
    [ 1581.927843] fe60: c0d85498 c1003048 c28dfe7c 00000002 00000000 c0d449bc 00000001 c0d85498
    [ 1581.936061] fe80: c10b10ec c2a34f90 00000000 c0165bac 00000007 c292b400 00000002 00000008
    [ 1581.944277] fea0: c0d85498 c016455c 00000008 c292b400 00000000 00000000 c2a34f80 c28dff20
    [ 1581.952494] fec0: c2a34f90 c02d8038 00000000 00000000 c2b263c0 00000000 00000008 00000000
    [ 1581.960711] fee0: c28dff78 00000000 00000000 c0243f24 00000008 c0265748 c2d43180 00000001
    [ 1581.968928] ff00: 00189550 00000008 00000005 00000000 00000000 c28dff08 00000000 00000001
    [ 1581.977145] ff20: c2b263c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000
    [ 1581.985362] ff40: 00000000 00000000 c2d43180 c1003048 00000002 c2b263c0 c2b263c0 00000000
    [ 1581.993579] ff60: 00000000 00189550 00000008 00000004 00000000 c02441ec 00000000 00000000
    [ 1582.001797] ff80: c404d840 c1003048 c2b263c0 00000008 00189550 b6f59230 00000004 c0100264
    [ 1582.010013] ffa0: c28de000 c0100060 00000008 00189550 00000001 00189550 00000008 00000000
    [ 1582.018230] ffc0: 00000008 00189550 b6f59230 00000004 00000008 00000000 00000000 00000000
    [ 1582.026446] ffe0: 00000004 bee36960 44b8d5b7 44b18d16 40070030 00000001 00000000 00000000
    [ 1582.034677] [<c08468d4>] (nvmem_device_get) from [<c0846944>] (devm_nvmem_device_get+0x38/0x70)
    [ 1582.043441] [<c0846944>] (devm_nvmem_device_get) from [<bf0a50dc>] (am33xx_pm_end+0x3c/0xe8 [pm33xx])
    [ 1582.052729] [<bf0a50dc>] (am33xx_pm_end [pm33xx]) from [<c01652f4>] (suspend_devices_and_enter+0x8c/0x620)
    [ 1582.062434] [<c01652f4>] (suspend_devices_and_enter) from [<c0165bac>] (pm_suspend+0x324/0x3a4)
    [ 1582.071173] [<c0165bac>] (pm_suspend) from [<c016455c>] (state_store+0x6c/0xcc)
    [ 1582.078529] [<c016455c>] (state_store) from [<c02d8038>] (kernfs_fop_write_iter+0x10c/0x1c4)
    [ 1582.087019] [<c02d8038>] (kernfs_fop_write_iter) from [<c0243f24>] (vfs_write+0x2d4/0x448)
    [ 1582.095326] [<c0243f24>] (vfs_write) from [<c02441ec>] (ksys_write+0x64/0xec)
    [ 1582.102499] [<c02441ec>] (ksys_write) from [<c0100060>] (ret_fast_syscall+0x0/0x54)
    [ 1582.110187] Exception stack(0xc28dffa8 to 0xc28dfff0)
    [ 1582.115262] ffa0:                   00000008 00189550 00000001 00189550 00000008 00000000
    [ 1582.123479] ffc0: 00000008 00189550 b6f59230 00000004 00000008 00000000 00000000 00000000
    [ 1582.131692] ffe0: 00000004 bee36960 44b8d5b7 44b18d16
    [ 1582.136773] Code: e1a00004 e8bd8010 e92d4010 e1a04001 (e5900194) 
    
    [ 1582.143080] ---[ end trace 98afeb9321e3c7f0 ]---
    
    [ 1584.809393] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control off
    

    The resume failed and I know why that is... it also has to do with the rtc access from the driver i.e.:

    nvmem = devm_nvmem_device_get(&omap_rtc->dev, "omap_rtc_scratch0");

    I fixed that in our upstream kernel v5.10.166 with this change:

    diff --git a/drivers/soc/ti/pm33xx.c b/drivers/soc/ti/pm33xx.c
    index dc21aa855..16455539b 100644
    --- a/drivers/soc/ti/pm33xx.c
    +++ b/drivers/soc/ti/pm33xx.c
    @@ -306,12 +306,12 @@ static void am33xx_pm_end(void)
     	u32 val = 0;
     	struct nvmem_device *nvmem;
     
    -	nvmem = devm_nvmem_device_get(&omap_rtc->dev, "omap_rtc_scratch0");
    -	if (IS_ERR(nvmem))
    -		return;
    -
     	m3_ipc->ops->finish_low_power(m3_ipc);
     	if (rtc_only_idle) {
    +		nvmem = devm_nvmem_device_get(&omap_rtc->dev, "omap_rtc_scratch0");
    +		if (IS_ERR(nvmem))
    +			return;
    +
     		if (retrigger_irq) {
     			/*
     			 * 32 bits of Interrupt Set-Pending correspond to 32

    I have also compared the kernel v5.10.100 from TI SDK release 08.02.00.24 with the upstream kernel v5.10.166. There isn't any major difference expect the wkup_m3_ipc driver in 5.10.100 kernel looks for the wkup_m3_scale_data binary for voltage scaling at suspend/resume. I will try porting changes from that to our kernel, but I doubt that that is the issue here...

    Could you please give us any pointers or things to look for to debug this issue?

  • Hi Arsalan,

    I don't quite follow your message above. Without get into details of my confusion, have you got TI SDK kernel suspend/resume working on your custom board or not?

  • Hi Bin,

    I now have the TI SDK kernel v5.10.100 from meta-ti tag 08.02.00.006 on our custom board.

    The suspend/resume behavior is exactly the same as it was on the upstream kernel v5.10.166 i.e. not working as documented by TI.

    See:

    root@lp-am3354:~# uname -a
    Linux lp-am3354 5.10.100-g7a7a3af903 #1 PREEMPT Thu Aug 31 08:35:36 UTC 2023 armv7l GNU/Linux
    root@lp-am3354:~# 
    
    root@lp-am3354:~# dmesg | grep -iE "remote|ipc|wkup|pm3"
    [   16.513694] remoteproc remoteproc0: wkup_m3 is available
    [   16.838229] remoteproc remoteproc0: powering up wkup_m3
    [   16.944567] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 243436
    [   17.128648] remoteproc remoteproc0: remote processor wkup_m3 is now up
    [   17.128674] wkup_m3_ipc 44e11324.wkup_m3_ipc: CM3 Firmware Version = 0x192
    root@lp-am3354:~# 
    
    root@lp-am3354:~# echo standby > /sys/power/state 
    [   94.339694] PM: suspend entry (shallow)
    [   94.359499] Filesystems sync: 0.015 seconds
    [   94.378379] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [   94.387335] OOM killer disabled.
    [   94.390751] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [   94.523483] cpsw 4a100000.ethernet eth0: Link is Down
    [   94.544004] pm33xx pm33xx: PM: Successfully put all powerdomains to target state
    [   94.544004] PM: Wakeup source MPU_WAKE
    �[   94.556577] cpsw 4a100000.ethernet: initializing cpsw version 1.12 (0)
    [   94.643703] Micrel KSZ8081 or KSZ8091 4a101000.mdio:05: attached PHY driver [Micrel KSZ8081 or KSZ8091] (mii_bus:phy_addr=4a101000.mdio:05, irq=POLL)
    [   94.661431] Micrel KSZ8081 or KSZ8091 4a101000.mdio:05: Micrel KSZ8081 or KSZ8091: resetting PHY on link down
    [   94.871307] OOM killer enabled.
    [   94.874558] Restarting tasks ... done.
    [   94.958914] PM: suspend exit
    root@lp-am3354:~# 
    
    root@lp-am3354:~# echo mem > /sys/power/state
    [  110.655139] PM: suspend entry (deep)
    [  110.665906] Filesystems sync: 0.006 seconds
    [  110.681142] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [  110.690284] OOM killer disabled.
    [  110.693527] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [  110.823224] cpsw 4a100000.ethernet eth0: Link is Down
    

    The suspend works in both standby and mem modes, but resume does not.

    1. When we add no_console_suspend to the kernel cmdline args:
    - The resume from standby mode works using the uart0. (although it shouldn't as per the documentation provided by TI)
    - The resume from mem mode DOES NOT work.

    2. When we DO NOT add no_console_suspend to the kernel cmdline args:
    - The resume from standby mode DOES NOT work.
    - The resume from mem mode DOES NOT work.

    This is exactly what we saw in upstream kernel v5.10.166 as well.

  • Hi Arsalan,

    Thanks for the details, now I understand the problems.

    First thing would be to try the use case on any AM335x EVM with the prebuilt image in SDK8.2, to see if suspend/resume has any issue. I could do this, but I am currently working on a critical task, and likely won't be able to get on this in at least a week.

    On your platform, I think you would have to connect to JTAG to understand where it got stuck, then debug from there.