• State
  • Locked Locked
  • Replies 9 replies
  • Subscribers 99 subscribers
  • Views 1595 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM335x cannot wake up from suspend on kernel 3.14

Bruno Morais
Prodigy 80 points
Other Parts Discussed in Thread: AM3354

Hello. I'm using a system on module device that contains the AM3354 processor. Unfortunately it seems that even using a kernel based on the TI SDK 8.0, it is impossible for me to recover from a suspend.

The CM3 firmware is properly loaded as can be seen in the system messages. The processor is unable to recover from an attempted suspend to ram OR standby issued to the sysfs power subsystem. Interestingly, NO events at all are able to wake up the processor, including events from the wake up domain -- GPIO0 , UART0.

This setup works fine on the kernel version 3.12 from previous SDK version, where I am able to suspend the system and bring it up again by means of any of the peripherals that are part of the wake up domain. I have tried various combinations, including different versions of the CM3 firmware, which produces the different outputs seen:

0x191 - unusable, fails

[ 2.192684] remoteproc0: wkup_m3 is available
[ 2.197396] remoteproc0: Note: remoteproc is still under development and considered experimental.
[ 2.206867] remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
[ 2.217953] remoteproc0: Direct firmware load failed with error -2
[ 2.224578] remoteproc0: Falling back to user helper
[ 2.878818] remoteproc0: powering up wkup_m3
[ 2.899980] remoteproc0: Booting fw image am335x-pm-firmware.elf, size 219663
[ 2.907641] remoteproc0: erroneous trace resource entry
[ 2.913298] remoteproc0: Failed to process resources: -22

0x190 - firmware loaded but cant resume

[ 2.192743] remoteproc0: wkup_m3 is available
[ 2.197458] remoteproc0: Note: remoteproc is still under development and considered experimental.
[ 2.206933] remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
[ 2.218014] remoteproc0: Direct firmware load failed with error -2
[ 2.224635] remoteproc0: Falling back to user helper
[ 3.142441] remoteproc0: powering up wkup_m3
[ 3.148058] remoteproc0: Booting fw image am335x-pm-firmware.elf, size 219827
[ 3.342381] remoteproc0: remote processor wkup_m3 is now up

0x189 - firmware loaded but unsupported

[ 2.202770] remoteproc0: wkup_m3 is available
[ 2.207482] remoteproc0: Note: remoteproc is still under development and considered experimental.
[ 2.216956] remoteproc0: THE BINARY FORMAT IS NOT YET FINALIZED, and backward compatibility isn't yet guaranteed.
[ 2.228040] remoteproc0: Direct firmware load failed with error -2
[ 2.234662] remoteproc0: Falling back to user helper
[ 2.883080] remoteproc0: powering up wkup_m3
[ 2.908671] remoteproc0: Booting fw image am335x-pm-firmware.elf, size 154768
[ 3.018387] remoteproc0: remote processor wkup_m3 is now up

(error)
[ 3.012994] PM: CM3 Firmware Version 189 not supported

Why I am not able to wake from a suspend attempt with kernel 3.14? I have also tried to use the kernel pm_test infrastructure to try and debug the suspending mechanisms, but all tests succeed without errors. I have also tried to do a RTC timed wakeup but nothing happens.

I've spent several hours researching every possible factor that could explain this behavior but I simply can't reach any conclusions. I absolutely would appreciate any input on this. 

  • 0
    TI__Guru**** 393215 points

    Hi Bruno,

    Can you please post the part of the log where you put the system in low-power mode? You can also check this presentation for debugging tips:

    sitara_boot_camp_am335x_pm_sw_training.pdf

  • 0 in reply to Biser Gatchev-XID
    Prodigy 80 points

    Hi, Biser, thank you for your reply.

    The log is mostly uninteresting, as it doesn't say much:

    [ 26.763980] PM: Syncing filesystems ... done.
    [ 26.823373] Freezing user space processes ... (elapsed 0.001 seconds) done.
    [ 26.832207] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
    [ 26.843889] wlcore: down
    [ 26.883843] PM: suspend of devices complete after 40.084 msecs
    [ 26.892468] PM: late suspend of devices complete after 2.374 msecs
    [ 26.901695] PM: noirq suspend of devices complete after 2.642 msecs

    Both "mem" and "standby" modes lead to this. Note: this is using no_console_suspend=1 in the kernel boot cmd, in the hope that more would be output, but to no luck.

    Also, I had already looked at the presentation you referred, but I was hoping someone would be able to have advice, before I start trying to go to that level of manual debugging, which will involve several kernel compilations and installations and is a very time consuming process.

    As it is right now I have no idea if the system is ever suspending or hanging while suspending OR hanging while trying to wake up. JTAG is not available at all to me.

  • 0 in reply to Biser Gatchev-XID
    Prodigy 80 points
    I have conducted more tests. It appears that the system hangs on wake up, not suspend. I have measured the power consumption while trying to suspend, and it is greatly reduced. When I try to use a GPIO to wake up the processor, the power consumption increases, suggesting that it is out of a low power sleep mode.

    I have also inserted various messages in the kernel source to trace the location where the processor hangs in code, but it led me to a low level assembly function of which I can't extract much data, and I am not very familiar with ARM assembly code so it is a bit cryptic.

    I still have no clue at all to why this functionality is broken.
  • 0 in reply to Bruno Morais
    TI__Guru**** 393215 points

    Hi Bruno,

    I have asked the power management experts to help on this, but it may take some time - seems they are quite busy right now. Anyway, thanks for updating. Any additional information will help locating the issue.

  • 0 in reply to Biser Gatchev-XID
    TI__Genius 15510 points

    Can you post more about the low-level assembly instructions?

    Steve K.

  • 0 in reply to Steve Kipisz
    Prodigy 80 points
    Hi, what would you like to know? I can trace the code up to a point where it enters a suspend function in the file sleep.S for the arm architecture in the kernel source.
    From there I am a bit confused to where it goes because I am not very familiar with ARM assembly and it seems to me that it branches to other places in code in a way I can't parse.
  • 0 in reply to Steve Kipisz
    Prodigy 80 points
    I've continued trying to debug this today. I can get to the point where the WFI process is done, but debugging the inside this function seems to be unfeasible and it is very unclear to me where is the return (wake up) entry point to see if I can extract any data pertaining to where would the CPU be in code, to try to debug it further.

    As it is right now, I have no idea at all on what could be breaking the wake up mechanism, and I'm sure there must be something very wrong going on --- this doesn't appear to be module-related at all.
  • 0 in reply to Bruno Morais
    Prodigy 10 points
    Hello,
    Did you manage to find a solution. I have exactly the same problem with AM3354 and kernel 3.14.26. I get the same messages as you and suspend/wakeup works fine on kernel 3.12 (M3 fw 0x186), but on kernel 3.14 (M3 fw 0x190) it seems to hang on wakeup (judging by power consumption).
  • 0 in reply to Ignas Jarusevicius41
    Prodigy 80 points
    Hi, no luck at all, and everyone else seems to say (mostly beaglebone) it works perfectly, so I don't really know what could be doing wrong.

    There are several differences from 3.12 to 3.14 so it's just hard / impossible to debug without having some kind of knowledge as I believe TI engineers must have.