• State TI Thinks Resolved
  • Locked Locked
  • Replies 10 replies
  • Answers 1 answer
  • Subscribers 100 subscribers
  • Views 3274 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728: Shutdown kernel panic

Kevin Shia
Prodigy 140 points
Part Number: AM5728

Hello, 

We currently have a board that is custom designed based on the AM5728 IDK. We wanted to have the board to have an automatic power-up when it is powered, so I followed the recommendations in this forum. I also posted some questions through this forum as well.

https://e2e.ti.com/support/power_management/pmu/f/200/t/555645 

We noticed that if we try to shutdown, it cases a kernel panic and I am wondering if the PMIC change has anything to do with this change. 

Regards,

Kevin

  • 0
    TI__Guru**** 393215 points
    Hi,

    How is the PWRHOLD signal connected?
  • 0 in reply to Biser Gatchev-XID
    Prodigy 140 points

    Hey Biser, 

    The POWERHOLD (GPIO_7) is currently pulled up to VRTC_OUT. 

    Regards,

    Kevin

  • 0 in reply to Kevin Shia
    TI__Guru**** 161600 points
    Hi Kevin,

    Can you share the dmesg log, where I can see the kernel panic & inspect the corresponding kernel sources?

    Best Regards,
    Yordan
  • 0 in reply to Yordan Kovachev
    Intellectual 280 points

    Hi Yordan,


    I work with Kevin, and these are the messages we see when a poweroff causes a panic:

    [  OK  ] Stopped target Sound Card.
             Stopping Save/Restore Sound Card State...
             Stopping Session c1 of user root.
    [  OK  ] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.         Stopping LSB: Start busybox udhcpd at boot time...
             Stopping LSB: Tool to automatically...ubmit kernel crash signatures...
             Stopping LSB: Start NTP daemon...
             Stopping Deferred execution scheduler...
             Stopping D-Bus System Message Bus...
             Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"...
             Stopping LSB: Advanced IEEE 802.11 management daemon...
             Stopping LSB: set CPUFreq kernel parameters...
             Stopping Zabbix Agent...
             Stopping LSB: Speech Dispatcher...
             Stopping OpenBSD Secure Shell server...
             Stopping LSB: automatic crash report generation...
             Stopping ACPI event daemon...
    [  OK  ] Stopped target Login Prompts.
             Stopping Getty on tty1...
             Stopping Serial Getty on ttyS0...
    [  OK  ] Stopped Deferred execution scheduler.
    [  OK  ] Stopped System Logging Service.
    [  OK  ] Stopped ACPI event daemon.
    [  OK  ] Stopped Cgroup management daemon.
    [  OK  ] Stopped Regular background program processing daemon.
    [  OK  ] Stopped OpenBSD Secure Shell server.
    [  OK  ] Stopped Serial Getty on ttyS0.
    [  OK  ] Stopped Getty on tty1.
    [  OK  ] Stopped User Manager for UID 0.
    [  OK  ] Stopped Zabbix Agent.
    [  OK  ] Stopped Save/Restore Sound Card State.
    [  OK  ] Stopped Session c1 of user root.
    [  OK  ] Stopped Session c2 of user root.
    [  OK  ] Stopped Session c3 of user root.
    [  OK  ] Stopped LSB: Tool to automatically ... submit kernel crash signatures.
    [  OK  ] Stopped D-Bus System Message Bus.
    [  OK  ] Stopped LSB: Advanced IEEE 802.11 management daemon.
    [  OK  ] Stopped LSB: Speech Dispatcher.
    [  OK  ] Stopped LSB: Start NTP daemon.
    [  OK  ] Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand".
    [  OK  ] Stopped LSB: set CPUFreq kernel parameters.
    [  OK  ] Stopped LSB: automatic crash report generation.
             Stopping LSB: Load kernel modules needed to enable cpufreq scaling...
    [  OK  ] Stopped User Manager for UID 0.
    [  OK  ] Stopped target System Time Synchronized.
             Stopping Figure4 Print Daemon...
    [  OK  ] Removed slice User Slice of root.
             Stopping Login Service...
    [  OK  ] Removed slice system-getty.slice.
             Stopping Permit User Sessions...
    [  OK  ] Removed slice system-serial\x2dgetty.slice.
    [  OK  ] Stopped /etc/rc.local Compatibility.
    [  OK  ] Stopped target Network is Online.
    [  OK  ] Stopped Login Service.
    [  OK  ] Stopped Figure4 Print Daemon.
    [  OK  ] Stopped Permit User Sessions.
    [  OK  ] Stopped LSB: Load kernel modules needed to enable cpufreq scaling.
             Stopping Parse journald log and push to zabbix trapper...
    [  OK  ] Stopped Parse journald log and push to zabbix trapper.
    [  OK  ] Stopped target Network.
             Stopping Raise network interfaces...
    [  OK  ] Stopped LSB: Start busybox udhcpd at boot time.
    [  OK  ] Stopped target Basic System.
    [  OK  ] Stopped target Slices.
    [  OK  ] Removed slice User and Session Slice.
    [  OK  ] Stopped target Sockets.
    [  OK  ] Closed Syslog Socket.
    [  OK  ] Closed ACPID Listen Socket.
    [  OK  ] Closed D-Bus System Message Bus Socket.
    [  OK  ] Stopped target Paths.
    [  OK  ] Stopped Forward Password Requests to Wall Directory Watch.
    [  OK  ] Stopped ACPI Events Check.
    [  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
    [  OK  ] Stopped target Remote File Systems.
    [  OK  ] Stopped target Remote File Systems (Pre).
    [  OK  ] Stopped target System Initialization.
             Stopping Load/Save Random Seed...
    [  OK  ] Stopped target Encrypted Volumes.
    [  OK  ] Stopped Create Volatile Files and Directories.
    [  OK  ] Stopped target Swap.
    [  OK  ] Stopped Load/Save Random Seed.
    [  OK  ] Stopped Raise network interfaces.
    [  OK  ] Stopped target Local File Systems.
             Unmounting /run/user/0...
             Unmounting /run/cgmanager/fs...
    [  OK  ] Stopped Apply Kernel Variables.
    [  OK  ] Stopped target Network (Pre).
    [  OK  ] Unmounted /run/user/0.
    [  OK  ] Unmounted /run/cgmanager/fs.
    [  OK  ] Reached target Unmount All Filesystems.
    [  OK  ] Stopped target Local File Systems (Pre).
    [  OK  ] Stopped Create Static Device Nodes in /dev.
    [  OK  ] Stopped Remount Root and Kernel File Systems.
    [  OK  ] Reached target Shutdown.
    [ 4914.198026] reboot: Power down
    [ 4914.202910] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
    [ 4914.202910]
    [ 4914.226530] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
    [ 4914.226530]

  • 0 in reply to Heather Rysenga
    Intellectual 280 points
    Is this enough information? The dmesg file gets wiped on each power up, but is there a way to have it persist?
  • 0 in reply to Yordan Kovachev
    Prodigy 140 points

    Hey Yordan, 

    Is the information from Heather sufficient? Please let us know if you need additional information.

    Regards,

    Kevin

  • 0 in reply to Kevin Shia
    TI__Guru**** 161600 points
    Hi Kevin,

    Sorry for the delay.

    From the log I see, that the kernel appears to recover from this panic, is this correct? Are you able to poweroff the board?
    How are you executing the shutdown on the board? Do you use poweroff, halt or anything else as a command?
    Do you see a problem in the suspend/resume mechanism?

    Best Regards,
    Yordan
  • 0 in reply to Yordan Kovachev
    Intellectual 280 points

    Hi Yordan,


    The log I included is what was printed out right after I sent a poweroff command, not when we were booting up. It seems to crash before it completes the shutdown process. We haven't tried suspending or resuming the board.

  • 0 in reply to Heather Rysenga
    TI__Guru**** 161600 points

    Hi, 

    Since I cannot reproduce this, could you please add debug prints in the following functions:  

    kernel/reboot.c:  

    void kernel_power_off(void)
    {
            kernel_shutdown_prepare(SYSTEM_POWER_OFF);
            if (pm_power_off_prepare)

                 ==> Add debug print here. 
                                pm_power_off_prepare();

            ==> Add debug print here
            migrate_to_reboot_cpu();

            ==> Add debug print here
            syscore_shutdown();                   ===> trace the original function (drivers/base/syscore.c) and add debug print in it. 
            pr_emerg("Power down\n");
            kmsg_dump(KMSG_DUMP_POWEROFF);
            machine_power_off();                 ===> trace the original function (arch/arm/kernel/reboot.c) and add debug print in it. 
    }

    arch/arm/kernel/reboot.c: 

    void machine_power_off(void)
    {

         ==> Add debug print here
         local_irq_disable();

          ==> Add debug print here
          smp_send_stop();

         ==> Add debug print here

          if (pm_power_off)
                 pm_power_off(); 

          ==> Add debug print here
    }

    drivers/base/syscore.c: 

    void syscore_shutdown(void)
    {
          struct syscore_ops *ops;

           ==> Add debug print here

          mutex_lock(&syscore_ops_lock);

          list_for_each_entry_reverse(ops, &syscore_ops_list, node)
          if (ops->shutdown) {

                     ==> Add debug print here
                   if (initcall_debug)
                          pr_info("PM: Calling %pF\n", ops->shutdown);
                          ops->shutdown();
          }

           ==> Add debug print here

          mutex_unlock(&syscore_ops_lock);

    }

    Hopefully this will provide more verbose output and help us track exactly at which point the kernel panics. 

    Best Regards, 
    Yordan

  • 0 in reply to Yordan Kovachev
    Intellectual 280 points

    The issue seems to happen right when the kernel_power_off function is exited. Here are my debug print statements:

    [  OK  ] Reached target Shutdown.
    [  202.845679] reboot: Before migrate_to_reboot_cpu()
    [  202.850552] syscore_shutdown: Before mutex_lock
    [  202.855101] syscore_shutdown: Before ops->shutdown
    [  202.859934] syscore_shutdown: Before ops->shutdown
    [  202.864744] syscore_shutdown: Before ops->shutdown
    [  202.870628] syscore_shutdown: Before mutex_unlock
    [  202.875354] reboot: Power down
    [  202.878439] machine_power_off: Before local_irq_disable
    [  202.883685] machine_power_off: Before smp_send_stop
    [  202.888584] machine_power_off: Before pm_power_off
    [  202.893391] machine_power_off: Calling pm_power_off
    [  202.898552] machine_power_off: After pm_power_off
    [  202.903274] reboot: Exiting kernel_power_off
    [  202.909154] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
    [  202.909154]
    [  202.918334] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000000
    [  202.918334]

    Is kernel_power_off the last function that runs before the system is powered off? When is the PMIC told to power off components?