• State
  • Locked Locked
  • Replies 3 replies
  • Subscribers 98 subscribers
  • Views 206 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5728: Kernel Panic On Reboot

Nathan Wright
Intellectual 782 points

Part Number: AM5728
Other Parts Discussed in Thread: TPD12S015

Tool/software:

Hello all,

We are using Arago to build an image for the AM57xx-evm.  We are getting a kernel panic on reboot which was halting the system.  We've worked around that with the reboot-on-panic kernel config option.  I am not sure how to proceed with root causing the panic though. 

Perhaps someone here can make heads or tail of this?

Broadcast message from root@am57xx-evm-ep00075 (Thu 2025-01-23 14:53:04 UTC):

The system will reboot now!

         Stopping [  OK  ] Stopped target Multi-User System.
[  OK  ] Stopped target Login Prompts.
[  OK  ] Stopped target Host and Network Name Lookups.
[  OK  ] Stopped target RPC Port Mapper.
[  OK  ] Stopped target System Time Set.
[  OK  ] Stopped target Timer Units.
[  OK  ] Stopped Daily Cleanup of Temporary Directories.
[  OK  ] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
         Stopping Avahi mDNS/DNS-SD Stack...
         Stopping containerd container runtime...
         Stopping Netperf Benchmark Server...
         Stopping Serial Getty on ttyS2...
         Stopping SWUpdate daemon...
         Stopping Load/Save OS Random Seed...
         Stopping Telnet Server...
[  OK  ] Stopped Avahi mDNS/DNS-SD Stack.
[  OK  ] Stopped SWUpdate daemon.
[  OK  ] Stopped Netperf Benchmark Server.
[  OK  ] Stopped containerd container runtime.
[  OK  ] Stopped Serial Getty on ttyS2.
[  OK  ] Stopped Load/Save OS Random Seed.
[  OK  ] Stopped Telnet Server.
[  OK  ] Stopped Session c1 of User root.
[  OK  ] Removed slice Slice /system/serial-getty.
         Stopping User Login Management...
         Stopping Permit User Sessions...
         Stopping User Manager for UID 0...
[  OK  ] Stopped Permit User Sessions.
[  OK  ] Stopped User Manager for UID 0.
[  OK  ] Stopped target Network.
[  OK  ] Stopped target Remote File Systems.
         Stopping Network Configuration...
         Stopping User Runtime Directory /run/user/0...
[  OK  ] Unmounted /run/user/0.
[  OK  ] Stopped User Login Management.
[  OK  ] Stopped Network Configuration.
[  OK  ] Stopped User Runtime Directory /run/user/0.
[  OK  ] Removed slice User Slice of UID 0.
[  OK  ] Stopped target Preparation for Network.
[  OK  ] Stopped IPv6 Packet Filtering Framework.
[  OK  ] Stopped IPv4 Packet Filtering Framework.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Path Units.
[  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
[  OK  ] Stopped Forward Password Requests to Wall Directory Watch.
[  OK  ] Stopped target Slice Units.
[  OK  ] Removed slice User and Session Slice.
[  OK  ] Stopped target Socket Units.
[  OK  ] Closed Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Closed Docker Socket for the API.
[  OK  ] Closed dropbear.socket.
[  OK  ] Closed SWUpdate socket listener.
[  OK  ] Closed Network Service Netlink Socket.
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped Generate network units from Kernel command line.
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Closed D-Bus System Message Bus Socket.
[  OK  ] Stopped target System Initialization.
         Stopping Network Name Resolution...
         Stopping Network Time Synchronization...
         Stopping Record System Boot/Shutdown in UTMP...
[  OK  ] Stopped Network Name Resolution.
[  OK  ] Stopped Network Time Synchronization.
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Closed Process Core Dump Socket.
[  OK  ] Stopped Load Kernel Modules.
[  OK  ] Stopped Record System Boot/Shutdown in UTMP.
[  OK  ] Stopped Create Volatile Files and Directories.
[  OK  ] Stopped target Local File Systems.
         Unmounting /media/ram...
         Unmounting /run/media/roota-mmcblk1p1...
[64301.246429] EXT4-fs (mmcblk1p1): unmounting filesystem 6b116e6e-5126-4b18-88d0-8e7ceffb07c6.

         Unmounting Temporary Directory /tmp...
[64301.294799] EXT4-fs (mmcblk1p3): unmounting filesystem 9d4e835f-dede-4c92-b42f-5144bb3c5627.
         Unmounting /var/volatile...
[  OK  ] Unmounted /media/ram.
[  OK  ] Unmounted /run/media/secret-mmcblk1p3.
[  OK  ] Unmounted Temporary Directory /tmp.
[  OK  ] Unmounted /var/volatile.
[  OK  ] Stopped target Swaps.
[  OK  ] Stopped File System Check on /dev/mmcblk1p3.
[  OK  ] Unmounted /run/media/roota-mmcblk1p1.
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Stopped File System Check on /dev/mmcblk1p1.
[  OK  ] Removed slice Slice /system/systemd-fsck.
[  OK  ] Stopped target Preparation for Local File Systems.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create Static Device Nodes in /dev gracefully.
[  OK  ] Reached target System Shutdown.
[  OK  ] Reached target Late Shutdown Services.
[  OK  ] Finished System Reboot.
[  OK  ] Reached target System Reboot.
[64302.072509] watchdog: watchdog0: watchdog did not stop!
[64302.090728] systemd-shutdown[1]: Using hardware watchdog 'OMAP Watchdog', version 0, device /dev/watchdog0
[64302.101745] systemd-shutdown[1]: Watchdog running with a timeout of 1min.
[64302.117645] systemd-shutdown[1]: Syncing filesystems and block devices.
[64302.129089] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[64302.148437] systemd-journald[109]: Received SIGTERM from PID 1 (systemd-shutdow).
[64302.188354] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[64302.204132] systemd-shutdown[1]: Unmounting file systems.
[64302.212463] (sd-remount)[1798]: Remounting '/' read-only with options ''.
[64302.264099] EXT4-fs (mmcblk1p2): re-mounted c6c50df7-ecc3-4794-ac1d-99489eebc2de ro. Quota mode: disabled.
[64302.280761] systemd-shutdown[1]: All filesystems unmounted.
[64302.286437] systemd-shutdown[1]: Deactivating swaps.
[64302.291534] systemd-shutdown[1]: All swaps deactivated.
[64302.296844] systemd-shutdown[1]: Detaching loop devices.
[64302.308654] systemd-shutdown[1]: All loop devices detached.
[64302.314270] systemd-shutdown[1]: Stopping MD devices.
[64302.320190] systemd-shutdown[1]: All MD devices stopped.
[64302.325592] systemd-shutdown[1]: Detaching DM devices.
[64302.331054] systemd-shutdown[1]: All DM devices detached.
[64302.336547] systemd-shutdown[1]: All filesystems, swaps, loop devices, MD devices and DM devices detached.
[64302.349212] systemd-shutdown[1]: Syncing filesystems and block devices.
[64302.356109] systemd-shutdown[1]: Rebooting.
[64302.482482] 8<--- cut here ---
[64302.485565] Unable to handle kernel NULL pointer dereference at virtual address 0000008c when read
[64302.494567] [0000008c] *pgd=fb067835
[64302.498168] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
[64302.503601] Modules linked in: sha256_generic libsha256 sha256_arm cfg80211 pwm_fan onboard_usb_hub pwm_tiehrpwm display_connector ti_tpd12s015 omap_remoteproc cryptodev(O)
[64302.519104] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: G           O       6.6.32-ti-01287-g6de6e418c80e-dirty #1
[64302.529602] Hardware name: Generic DRA74X (Flattened Device Tree)
[64302.535736] PC is at pci_generic_config_read+0x44/0x84
[64302.540893] LR is at pci_generic_config_read+0x14/0x84
[64302.546081] pc : [<c0609a80>]    lr : [<c0609a50>]    psr: 60030093
[64302.552368] sp : f000dd10  ip : f000dd20  fp : 000180b0
[64302.557617] r10: c189f3ec  r9 : 00000008  r8 : 00000000
[64302.562866] r7 : f000dd5a  r6 : c1be9c00  r5 : 40030093  r4 : 00000002
[64302.569427] r3 : c189e868  r2 : 0000008c  r1 : 00000000  r0 : 0000008c
[64302.575988] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[64302.583251] Control: 10c5387d  Table: aa8a406a  DAC: 00000051
[64302.589019] Register r0 information: non-paged memory
[64302.594085] Register r1 information: NULL pointer
[64302.598846] Register r2 information: non-paged memory
[64302.603912] Register r3 information: slab kmalloc-2k start c189e800 pointer offset 104 size 2048
[64302.612762] Register r4 information: non-paged memory
[64302.617828] Register r5 information: non-paged memory
[64302.622924] Register r6 information: slab kmalloc-1k start c1be9c00 pointer offset 0 size 1024
[64302.631591] Register r7 information: 2-page vmalloc region starting at 0xf000c000 allocated at kernel_clone+0xa4/0x3bc
[64302.642364] Register r8 information: NULL pointer
[64302.647094] Register r9 information: non-paged memory
[64302.652160] Register r10 information: slab kmalloc-2k start c189f000 pointer offset 1004 size 2048
[64302.661193] Register r11 information: non-paged memory
[64302.666351] Register r12 information: 2-page vmalloc region starting at 0xf000c000 allocated at kernel_clone+0xa4/0x3bc
[64302.677185] Process systemd-shutdow (pid: 1, stack limit = 0x1bb02f03)
[64302.683746] Stack: (0xf000dd10 to 0xf000e000)
[64302.688140] dd00:                                     c0609a3c c06097ec f000dd20 c01af080
[64302.696350] dd20: 00000000 d3280dd7 c2a75c60 0000001c c189f000 f000dd5a 00000000 c060a640
[64302.704589] dd40: c2a7fb80 c189f000 0000001c 60030093 00000000 c060a854 00005480 d3280dd7
[64302.712799] dd60: c2a7fb80 c1926000 c189f000 c1926054 00000044 c189f080 00000000 c0628540
[64302.721008] dd80: c1926010 00000000 c12fdee8 c06252f4 c1926010 c076d010 c1b45530 c1926010
[64302.729248] dda0: c1b4550c c1b45538 c1b45540 c076b60c c1926010 c1926054 00000000 c13b46e0
[64302.737457] ddc0: c1930000 c07669f8 eed11a94 c0cc9f70 c1bdc8b8 eed11a80 00000001 c01c53cc
[64302.745666] dde0: c1bdc8b8 d3280dd7 eed11a80 c1926010 00000000 c06253c8 c13b46e0 c189f0c4
[64302.753906] de00: c1373014 00000058 000180b0 c0766d18 00000000 c06253ec 00000000 c0764cc4
[64302.762115] de20: 000180b0 c2a47c00 c2a2bc14 d3280dd7 c189f000 c189f080 c189f000 c0625938
[64302.770324] de40: c189f080 c12fdd64 c189f000 c061845c c189f084 c1bea000 c189f080 c076a06c
[64302.778533] de60: 00000000 c121ef18 c121eb90 fee1dead 00000000 c1930000 00000058 c0159354
[64302.786773] de80: 01234567 c0159648 f000dee8 00000000 00000000 00000000 00000000 00000000
[64302.794982] dea0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64302.803192] dec0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64302.811431] dee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64302.819641] df00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64302.827850] df20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64302.836090] df40: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64302.844299] df60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[64302.852508] df80: 00000000 00000000 00000000 d3280dd7 00000000 00000000 00000000 00000058
[64302.860717] dfa0: c01002c4 c0100060 00000000 00000000 fee1dead 28121969 01234567 00000000
[64302.868957] dfc0: 00000000 00000000 00000000 00000058 000187bc 00000003 00000000 000180b0
[64302.877166] dfe0: 00000058 bea22c24 b6cd8fd5 b6c515e6 60030030 fee1dead 00000000 00000000
[64302.885375]  pci_generic_config_read from pci_bus_read_config_word+0x6c/0xb4
[64302.892486]  pci_bus_read_config_word from pcie_capability_read_word+0x90/0xb0
[64302.899780]  pcie_capability_read_word from pcie_capability_clear_and_set_word_locked+0x48/0xa0
[64302.908538]  pcie_capability_clear_and_set_word_locked from pcie_pme_remove+0x2c/0x64
[64302.916412]  pcie_pme_remove from pcie_port_remove_service+0x34/0x44
[64302.922821]  pcie_port_remove_service from device_release_driver_internal+0x184/0x1f8
[64302.930694]  device_release_driver_internal from bus_remove_device+0xc0/0xe4
[64302.937805]  bus_remove_device from device_del+0x14c/0x460
[64302.943328]  device_del from device_unregister+0xc/0x20
[64302.948577]  device_unregister from remove_iter+0x24/0x2c
[64302.954010]  remove_iter from device_for_each_child+0x64/0xa4
[64302.959808]  device_for_each_child from pcie_portdrv_shutdown+0x2c/0x6c
[64302.966461]  pcie_portdrv_shutdown from pci_device_shutdown+0x34/0x60
[64302.972930]  pci_device_shutdown from device_shutdown+0x164/0x23c
[64302.979064]  device_shutdown from kernel_restart+0x38/0x90
[64302.984588]  kernel_restart from __do_sys_reboot+0x164/0x210
[64302.990264]  __do_sys_reboot from ret_fast_syscall+0x0/0x54
[64302.995880] Exception stack(0xf000dfa8 to 0xf000dff0)
[64303.000946] dfa0:                   00000000 00000000 fee1dead 28121969 01234567 00000000
[64303.009185] dfc0: 00000000 00000000 00000000 00000058 000187bc 00000003 00000000 000180b0
[64303.017395] dfe0: 00000058 bea22c24 b6cd8fd5 b6c515e6
[64303.022460] Code: e59d2008 e3a00000 e5823000 e8bd8010 (e1d030b0)
[64303.028594] ---[ end trace 0000000000000000 ]---
[64303.033233] note: systemd-shutdow[1] exited with irqs disabled
[64303.039154] note: systemd-shutdow[1] exited with preempt_count 3
[64303.045196] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[64303.052886] Rebooting in 1 seconds..

  • 0
    TI__Mastermind 36266 points

    Hello Nathan,

    Can you share some more details please?

    • What SDK version are you using?
    • Are you using a TI EVM or is it a custom board?
    • Can you confirm what kernel version you are using?

    I believe I see that you are using kernel  6.6.32-ti-01287-g6de6e418c80e-dirty. Can you confirm?

    Unfortunately we have not validated Linux kernel beyond 6.1 so we cannot be of any help.

    -Josue

  • 0 in reply to Josue Zamitiz-Ayala
    Intellectual 782 points

    Sure, this is a AM5728-GP-EVM, aka the BeagleBoard X-15 sitting on a custom carrier board.

    We are also duplicating the design for an AM62X platform from a partner.  We are using their config, (minus their layer), and trying to bring up the am57x board.

    bitbake,https://git.openembedded.org/bitbake,2.8,11d83170922a2c6b9db1f6e8c23e533526984b2c
meta-tisdk,https://github.com/TexasInstruments/meta-tisdk.git,scarthgap,10.00.07.04,layers=
meta-arago,https://git.yoctoproject.org/meta-arago,scarthgap,10.00.07,layers=meta-arago-distro:meta-arago-extras:meta-arago-demos:meta-arago-test
meta-qt5,https://github.com/meta-qt5/meta-qt5.git,scarthgap,eb828418264a49b8d00035cb3d7b12fcea3be801,layers=
meta-virtualization,https://git.yoctoproject.org/meta-virtualization,scarthgap,6a80f140e387621f62964209a2e07d3bcfb125ce,layers=
meta-openembedded,https://git.openembedded.org/meta-openembedded,scarthgap,18f939a5fb37528a5415b05077ece383c346a119,layers=meta-networking:meta-python:meta-oe:meta-gnome:meta-filesystems:meta-multimedia
meta-ti,https://git.yoctoproject.org/meta-ti,scarthgap,10.00.07,layers=meta-ti-extras:meta-ti-bsp
meta-arm,https://git.yoctoproject.org/meta-arm,scarthgap,1b85bbb4cab9658da3cd926c62038b8559c5c64e,layers=meta-arm:meta-arm-toolchain
meta-clang,https://github.com/kraj/meta-clang,scarthgap,df21b1563910c80d7e2964971b7c5b79b5186922,layers=
oe-core,https://git.openembedded.org/openembedded-core,scarthgap,c00a9a8f25b0042b0723a5d20b88e92d473d22bf,layers=meta

  • 0 in reply to Nathan Wright
    TI__Mastermind 36266 points

    Nathan,

    I will reiterate that this Linux version is not officially supported. One think you could try is to try to build with the most stable configuration, which after conferring with our local Yocto application engineer is the following: 
    https://git.ti.com/cgit/arago-project/oe-layersetup/tree/configs/arago-scarthgap-config.txt

    You can try building with this one or, based on the oops logs, maybe just disable the PCIe and see if that keeps you from seeing the kernel panic.

    -Josue