Other Parts Discussed in Thread: SK-AM64B
Hi,
I want to use KGDB to debug linux kernel, and after some days work, there are still a problem I couldn't solve.
The hardware ,software(SDK, kernel, etc), kernel options, kernel params and logs which I used will be mention at last.
Problem:
For using KGDB, I added some kernel configure options and rebuild the kernel.
Then, I can start KGDB after Linux has been started like this, and connect to target board by GDB from host ubuntu PC.
$ echo ttyS2 > /sys/module/kgdboc/parameters/kgdboc
$ echo g > /proc/sysrq-trigger
But when I using kernel params to start KGDB at kernel booting time, a kernel panic has been happend.
For some days debug, I know that kgdb_register_io_module() -> kgdb_initial_breakpoint() -> kgdb_breakpoint() -> arch_kgdb_breakpoint() has been called,
and kernel panic happend in arch_kgdb_breakpoint(). What arch_kgdb_breakpoint() do is just set a init breakpoint.
Call arch_kgdb_breakpoint() after linux started is just fine.
Are there some registers must be setted before arch_kgdb_breakpoint()? Or some other init processes?
Or some thing I don't know?
Needs help, please, Thanks!
Hardware:
SK-AM64B
Software:
ti-processor-sdk-linux-rt-am64xx-evm-08.05.00.21
linux-rt-5.10.153+gitAUTOINC+29dbc132eb-g29dbc132eb
Kernel Options:
CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_DEFAULT_ENABLE=0x1
CONFIG_RANDOMIZE_BASE=n
CONFIG_WATCHDOG=n
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_GDB_SCRIPTS=y
Kernel params:
earlycon=ns16550a,mmio32,0x02800000 console=ttyS2,115200n8 kgdboc=ttyS2 kgdbwait
And I have been tried a lots of params, like " kgdboc=ttyS2,115200" " kgdboc=ttyS2,115200n8", still same problem.
Logs:
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[ 0.000000] Linux version 5.10.153-rt76-g29dbc132eb (p3uplus@p3uplus-pc) (aarch64-none-linux-gnu-gcc (GNU Toolchain for the A-profile Architecture 9.2-2019.12 (a3
[ 0.000000] Machine model: Texas Instruments AM642 SK
[ 0.000000] earlycon: ns16550a0 at MMIO32 0x0000000002800000 (options '')
[ 0.000000] printk: bootconsole [ns16550a0] enabled
[ 0.000000] efi: UEFI not found.
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a0000000, size 1 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-dma-memory@a0000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a0100000, size 15 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-memory@a0100000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a1000000, size 1 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-dma-memory@a1000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a1100000, size 15 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-memory@a1100000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a2000000, size 1 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-dma-memory@a2000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a2100000, size 15 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-memory@a2100000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a3000000, size 1 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-dma-memory@a3000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a3100000, size 15 MiB
[ 0.000000] OF: reserved mem: initialized node r5f-memory@a3100000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a4000000, size 1 MiB
[ 0.000000] OF: reserved mem: initialized node m4f-dma-memory@a4000000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created DMA memory pool at 0x00000000a4100000, size 15 MiB
[ 0.000000] OF: reserved mem: initialized node m4f-memory@a4100000, compatible id shared-dma-pool
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000080000000-0x00000000ffffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x000000009e7fffff]
[ 0.000000] node 0: [mem 0x000000009e800000-0x00000000a57fffff]
[ 0.000000] node 0: [mem 0x00000000a5800000-0x00000000ffffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffffff]
[ 0.000000] On node 0 totalpages: 524288
[ 0.000000] DMA zone: 8192 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 524288 pages, LIFO batch:63
[ 0.000000] cma: Reserved 512 MiB at 0x00000000dd000000
[ 0.000000] psci: probing for conduit method from DT.
[ 0.000000] psci: PSCIv1.1 detected in firmware.
[ 0.000000] psci: Using standard PSCI v0.2 function IDs
[ 0.000000] psci: Trusted OS migration not required
[ 0.000000] psci: SMC Calling Convention v1.2
[ 0.000000] percpu: Embedded 18 pages/cpu s35136 r8192 d30400 u73728
[ 0.000000] pcpu-alloc: s35136 r8192 d30400 u73728 alloc=18*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Detected VIPT I-cache on CPU0
[ 0.000000] CPU features: detected: ARM erratum 845719
[ 0.000000] CPU features: detected: GIC system register CPU interface
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 516096
[ 0.000000] Kernel command line: earlycon=ns16550a,mmio32,0x02800000 console=ttyS2,115200n8 kgdboc=ttyS2 kgdbwait root=/dev/nfs nfsroot=10.177.236.114:/home/p3upf
[ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 1399120K/2097152K available (10816K kernel code, 1198K rwdata, 4228K rodata, 1792K init, 749K bss, 173744K reserved, 524288K cma-reserved)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] rcu: Preemptible hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
[ 0.000000] rcu: RCU priority boosting: priority 1 delay 500 ms.
[ 0.000000] rcu: RCU_SOFTIRQ processing moved to rcuc kthreads.
[ 0.000000] No expedited grace period (rcu_normal_after_boot).
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[ 0.000000] GICv3: 256 SPIs implemented
[ 0.000000] GICv3: 0 Extended SPIs implemented
[ 0.000000] GICv3: Distributor has no Range Selector support
[ 0.000000] GICv3: 16 PPIs implemented
[ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000001840000
[ 0.000000] ITS [mem 0x01820000-0x0182ffff]
[ 0.000000] GIC: enabling workaround for ITS: Socionext Synquacer pre-ITS
[ 0.000000] ITS@0x0000000001820000: Devices Table too large, reduce ids 20->19
[ 0.000000] ITS@0x0000000001820000: allocated 524288 Devices @80800000 (flat, esz 8, psz 64K, shr 0)
[ 0.000000] ITS: using cache flushing for cmd queue
[ 0.000000] GICv3: using LPI property table @0x0000000080030000
[ 0.000000] GIC: using cache flushing for LPI property table
[ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000080040000
[ 0.000000] arch_timer: cp15 timer(s) running at 200.00MHz (phys).
[ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x2e2049d3e8, max_idle_ns: 440795210634 ns
[ 0.000001] sched_clock: 56 bits at 200MHz, resolution 5ns, wraps every 4398046511102ns
[ 0.008565] Console: colour dummy device 80x25
[ 0.013170] Calibrating delay loop (skipped), value calculated using timer frequency.. 400.00 BogoMIPS (lpj=800000)
[ 0.023845] pid_max: default: 32768 minimum: 301
[ 0.028696] LSM: Security Framework initializing
[ 0.033564] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.041144] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.051363] rcu: Hierarchical SRCU implementation.
[ 0.057151] Platform MSI: msi-controller@1820000 domain created
[ 0.063631] PCI/MSI: /bus@f4000/interrupt-controller@1800000/msi-controller@1820000 domain created
[ 0.072970] EFI services will not be available.
[ 0.077977] smp: Bringing up secondary CPUs ...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
[ 0.091819] Detected VIPT I-cache on CPU1
[ 0.095964] GICv3: CPU1: found redistributor 1 region 0:0x0000000001860000
[ 0.102999] GICv3: CPU1: using allocated LPI pending table @0x0000000080050000
[ 0.110436] CPU1: Booted secondary processor 0x0000000001 [0x410fd034]
[ 0.117273] smp: Brought up 1 node, 2 CPUs
[ 0.121506] SMP: Total of 2 processors activated.
[ 0.126330] CPU features: detected: 32-bit EL0 Support
[ 0.131596] CPU features: detected: CRC32 instructions
[ 0.144799] CPU: All CPU(s) started at EL2
[ 0.149008] alternatives: patching kernel code
[ 0.154921] devtmpfs: initialized
[ 0.167139] KASLR disabled due to lack of seed
[ 0.171935] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[ 0.181911] futex hash table entries: 512 (order: 4, 65536 bytes, linear)
[ 0.206979] pinctrl core: initialized pinctrl subsystem
[ 0.213043] DMI not present or invalid.
[ 0.217828] NET: Registered protocol family 16
[ 0.224731] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations
[ 0.232183] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[ 0.240318] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[ 0.249242] thermal_sys: Registered thermal governor 'step_wise'
[ 0.249993] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[ 0.263187] ASID allocator initialised with 65536 entries
[ 0.299645] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.306604] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[ 0.313496] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.320361] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[ 0.328647] cryptd: max_cpu_qlen set to 1000
[ 0.337200] k3-chipinfo 43000014.chipid: Family:AM64X rev:SR2.0 JTAGID[0x1bb3802f] Detected
[ 0.346575] vcc_3v3_sys: supplied by vusb_main5v0
[ 0.353127] iommu: Default domain type: Translated
[ 0.358717] SCSI subsystem initialized
[ 0.363435] mc: Linux media interface: v0.10
[ 0.367899] videodev: Linux video capture interface: v2.00
[ 0.373669] pps_core: LinuxPPS API ver. 1 registered
[ 0.378761] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.388121] PTP clock support registered
[ 0.392185] EDAC MC: Ver: 3.0.0
[ 0.396390] omap-mailbox 29020000.mailbox: omap mailbox rev 0x66fc9100
[ 0.403354] omap-mailbox 29040000.mailbox: omap mailbox rev 0x66fc9100
[ 0.410198] omap-mailbox 29060000.mailbox: omap mailbox rev 0x66fc9100
[ 0.417787] FPGA manager framework
[ 0.421445] Advanced Linux Sound Architecture Driver Initialized.
[ 0.429045] clocksource: Switched to clocksource arch_sys_counter
[ 0.435570] VFS: Disk quotas dquot_6.6.0
[ 0.439658] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.454329] NET: Registered protocol family 2
[ 0.459157] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)
[ 0.468366] tcp_listen_portaddr_hash hash table entries: 1024 (order: 4, 65536 bytes, linear)
[ 0.477240] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 0.485539] TCP bind hash table entries: 16384 (order: 7, 917504 bytes, linear)
[ 0.494149] TCP: Hash tables configured (established 16384 bind 16384)
[ 0.501303] UDP hash table entries: 1024 (order: 5, 131072 bytes, linear)
[ 0.508458] UDP-Lite hash table entries: 1024 (order: 5, 131072 bytes, linear)
[ 0.516326] NET: Registered protocol family 1
[ 0.521546] RPC: Registered named UNIX socket transport module.
[ 0.527664] RPC: Registered udp transport module.
[ 0.532472] RPC: Registered tcp transport module.
[ 0.537280] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 0.543890] PCI: CLS 0 bytes, default 64
[ 0.548823] hw perfevents: enabled with armv8_cortex_a53 PMU driver, 7 counters available
[ 0.562246] Initialise system trusted keyrings
[ 0.567216] workingset: timestamp_bits=46 max_order=19 bucket_order=0
[ 0.579824] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.586759] NFS: Registering the id_resolver key type
[ 0.592318] Key type id_resolver registered
[ 0.596650] Key type id_legacy registered
[ 0.600897] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 0.607785] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[ 0.615796] 9p: Installing v9fs 9p2000 file system support
[ 0.664774] Key type asymmetric registered
[ 0.669078] Asymmetric key parser 'x509' registered
[ 0.674220] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[ 0.681794] io scheduler mq-deadline registered
[ 0.686438] io scheduler kyber registered
[ 0.693365] pinctrl-single 4084000.pinctrl: 33 pins, size 132
[ 0.699869] pinctrl-single f4000.pinctrl: 180 pins, size 720
[ 0.706917] pinctrl-single a40000.timesync-router: 512 pins, size 2048
[ 0.724729] Serial: 8250/16550 driver, 10 ports, IRQ sharing enabled
[ 0.735590] name:ttyLF driver name:fsl-linflexuart
[ 0.740630] name:ttyLP driver name:fsl-lpuart
[ 0.745090] name:ttyS driver name:serial
[ 0.749104] KGDB: Registered I/O driver kgdboc
[ 0.753802] KGDB: Waiting for connection from remote gdb...
[ 0.760615] Unable to handle kernel paging request at virtual address fffffdfffec00001
[ 0.768702] Mem abort info:
[ 0.771547] ESR = 0x96000006
[ 0.774659] EC = 0x25: DABT (current EL), IL = 32 bits
[ 0.780080] SET = 0, FnV = 0
[ 0.783190] EA = 0, S1PTW = 0
[ 0.786390] Data abort info:
[ 0.789323] ISV = 0, ISS = 0x00000006
[ 0.793233] CM = 0, WnR = 0
[ 0.796258] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000082ec6000
[ 0.803099] [fffffdfffec00001] pgd=00000000831bc003, p4d=00000000831bc003, pud=00000000831bd003, pmd=0000000000000000
[ 0.813950] Internal error: Oops: 96000006 [#1] PREEMPT_RT SMP
[ 0.819928] KGDB: re-enter exception: ALL breakpoints killed
[ 0.825712] ---[ end trace 0000000000000001 ]---
[ 0.830465] note: swapper/0[1] exited with preempt_count 4
[ 0.836069] Kernel panic - not syncing:
[ 0.839978] Attempted to kill init! exitcode=0x00000009
[ 0.845347] Unable to handle kernel paging request at virtual address fffffdfffec00001
[ 0.853431] Mem abort info:
[ 0.856273] ESR = 0x96000006
[ 0.859383] EC = 0x25: DABT (current EL), IL = 32 bits
[ 0.864804] SET = 0, FnV = 0
[ 0.867915] EA = 0, S1PTW = 0
[ 0.871115] Data abort info:
[ 0.874048] ISV = 0, ISS = 0x00000006
[ 0.877957] CM = 0, WnR = 0
[ 0.880978] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000082ec6000
[ 0.887817] [fffffdfffec00001] pgd=00000000831bc003, p4d=00000000831bc003, pud=00000000831bd003, pmd=0000000000000000
[ 0.898662] Internal error: Oops: 96000006 [#2] PREEMPT_RT SMP
[ 0.904636] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G D 5.10.153-rt76-g29dbc132eb #10
[ 0.913876] Hardware name: Texas Instruments AM642 SK (DT)
[ 0.919475] Call trace:
[ 0.921965] dump_backtrace+0x0/0x1a0
[ 0.925716] show_stack+0x18/0x68
[ 0.929101] dump_stack+0xd0/0x12c
[ 0.932579] kgdb_reenter_check+0x128/0x150
[ 0.936853] kgdb_handle_exception+0x9c/0x120
[ 0.941299] kgdb_notify+0x34/0x70
[ 0.944771] atomic_notifier_call_chain+0x60/0x90
[ 0.949577] notify_die+0x48/0x70
[ 0.952960] die+0xa4/0x268
[ 0.955809] die_kernel_fault+0x64/0x78
[ 0.959724] __do_kernel_fault+0x74/0x148
[ 0.963816] do_bad_area+0x5c/0x68
[ 0.967287] do_translation_fault+0x38/0x70
[ 0.971558] do_mem_abort+0x40/0xa0
[ 0.975116] el1_abort+0x48/0x70
[ 0.978412] el1_sync_handler+0xac/0xc8
[ 0.982328] el1_sync+0x88/0x140
[ 0.985621] io_serial_in+0x24/0x48
[ 0.989182] uart_poll_put_char+0x9c/0x100
[ 0.993365] kgdboc_put_char+0x30/0x40
[ 0.997192] kdb_msg_write.part.0+0x50/0xd8
[ 1.001462] vkdb_printf+0x1d8/0x7f0
[ 1.005112] kdb_printf+0x68/0x90
[ 1.008494] kgdb_panic+0x54/0x68
[ 1.011875] panic+0x1a8/0x368
[ 1.014992] do_exit+0x9a8/0x9e0
[ 1.018285] die_kernel_fault+0x74/0x78
[ 1.022200] __do_kernel_fault+0x74/0x148
[ 1.026292] do_bad_area+0x5c/0x68
[ 1.029763] do_translation_fault+0x38/0x70
[ 1.034032] do_mem_abort+0x40/0xa0
[ 1.037591] el1_abort+0x48/0x70
[ 1.040885] el1_sync_handler+0xac/0xc8
[ 1.044800] el1_sync+0x88/0x140
[ 1.048093] io_serial_in+0x24/0x48
[ 1.051652] uart_poll_put_char+0x88/0x100
[ 1.055833] kgdboc_put_char+0x30/0x40
[ 1.059658] kdb_msg_write.part.0+0x50/0xd8
[ 1.063929] vkdb_printf+0x1d8/0x7f0
[ 1.067580] kdb_printf+0x68/0x90
[ 1.070961] kdb_main_loop+0x168/0x888
[ 1.074789] kdb_stub+0x254/0x438
[ 1.078173] kgdb_cpu_enter+0x1a0/0x708
[ 1.082090] kgdb_handle_exception+0xd0/0x120
[ 1.086538] kgdb_compiled_brk_fn+0x28/0x38
[ 1.090809] call_break_hook+0x68/0x80
[ 1.094634] brk_handler+0x1c/0x60
[ 1.098104] do_debug_exception+0xa0/0x168
[ 1.102285] el1_dbg+0x38/0x58
[ 1.105402] el1_sync_handler+0x9c/0xc8
[ 1.109316] el1_sync+0x88/0x140
[ 1.112609] kgdb_breakpoint+0x38/0x78
[ 1.116434] configure_kgdboc+0x1a4/0x1e8
[ 1.120527] kgdboc_probe+0x54/0x68
[ 1.124085] platform_drv_probe+0x54/0xa8
[ 1.128180] really_probe+0xec/0x3e0
[ 1.131828] driver_probe_device+0x58/0xb8
[ 1.136009] __device_attach_driver+0xb8/0xe0
[ 1.140456] bus_for_each_drv+0x78/0xc8
[ 1.144373] __device_attach+0xf8/0x188
[ 1.148289] device_initial_probe+0x14/0x20
[ 1.152558] bus_probe_device+0x9c/0xa8
[ 1.156474] device_add+0x350/0x728
[ 1.160034] platform_device_add+0x100/0x238
[ 1.164393] init_kgdboc+0x54/0x8c
[ 1.167869] do_one_initcall+0x54/0x1b8
[ 1.171783] kernel_init_freeable+0x220/0x2a0
[ 1.176229] kernel_init+0x14/0x11c
[ 1.179793] ret_from_fork+0x10/0x30
[ 1.183442] Kernel panic - not syncing:
[ 1.187352] Recursive entry to debugger
[ 1.187404] SMP: stopping secondary CPUs
[ 2.300769] SMP: failed to stop secondary CPUs 0-1
[ 2.305660] Kernel Offset: disabled
[ 2.309216] CPU features: 0x00040002,2000200c
[ 2.313660] Memory Limit: none
[ 2.316774] ---[ end Kernel panic - not syncing: Recursive entry to debugger ]---
