Linux 2.6.32 kernel freeze issue

1. power management
2. function tracer
3. stack usage
4. kernel object
5. kernel memory leak detector
6. verbose kernel error logs
7. virtual memory debug
8. shared irq
9. spinlock and mutex 

Sys Boot Pin Values = 0x3b
Current Trace word 1 = 0x0045085f
Current Trace word 2 = 0x00020004
Current copy of PRM_RSTST = 0x00000001
Cold reset trace word 1 = 0x00000000
Cold reset trace word 2 = 0x00000000
Software Boot Config Pointer = 0x0

Texas Instruments X-Loader 1.4.3 (Sep 27 2013 - 03:34:16)
current boot 1 = 0x2040081b
current boot 2 = 0x00000040
prmRstSt = 0x00000002
coldBoot1 = 0x0045085f
coldBoot2 = 0x00020004
Reading boot sector
Loading u-boot.bin from mmc

U-Boot 2012.10-rc3 (May 01 2015 - 04:42:10)

OMAP3503-GP ES3.1.2, CPU-OPP2, L3-165MHz, Max CPU Clock 600 mHz
Gumstix Overo board + LPDDR/NAND
I2C: ready
configured
DRAM: 256 MiB
relocate
board_init_r
NAND: 256 MiB
MMC: OMAP SD/MMC: 0
Using default environment

In: serial
Out: serial
Err: serial
Board revision: 4
Direct connection on mmc2
timed out in wait_for_bb: I2C_STAT=1000
No EEPROM on expansion board
Die ID #7fdc000700000000015154160a014012
Net: Board Net Initialization
smc911x-0
Warning: smc911x-0 using MAC address from net device

Hit any key to stop autoboot: 1 0
info->boot_source = mmc
reading ctrlblock.bin
info->boot_mode = 00005678
info->upgrade_type = 0000ef01
info->boot_tracker = 0
reading uimage-system

3160356 bytes read
Booting from system image from mmc ...
## Booting kernel from Legacy Image at 85000000 ...
Image Name: Linux-2.6.32
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 3160292 Bytes = 3 MiB
Load Address: 80008000
Entry Point: 80008000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK

Starting kernel ...

Uncompressing Linux........................................................................................................................................................................................................................................................................ done, booting the kernel.
on 6.0, musb-dma, host, debug=0
musb_hdrc: USB Host mode controller at fa0ab000 using DMA, IRQ 92
musb_hdrc musb_hdrc: MUSB HDRC host driver
musb_hdrc musb_hdrc: new USB bus registered, assigned bus number 1
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: MUSB HDRC host driver
usb usb1: Manufacturer: Linux 2.6.32 musb-hcd
usb usb1: SerialNumber: musb_hdrc
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 6, 327680 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Slow work thread pool: Starting up
Slow work thread pool: Ready
JFFS2 version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
msgmni has been set to 457
cryptomgr_test used greatest stack depth: 6188 bytes left
cryptomgr_test used greatest stack depth: 5868 bytes left
cryptomgr_test used greatest stack depth: 5308 bytes left
alg: No test for stdrng (krng)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
OMAP DSS rev 2.0
OMAP DISPC rev 3.0
OMAP VENC rev 2
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250.0: ttyS0 at MMIO 0x4806a000 (irq = 72) is a ST16654
serial8250.1: ttyS1 at MMIO 0x4806c000 (irq = 73) is a ST16654
console [ttyS1] enabled
serial8250.2: ttyS2 at MMIO 0x49020000 (irq = 74) is a ST16654
brd: module loaded
loop: module loaded
omap2-nand driver initializing
ONFI flash detected
NAND device: Manufacturer ID: 0x2c, Chip ID: 0xba (Micron )
cmdlinepart partition parsing not available
Creating 6 MTD partitions on "omap2-nand.0":
0x000000000000-0x000000300000 : "boot"
0x000000300000-0x000000e00000 : "nandkernel"
0x000000e00000-0x000001900000 : "sdkernel"
0x000001900000-0x000007500000 : "bootstrap"
0x000007500000-0x000008000000 : "scratch"
0x000008000000-0x000008400000 : "controlblock"
smsc911x: Driver version 2008-10-21.
Detected SMSC9221!
smsc911x-mdio: probed
eth0: attached PHY driver [Generic PHY] (mii_bus:phy_addr=0:01, irq=-1)
net eth0: MAC Address: 00:07:b0:0c:90:1f
usbcore: registered new interface driver asix
usbcore: registered new interface driver cdc_ether
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-omap ehci-omap.0: OMAP-EHCI Host Controller
ehci-omap ehci-omap.0: new USB bus registered, assigned bus number 2
ehci-omap ehci-omap.0: irq 77, io mem 0x48064800
ehci-omap ehci-omap.0: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: OMAP-EHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32 ehci_hcd
usb usb2: SerialNumber: ehci-omap.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
mice: PS/2 mouse device common for all mice
Registering Panjit TS
MUX VALUE = 01180118
Successfully initialized Panjit TS!
input: panjit_ts as /devices/platform/i2c_omap.2/i2c-2/2-0003/input/input0
twl4030_rtc twl4030_rtc: rtc core: registered twl4030_rtc as rtc0
i2c /dev entries driver
Linux video capture interface: v2.00
OMAP Watchdog Timer Rev 0x31: initial timeout 60 sec
omap_pwm_led omap_pwm_led.0: OMAP PWM LED (red_led) at GP timer 10/0
Registered led device: red_led
Sys_rate = 13000000
omap_pwm_led omap_pwm_led.1: OMAP PWM LED (green_led) at GP timer 11/0
Registered led device: green_led
Sys_rate = 13000000
omap_pwm_led omap_pwm_led.2: OMAP PWM LED (backlight) at GP timer 9/0
Registered led device: backlight
Sys_rate = 13000000
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
Advanced Linux Sound Architecture Driver Version 1.0.21.
usbcore: registered new interface driver snd-usb-audio
No device for DAI twl4030
No device for DAI twl4030 Voice
No device for DAI omap-mcbsp-dai-0
No device for DAI omap-mcbsp-dai-1
No device for DAI omap-mcbsp-dai-2
No device for DAI omap-mcbsp-dai-3
No device for DAI omap-mcbsp-dai-4
overo SoC init
TWL4030 Audio Codec init
asoc: twl4030 <-> omap-mcbsp-dai-0 mapping ok
ALSA device list:
#0: overo (twl4030)
oprofile: using arm/armv7
TCP cubic registered
NET: Registered protocol family 17
NET: Registered protocol family 15
ThumbEE CPU extension supported.
Power Management for TI OMAP3.
VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 1
kmemleak: Kernel memory leak detector initialized
kmemleak: Automatic memory scanning thread started
Console: switching to colour frame buffer device 100x30
omapdss DPI: Could not find exact pixel clock. Requested 33260 kHz, got 33230 kHz
regulator_init_complete: incomplete constraints, leaving VDVI on
regulator_init_complete: incomplete constraints, leaving VDAC on
twl4030_rtc twl4030_rtc: setting system clock to 2016-01-28 07:14:45 UTC (1453965285)
omapdss DPI error: display already enabled
omap_vout omap_vout: 'lcd' Display already enabled
omapdss DPI error: display already enabled
omap_vout omap_vout: 'lcd' Display already enabled
omap_voutBuffer Size = 3686400
omap_vout: registered and initialized video device 0 [v4l2]
Waiting for root device /dev/mmcblk0p6...
mmc0: host does not support reading read-only switch. assuming write-enable.
mmc0: new high speed SDHC card at address b368
mmcblk0: mmc0:b368 4GB 3.84 GiB
mmcblk0: p1 p2 p3 p4 < p5 p6 p7 >
EXT4-fs (mmcblk0p6): warning: checktime reached, running e2fsck is recommended
EXT4-fs (mmcblk0p6): mounted filesystem with ordered data mode
VFS: Mounted root (ext4 filesystem) on device 179:6.
devtmpfs: mounted
Freeing init memory: 588K

INIT: version 2.86 booting

mount: no /proc/mounts
cut used greatest stack depth: 4852 bytes left
EXT4-fs (mmcblk0p5): warning: checktime reached, running e2fsck is recommended
EXT4-fs (mmcblk0p5): recovery complete
EXT4-fs (mmcblk0p5): mounted filesystem with ordered data mode
mount used greatest stack depth: 4636 bytes left
upgrade_logger. used greatest stack depth: 4108 bytes left
Finished KILL scripts for previous runlevel
Running START scripts for new runlevel
EXT4-fs (mmcblk0p2): warning: checktime reached, running e2fsck is recommended
EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode
Mounting sysfs filesystem
Starting udev
udevd[316]: SYSFS{}= will be removed in a future udev version, please use ATTR{}= to match the event device, or ATTRS{}= to match a parent device, in /etc/udev/rules.d/local.rules:35

udevd[316]: SYSFS{}= will be removed in a future udev version, please use ATTR{}= to match the event device, or ATTRS{}= to match a pudev: starting version 151
arent device, in /etc/udev/rules.d/local.rules:36

udevd[316]: NAME="%k" is superfluous and breaks kernel supplied names, please remove it from /etc/udev/rules.d/udev.rules:18

udevd[316]: NAME="%k" is superfluous and breaks kernel supplied names, please remove it from /etc/udev/rules.d/udev.rules:106

udevadm settle - timeout of 3 seconds reached, the event queue contains:
/sys/devices/platform/ehci-omap.0/usb2/2-0:1.0 (545)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0 (570)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0/mmcblk0p1 (571)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0/mmcblk0p2 (572)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0/mmcblk0p3 (573)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0/mmcblk0p4 (574)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0/mmcblk0p5 (575)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0/mmcblk0p6 (576)
/sys/devices/platform/mmci-omap-hs.0/mmc_host/mmc0/mmc0:b368/block/mmcblk0/mmcblk0p7 (577)
/sys/devices/platform/musb_hdrc/usb1/1-0:1.0 (580)
/sys/devices/virtual/block/loop4 (633)
/sys/devices/virtual/block/loop5 (634)
/sys/devices/virtual/block/loop6 (635)
/sys/devices/virtual/block/loop7 (636)
/sys/devices/virtual/block/ram0 (637)
/sys/devices/virtual/block/ram1 (638)
/sys/devices/virtual/block/ram10 (639)
/sys/devices/virtual/block/ram11 (640)
/sys/devices/virtual/block/ram12 (641)
/sys/devices/virtual/block/ram13 (642)
/sys/devices/virtual/block/ram14 (643)
/sys/devices/virtual/block/ram15 (644)
/sys/devices/virtual/block/ram2 (645)
/sys/devices/virtual/block/ram3 (646)
/sys/devices/virtual/block/ram4 (647)
/sys/devices/virtual/block/ram5 (648)
/sys/devices/virtual/block/ram6 (649)
/sys/devices/virtual/block/ram7 (650)
/sys/devices/virtual/block/ram8 (651)
/sys/devices/virtual/block/ram9 (652)
/sys/devices/virtual/mtd/mtd0/mtdblock0 (670)
/sys/devices/virtual/mtd/mtd1/mtdblock1 (673)
/sys/devices/virtual/mtd/mtd2/mtdblock2 (676)
/sys/devices/virtual/mtd/mtd3/mtdblock3 (679)
/sys/devices/virtual/mtd/mtd4/mtdblock4 (682)
/sys/devices/virtual/mtd/mtd5/mtdblock5 (685)
/sys/devices/virtual/net/lo (687)
Machine hardware name = 'armv7l' : Setting /proc/cpu/alignment = 0
Adding 257000k swap on /dev/mmcblk0p7. Priority:-1 extents:1 across:257000k SS
Remounting root file system...
Caching udev devnodes
Populating dev cachemv: cannot stat `/tmp/uname': No such file or directory
Running : '/sbin/sysctl -q -p /etc/sysctl.conf'
Unknown HZ value! (91) Assume 100.
Mounting local filesystems
EXT4-fs (mmcblk0p3): warning: checktime reached, running e2fsck is recommended
EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode
root: mount: mounting /dev/mmcblk0p7 on swap failed: No such file or directory
Configuring network interfaces...
done.
Starting syslogd/klog..d done
Starting Networking scripts
net eth0: SMSC911x/921x identified at 0xd08c8000, IRQ: 178
Unknown HZ value! (93) Assume 100.
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
Current Locale : ja_JP
INIT: Entering runlevel: 5
Running KILL scripts for previous runlevel
Running START scripts for new runlevel
Starting system message bus
Starting Hardware abstraction layer hald
Starting OpenBSD Secure Shell server: sshd
Starting ntpd
starting FTP Server: vsftpd...
Starting Xorg
Starting Web Server (empty hook)

upgrade_logger. used greatest stack depth: 4044 bytes left
overo login: splash-write used greatest stack depth: 3884 bytes left
Sys_rate = 13000000
Sys_rate = 13000000
kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
overo login:
overo login:
overo login: root
Password:
root@overo:~#
root@overo:~#
root@overo:~#
root@overo:~# kmemleak: 3 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
root@overo:~#
root@overo:~#
root@overo:~# kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
root@overo:~# kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 16 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 5 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
dig used greatest stack depth: 3484 bytes left
root@overo:~#
root@overo:~#
root@overo:~# kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
Spurious irq 95: 0xffffffdf, please flush posted write for irq 56
Spurious irq 95: 0xffffffdf, please flush posted write for irq 56
kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
Sys_rate = 13000000
Sys_rate = 13000000
kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 3 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemlea
root@overo:~#
root@overo:~#
root@overo:~#
root@overo:~#
root@overo:~# cd /var
root@overo:/var# kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
root@overo:/var#
root@overo:/var#
root@overo:/var#
root@overo:/var#
root@overo:/var#
root@overo:/var# kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
Sys_rate = 13000000