On both B and C revisions of Beaglebone black, our application (running under either the shipped Angstrom or Debian distros) locks up after random amounts of time. The debug port reports the following problems:
[ 752.038977] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:1:15]
[ 752.063631] BUG: scheduling while atomic: kworker/0:1/15/0x40010100
[ 754.935632] Kernel panic - not syncing: softlockup: hung tasks
[ 754.946347] [<c0010443>] (unwind_backtrace+0x1/0x8a) from [<c0455ced>] (panic
+0x51/0x148)
[ 754.960814] [<c0455ced>] (panic+0x51/0x148) from [<c00678df>] (watchdog_timer
_fn+0xc3/0xf8)
[ 754.975585] [<c00678df>] (watchdog_timer_fn+0xc3/0xf8) from [<c00421f3>] (__r
un_hrtimer+0xb3/0x140)
[ 754.991496] [<c00421f3>] (__run_hrtimer+0xb3/0x140) from [<c004295b>] (hrtime
r_interrupt+0xd7/0x1c8)
[ 755.007625] [<c004295b>] (hrtimer_interrupt+0xd7/0x1c8) from [<c001c8ed>] (om
ap2_gp_timer_interrupt+0x11/0x1c)
[ 755.025164] [<c001c8ed>] (omap2_gp_timer_interrupt+0x11/0x1c) from [<c0067fa9
>] (handle_irq_event_percpu+0x4d/0x174)
[ 755.043502] [<c0067fa9>] (handle_irq_event_percpu+0x4d/0x174) from [<c00680f9
>] (handle_irq_event+0x29/0x3c)
[ 755.060694] [<c00680f9>] (handle_irq_event+0x29/0x3c) from [<c0069b17>] (hand
le_level_irq+0x6f/0x84)
[ 755.076699] [<c0069b17>] (handle_level_irq+0x6f/0x84) from [<c0067bff>] (gene
ric_handle_irq+0x13/0x1c)
[ 755.092998] [<c0067bff>] (generic_handle_irq+0x13/0x1c) from [<c000c8cb>] (ha
ndle_IRQ+0x3b/0x5c)
[ 755.108422] [<c000c8cb>] (handle_IRQ+0x3b/0x5c) from [<c0008551>] (omap3_intc
_handle_irq+0x39/0x58)
[ 755.124243] [<c0008551>] (omap3_intc_handle_irq+0x39/0x58) from [<c045b95b>]
(__irq_svc+0x3b/0x5c)
[ 755.139692] Exception stack(0xdf117cf8 to 0xdf117d40)
[ 755.148646] 7ce0: 00000
000 c0849ac0
[ 755.162967] 7d00: 00000000 00000000 c07a2080 00000000 df116000 00000002 df117
db0 00000003
[ 755.177335] 7d20: c0812610 df116000 edb88320 df117d40 c0031241 c0030f4e 40000
133 ffffffff
[ 755.191758] [<c045b95b>] (__irq_svc+0x3b/0x5c) from [<c0030f4e>] (__do_softir
q+0x46/0x174)
[ 755.206357] [<c0030f4e>] (__do_softirq+0x46/0x174) from [<c0031241>] (irq_exi
t+0x29/0x50)
[ 755.220773] [<c0031241>] (irq_exit+0x29/0x50) from [<c000c8cf>] (handle_IRQ+0
x3f/0x5c)
[ 755.234735] [<c000c8cf>] (handle_IRQ+0x3f/0x5c) from [<c0008551>] (omap3_intc
_handle_irq+0x39/0x58)
[ 755.250549] [<c0008551>] (omap3_intc_handle_irq+0x39/0x58) from [<c045b95b>]
(__irq_svc+0x3b/0x5c)
[ 755.265995] Exception stack(0xdf117db0 to 0xdf117df8)
[ 755.274974] 7da0: 00000002 00000000 00007
d00 00000000
[ 755.289328] 7dc0: c07c81d0 c07c81d0 c07c75dc 00007d02 0000007d 00000003 c0812
610 de5f4b40
[ 755.303615] 7de0: 00000100 df117df8 c0025b2d c0025bea 00000133 ffffffff
[ 755.315416] [<c045b95b>] (__irq_svc+0x3b/0x5c) from [<c0025bea>] (omap3_nonco
re_dpll_set_rate+0x1f2/0x330)
[ 755.332303] [<c0025bea>] (omap3_noncore_dpll_set_rate+0x1f2/0x330) from [<c03
83273>] (clk_change_rate+0x1b/0x52)
[ 755.350016] [<c0383273>] (clk_change_rate+0x1b/0x52) from [<c03832fb>] (clk_s
et_rate+0x51/0x72)
[ 755.365316] [<c03832fb>] (clk_set_rate+0x51/0x72) from [<c034ba29>] (cpu0_set
_target+0xf9/0x198)
[ 755.380787] [<c034ba29>] (cpu0_set_target+0xf9/0x198) from [<c0348c5d>] (__cp
ufreq_driver_target+0x4d/0x70)
[ 755.397849] [<c0348c5d>] (__cpufreq_driver_target+0x4d/0x70) from [<c034b33b>
] (dbs_check_cpu+0x123/0x134)
[ 755.414765] [<c034b33b>] (dbs_check_cpu+0x123/0x134) from [<c034ad31>] (od_db
s_timer+0x4d/0xb0)
[ 755.430064] [<c034ad31>] (od_dbs_timer+0x4d/0xb0) from [<c003c8c5>] (process_
one_work+0x1b5/0x2c0)
[ 755.445787] [<c003c8c5>] (process_one_work+0x1b5/0x2c0) from [<c003cca3>] (wo
rker_thread+0x19b/0x258)
[ 755.461985] [<c003cca3>] (worker_thread+0x19b/0x258) from [<c003fb8f>] (kthre
ad+0x67/0x74)
[ 755.476573] [<c003fb8f>] (kthread+0x67/0x74) from [<c000c0dd>] (ret_from_fork
+0x11/0x34)
[ 755.490617] drm_kms_helper: panic occurred, switching back to text console
[ 696.382258] CAUTION: musb: Babble Interrupt Occurred
[ 696.422949] CAUTION: musb: Babble Interrupt Occurred
[ 696.483459] gadget: high-speed config #1: Multifunction with RNDIS
[ 702.376300] musb_g_ep0_irq 710: SetupEnd came in a wrong ep0stage wait
[ 775.658309] CAUTION: musb: Babble Interrupt Occurred
The application performs a modest amount of IO using FTDI through a USB hub. The more USB devices we service, the faster the freeze appears, though it can happen soon after the code runs, or as much as 16 hours later.
This very much appears to be a kernel or driver issue. Can it be diagnosed, and is there a way of fixing it?
