This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM335X: Kernel crashes with network traffic

Other Parts Discussed in Thread: AM3352

Hi all,

I have a custom board with AM335x and a Micrel 9031 phy.

Occasionally we have kernel crashes without any special user action.
See the 3 Crashlogs below.
Each backtrace shows cpsw_rx_handler() and skb_put()  with device eth0.

Here https://e2e.ti.com/support/wireless_connectivity/wilink_wifi_bluetooth/f/307/t/514808
is a similar crash description, caused by malformed uart-packets. This lkml.iu.edu/.../00673.html fix was recommended.

Might be that we do also receive malformed network packets?
If yes, is there a patch available?
I'm running linux  3.12.10-ti2013.12.01 from SDK 07.00.00.00.

Kind Regards,

Roland

Crash #1:

 root@am335x-evm:/home/testApp# [  131.750162] skbuff: skb_over_panic: text:c03f58e4 len:2043 put:2043 head:ebdd2000 data:ebdd2042 tail:0xebdd283d end:0xebdd2740 dev:eth0
[  131.763048] ------------[ cut here ]------------
[  131.767902] Kernel BUG at c05d1780 [verbose debug info unavailable]
[  131.774490] Internal error: Oops - BUG: 0 [#1] ARM
[  131.779531] Modules linked in: cfg80211 g_mass_storage usb_f_mass_storage libcomposite configfs bufferclass_ti(O) omaplfb(O) cryptodev(O) musb_dsps musb_hdrc wlcore_sdio pvrsrvkm(O) musb_am335x btwilink bluetooth
[  131.799456] CPU: 0 PID: 0 Comm: swapper Tainted: G        W  O 3.12.10-ti2013.12.01 #1
[  131.807786] task: c08faaa0 ti: c08f0000 task.ti: c08f0000
[  131.813488] PC is at skb_panic+0x5c/0x68
[  131.817626] LR is at console_unlock+0x1c8/0x3fc
[  131.822389] pc : [<c05d1780>]    lr : [<c006f3fc>]    psr: 600f0113
[  131.822389] sp : c08f1cd0  ip : c08f1bf8  fp : c08f1d04
[  131.834460] r10: ed350940  r9 : 00000000  r8 : 000007fb
[  131.839951] r7 : c078f02c  r6 : ebdd2000  r5 : 000007fb  r4 : ebdd2042
[  131.846813] r3 : c094523c  r2 : c094523c  r1 : 00000000  r0 : 0000007b
[  131.853677] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  131.861361] Control: 10c5387d  Table: ab94c019  DAC: 00000017
[  131.867401] Process swapper (pid: 0, stack limit = 0xc08f0240)
[  131.873533] Stack: (0xc08f1cd0 to 0xc08f2000)
[  131.878115] 1cc0:                                     000007fb ebdd2000 ebdd2042 ebdd283d
[  131.886716] 1ce0: ebdd2740 ed326000 000007fb ebdd283d ebdd2740 00000000 c08f1d24 c08f1d08
[  131.895318] 1d00: c051890c c05d1730 ebae72c0 ed326440 ed326000 ed38d200 c08f1d54 c08f1d28
[  131.903919] 1d20: c03f58e4 c05188c4 800f0113 00020000 c08f1d54 f0110000 ed3499c0 00020000
[  131.912522] 1d40: 000007fb ed38d200 c08f1d8c c08f1d58 c03f1bc0 c03f584c 00000000 c08f1de8
[  131.921122] 1d60: 00000000 00020000 200f0113 0000003f 3a212000 f0110000 c097e000 4a102000
[  131.929723] 1d80: c08f1db4 c08f1d90 c03f1cb4 c03f1b34 00000000 ed3499c0 00000040 0000012c
[  131.938323] 1da0: c097e000 c0943d80 c08f1dd4 c08f1db8 c03f1d34 c03f1be4 c097e008 ed326448
[  131.946924] 1dc0: ed326448 00000040 c08f1df4 c08f1dd8 c03f57ac c03f1cfc c097e008 c03f5790
[  131.955525] 1de0: ed326448 00000040 c08f1e2c c08f1df8 c0523864 c03f579c c097e008 ffffbe48
[  131.964125] 1e00: 00000001 00000001 0000000c c0980050 c0980040 c08f0000 00000003 00000100
[  131.972725] 1e20: c08f1e74 c08f1e30 c004a35c c05237cc 00000001 ed005e40 0000000c 00200000
[  131.981326] 1e40: ffffbe47 0000000a 413fc082 600f0193 00000039 00000000 00000039 00000000
[  131.989927] 1e60: 413fc082 c097e0d4 c08f1e8c c08f1e78 c004a4e0 c004a2a0 00000000 c08f0000
[  131.998528] 1e80: c08f1ea4 c08f1e90 c004a758 c004a49c 00000112 c094d340 c08f1ec4 c08f1ea8
[  132.007128] 1ea0: c0015564 c004a6c8 00000080 fa200000 c08f1ee8 c097ef14 c08f1ee4 c08f1ec8
[  132.015730] 1ec0: c000879c c0015534 c04a04cc 200f0013 ffffffff c08f1f1c c08f1f5c c08f1ee8
[  132.024330] 1ee0: c05d47c0 c0008740 c08f1f30 3b9aca00 00ad4bc5 0000001e abcdf7aa 0000001e
[  132.032931] 1f00: c096b510 00000001 00000000 413fc082 c097e0d4 c08f1f5c 0000001a c08f1f30
[  132.041531] 1f20: c00779a8 c04a04cc 200f0013 ffffffff ac7b436f 0000001e 00000000 c08fdf28
[  132.050131] 1f40: c096b510 00000001 c08fdf28 c09b081c c08f1f84 c08f1f60 c04a0604 c04a047c
[  132.058734] 1f60: 00000000 c08f0000 00000000 c08f8104 c097e0d4 00000001 c08f1f94 c08f1f88
[  132.067334] 1f80: c0015730 c04a056c c08f1fac c08f1f98 c0070df0 c001572c 00000000 c11cb400
[  132.075934] 1fa0: c08f1fbc c08f1fb0 c05cb62c c0070da4 c08f1ff4 c08f1fc0 c07d8a48 c05cb5d4
[  132.084535] 1fc0: ffffffff ffffffff c07d8544 00000000 00000000 c0815c60 10c53c7d c08f80a8
[  132.093135] 1fe0: c0815c5c c08fbb40 00000000 c08f1ff8 80008070 c07d87b4 00000000 00000000
[  132.101730] Backtrace:
[  132.104331] [<c05d1724>] (skb_panic+0x0/0x68) from [<c051890c>] (skb_put+0x54/0x58)
[  132.112384]  r7:00000000 r6:ebdd2740 r5:ebdd283d r4:000007fb
[  132.118366] [<c05188b8>] (skb_put+0x0/0x58) from [<c03f58e4>] (cpsw_rx_handler+0xa4/0x174)
[  132.127054]  r7:ed38d200 r6:ed326000 r5:ed326440 r4:ebae72c0
[  132.133026] [<c03f5840>] (cpsw_rx_handler+0x0/0x174) from [<c03f1bc0>] (__cpdma_chan_free+0x98/0xb0)
[  132.142627]  r8:ed38d200 r7:000007fb r6:00020000 r5:ed3499c0 r4:f0110000
[  132.149696] [<c03f1b28>] (__cpdma_chan_free+0x0/0xb0) from [<c03f1cb4>] (__cpdma_chan_process+0xdc/0x118)
[  132.159760] [<c03f1bd8>] (__cpdma_chan_process+0x0/0x118) from [<c03f1d34>] (cpdma_chan_process+0x44/0x5c)
[  132.169915] [<c03f1cf0>] (cpdma_chan_process+0x0/0x5c) from [<c03f57ac>] (cpsw_poll+0x1c/0xb0)
[  132.178969]  r6:00000040 r5:ed326448 r4:ed326448 r3:c097e008
[  132.184955] [<c03f5790>] (cpsw_poll+0x0/0xb0) from [<c0523864>] (net_rx_action+0xa4/0x1a8)
[  132.193643]  r6:00000040 r5:ed326448 r4:c03f5790 r3:c097e008
[  132.199627] [<c05237c0>] (net_rx_action+0x0/0x1a8) from [<c004a35c>] (__do_softirq+0xc8/0x1ac)
[  132.208688] [<c004a294>] (__do_softirq+0x0/0x1ac) from [<c004a4e0>] (do_softirq+0x50/0x5c)
[  132.217382] [<c004a490>] (do_softirq+0x0/0x5c) from [<c004a758>] (irq_exit+0x9c/0xf0)
[  132.225612]  r4:c08f0000 r3:00000000
[  132.229403] [<c004a6bc>] (irq_exit+0x0/0xf0) from [<c0015564>] (handle_IRQ+0x3c/0x8c)
[  132.237634]  r4:c094d340 r3:00000112
[  132.241403] [<c0015528>] (handle_IRQ+0x0/0x8c) from [<c000879c>] (omap3_intc_handle_irq+0x68/0x7c)
[  132.250822]  r6:c097ef14 r5:c08f1ee8 r4:fa200000 r3:00000080
[  132.256794] [<c0008734>] (omap3_intc_handle_irq+0x0/0x7c) from [<c05d47c0>] (__irq_svc+0x40/0x54)
[  132.266123] Exception stack(0xc08f1ee8 to 0xc08f1f30)
[  132.271435] 1ee0:                   c08f1f30 3b9aca00 00ad4bc5 0000001e abcdf7aa 0000001e
[  132.280036] 1f00: c096b510 00000001 00000000 413fc082 c097e0d4 c08f1f5c 0000001a c08f1f30
[  132.288634] 1f20: c00779a8 c04a04cc 200f0013 ffffffff
[  132.293941]  r7:c08f1f1c r6:ffffffff r5:200f0013 r4:c04a04cc
[  132.299913] [<c04a0470>] (cpuidle_enter_state+0x0/0xf0) from [<c04a0604>] (cpuidle_idle_call+0xa4/0x148)
[  132.309880]  r7:c09b081c r6:c08fdf28 r5:00000001 r4:c096b510
[  132.315850] [<c04a0560>] (cpuidle_idle_call+0x0/0x148) from [<c0015730>] (arch_cpu_idle+0x10/0x4c)
[  132.325269]  r8:00000001 r7:c097e0d4 r6:c08f8104 r5:00000000 r4:c08f0000
r3:00000000
[  132.333537] [<c0015720>] (arch_cpu_idle+0x0/0x4c) from [<c0070df0>] (cpu_startup_entry+0x58/0xf4)
[  132.342874] [<c0070d98>] (cpu_startup_entry+0x0/0xf4) from [<c05cb62c>] (rest_init+0x64/0x7c)
[  132.351837]  r7:c11cb400 r3:00000000
[  132.355624] [<c05cb5c8>] (rest_init+0x0/0x7c) from [<c07d8a48>] (start_kernel+0x2a0/0x2fc)
[  132.364325] [<c07d87a8>] (start_kernel+0x0/0x2fc) from [<80008070>] (0x80008070)
[  132.372106] Code: e58d4008 e58de00c e59f0008 ebfff524 (e7f001f2)
[  132.378528] ---[ end trace 1b75b31a2719ed1e ]---
[  132.383385] Kernel panic - not syncing: Fatal exception in interrupt


Crash #2:

root@am335x-evm:/home/testApp# [   48.361261] skbuff: skb_over_panic: text:c03f58e4 len:2043 put:2043 head:ec4b6800 data:ec4b6842 tail:0xec4b703d end:0xec4b6f40 dev:eth0
[   48.374256] ------------[ cut here ]------------
[   48.379115] Kernel BUG at c05d1e24 [verbose debug info unavailable]
[   48.385705] Internal error: Oops - BUG: 0 [#1] ARM

[   48.390748] Modules linked in: g_mass_storage usb_f_mass_storage libcomposite configfs bufferclass_ti(O) omaplfb(O) cryptodev(O) wl18xx wlcore mac80211 cfg80211 musb_dsps musb_hdrc pvrsrvkm(O) wlcore_sdio btwilink musb_am335x bluetooth
[   48.412974] CPU: 0 PID: 1587 Comm: QQmlThread Tainted: G        W  O 3.12.10-ti2013.12.01 #1
[   48.421852] task: ec3066c0 ti: ec55c000 task.ti: ec55c000
[   48.427559] PC is at skb_panic+0x5c/0x68
[   48.431693] LR is at console_unlock+0x1c8/0x3fc
[   48.436455] pc : [<c05d1e24>]    lr : [<c006f3fc>]    psr: 600f0113
[   48.436455] sp : ec55dd98  ip : ec55dcc0  fp : ec55ddcc
[   48.448525] r10: ed34aa00  r9 : 00000000  r8 : 000007fb
[   48.454016] r7 : c078f930  r6 : ec4b6800  r5 : 000007fb  r4 : ec4b6842
[   48.460877] r3 : c094523c  r2 : c094523c  r1 : 00000000  r0 : 0000007b
[   48.467743] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   48.475244] Control: 10c5387d  Table: ac48c019  DAC: 00000015
[   48.481283] Process QQmlThread (pid: 1587, stack limit = 0xec55c240)
[   48.487963] Stack: (0xec55dd98 to 0xec55e000)
[   48.492545] dd80:                                                       000007fb ec4b6800
[   48.501145] dda0: ec4b6842 ec4b703d ec4b6f40 ed32f000 000007fb ec4b703d ec4b6f40 00000000
[   48.509746] ddc0: ec55ddec ec55ddd0 c0518fb0 c05d1dd4 ed511200 ed32f440 ed32f000 ed61b6c0
[   48.518347] dde0: ec55de1c ec55ddf0 c03f58e4 c0518f68 600f0113 00020000 ec55de1c f010c040
[   48.526948] de00: ed33b6c0 00020000 000007fb ed61b6c0 ec55de54 ec55de20 c03f1bc0 c03f584c
[   48.535548] de20: 00000000 c08fd480 00000000 00020000 200f0113 0000003f 3a20e040 f010c000
[   48.544148] de40: c097e080 4a102000 ec55de7c ec55de58 c03f1cb4 c03f1b34 00000000 ed33b6c0
[   48.552749] de60: 00000040 0000012c c097e080 c0943d80 ec55de9c ec55de80 c03f1d34 c03f1be4
[   48.561349] de80: c097e088 ed32f448 ed32f448 00000040 ec55debc ec55dea0 c03f57ac c03f1cfc
[   48.569948] dea0: c097e088 c03f5790 ed32f448 00000040 ec55def4 ec55dec0 c0523f08 c03f579c
[   48.578548] dec0: c097e088 ffff9db5 00000001 00000001 0000000c c09800d0 c09800c0 ec55c000
[   48.587148] dee0: 00000003 00000100 ec55df3c ec55def8 c004a35c c0523e70 ec55df24 ed005e40
[   48.595748] df00: 0000000c 00400040 ffff9db4 0000000a 0000f04f 600f0193 00000039 00000000
[   48.604348] df20: 00000039 00000c20 0000f04f ad44da38 ec55df54 ec55df40 c004a4e0 c004a2a0
[   48.612948] df40: 00000000 ec55c000 ec55df6c ec55df58 c004a758 c004a49c 00000112 c094d340
[   48.621548] df60: ec55df8c ec55df70 c0015564 c004a6c8 00000080 fa200000 ec55dfb0 c097ef94
[   48.630148] df80: ec55dfac ec55df90 c000879c c0015534 b6580780 400f0030 ffffffff 00000000
[   48.638748] dfa0: 00000000 ec55dfb0 c05d5020 c0008740 ad108c40 ad4946b0 b66c7e8c b660f099
[   48.647348] dfc0: ae97c39c 00000000 0000000a 00000000 00000c20 0000f04f ad44da38 00000000
[   48.655948] dfe0: b6716d6c ae97c35c b6580841 b6580780 400f0030 ffffffff 00000000 00000000
[   48.664544] Backtrace:
[   48.667144] [<c05d1dc8>] (skb_panic+0x0/0x68) from [<c0518fb0>] (skb_put+0x54/0x58)
[   48.675194]  r7:00000000 r6:ec4b6f40 r5:ec4b703d r4:000007fb
[   48.681175] [<c0518f5c>] (skb_put+0x0/0x58) from [<c03f58e4>] (cpsw_rx_handler+0xa4/0x174)
[   48.689867]  r7:ed61b6c0 r6:ed32f000 r5:ed32f440 r4:ed511200
[   48.695839] [<c03f5840>] (cpsw_rx_handler+0x0/0x174) from [<c03f1bc0>] (__cpdma_chan_free+0x98/0xb0)
[   48.705442]  r8:ed61b6c0 r7:000007fb r6:00020000 r5:ed33b6c0 r4:f010c040
[   48.712513] [<c03f1b28>] (__cpdma_chan_free+0x0/0xb0) from [<c03f1cb4>] (__cpdma_chan_process+0xdc/0x118)
[   48.722580] [<c03f1bd8>] (__cpdma_chan_process+0x0/0x118) from [<c03f1d34>] (cpdma_chan_process+0x44/0x5c)
[   48.732737] [<c03f1cf0>] (cpdma_chan_process+0x0/0x5c) from [<c03f57ac>] (cpsw_poll+0x1c/0xb0)
[   48.741792]  r6:00000040 r5:ed32f448 r4:ed32f448 r3:c097e088
[   48.747780] [<c03f5790>] (cpsw_poll+0x0/0xb0) from [<c0523f08>] (net_rx_action+0xa4/0x1a8)
[   48.756470]  r6:00000040 r5:ed32f448 r4:c03f5790 r3:c097e088
[   48.762457] [<c0523e64>] (net_rx_action+0x0/0x1a8) from [<c004a35c>] (__do_softirq+0xc8/0x1ac)
[   48.771519] [<c004a294>] (__do_softirq+0x0/0x1ac) from [<c004a4e0>] (do_softirq+0x50/0x5c)
[   48.780216] [<c004a490>] (do_softirq+0x0/0x5c) from [<c004a758>] (irq_exit+0x9c/0xf0)
[   48.788450]  r4:ec55c000 r3:00000000
[   48.792238] [<c004a6bc>] (irq_exit+0x0/0xf0) from [<c0015564>] (handle_IRQ+0x3c/0x8c)
[   48.800471]  r4:c094d340 r3:00000112
[   48.804240] [<c0015528>] (handle_IRQ+0x0/0x8c) from [<c000879c>] (omap3_intc_handle_irq+0x68/0x7c)
[   48.813659]  r6:c097ef94 r5:ec55dfb0 r4:fa200000 r3:00000080
[   48.819636] [<c0008734>] (omap3_intc_handle_irq+0x0/0x7c) from [<c05d5020>] (__irq_usr+0x40/0x60)
[   48.828965] Exception stack(0xec55dfb0 to 0xec55dff8)
[   48.834276] dfa0:                                     ad108c40 ad4946b0 b66c7e8c b660f099
[   48.842877] dfc0: ae97c39c 00000000 0000000a 00000000 00000c20 0000f04f ad44da38 00000000
[   48.851478] dfe0: b6716d6c ae97c35c b6580841 b6580780 400f0030 ffffffff
[   48.858430]  r7:00000000 r6:ffffffff r5:400f0030 r4:b6580780
[   48.864397] Code: e58d4008 e58de00c e59f0008 ebfff524 (e7f001f2)
[   48.870816] ---[ end trace 1b75b31a2719ed1e ]---
[   48.875672] Kernel panic - not syncing: Fatal exception in interrupt

Crash #3

[   73.747547] skbuff: skb_over_panic: text:c03f3df0 len:2043 put:2043 head:ec343800 data:ec343842 tail:0xec34403d end:0xec343f40 dev:eth0
[   73.760906] ------------[ cut here ]------------
[   73.765944] Kernel BUG at c05ccae8 [verbose debug info unavailable]
[   73.772781] Internal error: Oops - BUG: 0 [#1] ARM
[   73.778012] Modules linked in: g_mass_storage usb_f_mass_storage libcomposite configfs bufferclass_ti(O) omaplfb(O) cryptodev(O) goodix wl18xx wlcore mac80211 cfg80211 musb_dsps musb_hdrc pvrsrvkm(O) btwilinkh
[   73.801594] CPU: 0 PID: 0 Comm: swapper Tainted: G        W  O 3.12.10-ti2013.12.01 #1
[   73.810238] task: c08f2aa0 ti: c08e8000 task.ti: c08e8000
[   73.816155] PC is at skb_panic+0x5c/0x68
[   73.820446] LR is at console_unlock+0x1c8/0x3fc
[   73.825389] pc : [<c05ccae8>]    lr : [<c006f9cc>]    psr: 600f0113
[   73.825389] sp : c08e9cd0  ip : c08e9bf8  fp : c08e9d04
[   73.837915] r10: ed34e5c0  r9 : 00000000  r8 : 000007fb
[   73.843614] r7 : c0787e50  r6 : ec343800  r5 : 000007fb  r4 : ec343842
[   73.850736] r3 : c093d238  r2 : c093d238  r1 : 00000000  r0 : 0000007b
[   73.857861] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[   73.865836] Control: 10c5387d  Table: ac088019  DAC: 00000017
[   73.872104] Process swapper (pid: 0, stack limit = 0xc08e8240)
[   73.878468] Stack: (0xc08e9cd0 to 0xc08ea000)
[   73.883224] 9cc0:                                     000007fb ec343800 ec343842 ec34403d
[   73.892151] 9ce0: ec343f40 ed328000 000007fb ec34403d ec343f40 00000000 c08e9d24 c08e9d08
[   73.901076] 9d00: c0513c74 c05cca98 ed493480 ed328440 ed328000 ed493300 c08e9d54 c08e9d28
[   73.910002] 9d20: c03f3df0 c0513c2c 600f0113 00020000 c08e9d54 f0110020 ed3416c0 00020000
[   73.918927] 9d40: 000007fb ed493300 c08e9d8c c08e9d58 c03f00cc c03f3d58 00000000 c08e9d68
[   73.927853] 9d60: c00589d8 00020000 200f0113 0000003f 3a212020 f0110000 c0975bc0 4a102000
[   73.936779] 9d80: c08e9db4 c08e9d90 c03f01c0 c03f0040 00000000 ed3416c0 00000040 0000012c
[   73.945704] 9da0: c0975bc0 c093bd40 c08e9dd4 c08e9db8 c03f0240 c03f00f0 c0975bc8 ed328448
[   73.954630] 9dc0: ed328448 00000040 c08e9df4 c08e9dd8 c03f3cb8 c03f0208 c0975bc8 c03f3c9c
[   73.963555] 9de0: ed328448 00000040 c08e9e2c c08e9df8 c051ebcc c03f3ca8 c0975bc8 ffffa7a0
[   73.972480] 9e00: 00000001 00000001 0000000c c0977c10 c0977c00 c08e8000 00000003 00000100
[   73.981405] 9e20: c08e9e74 c08e9e30 c004a2d0 c051eb34 00000001 ed005e40 0000000c 00200000
[   73.990331] 9e40: ffffa79f 0000000a 413fc082 600f0193 00000039 00000000 00000039 00000000
[   73.999256] 9e60: 413fc082 c0975c94 c08e9e8c c08e9e78 c004a454 c004a214 00000000 c08e8000
[   74.008181] 9e80: c08e9ea4 c08e9e90 c004a6cc c004a410 00000112 c0945308 c08e9ec4 c08e9ea8
[   74.017107] 9ea0: c0015564 c004a63c 00000080 fa200000 c08e9ee8 c0976ad4 c08e9ee4 c08e9ec8
[   74.026032] 9ec0: c000879c c0015534 c049b834 200f0013 ffffffff c08e9f1c c08e9f5c c08e9ee8
[   74.034958] 9ee0: c05cfb40 c0008740 c08e9f30 3b9aca00 000aca0c 00000011 2b411560 00000011
[   74.043884] 9f00: c09630b8 00000001 00000000 413fc082 c0975c94 c08e9f5c 0000001a c08e9f30
[   74.052809] 9f20: c0077f78 c049b834 200f0013 ffffffff 2b4bdf6c 00000011 00000000 c08f5f28
[   74.061734] 9f40: c09630b8 00000001 c08f5f28 c09a83b8 c08e9f84 c08e9f60 c049b96c c049b7e4
[   74.070659] 9f60: 00000000 c08e8000 00000000 c08f0104 c0975c94 00000001 c08e9f94 c08e9f88
[   74.079584] 9f80: c0015730 c049b8d4 c08e9fac c08e9f98 c00713c0 c001572c 00000000 c11c3500
[   74.088509] 9fa0: c08e9fbc c08e9fb0 c05c6994 c0071374 c08e9ff4 c08e9fc0 c07d0a48 c05c693c
[   74.097434] 9fc0: ffffffff ffffffff c07d0544 00000000 00000000 c080da50 10c53c7d c08f00a8
[   74.106360] 9fe0: c080da4c c08f3b40 00000000 c08e9ff8 80008070 c07d07b4 00000000 00000000
[   74.115280] Backtrace: 
[   74.117971] [<c05cca8c>] (skb_panic+0x0/0x68) from [<c0513c74>] (skb_put+0x54/0x58)
[   74.126325]  r7:00000000 r6:ec343f40 r5:ec34403d r4:000007fb
[   74.132541] [<c0513c20>] (skb_put+0x0/0x58) from [<c03f3df0>] (cpsw_rx_handler+0xa4/0x174)
[   74.141558]  r7:ed493300 r6:ed328000 r5:ed328440 r4:ed493480
[   74.147763] [<c03f3d4c>] (cpsw_rx_handler+0x0/0x174) from [<c03f00cc>] (__cpdma_chan_free+0x98/0xb0)
[   74.157726]  r8:ed493300 r7:000007fb r6:00020000 r5:ed3416c0 r4:f0110020
[   74.165074] [<c03f0034>] (__cpdma_chan_free+0x0/0xb0) from [<c03f01c0>] (__cpdma_chan_process+0xdc/0x118)
[   74.175518] [<c03f00e4>] (__cpdma_chan_process+0x0/0x118) from [<c03f0240>] (cpdma_chan_process+0x44/0x5c)
[   74.186056] [<c03f01fc>] (cpdma_chan_process+0x0/0x5c) from [<c03f3cb8>] (cpsw_poll+0x1c/0xb0)
[   74.195451]  r6:00000040 r5:ed328448 r4:ed328448 r3:c0975bc8
[   74.201663] [<c03f3c9c>] (cpsw_poll+0x0/0xb0) from [<c051ebcc>] (net_rx_action+0xa4/0x1a8)
[   74.210679]  r6:00000040 r5:ed328448 r4:c03f3c9c r3:c0975bc8
[   74.216896] [<c051eb28>] (net_rx_action+0x0/0x1a8) from [<c004a2d0>] (__do_softirq+0xc8/0x1ac)
[   74.226299] [<c004a208>] (__do_softirq+0x0/0x1ac) from [<c004a454>] (do_softirq+0x50/0x5c)
[   74.235322] [<c004a404>] (do_softirq+0x0/0x5c) from [<c004a6cc>] (irq_exit+0x9c/0xf0)
[   74.243863]  r4:c08e8000 r3:00000000
[   74.247795] [<c004a630>] (irq_exit+0x0/0xf0) from [<c0015564>] (handle_IRQ+0x3c/0x8c)
[   74.256337]  r4:c0945308 r3:00000112
[   74.260258] [<c0015528>] (handle_IRQ+0x0/0x8c) from [<c000879c>] (omap3_intc_handle_irq+0x68/0x7c)
[   74.270032]  r6:c0976ad4 r5:c08e9ee8 r4:fa200000 r3:00000080
[   74.276243] [<c0008734>] (omap3_intc_handle_irq+0x0/0x7c) from [<c05cfb40>] (__irq_svc+0x40/0x54)
[   74.285924] Exception stack(0xc08e9ee8 to 0xc08e9f30)
[   74.291436] 9ee0:                   c08e9f30 3b9aca00 000aca0c 00000011 2b411560 00000011
[   74.300362] 9f00: c09630b8 00000001 00000000 413fc082 c0975c94 c08e9f5c 0000001a c08e9f30
[   74.309285] 9f20: c0077f78 c049b834 200f0013 ffffffff
[   74.314794]  r7:c08e9f1c r6:ffffffff r5:200f0013 r4:c049b834
[   74.321017] [<c049b7d8>] (cpuidle_enter_state+0x0/0xf0) from [<c049b96c>] (cpuidle_idle_call+0xa4/0x148)
[   74.331360]  r7:c09a83b8 r6:c08f5f28 r5:00000001 r4:c09630b8
[   74.337567] [<c049b8c8>] (cpuidle_idle_call+0x0/0x148) from [<c0015730>] (arch_cpu_idle+0x10/0x4c)
[   74.347342]  r8:00000001 r7:c0975c94 r6:c08f0104 r5:00000000 r4:c08e8000
r3:00000000
[   74.355934] [<c0015720>] (arch_cpu_idle+0x0/0x4c) from [<c00713c0>] (cpu_startup_entry+0x58/0xf4)
[   74.365633] [<c0071368>] (cpu_startup_entry+0x0/0xf4) from [<c05c6994>] (rest_init+0x64/0x7c)
[   74.374934]  r7:c11c3500 r3:00000000
[   74.378872] [<c05c6930>] (rest_init+0x0/0x7c) from [<c07d0a48>] (start_kernel+0x2a0/0x2fc)
[   74.387908] [<c07d07a8>] (start_kernel+0x0/0x2fc) from [<80008070>] (0x80008070)
[   74.395980] Code: e58d4008 e58de00c e59f0008 ebfff524 (e7f001f2) 
[   74.402641] ---[ end trace 1b75b31a2719ed1e ]---
[   74.407680] Kernel panic - not syncing: Fatal exception in interrupt

  • Hi,

    I will forward this to the software team.
  • Looking at the console logs in each case looks it looks like a packet of 2043 was sent to the am3352.

    Could you go into more detail about the malformed packets? Is this testing that is being performed?

    Have any modifications been done to the kernel? What are the differences between the default kernel configuration from the SDK and what is being used here?
  • we did not modify the kernel itself.
    All we did is adaption to our hardware.

    The boards are connected to a router with dhcp.
    Developers access the boards via ssh and scp.

    We did not test the network explicitly.
    The crashes occur without any special user action or test conditions.
    So I have no information about the type or content of the network packets.
    (If my assumption is correct that some network traffic is the reason for the crashes)

  • Hello Schuyler,

    is there any chance to fix this issue?
    Or to do more debugging ?
    When the exception is detected in skb_put(), there is the chance to print out the packet or sth. else. Please advice.


    Best Regards,
    Roland
  • Putting a printk to print out the packet causing the problem might help, perhaps it is an IP packets. Set a qualifier whenever the packet len is greater than 1522. The packets received should not be larger than this as the part does not support jumbo packets.

    Is the same kernel config being used as what was provided with the SDK?
    What other network interfaces are in the system? How many USB interfaces?
  • I am wondering if you found what caused the problem? We are experiencing similar issue. It only happens in 1Gb mode though.
    I know that this topic is one year old, but there was no closure.

    [ 138.929732] skbuff: skb_over_panic: text:c0417c04 len:1709 put:1709 head:dc786b00 data:dc786b42 tail:0xdc7871ef end:0xdc787140 dev:eth1
    [ 138.942042] -----------[ cut here ]-----------
    [ 138.946677] Kernel BUG at c04bde4c [verbose debug info unavailable]
    [ 138.952967] Internal error: Oops - BUG: 0 1 PREEMPT ARM
    [ 138.958474] Modules linked in: sha512_generic sha512_arm sha1_generic sha1_arm_neon sha1_arm md5 jitterentropy_rng hmac drbg des_generic cbc bc_example(O) evdev
    joydev musb_dsps musb_hdrc mousedev udc_core usbcore ti_am335x_adc phy_am335x phy_am335x_control kfifo_buf industrialio pm33xx wkup_m3_rproc wkup_m3_ipc pvrsrvkm(
    O) remoteproc virtio virtio_ring omap_aes_driver omap_sham c_can_platform c_can omap_rng rng_core can_dev ti_am335x_tsc musb_am335x omap_wdt ti_am335x_tscadc sch_f
    q_codel cryptodev(O) autofs4
    [ 139.004024] CPU: 0 PID: 0 Comm: swapper Tainted: G O 4.4.19 #1
    [ 139.011011] Hardware name: Generic AM33XX (Flattened Device Tree)
    [ 139.017127] task: c07c2178 ti: c07be000 task.ti: c07be000
    [ 139.022562] PC is at skb_panic+0x68/0x6c
    [ 139.026511] LR is at irq_work_queue+0xf4/0xfc
    [ 139.030884] pc : [<c04bde4c>] lr : [<c009e660>] psr: 20000113
    [ 139.030884] sp : c07bfc90 ip : c07bfb60 fp : c07bfcbc
    [ 139.042407] r10: c001a40c r9 : 00020000 r8 : 000006ad
    [ 139.047651] r7 : c05e7028 r6 : dc786b00 r5 : dc786b42 r4 : dc7871ef
    [ 139.054201] r3 : c07be000 r2 : 00000101 r1 : c07cb1a4 r0 : 0000007b
    [ 139.060754] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
    [ 139.067916] Control: 10c5387d Table: 9c698019 DAC: 00000051
    [ 139.073682] Process swapper (pid: 0, stack limit = 0xc07be210)
    [ 139.079537] Stack: (0xc07bfc90 to 0xc07c0000)
    [ 139.083912] fc80: 000006ad dc786b00 dc786b42 dc7871ef
    [ 139.092126] fca0: dc787140 dd1d3800 c07bfcdc dc787140 c07bfcdc c07bfcc0 c04bebe8 c04bddf0
    [ 139.100340] fcc0: dd1d3800 dc26fb40 dd7e8a80 dd1d3c60 c07bfd0c c07bfce0 c0417c04 c04beb9c
    [ 139.108553] fce0: c02bf9b4 c006cdcc e0958a60 e0958a60 dd4f2310 dd4c8310 dd7e8a80 000006ad
    [ 139.116766] fd00: c07bfd44 c07bfd10 c0412a5c c0417b78 00000000 c0020ac8 3f8ec080 00020000
    [ 139.124980] fd20: e0958a60 000006ad c07cc340 0000012c c07bfdc8 ffffc116 c07bfd6c c07bfd48
    [ 139.133193] fd40: c0412ba8 c04129c0 c07bfd84 c07bfd58 c0083b10 00000025 dd4c8310 00000040
    [ 139.141406] fd60: c07bfd8c c07bfd70 c0412c60 c0412a84 dd1d2c68 dd1d2c68 00000040 c07cc340
    [ 139.149620] fd80: c07bfdac c07bfd90 c0419b84 c0412c28 dd1d2c68 00000001 00000040 c07cc340
    [ 139.157833] fda0: c07bfe04 c07bfdb0 c04ce068 c0419b74 c07382b4 c073c574 c0737e50 c07c07d0
    [ 139.166045] fdc0: c07f2e2c c07f2b40 c07bfdc8 c07bfdc8 c07bfdd0 c07bfdd0 00010000 00000000
    [ 139.174258] fde0: 00000003 c07f5e0c c07be000 00000101 c07f5e00 40000003 c07bfe64 c07bfe08
    [ 139.182472] fe00: c0034aac c04cde9c c06d84a4 c06d847c 20000193 00200000 c07cc340 ffffc115
    [ 139.190685] fe20: 0000000a c058cfe4 ffffe000 c07f5dc0 c07ca6c8 c07f5e00 c0066250 c07d5ac4
    [ 139.198898] fe40: 000000b4 00000000 00000000 dd006000 00000020 00000000 c07bfe7c c07bfe68
    [ 139.207112] fe60: c0034ef4 c00349a8 c07d5ac4 000000b4 c07bfea4 c07bfe80 c0065a6c c0034e14
    [ 139.215325] fe80: c082c7c0 20000013 ffffffff c07bfef4 58b573f6 00000020 c07bfebc c07bfea8
    [ 139.223538] fea0: c0009468 c0065a18 c043d1bc 20000013 c07bff4c c07bfec0 c0014194 c0009438
    [ 139.231752] fec0: 00000000 00000000 58b830f8 c07ca1a8 0002bd02 00000020 00000000 dd4f6200
    [ 139.239965] fee0: 58b573f6 00000020 00000000 c07bff4c c07bff10 c07bff10 c043d1b4 c043d1bc
    [ 139.248178] ff00: 20000013 ffffffff 00000051 c005cc08 00000023 c07e7d28 58b830f8 00000020
    [ 139.256391] ff20: 00000023 00000000 dd4f6200 c07e7d90 c07e7938 c07c0124 c07f2c90 c07c011c
    [ 139.264605] ff40: c07bff5c c07bff50 c043d338 c043d048 c07bff74 c07bff60 c005cce0 c043d328
    [ 139.272818] ff60: c07be000 dd4f6200 c07bff94 c07bff78 c005ce24 c005ccbc c05862e8 c06d74bc
    [ 139.281032] ff80: 00000002 c07c0040 c07bffac c07bff98 c0584610 c005cd10 c07f504c c07f5000
    [ 139.289245] ffa0: c07bfff4 c07bffb0 c077bd40 c0584590 ffffffff ffffffff 00000000 c077b6e8
    [ 139.297459] ffc0: 00000000 c07ada28 00000000 c07f51d4 c07c00b8 c07ada24 c07c3354 80004059
    [ 139.305672] ffe0: 413fc082 00000000 00000000 c07bfff8 80008078 c077b990 00000000 00000000
    [ 139.313878] Backtrace:
    [ 139.316344] [<c04bdde4>] (skb_panic) from [<c04bebe8>] (skb_put+0x58/0x5c)
    [ 139.323244] r7:dc787140
    [ 139.325799] [<c04beb90>] (skb_put) from [<c0417c04>] (cpsw_rx_handler+0x98/0x1f8)
    [ 139.333309] r7:dd1d3c60 r6:dd7e8a80 r5:dc26fb40 r4:dd1d3800
    [ 139.339024] [<c0417b6c>] (cpsw_rx_handler) from [<c0412a5c>] (__cpdma_chan_free+0xa8/0xc4)
    [ 139.347318] r8:000006ad r7:dd7e8a80 r6:dd4c8310 r5:dd4f2310 r4:e0958a60
    [ 139.354078] [<c04129b4>] (_cpdma_chan_free) from [<c0412ba8>] (_cpdma_chan_process+0x130/0x1a4)
    [ 139.362983] r10:ffffc116 r9:c07bfdc8 r8:0000012c r7:c07cc340 r6:000006ad r5:e0958a60
    [ 139.370869] r4:00020000
    [ 139.373417] [<c0412a78>] (__cpdma_chan_process) from [<c0412c60>] (cpdma_chan_process+0x44/0x6c)
    [ 139.382234] r6:00000040 r5:dd4c8310 r4:00000025
    [ 139.386889] [<c0412c1c>] (cpdma_chan_process) from [<c0419b84>] (cpsw_rx_poll+0x1c/0x9c)
    [ 139.395009] r7:c07cc340 r6:00000040 r5:dd1d2c68 r4:dd1d2c68
    [ 139.400726] [<c0419b68>] (cpsw_rx_poll) from [<c04ce068>] (net_rx_action+0x1d8/0x2b4)
    [ 139.408584] r7:c07cc340 r6:00000040 r5:00000001 r4:dd1d2c68
    [ 139.414296] [<c04cde90>] (net_rx_action) from [<c0034aac>] (__do_softirq+0x110/0x258)
    [ 139.422154] r10:40000003 r9:c07f5e00 r8:00000101 r7:c07be000 r6:c07f5e0c r5:00000003
    [ 139.430039] r4:00000000
    [ 139.432588] [<c003499c>] (__do_softirq) from [<c0034ef4>] (irq_exit+0xec/0x120)
    [ 139.439923] r10:00000000 r9:00000020 r8:dd006000 r7:00000000 r6:00000000 r5:000000b4
    [ 139.447808] r4:c07d5ac4
    [ 139.450360] [<c0034e08>] (irq_exit) from [<c0065a6c>] (__handle_domain_irq+0x60/0xb0)
    [ 139.458218] r5:000000b4 r4:c07d5ac4
    [ 139.461819] [<c0065a0c>] (__handle_domain_irq) from [<c0009468>] (omap_intc_handle_irq+0x3c/0x98)
    [ 139.470724] r9:00000020 r8:58b573f6 r7:c07bfef4 r6:ffffffff r5:20000013 r4:c082c7c0
    [ 139.478528] [<c000942c>] (omap_intc_handle_irq) from [<c0014194>] (__irq_svc+0x54/0x90)
    [ 139.486562] Exception stack(0xc07bfec0 to 0xc07bff08)
    [ 139.491635] fec0: 00000000 00000000 58b830f8 c07ca1a8 0002bd02 00000020 00000000 dd4f6200
    [ 139.499849] fee0: 58b573f6 00000020 00000000 c07bff4c c07bff10 c07bff10 c043d1b4 c043d1bc
    [ 139.508058] ff00: 20000013 ffffffff
    [ 139.511555] r5:20000013 r4:c043d1bc
    [ 139.515163] [<c043d03c>] (cpuidle_enter_state) from [<c043d338>] (cpuidle_enter+0x1c/0x20)
    [ 139.523457] r10:c07c011c r9:c07f2c90 r8:c07c0124 r7:c07e7938 r6:c07e7d90 r5:dd4f6200
    [ 139.531341] r4:00000000
    [ 139.533895] [<c043d31c>] (cpuidle_enter) from [<c005cce0>] (call_cpuidle+0x30/0x54)
    [ 139.541586] [<c005ccb0>] (call_cpuidle) from [<c005ce24>] (cpu_startup_entry+0x120/0x1a8)
    [ 139.549793] r5:dd4f6200 r4:c07be000
    [ 139.553399] [<c005cd04>] (cpu_startup_entry) from [<c0584610>] (rest_init+0x8c/0x90)
    [ 139.561170] r7:c07c0040
    [ 139.563728] [<c0584584>] (rest_init) from [<c077bd40>] (start_kernel+0x3bc/0x3c8)
    [ 139.571238] r5:c07f5000 r4:c07f504c
    [ 139.574838] [<c077b984>] (start_kernel) from [<80008078>] (0x80008078)
    [ 139.581396] Code: e58d5008 e58d400c e58de010 ebefc9ce (e7f001f2)
    [ 139.587516] --[ end trace 538846175bc63939 ]--
    [ 139.592151] Kernel panic - not syncing: Fatal exception in interrupt
    [ 139.598532] ---[ end Kernel panic - not syncing: Fatal exception in interrupt