Linux/PROCESSOR-SDK-AM335X: Kernel issue with USB WiFi dongle

Craig McQueen

Tool/software: Linux

I am doing development for a BeagleBone Black based system, built with Yocto morty using linux-ti-staging from meta-mono (kernel 4.9.63-ge30fbd4352).

The system optionally supports a USB Wi-Fi dongle based on the RT2800 chipset (with RT chipset 5390, rev 0502, and with RF chipset 5370 — e.g. an Edup EP-N8537).

The system has been working for a while with 3.14.x and 4.4.x kernels.

Recently I've been updating it to a 4.9.x kernel. I've noticed that if it boots with the Wi-Fi device plugged in, about 50% of the time, just after the Wi-Fi initialises, I get an error from cppi41.c about a "Unable to handle kernel NULL pointer dereference at virtual address 00000104". Here is a serial debug console sample output:

Starting Connection Manager
Monitor for upgrade file on USB device
Starting Inner Range user reset button: [   19.202475] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
ir-user-reset.
[   19.677027] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Starting Dropbear SSH server: [   19.812796] 8021q: 802.1Q VLAN Support v1.8
[   19.846468] 8021q: adding VLAN 0 to HW filter on device eth0
dropbear.
Starting watchdog: [  OK  ]
[   20.239074] ieee80211 phy0: rt2x00lib_request_firmware: Info - Loading firmware file 'rt2870.bin'
[   20.272823] ieee80211 phy0: rt2x00lib_request_firmware: Info - Firmware detected - version: 0.36
Starting advanced power management daemon: No APM support in kernel
(failed.)
starting Busybox Periodic Command Scheduler: crond... done.
crond[974]: crond (busybox 1.24.1) started, log level 8

Starting syslogd/klogd: [   20.649326] ------------[ cut here ]------------
[   20.654211] WARNING: CPU: 0 PID: 4 at /home/teamcity/buildAgent/work/ee889e486c4390d4/poky/build/tmp/work-shared/webtegriti/kernel-source/drivers/dma/cppi41.c:466 cppi41_run_queue+0x100/0x10c
[   20.672058] Modules linked in: 8021q bridge stp llc xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_REDIRECT nf_nat_redirect nf_nat nf_conntrack iptable_filter ip_tables x_tables arc4 rt2800usb rt2x00usb rt2800lib rt2x00lib crc_ccitt mac80211 cfg80211 pru_rproc pruss_intc musb_dsps musb_hdrc usbcore phy_am335x phy_generic phy_am335x_control usb_common ti_am335x_adc omap_aes_driver pm33xx wkup_m3_ipc wkup_m3_rproc crypto_engine omap_sham pruss_soc_bus ti_emif_sram musb_am335x rtc_omap ir_lan_bbb omap_wdt ti_am335x_tscadc at24 pruss remoteproc
[   20.724525] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 4.9.63-ge30fbd4352 #2
[   20.731996] Hardware name: Generic AM33XX (Flattened Device Tree)
[   20.738389] Workqueue: pm pm_runtime_work
[   20.742587] Backtrace:
[   20.745167] [<c010b690>] (dump_backtrace) from [<c010b94c>] (show_stack+0x18/0x1c)
[   20.753096]  r7:00000009 r6:00000000 r5:c085f348 r4:00000000
[   20.759032] [<c010b934>] (show_stack) from [<c037852c>] (dump_stack+0x24/0x28)
[   20.766606] [<c0378508>] (dump_stack) from [<c012b4bc>] (__warn+0xe8/0x100)
[   20.773901] [<c012b3d4>] (__warn) from [<c012b58c>] (warn_slowpath_null+0x28/0x30)
[   20.781831]  r9:dc4dd410 r8:81004000 r7:dc4dd700 r6:00000078 r5:dc4dd490 r4:da9d5210
[   20.789943] [<c012b564>] (warn_slowpath_null) from [<c03f84c8>] (cppi41_run_queue+0x100/0x10c)
[   20.798965] [<c03f83c8>] (cppi41_run_queue) from [<c03f8518>] (cppi41_runtime_resume+0x44/0x7c)
[   20.808078]  r10:00000008 r9:c01593d8 r8:c044619c r7:00000000 r6:c044619c r5:a00c0013
[   20.816276]  r4:ffffe000
[   20.818928] [<c03f84d4>] (cppi41_runtime_resume) from [<c04461d0>] (pm_generic_runtime_resume+0x34/0x40)
[   20.828856]  r5:00000000 r4:da86b810
[   20.832600] [<c044619c>] (pm_generic_runtime_resume) from [<c0449080>] (__rpm_callback+0x124/0x26c)
[   20.842077] [<c0448f5c>] (__rpm_callback) from [<c04491f0>] (rpm_callback+0x28/0x88)
[   20.850189]  r10:00000008 r9:c01593d8 r8:c044619c r7:00000000 r6:dc145c10 r5:da86b810
[   20.858387]  r4:da86b810
[   20.861039] [<c04491c8>] (rpm_callback) from [<c0448c14>] (rpm_resume+0x3f4/0x69c)
[   20.868967]  r7:00000000 r6:dc145c10 r5:da86b810 r4:c0d13340
[   20.874894] [<c0448820>] (rpm_resume) from [<c0449b4c>] (pm_runtime_work+0x78/0xe0)
[   20.882915]  r9:00000000 r8:c0d0dc88 r7:dcb36c00 r6:00000000 r5:dc028600 r4:da86b8b4
[   20.891032] [<c0449ad4>] (pm_runtime_work) from [<c01424ec>] (process_one_work+0x1f8/0x424)
[   20.899780] [<c01422f4>] (process_one_work) from [<c0142784>] (worker_thread+0x6c/0x63c)
[   20.908257]  r10:00000008 r9:c0d0dc9c r8:c0d13340 r7:dc06c000 r6:c0d0dc88 r5:dc028618
[   20.916455]  r4:dc028600
[   20.919112] [<c0142718>] (worker_thread) from [<c0148374>] (kthread+0xf8/0x110)
[   20.926770]  r10:00000000 r9:00000000 r8:c0142718 r7:dc028600 r6:dc06c000 r5:dc02a400
[   20.934968]  r4:00000000
[   20.937621] [<c014827c>] (kthread) from [<c0107c38>] (ret_from_fork+0x14/0x3c)
[   20.945186]  r8:00000000 r7:00000000 r6:00000000 r5:c014827c r4:dc02a400
[   20.952202] ---[ end trace e9cb429abb495b1d ]---
[   20.957060] Unable to handle kernel NULL pointer dereference at virtual address 00000104
[   20.965533] pgd = c0004000
[   20.968362] [00000104] *pgd=00000000
[   20.972109] Internal error: Oops: 817 [#1] PREEMPT ARM
[   20.977487] Modules linked in: 8021q bridge stp llc xt_tcpudp iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_REDIRECT nf_nat_redirect nf_nat nf_conntrack iptable_filter ip_tables x_tables arc4 rt2800usb rt2x00usb rt2800lib rt2x00lib crc_ccitt mac80211 cfg80211 pru_rproc pruss_intc musb_dsps musb_hdrc usbcore phy_am335x phy_generic phy_am335x_control usb_common ti_am335x_adc omap_aes_driver pm33xx wkup_m3_ipc wkup_m3_rproc crypto_engine omap_sham pruss_soc_bus ti_emif_sram musb_am335x rtc_omap ir_lan_bbb omap_wdt ti_am335x_tscadc at24 pruss remoteproc
[   21.029922] CPU: 0 PID: 4 Comm: kworker/0:0 Tainted: G        W       4.9.63-ge30fbd4352 #2
[   21.038667] Hardware name: Generic AM33XX (Flattened Device Tree)
[   21.045051] Workqueue: pm pm_runtime_work
[   21.049249] task: dc055080 task.stack: dc06c000
[   21.053991] PC is at cppi41_run_queue+0x74/0x10c
[   21.058825] LR is at add_taint+0x28/0x64
[   21.062930] pc : [<c03f843c>]    lr : [<c012af90>]    psr: 600c0093
[   21.062930] sp : dc06dde8  ip : 00000000  fp : dc06de1c
[   21.074953] r10: 9d041402  r9 : dc4dd410  r8 : 81004000
[   21.080422] r7 : dc4dd700  r6 : 00000078  r5 : dc4dd490  r4 : da9d5210
[   21.087256] r3 : 00000200  r2 : 00000100  r1 : 00000200  r0 : 00000100
[   21.094092] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   21.101655] Control: 10c5387d  Table: 9ab54019  DAC: 00000051
[   21.107670] Process kworker/0:0 (pid: 4, stack limit = 0xdc06c208)
[   21.114140] Stack: (0xdc06dde8 to 0xdc06e000)
[   21.118704] dde0:                   c0d0e8e8 c085f348 dc0550b0 ffffe000 a00c0013 c044619c
[   21.127271] de00: 00000000 c044619c c01593d8 00000008 dc06de34 dc06de20 c03f8518 c03f83d4
[   21.135839] de20: da86b810 00000000 dc06de44 dc06de38 c04461d0 c03f84e0 dc06de7c dc06de48
[   21.144406] de40: c0449080 c04461a8 dc0550b0 dc0553c0 ffffffff da86b810 da86b810 dc145c10
[   21.152974] de60: 00000000 c044619c c01593d8 00000008 dc06de9c dc06de80 c04491f0 c0448f68
[   21.161541] de80: c0d13340 da86b810 dc145c10 00000000 dc06dedc dc06dea0 c0448c14 c04491d4
[   21.170109] dea0: da8eec40 00000004 dc06df0c dc06deb8 c06379c0 c014deb0 da86b8b4 dc028600
[   21.178676] dec0: 00000000 dcb36c00 c0d0dc88 00000000 dc06deec dc06dee0 c0449b4c c044882c
[   21.187244] dee0: dc06df24 dc06def0 c01424ec c0449ae0 dc06c000 c0d13340 c0d0dc9c dc028600
[   21.195811] df00: dc028618 c0d0dc88 dc06c000 c0d13340 c0d0dc9c 00000008 dc06df5c dc06df28
[   21.204378] df20: c0142784 c0142300 dc028600 c0d0dccc 00000000 00000000 dc02a400 dc06c000
[   21.212945] df40: dc028600 c0142718 00000000 00000000 dc06dfac dc06df60 c0148374 c0142724
[   21.221513] df60: dc06df60 00000000 c01394d0 dc028600 00000000 dc06df74 dc06df74 00000000
[   21.230080] df80: dc06df80 dc06df80 dc06dfac dc02a400 c014827c 00000000 00000000 00000000
[   21.238647] dfa0: 00000000 dc06dfb0 c0107c38 c0148288 00000000 00000000 00000000 00000000
[   21.247214] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   21.255781] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[   21.264342] Backtrace:
[   21.266905] [<c03f83c8>] (cppi41_run_queue) from [<c03f8518>] (cppi41_runtime_resume+0x44/0x7c)
[   21.276018]  r10:00000008 r9:c01593d8 r8:c044619c r7:00000000 r6:c044619c r5:a00c0013
[   21.284217]  r4:ffffe000
[   21.286868] [<c03f84d4>] (cppi41_runtime_resume) from [<c04461d0>] (pm_generic_runtime_resume+0x34/0x40)
[   21.296797]  r5:00000000 r4:da86b810
[   21.300540] [<c044619c>] (pm_generic_runtime_resume) from [<c0449080>] (__rpm_callback+0x124/0x26c)
[   21.310018] [<c0448f5c>] (__rpm_callback) from [<c04491f0>] (rpm_callback+0x28/0x88)
[   21.318129]  r10:00000008 r9:c01593d8 r8:c044619c r7:00000000 r6:dc145c10 r5:da86b810
[   21.326328]  r4:da86b810
[   21.328979] [<c04491c8>] (rpm_callback) from [<c0448c14>] (rpm_resume+0x3f4/0x69c)
[   21.336907]  r7:00000000 r6:dc145c10 r5:da86b810 r4:c0d13340
[   21.342835] [<c0448820>] (rpm_resume) from [<c0449b4c>] (pm_runtime_work+0x78/0xe0)
[   21.350856]  r9:00000000 r8:c0d0dc88 r7:dcb36c00 r6:00000000 r5:dc028600 r4:da86b8b4
[   21.358968] [<c0449ad4>] (pm_runtime_work) from [<c01424ec>] (process_one_work+0x1f8/0x424)
[   21.367717] [<c01422f4>] (process_one_work) from [<c0142784>] (worker_thread+0x6c/0x63c)
[   21.376192]  r10:00000008 r9:c0d0dc9c r8:c0d13340 r7:dc06c000 r6:c0d0dc88 r5:dc028618
[   21.384391]  r4:dc028600
[   21.387042] [<c0142718>] (worker_thread) from [<c0148374>] (kthread+0xf8/0x110)
[   21.394699]  r10:00000000 r9:00000000 r8:c0142718 r7:dc028600 r6:dc06c000 r5:dc02a400
[   21.402897]  r4:00000000
[   21.405550] [<c014827c>] (kthread) from [<c0107c38>] (ret_from_fork+0x14/0x3c)
[   21.413115]  r8:00000000 r7:00000000 r6:00000000 r5:c014827c r4:dc02a400
[   21.420136] Code: e594308c e3a00c01 e5942088 e3a01c02 (e5823004)
[   21.426517] ---[ end trace e9cb429abb495b1e ]---
[   21.431354] note: kworker/0:0[4] exited with preempt_count 1

This repeats apparently indefinitely, until the watchdog resets the system. A longer console capture is attached.

2017-11-28_4_9_kernel_oops.txt

over 8 years ago

0 Biser Gatchev-XID over 8 years ago

TI__Guru**** 393215 points

Hi,

The current AM335x Processor SDK is kernel v4.9.41: www.ti.com/.../PROCESSOR-SDK-AM335X Have you tried with it?

0 Craig McQueen over 8 years ago in reply to Biser Gatchev-XID

Intellectual 850 points

It looks as though that corresponds to kernel tag ti2017.03, which corresponds to meta-ti commit cbeff59a. I'll plan to try it out early next week.

0 Craig McQueen over 8 years ago in reply to Biser Gatchev-XID

Intellectual 850 points

I have now tried with meta-ti commit cbeff59a, which contains Linux kernel commit 7448cce4 tagged ti2017.03. When I boot it, the version is shown as 4.9.41-g7448cce4e5.

This Wi-Fi problem is also present in that version.

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Craig,

The kernel crash log tells the issue is related to runtime PM in the usb driver, but I don't have such setup to replicate the issue, is it possible for you to test with the community latest v4.14.3 kernel? I first want to check if the issue is already solved in the latest kernel.

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

I normally use the Yocto build meta-ti layer to get the TI kernel. That has a Yocto recipe for 4.9 but not 4.14. The 4.9 recipe is typically updated by Dan Murphy and signed-off by Denys Dmytriyenko at TI.

But perhaps I can create my own Yocto recipe for 4.14. I see the TI kernel git repository has a ti-linux-4.14.y branch; I assume I'd refer to the latest commit on that branch. I should be able to try it.

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Yes, you can use the ti-linux-4.14.y branch, it is sync'd with the upstream.

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

Now I'm looking into trying it, I see that the Yocto meta-ti layer, recipe linux-ti-staging_4.9.bb, actually points to branch ti-lsk-linux-4.9.y rather than ti-linux-4.9.y. What is the difference between the two?

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

I've made a Yocto recipe for the TI 4.14.3 kernel, based on branch ti-lsk-linux-4.14.y. I tested a boot-up with the Wi-Fi device plugged in, and it also had a similar kernel crash. Although in this case, the behaviour was somewhat different: The kernel messages didn't continue continuously (as they did in a 4.9 kernel), and it was able to complete the boot initscripts. But, the networking didn't fully initialise, so if I tried to login and do "ip address" command, it locked up and never displayed any information. A number of processes (which do network functions) were stuck in the "D" state according to the "ps" command.

(I've done this investigation on a separate project which is closer to a base BeagleBone Black, because it didn't require so many custom kernel patches and device tree, making this investigation faster to implement. See the attached log from the serial debug port.)

Fullscreen 2017-12-06_laura_wifi_kernel_4_14_3_crash.txt Download

U-Boot SPL 2016.05-dirty (Aug 21 2017 - 12:19:14)
Trying to boot from MMC2
Card did not respond to voltage select!
*** Warning - MMC init failed, using default environment

reading u-boot.img
reading u-boot.img


U-Boot 2016.05-dirty (Aug 21 2017 - 12:19:14 +1000)

       Watchdog enabled
I2C:   ready
DRAM:  512 MiB
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
<ethaddr> not set. Validating first E-fuse MAC
Net:   cpsw, usb_ether
Autoboot in 1 seconds; press SPACE to abort
Card did not respond to voltage select!
Card did not respond to voltage select!
Card did not respond to voltage select!
Card did not respond to voltage select!
Card did not respond to voltage select!
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
switch to partitions #0, OK
mmc1(part 0) is current device
SD/MMC found on device 1
switch to partitions #0, OK
mmc1(part 0) is current device
SD/MMC found on device 1
reading boot.scr
** Unable to read file boot.scr **
reading uEnv.txt
507 bytes read in 3 ms (165 KiB/s)
Loaded env from uEnv.txt
Importing environment from mmc1 ...
Running uenvcmd ...
Boot slot 1
5538304 bytes read in 317 ms (16.7 MiB/s)
36777 bytes read in 13 ms (2.7 MiB/s)
Kernel image @ 0x82000000 [ 0x000000 - 0x548200 ]
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 8fff4000, end 8fffffa8 ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.3-g74d9ccee38 (craigm@craig-linux) (gcc version 6.2.0 (GCC)) #2 PREEMPT Wed Dec 6 11:53:08 AEDT 2017
[    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: LAURA on TI AM335x BeagleBone Black
[    0.000000] Memory policy: Data cache writeback
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: UEFI not found.
[    0.000000] cma: Reserved 48 MiB at 0x9d000000
[    0.000000] CPU: All CPU(s) started in SVC mode.
[    0.000000] AM335X ES2.1 (sgx neon)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129920
[    0.000000] Kernel command line: console=ttyO0,115200n8 mmcdev=1 bootnum=1
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 452900K/524288K available (6144K kernel code, 276K rwdata, 1852K rodata, 8192K init, 238K bss, 22236K reserved, 49152K cma-reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xe0800000 - 0xff800000   ( 496 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0008000 - 0xc0700000   (7136 kB)
[    0.000000]       .init : 0xc0900000 - 0xc1100000   (8192 kB)
[    0.000000]       .data : 0xc1100000 - 0xc1145318   ( 277 kB)
[    0.000000]        .bss : 0xc1145318 - 0xc1180ed4   ( 239 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000]  Tasks RCU enabled.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
[    0.000000] OMAP clockevent source: timer2 at 24000000 Hz
[    0.000014] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[    0.000033] clocksource: timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[    0.000043] OMAP clocksource: timer1 at 24000000 Hz
[    0.000201] timer_probe: no matching timers found
[    0.000289] Calibrating delay loop... 996.14 BogoMIPS (lpj=4980736)
[    0.089192] pid_max: default: 32768 minimum: 301
[    0.089376] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.089390] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.090140] CPU: Testing write buffer coherency: ok
[    0.090785] Setting up static identity map for 0x80100000 - 0x80100060
[    0.090918] Hierarchical SRCU implementation.
[    0.091190] EFI services will not be available.
[    0.092285] devtmpfs: initialized
[    0.098713] random: get_random_u32 called from bucket_table_alloc+0x8c/0x1c4 with crng_init=0
[    0.099123] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[    0.099483] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.099503] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.102842] pinctrl core: initialized pinctrl subsystem
[    0.103501] DMI not present or invalid.
[    0.103867] NET: Registered protocol family 16
[    0.105784] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.118258] omap_hwmod: debugss: _wait_target_disable failed
[    0.170328] cpuidle: using governor ladder
[    0.170359] cpuidle: using governor menu
[    0.173967] OMAP GPIO hardware version 0.1
[    0.182343] No ATAGs?
[    0.182360] hw-breakpoint: debug architecture 0x4 unsupported.
[    0.191867] edma 49000000.edma: TI EDMA DMA engine driver
[    0.193198] vgaarb: loaded
[    0.193741] omap_i2c 44e0b000.i2c: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/pinmux_i2c0_pins, deferring probe
[    0.193797] omap_i2c 4819c000.i2c: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/pinmux_i2c2_pins, deferring probe
[    0.193930] pps_core: LinuxPPS API ver. 1 registered
[    0.193938] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.193957] PTP clock support registered
[    0.193988] EDAC MC: Ver: 3.0.0
[    0.194387] dmi: Firmware registration failed.
[    0.194766] omap-mailbox 480c8000.mailbox: omap mailbox rev 0x400
[    0.195941] clocksource: Switched to clocksource timer1
[    0.197498] NET: Registered protocol family 2
[    0.198147] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    0.198190] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
[    0.198228] TCP: Hash tables configured (established 4096 bind 4096)
[    0.198317] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.198334] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.198462] NET: Registered protocol family 1
[    0.266588] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[    0.266712] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[    0.268849] workingset: timestamp_bits=14 max_order=17 bucket_order=3
[    0.272505] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.273459] random: fast init done
[    0.274429] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[    0.274444] io scheduler noop registered
[    0.274451] io scheduler deadline registered
[    0.274642] io scheduler cfq registered (default)
[    0.274652] io scheduler mq-deadline registered
[    0.274659] io scheduler kyber registered
[    0.275793] pinctrl-single 44e10800.pinmux: 142 pins at pa f9e10800 size 568
[    0.279527] omap_uart 44e09000.serial: no wakeirq for uart0
[    0.279616] 44e09000.serial: ttyO0 at MMIO 0x44e09000 (irq = 30, base_baud = 3000000) is a OMAP UART0
[    0.856153] console [ttyO0] enabled
[    0.860291] omap_uart 48022000.serial: no wakeirq for uart1
[    0.866288] 48022000.serial: ttyO1 at MMIO 0x48022000 (irq = 31, base_baud = 3000000) is a OMAP UART1
[    0.876329] omap_uart 48024000.serial: no wakeirq for uart2
[    0.882250] 48024000.serial: ttyO2 at MMIO 0x48024000 (irq = 32, base_baud = 3000000) is a OMAP UART2
[    0.893151] omap_rng 48310000.rng: Random Number Generator ver. 20
[    0.899748] random: crng init done
[    0.912113] brd: module loaded
[    0.921352] loop: module loaded
[    0.926322] libphy: Fixed MDIO Bus: probed
[    0.996019] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6, bus freq 1000000
[    1.004035] davinci_mdio 4a101000.mdio: detected phy mask fffffffe
[    1.017013] libphy: 4a101000.mdio: probed
[    1.021229] davinci_mdio 4a101000.mdio: phy[0]: device 4a101000.mdio:00, driver SMSC LAN8710/LAN8720
[    1.031654] cpsw 4a100000.ethernet: Detected MACID = d0:39:72:53:ef:0e
[    1.038821] cpsw 4a100000.ethernet: cpts: overflow check period 500 (jiffies)
[    1.047590] i2c /dev entries driver
[    1.051574] IR NEC protocol handler initialized
[    1.056421] IR RC5(x/sz) protocol handler initialized
[    1.061699] IR RC6 protocol handler initialized
[    1.066447] IR JVC protocol handler initialized
[    1.071176] IR Sony protocol handler initialized
[    1.076007] IR SANYO protocol handler initialized
[    1.080917] IR Sharp protocol handler initialized
[    1.085826] IR MCE Keyboard/mouse protocol handler initialized
[    1.091928] IR XMP protocol handler initialized
[    1.097585] device-mapper: ioctl: 4.37.0-ioctl (2017-09-20) initialised: dm-devel@redhat.com
[    1.107263] cpuidle: enable-method property 'ti,am3352' found operations
[    1.115048] omap_hsmmc 48060000.mmc: Got CD GPIO
[    1.237534] ledtrig-cpu: registered to indicate activity on CPUs
[    1.244085] prcm_reset_reason 44e00f08.prcm_reset_reason: Reset reasons: software-reset
[    1.257030] NET: Registered protocol family 10
[    1.262928] Segment Routing with IPv6
[    1.266948] NET: Registered protocol family 17
[    1.271831] Key type dns_resolver registered
[    1.276452] omap_voltage_late_init: Voltage driver support not added
[    1.325588] tps65217 0-0024: TPS65217 ID 0xe version 1.2
[    1.331733] omap_i2c 44e0b000.i2c: bus 0 rev0.11 at 400 kHz
[    1.339361] omap_i2c 4819c000.i2c: bus 2 rev0.11 at 100 kHz
[    1.348134] input: buttons as /devices/platform/buttons/input/input0
[    1.355295] hctosys: unable to open rtc device (rtc0)
[    1.360750] mmc1: new high speed MMC card at address 0001
[    1.367018] mmcblk1: mmc1:0001 MMC04G 3.60 GiB
[    1.367202] mmcblk1boot0: mmc1:0001 MMC04G partition 1 2.00 MiB
[    1.367339] mmcblk1boot1: mmc1:0001 MMC04G partition 2 2.00 MiB
[    1.378434] mmcblk1rpmb: mmc1:0001 MMC04G partition 3 128 KiB
[    1.385431]  mmcblk1: p1 p2 p3 p4
[    1.406299] Freeing unused kernel memory: 8192K
[    4.388742] EXT4-fs (dm-0): recovery complete
[    4.395548] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[    6.865038] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
INIT: version 2.88 booting
Starting udev
[    8.530693] udevd[267]: starting version 3.2
[    8.666437] udevd[268]: starting eudev-3.2
[    9.050176] omap_rtc 44e3e000.rtc: already running
[    9.093600] at24 0-0050: 32768 byte 24c256 EEPROM, writable, 1 bytes/write
[    9.111799] omap_rtc 44e3e000.rtc: registered as rtc0
[    9.114713] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[    9.348520] omap-sham 53100000.sham: hw accel on OMAP rev 4.3
[    9.424461] remoteproc remoteproc0: wkup_m3 is available
[    9.453988] omap-aes 53500000.aes: OMAP AES hw accel rev: 3.2
[    9.462168] omap-aes 53500000.aes: will run requests pump with realtime priority
[    9.474869] wkup_m3_ipc 44e11324.wkup_m3_ipc: could not get rproc handle
[    9.523357] wkup_m3_ipc 44e11324.wkup_m3_ipc: could not get rproc handle
[    9.534808] PM: Cannot get wkup_m3_ipc handle
[    9.808399] PM: Cannot get wkup_m3_ipc handle
[    9.813045] remoteproc remoteproc0: powering up wkup_m3
[   10.015150] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 224228
[   10.074990] remoteproc remoteproc0: remote processor wkup_m3 is now up
[   10.075021] wkup_m3_ipc 44e11324.wkup_m3_ipc: CM3 Firmware Version = 0x192
[   11.704432] PM: bootloader does not support rtc-only!
[   11.803066] am335x-phy-driver 47401b00.usb-phy: 47401b00.usb-phy supply vcc not found, using dummy regulator
[   11.875914] usbcore: registered new interface driver usbfs
[   11.906389] usbcore: registered new interface driver hub
[   11.944932] usbcore: registered new device driver usb
[   12.074545] musb-hdrc musb-hdrc.1: MUSB HDRC host driver
[   12.087933] musb-hdrc musb-hdrc.1: new USB bus registered, assigned bus number 1
[   12.117007] hub 1-0:1.0: USB hub found
[   12.121002] hub 1-0:1.0: 1 port detected
[   12.576147] usb 1-1: new high-speed USB device number 2 using musb-hdrc
0
[   13.097846] usbcore: registered new interface driver rt2800usb
[   13.266356] usb 1-1: reset high-speed USB device number 2 using musb-hdrc
[   13.477193] ieee80211 phy0: rt2x00_set_rt: Info - RT chipset 5390, rev 0502 detected
[   13.511832] ieee80211 phy0: rt2x00_set_rf: Info - RF chipset 5370 detected
Starting Simple watchdog daemon: wd_keepalive: NOTE: Using --loop-exit so daemon will exit after 60 time intervals
wd_keepalive.
INIT: Entering runlevel: 5
Configuring network interfaces... [   14.408880] net eth0: initializing cpsw version 1.12 (0)
[   14.414458] cpsw 4a100000.ethernet: initialized cpsw ale version 1.4
[   14.422338] cpsw 4a100000.ethernet: ALE Table size 1024
[   14.527594] SMSC LAN8710/LAN8720 4a101000.mdio:00: attached PHY driver [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=4a101000.mdio:00, irq=POLL)
[   14.551071] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
done.
Starting system message bus: dbus.
[   14.834043] ip_tables: (C) 2000-2006 Netfilter Core Team
Starting Connection Manager
Monitor for upgrade file on USB device
Starting Inner Range user reset button: [   15.707903] nf_conntrack version 0.5.0 (8192 buckets, 32768 max)
No input device with buzzer found.
ir-user-reset.
Starting Dropbear SSH server: dropbear.
Starting watchdog: [  OK  ]
[   16.146812] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Starting advanced power management daemon: No APM support in kernel
(failed.)
starting Busybox Periodic Command Scheduler: crond... done.
crond[882]: crond (busybox 1.24.1) started, log level 8

Starting syslogd/klogd: [   16.491592] ieee80211 phy0: rt2x00lib_request_firmware: Info - Loading firmware file 'rt2870.bin'
done
[   16.507551] ieee80211 phy0: rt2x00lib_request_firmware: Info - Firmware detected - version: 0.36
 * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon
[   16.647326] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[   16.812136] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   16.818519] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   16.846380] ------------[ cut here ]------------
[   16.851271] WARNING: CPU: 0 PID: 38 at /home/craigm/yocto-laura/poky/build/tmp/work-shared/laura/kernel-source/drivers/dma/cppi41.c:466 cppi41_run_queue+0x100/0x10c
[   16.866664] Modules linked in: bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_filter ip_tables x_tables arc4 rt2800usb rt2x00usb rt2800lib crc_ccitt rt2x00lib mac80211 cfg80211 musb_dsps musb_hdrc usbcore phy_am335x phy_generic usb_common phy_am335x_control pm33xx wkup_m3_ipc wkup_m3_rproc remoteproc omap_aes_driver crypto_engine omap_sham omap_crypto ti_emif_sram at24 musb_am335x omap_wdt rtc_omap
[   16.909493] CPU: 0 PID: 38 Comm: kworker/0:2 Not tainted 4.14.3-g74d9ccee38 #2
[   16.917056] Hardware name: Generic AM33XX (Flattened Device Tree)
[   16.923448] Workqueue: pm pm_runtime_work
[   16.927647] Backtrace:
[   16.930232] [<c010bb08>] (dump_backtrace) from [<c010bde8>] (show_stack+0x18/0x1c)
[   16.938166]  r7:00000009 r6:00000000 r5:c085f080 r4:00000000
[   16.944104] [<c010bdd0>] (show_stack) from [<c069e7c8>] (dump_stack+0x24/0x28)
[   16.951682] [<c069e7a4>] (dump_stack) from [<c0129f54>] (__warn+0xe8/0x100)
[   16.958980] [<c0129e6c>] (__warn) from [<c012a024>] (warn_slowpath_null+0x28/0x30)
[   16.966914]  r9:db1d9010 r8:81004000 r7:db1d92fc r6:00000078 r5:db1d9090 r4:db1bd210
[   16.975032] [<c0129ffc>] (warn_slowpath_null) from [<c0413548>] (cppi41_run_queue+0x100/0x10c)
[   16.984058] [<c0413448>] (cppi41_run_queue) from [<c0413598>] (cppi41_runtime_resume+0x44/0x7c)
[   16.993176]  r10:c110e810 r9:c015ad3c r8:c045fc90 r7:00000000 r6:c045fc90 r5:a0040113
[   17.001377]  r4:ffffe000
[   17.004043] [<c0413554>] (cppi41_runtime_resume) from [<c045fcc4>] (pm_generic_runtime_resume+0x34/0x40)
[   17.013975]  r5:00000000 r4:db10a810
[   17.017728] [<c045fc90>] (pm_generic_runtime_resume) from [<c0462b90>] (__rpm_callback+0x124/0x26c)
[   17.027211] [<c0462a6c>] (__rpm_callback) from [<c0462d00>] (rpm_callback+0x28/0x88)
[   17.035327]  r10:c110e810 r9:c015ad3c r8:c045fc90 r7:00000000 r6:dc107010 r5:db10a810
[   17.043528]  r4:db10a810
[   17.046185] [<c0462cd8>] (rpm_callback) from [<c04626ac>] (rpm_resume+0x3f4/0x69c)
[   17.054116]  r7:00000000 r6:dc107010 r5:db10a810 r4:c1114fc0
[   17.060048] [<c04622b8>] (rpm_resume) from [<c04635e0>] (pm_runtime_work+0x78/0xe0)
[   17.068073]  r9:00000000 r8:c110e7fc r7:dcb3ae00 r6:00000000 r5:dc434000 r4:db10a8b4
[   17.076194] [<c0463568>] (pm_runtime_work) from [<c0140f68>] (process_one_work+0x1f8/0x448)
[   17.084947] [<c0140d70>] (process_one_work) from [<c0141224>] (worker_thread+0x6c/0x63c)
[   17.093427]  r10:c110e810 r9:c1114fc0 r8:00000008 r7:dc44c000 r6:dc434018 r5:c110e7fc
[   17.101628]  r4:dc434000
[   17.104290] [<c01411b8>] (worker_thread) from [<c0146f9c>] (kthread+0x130/0x160)
[   17.112042]  r10:dc057ea4 r9:dc4330d8 r8:dc434000 r7:dc44c000 r6:dc433080 r5:00000000
[   17.120243]  r4:dc4330c0
[   17.122900] [<c0146e6c>] (kthread) from [<c0107e70>] (ret_from_fork+0x14/0x24)
[   17.130470]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0146e6c
[   17.138671]  r4:dc433080
[   17.141321] ---[ end trace 2ef0867a0003cb95 ]---
[   17.146166] Unable to handle kernel NULL pointer dereference at virtual address 00000104
[   17.154641] pgd = c0004000
[   17.157472] [00000104] *pgd=00000000
[   17.161221] Internal error: Oops: 817 [#1] PREEMPT ARM
[   17.166601] Modules linked in: bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_filter ip_tables x_tables arc4 rt2800usb rt2x00usb rt2800lib crc_ccitt rt2x00lib mac80211 cfg80211 musb_dsps musb_hdrc usbcore phy_am335x phy_generic usb_common phy_am335x_control pm33xx wkup_m3_ipc wkup_m3_rproc remoteproc omap_aes_driver crypto_engine omap_sham omap_crypto ti_emif_sram at24 musb_am335x omap_wdt rtc_omap
[   17.209401] CPU: 0 PID: 38 Comm: kworker/0:2 Tainted: G        W       4.14.3-g74d9ccee38 #2
[   17.218238] Hardware name: Generic AM33XX (Flattened Device Tree)
[   17.224624] Workqueue: pm pm_runtime_work
[   17.228824] task: dc245800 task.stack: dc44c000
[   17.233570] PC is at cppi41_run_queue+0x74/0x10c
[   17.238406] LR is at add_taint+0x28/0x64
[   17.242512] pc : [<c04134bc>]    lr : [<c0129a38>]    psr: 60040193
[   17.249075] sp : dc44de00  ip : 00000000  fp : dc44de34
[   17.254546] r10: 9d041402  r9 : db1d9010  r8 : 81004000
[   17.260017] r7 : db1d92fc  r6 : 00000078  r5 : db1d9090  r4 : db1bd210
[   17.266854] r3 : 00000200  r2 : 00000100  r1 : 00000200  r0 : 00000100
[   17.273693] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   17.281258] Control: 10c5387d  Table: 9c1bc019  DAC: 00000051
[   17.287276] Process kworker/0:2 (pid: 38, stack limit = 0xdc44c208)
[   17.293840] Stack: (0xdc44de00 to 0xdc44e000)
[   17.298408] de00: c015350c c085f080 c110f5a8 ffffe000 a0040113 c045fc90 00000000 c045fc90
[   17.306980] de20: c015ad3c c110e810 dc44de4c dc44de38 c0413598 c0413454 db10a810 00000000
[   17.315552] de40: dc44de5c dc44de50 c045fcc4 c0413560 dc44de94 dc44de60 c0462b90 c045fc9c
[   17.324123] de60: c0544d7c dc245ba8 dc44dea4 db10a810 db10a810 dc107010 00000000 c045fc90
[   17.332695] de80: c015ad3c c110e810 dc44deb4 dc44de98 c0462d00 c0462a78 c1114fc0 db10a810
[   17.341266] dea0: dc107010 00000000 dc44def4 dc44deb8 c04626ac c0462ce4 c110f578 db1a51c0
[   17.349838] dec0: db1a51c0 00000008 dc44df24 dc44ded8 db10a8b4 dc434000 00000000 dcb3ae00
[   17.358409] dee0: c110e7fc 00000000 dc44df04 dc44def8 c04635e0 c04622c4 dc44df3c dc44df08
[   17.366981] df00: c0140f68 c0463574 dc44c000 00000008 c1114fc0 dc434000 c110e7fc dc434018
[   17.375552] df20: dc44c000 00000008 c1114fc0 c110e810 dc44df74 dc44df40 c0141224 c0140d7c
[   17.384123] df40: ffffe000 dc434000 dc4330d8 dc4330c0 00000000 dc433080 dc44c000 dc434000
[   17.392695] df60: dc4330d8 dc057ea4 dc44dfac dc44df78 c0146f9c c01411c4 00000000 c01411b8
[   17.401265] df80: 00000000 dc433080 c0146e6c 00000000 00000000 00000000 00000000 00000000
[   17.409837] dfa0: 00000000 dc44dfb0 c0107e70 c0146e78 00000000 00000000 00000000 00000000
[   17.418407] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   17.426978] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 ffff0000 ffff0000
[   17.435540] Backtrace:
[   17.438109] [<c0413448>] (cppi41_run_queue) from [<c0413598>] (cppi41_runtime_resume+0x44/0x7c)
[   17.447227]  r10:c110e810 r9:c015ad3c r8:c045fc90 r7:00000000 r6:c045fc90 r5:a0040113
[   17.455428]  r4:ffffe000
[   17.458086] [<c0413554>] (cppi41_runtime_resume) from [<c045fcc4>] (pm_generic_runtime_resume+0x34/0x40)
[   17.468017]  r5:00000000 r4:db10a810
[   17.471767] [<c045fc90>] (pm_generic_runtime_resume) from [<c0462b90>] (__rpm_callback+0x124/0x26c)
[   17.481250] [<c0462a6c>] (__rpm_callback) from [<c0462d00>] (rpm_callback+0x28/0x88)
[   17.489366]  r10:c110e810 r9:c015ad3c r8:c045fc90 r7:00000000 r6:dc107010 r5:db10a810
[   17.497567]  r4:db10a810
[   17.500223] [<c0462cd8>] (rpm_callback) from [<c04626ac>] (rpm_resume+0x3f4/0x69c)
[   17.508155]  r7:00000000 r6:dc107010 r5:db10a810 r4:c1114fc0
[   17.514087] [<c04622b8>] (rpm_resume) from [<c04635e0>] (pm_runtime_work+0x78/0xe0)
[   17.522111]  r9:00000000 r8:c110e7fc r7:dcb3ae00 r6:00000000 r5:dc434000 r4:db10a8b4
[   17.530227] [<c0463568>] (pm_runtime_work) from [<c0140f68>] (process_one_work+0x1f8/0x448)
[   17.538980] [<c0140d70>] (process_one_work) from [<c0141224>] (worker_thread+0x6c/0x63c)
[   17.547460]  r10:c110e810 r9:c1114fc0 r8:00000008 r7:dc44c000 r6:dc434018 r5:c110e7fc
[   17.555661]  r4:dc434000
[   17.558316] [<c01411b8>] (worker_thread) from [<c0146f9c>] (kthread+0x130/0x160)
[   17.566068]  r10:dc057ea4 r9:dc4330d8 r8:dc434000 r7:dc44c000 r6:dc433080 r5:00000000
[   17.574269]  r4:dc4330c0
[   17.576926] [<c0146e6c>] (kthread) from [<c0107e70>] (ret_from_fork+0x14/0x24)
[   17.584495]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0146e6c
[   17.592695]  r4:dc433080
[   17.595352] Code: e594308c e3a00c01 e5942088 e3a01c02 (e5823004)
[   17.601738] ---[ end trace 2ef0867a0003cb96 ]---
[   17.606902] note: kworker/0:2[38] exited with preempt_count 1
   ...done.
[   17.646606] ------------[ cut here ]------------
[   17.651489] WARNING: CPU: 0 PID: 832 at /home/craigm/yocto-laura/poky/build/tmp/work-shared/laura/kernel-source/drivers/dma/cppi41.c:466 cppi41_run_queue+0x100/0x10c
[   17.666931] Modules linked in: bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_filter ip_tables x_tables arc4 rt2800usb rt2x00usb rt2800lib crc_ccitt rt2x00lib mac80211 cfg80211 musb_dsps musb_hdrc usbcore phy_am335x phy_generic usb_common phy_am335x_control pm33xx wkup_m3_ipc wkup_m3_rproc remoteproc omap_aes_driver crypto_engine omap_sham omap_crypto ti_emif_sram at24 musb_am335x omap_wdt rtc_omap
[   17.709628] CPU: 0 PID: 832 Comm: connmand Tainted: G      D W       4.14.3-g74d9ccee38 #2
[   17.718261] Hardware name: Generic AM33XX (Flattened Device Tree)
[   17.724624] Backtrace:
[   17.727200] [<c010bb08>] (dump_backtrace) from [<c010bde8>] (show_stack+0x18/0x1c)
[   17.735110]  r7:00000009 r6:00000000 r5:c085f080 r4:00000000
[   17.741030] [<c010bdd0>] (show_stack) from [<c069e7c8>] (dump_stack+0x24/0x28)
[   17.748585] [<c069e7a4>] (dump_stack) from [<c0129f54>] (__warn+0xe8/0x100)
[   17.755862] [<c0129e6c>] (__warn) from [<c012a024>] (warn_slowpath_null+0x28/0x30)
[   17.763773]  r9:db1d9010 r8:81004000 r7:db1d92fc r6:00000078 r5:db1d9090 r4:db1bd210
[   17.771867] [<c0129ffc>] (warn_slowpath_null) from [<c0413548>] (cppi41_run_queue+0x100/0x10c)
[   17.780868] [<c0413448>] (cppi41_run_queue) from [<c041400c>] (cppi41_dma_issue_pending+0x128/0x12c)
[   17.790414]  r10:db1d1b80 r9:9b71a04c r8:db1bd210 r7:00000000 r6:60010193 r5:db1bd210
[   17.798593]  r4:db1d9010
[   17.801305] [<c0413ee4>] (cppi41_dma_issue_pending) from [<bf112518>] (cppi41_dma_channel_program+0x1c8/0x294 [musb_hdrc])
[   17.812846]  r7:00000000 r6:c0413e68 r5:dc622048 r4:dc622048
[   17.818791] [<bf112350>] (cppi41_dma_channel_program [musb_hdrc]) from [<bf10d870>] (musb_start_urb+0x80c/0xd78 [musb_hdrc])
[   17.830513]  r10:dc5604c8 r9:bf115484 r8:db5c2a00 r7:dc622048 r6:00000200 r5:bf112350
[   17.838691]  r4:dc560010
[   17.841366] [<bf10d064>] (musb_start_urb [musb_hdrc]) from [<bf10df40>] (musb_advance_schedule+0x164/0x2bc [musb_hdrc])
[   17.852636]  r10:00000000 r9:00000000 r8:dc5604c8 r7:dc560010 r6:dc5604c8 r5:00000080
[   17.860814]  r4:db690e80
[   17.863490] [<bf10dddc>] (musb_advance_schedule [musb_hdrc]) from [<bf10e224>] (musb_cleanup_urb+0x18c/0x1e4 [musb_hdrc])
[   17.874940]  r10:dc622048 r9:dc560010 r8:e0b8dc10 r7:00000000 r6:db5c2980 r5:00000080
[   17.883120]  r4:dc5604c8
[   17.885795] [<bf10e098>] (musb_cleanup_urb [musb_hdrc]) from [<bf10e980>] (musb_urb_dequeue+0x14c/0x154 [musb_hdrc])
[   17.896792]  r10:00000001 r9:db690e80 r8:00000000 r7:c0008280 r6:dc560010 r5:a0010113
[   17.904970]  r4:db5c2980
[   17.907870] [<bf10e834>] (musb_urb_dequeue [musb_hdrc]) from [<bf0cbaf4>] (unlink1+0x34/0x144 [usbcore])
[   17.917779]  r10:dc5629ac r9:00000001 r8:60010113 r7:fffffffe r6:dc4bb800 r5:db5c2980
[   17.925958]  r4:db5c2980 r3:bf10e834
[   17.929840] [<bf0cbac0>] (unlink1 [usbcore]) from [<bf0cdea0>] (usb_hcd_unlink_urb+0xa0/0xc0 [usbcore])
[   17.939659]  r9:00000001 r8:60010113 r7:fffffffe r6:dc6cf000 r5:db5c2980 r4:00000000
[   17.947882] [<bf0cde00>] (usb_hcd_unlink_urb [usbcore]) from [<bf0cf174>] (usb_kill_urb.part.3+0x38/0xb0 [usbcore])
[   17.958789]  r9:00000001 r8:bf231a40 r7:00000000 r6:dc562d20 r5:db5c298c r4:db5c2980
[   17.967010] [<bf0cf13c>] (usb_kill_urb.part.3 [usbcore]) from [<bf0cf21c>] (usb_kill_urb+0x30/0x34 [usbcore])
[   17.977370]  r6:dc562d20 r5:db18ee78 r4:db18e348
[   17.982277] [<bf0cf1ec>] (usb_kill_urb [usbcore]) from [<bf231a78>] (rt2x00usb_flush_entry+0x38/0x60 [rt2x00usb])
[   17.993069] [<bf231a40>] (rt2x00usb_flush_entry [rt2x00usb]) from [<bf1f8cf8>] (rt2x00queue_for_each_entry+0x108/0x194 [rt2x00lib])
[   18.005427]  r7:00000000 r6:00000364 r5:db1f4e00 r4:0000001f
[   18.011363] [<bf1f8bf0>] (rt2x00queue_for_each_entry [rt2x00lib]) from [<bf231a3c>] (rt2x00usb_flush_queue+0xa4/0xa8 [rt2x00usb])
[   18.023540]  r9:00000000 r8:dc562a48 r7:db18c4e0 r6:00000001 r5:db1f4e00 r4:db1f4e00
[   18.031655] [<bf231998>] (rt2x00usb_flush_queue [rt2x00usb]) from [<bf1f91d4>] (rt2x00queue_flush_queue+0x4c/0xb0 [rt2x00lib])
[   18.043557]  r6:00000001 r5:dc562d20 r4:db1f4e00
[   18.048413] [<bf1f9188>] (rt2x00queue_flush_queue [rt2x00lib]) from [<bf1f92a4>] (rt2x00queue_flush_queues+0x6c/0x70 [rt2x00lib])
[   18.060586]  r4:db1f4f80
[   18.063264] [<bf1f9238>] (rt2x00queue_flush_queues [rt2x00lib]) from [<bf1f6354>] (rt2x00lib_disable_radio+0x50/0x9c [rt2x00lib])
[   18.075439]  r7:db18c4e0 r6:0000000c r5:dc562440 r4:dc562d20
[   18.081384] [<bf1f6304>] (rt2x00lib_disable_radio [rt2x00lib]) from [<bf1f6dc8>] (rt2x00lib_stop+0x34/0x48 [rt2x00lib])
[   18.092651]  r5:dc562440 r4:dc562d20
[   18.096418] [<bf1f6d94>] (rt2x00lib_stop [rt2x00lib]) from [<bf1f7404>] (rt2x00mac_stop+0x20/0x24 [rt2x00lib])
[   18.106868]  r5:dc562440 r4:dc562440
[   18.110921] [<bf1f73e4>] (rt2x00mac_stop [rt2x00lib]) from [<bf181fcc>] (drv_stop+0x28/0x7c [mac80211])
[   18.120919] [<bf181fa4>] (drv_stop [mac80211]) from [<bf1afc2c>] (ieee80211_stop_device+0x40/0x44 [mac80211])
[   18.131279]  r5:dc562440 r4:dc562440
[   18.135179] [<bf1afbec>] (ieee80211_stop_device [mac80211]) from [<bf19403c>] (ieee80211_do_stop+0x68c/0x914 [mac80211])
[   18.146536]  r5:dc562440 r4:db18c508
[   18.150435] [<bf1939b0>] (ieee80211_do_stop [mac80211]) from [<bf1942dc>] (ieee80211_stop+0x18/0x20 [mac80211])
[   18.160978]  r10:00000000 r9:00000000 r8:be93c9e0 r7:00009003 r6:00000000 r5:dc1c5e18
[   18.169156]  r4:db18c000
[   18.171888] [<bf1942c4>] (ieee80211_stop [mac80211]) from [<c05a7110>] (__dev_close_many+0x98/0xdc)
[   18.181350] [<c05a7078>] (__dev_close_many) from [<c05af744>] (__dev_change_flags+0xa8/0x178)
[   18.190257]  r5:ffff9002 r4:db18c000
[   18.193993] [<c05af69c>] (__dev_change_flags) from [<c05af834>] (dev_change_flags+0x20/0x50)
[   18.202811]  r8:be93c9e0 r7:db18c140 r6:00000000 r5:00009003 r4:db18c000
[   18.209825] [<c05af814>] (dev_change_flags) from [<c06251fc>] (devinet_ioctl+0x6ec/0x7ac)
[   18.218370]  r9:00000000 r8:be93c9e0 r7:db3f9a0c r6:00000000 r5:dc1c5e90 r4:00000000
[   18.226464] [<c0624b10>] (devinet_ioctl) from [<c0627458>] (inet_ioctl+0x19c/0x1c8)
[   18.234467]  r10:00000000 r9:dc1c4000 r8:be93c9e0 r7:c113d178 r6:be93c9e0 r5:dbb5e920
[   18.242645]  r4:00008914
[   18.245292] [<c06272bc>] (inet_ioctl) from [<c058d5c4>] (sock_ioctl+0x154/0x2f8)
[   18.253028] [<c058d470>] (sock_ioctl) from [<c02335c4>] (do_vfs_ioctl+0xa8/0x948)
[   18.260848]  r7:00000010 r6:db680540 r5:dbb5e920 r4:be93c9e0
[   18.266763] [<c023351c>] (do_vfs_ioctl) from [<c0233ea0>] (SyS_ioctl+0x3c/0x64)
[   18.274401]  r10:00000000 r9:dc1c4000 r8:be93c9e0 r7:00008914 r6:db680540 r5:00000010
[   18.282579]  r4:db680540
[   18.285229] [<c0233e64>] (SyS_ioctl) from [<c0107da0>] (ret_fast_syscall+0x0/0x48)
[   18.293140]  r9:dc1c4000 r8:c0107f64 r7:00000036 r6:00000003 r5:00000010 r4:be93c9e0
[   18.301230] ---[ end trace 2ef0867a0003cb97 ]---
[   18.306062] Unable to handle kernel NULL pointer dereference at virtual address 00000104
[   18.314515] pgd = dc1bc000
[   18.317336] [00000104] *pgd=9b55f831, *pte=00000000, *ppte=00000000
[   18.323899] Internal error: Oops: 817 [#2] PREEMPT ARM
[   18.329265] Modules linked in: bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_filter ip_tables x_tables arc4 rt2800usb rt2x00usb rt2800lib crc_ccitt rt2x00lib mac80211 cfg80211 musb_dsps musb_hdrc usbcore phy_am335x phy_generic usb_common phy_am335x_control pm33xx wkup_m3_ipc wkup_m3_rproc remoteproc omap_aes_driver crypto_engine omap_sham omap_crypto ti_emif_sram at24 musb_am335x omap_wdt rtc_omap
[   18.371944] CPU: 0 PID: 832 Comm: connmand Tainted: G      D W       4.14.3-g74d9ccee38 #2
[   18.380577] Hardware name: Generic AM33XX (Flattened Device Tree)
[   18.386941] task: db20aa00 task.stack: dc1c4000
[   18.391676] PC is at cppi41_run_queue+0x74/0x10c
[   18.396503] LR is at add_taint+0x28/0x64
[   18.400596] pc : [<c04134bc>]    lr : [<c0129a38>]    psr: 60010193
[   18.407141] sp : dc1c59d8  ip : 00000000  fp : dc1c5a0c
[   18.412597] r10: 9d041402  r9 : db1d9010  r8 : 81004000
[   18.418052] r7 : db1d92fc  r6 : 00000078  r5 : db1d9090  r4 : db1bd210
[   18.424870] r3 : 00000200  r2 : 00000100  r1 : 00000200  r0 : 00000100
[   18.431690] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   18.439234] Control: 10c5387d  Table: 9c1bc019  DAC: 00000051
[   18.445235] Process connmand (pid: 832, stack limit = 0xdc1c4208)
[   18.451598] Stack: (0xdc1c59d8 to 0xdc1c6000)
[   18.456150] 59c0:                                                       db1bd210 c085f080
[   18.464698] 59e0: dc1c5a0c db1d9010 db1bd210 60010193 00000000 db1bd210 9b71a04c db1d1b80
[   18.473246] 5a00: dc1c5a2c dc1c5a10 c041400c c0413454 dc622048 dc622048 c0413e68 00000000
[   18.481793] 5a20: dc1c5a7c dc1c5a30 bf112518 c0413ef0 00000003 00000000 dc1c5a9c 00000000
[   18.490340] 5a40: 00000002 00000000 00000200 9b71a04c db5c2980 dc560010 bf112350 00000200
[   18.498886] 5a60: dc622048 db5c2a00 bf115484 dc5604c8 dc1c5b14 dc1c5a80 bf10d870 bf11235c
[   18.507433] 5a80: 00000f14 00000003 00000002 00000001 bf11409c 00000000 00000000 00000f14
[   18.515980] 5aa0: db5c2980 60010193 dc1c5ad4 dc1c5ab8 bf0cb6b4 00000000 e0b8dc00 00000000
[   18.524527] 5ac0: 00000200 db1d1b80 db71a04c e0b8dc10 00000f14 e0b8dc00 00000080 bf11548c
[   18.533073] 5ae0: dc622048 00000002 dc1c5b14 db690e80 00000080 dc5604c8 dc560010 dc5604c8
[   18.541619] 5b00: 00000000 00000000 dc1c5b54 dc1c5b18 bf10df40 bf10d070 db5c2a00 bf115484
[   18.550165] 5b20: e0b8dc10 00000001 dc1c5b54 dc5604c8 00000080 db5c2980 00000000 e0b8dc10
[   18.558713] 5b40: dc560010 dc622048 dc1c5b94 dc1c5b58 bf10e224 bf10dde8 db5c2980 00000000
[   18.567259] 5b60: db5c2900 a0040113 dc1c5b84 db5c2980 a0010113 dc560010 c0008280 00000000
[   18.575806] 5b80: db690e80 00000001 dc1c5bc4 dc1c5b98 bf10e980 bf10e0a4 bf10e834 db5c2980
[   18.584352] 5ba0: db5c2980 dc4bb800 fffffffe 60010113 00000001 dc5629ac dc1c5bec dc1c5bc8
[   18.592899] 5bc0: bf0cbaf4 bf10e840 00000000 db5c2980 dc6cf000 fffffffe 60010113 00000001
[   18.601445] 5be0: dc1c5c14 dc1c5bf0 bf0cdea0 bf0cbacc db5c2980 db5c298c dc562d20 00000000
[   18.609992] 5c00: bf231a40 00000001 dc1c5c4c dc1c5c18 bf0cf174 bf0cde0c 00000040 00000006
[   18.618538] 5c20: 00001004 00000040 dc1c5c6c dc1c5c38 bf231348 db18e348 db18ee78 dc562d20
[   18.627085] 5c40: dc1c5c5c dc1c5c50 bf0cf21c bf0cf148 dc1c5c7c dc1c5c60 bf231a78 bf0cf1f8
[   18.635632] 5c60: 0000001f db1f4e00 00000364 00000000 dc1c5cac dc1c5c80 bf1f8cf8 bf231a4c
[   18.644178] 5c80: dc1c5cbc dc563058 db1f4e00 db1f4e00 00000001 db18c4e0 dc562a48 00000000
[   18.652725] 5ca0: dc1c5cd4 dc1c5cb0 bf231a3c bf1f8bfc bf231a40 bf231360 dc1c5cc4 db1f4e00
[   18.661271] 5cc0: dc562d20 00000001 dc1c5cf4 dc1c5cd8 bf1f91d4 bf2319a4 dc1c5cfc dc1c5ce8
[   18.669818] 5ce0: bf1f8dec db1f4f80 dc1c5d14 dc1c5cf8 bf1f92a4 bf1f9194 dc562d20 dc562440
[   18.678365] 5d00: 0000000c db18c4e0 dc1c5d2c dc1c5d18 bf1f6354 bf1f9244 dc562d20 dc562440
[   18.686911] 5d20: dc1c5d44 dc1c5d30 bf1f6dc8 bf1f6310 dc562440 dc562440 dc1c5d54 dc1c5d48
[   18.695458] 5d40: bf1f7404 bf1f6da0 dc1c5d6c dc1c5d58 bf181fcc bf1f73f0 dc562440 dc562440
[   18.704005] 5d60: dc1c5d84 dc1c5d70 bf1afc2c bf181fb0 db18c508 dc562440 dc1c5de4 dc1c5d88
[   18.712551] 5d80: bf19403c bf1afbf8 c06b4118 00000000 00000000 db18c4e0 600b0013 00000001
[   18.721099] 5da0: c010c9d0 00000010 db18c000 001fff00 00000000 bf1942c4 db18c000 db18c000
[   18.729645] 5dc0: dc1c5e18 00000000 00009003 be93c9e0 00000000 00000000 dc1c5df4 dc1c5de8
[   18.738192] 5de0: bf1942dc bf1939bc dc1c5e14 dc1c5df8 c05a7110 bf1942d0 c05af3b4 db18c000
[   18.746738] 5e00: db18c000 ffff9002 dc1c5e44 dc1c5e18 c05af744 c05a7084 db18c04c db18c04c
[   18.755284] 5e20: dc1c5e3c db18c000 00009003 00000000 db18c140 be93c9e0 dc1c5e6c dc1c5e48
[   18.763830] 5e40: c05af834 c05af6a8 00000000 dc1c5e90 00000000 db3f9a0c be93c9e0 00000000
[   18.772376] 5e60: dc1c5edc dc1c5e70 c06251fc c05af820 dc1c5e94 db18c000 be93c9e0 db3f9a0c
[   18.780922] 5e80: db3f9a00 00000014 db18c000 00008914 6e616c77 00000030 00000000 00000000
[   18.789468] 5ea0: 00009002 00000000 00000000 00000000 00009003 00008914 dbb5e920 be93c9e0
[   18.798015] 5ec0: c113d178 be93c9e0 dc1c4000 00000000 dc1c5eec dc1c5ee0 c0627458 c0624b1c
[   18.806561] 5ee0: dc1c5f0c dc1c5ef0 c058d5c4 c06272c8 be93c9e0 dbb5e920 db680540 00000010
[   18.815108] 5f00: dc1c5f7c dc1c5f10 c02335c4 c058d47c c0221a2c c03a7e14 c1103498 00080000
[   18.823654] 5f20: dbb5e900 dbb5e900 dc1c5f6c dc1c5f38 c058d2d4 c02219e0 dc006e10 00000000
[   18.832200] 5f40: 00080000 00000010 dbb5e900 c0107f64 dc1c5f6c db680540 00000010 db680540
[   18.840746] 5f60: 00008914 be93c9e0 dc1c4000 00000000 dc1c5fa4 dc1c5f80 c0233ea0 c0233528
[   18.849292] 5f80: be93c9e0 00000010 00000003 00000036 c0107f64 dc1c4000 00000000 dc1c5fa8
[   18.857838] 5fa0: c0107da0 c0233e70 be93c9e0 00000010 00000010 00008914 be93c9e0 ffff9002
[   18.866384] 5fc0: be93c9e0 00000010 00000003 00000036 000cf408 000d40e8 00000000 00000000
[   18.874930] 5fe0: 000ae8a0 be93c9dc 00058f04 b6b85e6c 200f0010 00000010 00000000 00000000
[   18.883470] Backtrace:
[   18.886030] [<c0413448>] (cppi41_run_queue) from [<c041400c>] (cppi41_dma_issue_pending+0x128/0x12c)
[   18.895575]  r10:db1d1b80 r9:9b71a04c r8:db1bd210 r7:00000000 r6:60010193 r5:db1bd210
[   18.903753]  r4:db1d9010
[   18.906438] [<c0413ee4>] (cppi41_dma_issue_pending) from [<bf112518>] (cppi41_dma_channel_program+0x1c8/0x294 [musb_hdrc])
[   18.917978]  r7:00000000 r6:c0413e68 r5:dc622048 r4:dc622048
[   18.923922] [<bf112350>] (cppi41_dma_channel_program [musb_hdrc]) from [<bf10d870>] (musb_start_urb+0x80c/0xd78 [musb_hdrc])
[   18.935646]  r10:dc5604c8 r9:bf115484 r8:db5c2a00 r7:dc622048 r6:00000200 r5:bf112350
[   18.943824]  r4:dc560010
[   18.946501] [<bf10d064>] (musb_start_urb [musb_hdrc]) from [<bf10df40>] (musb_advance_schedule+0x164/0x2bc [musb_hdrc])
[   18.957770]  r10:00000000 r9:00000000 r8:dc5604c8 r7:dc560010 r6:dc5604c8 r5:00000080
[   18.965948]  r4:db690e80
[   18.968624] [<bf10dddc>] (musb_advance_schedule [musb_hdrc]) from [<bf10e224>] (musb_cleanup_urb+0x18c/0x1e4 [musb_hdrc])
[   18.980076]  r10:dc622048 r9:dc560010 r8:e0b8dc10 r7:00000000 r6:db5c2980 r5:00000080
[   18.988254]  r4:dc5604c8
[   18.990930] [<bf10e098>] (musb_cleanup_urb [musb_hdrc]) from [<bf10e980>] (musb_urb_dequeue+0x14c/0x154 [musb_hdrc])
[   19.001926]  r10:00000001 r9:db690e80 r8:00000000 r7:c0008280 r6:dc560010 r5:a0010113
[   19.010104]  r4:db5c2980
[   19.012907] [<bf10e834>] (musb_urb_dequeue [musb_hdrc]) from [<bf0cbaf4>] (unlink1+0x34/0x144 [usbcore])
[   19.022817]  r10:dc5629ac r9:00000001 r8:60010113 r7:fffffffe r6:dc4bb800 r5:db5c2980
[   19.030996]  r4:db5c2980 r3:bf10e834
[   19.034865] [<bf0cbac0>] (unlink1 [usbcore]) from [<bf0cdea0>] (usb_hcd_unlink_urb+0xa0/0xc0 [usbcore])
[   19.044683]  r9:00000001 r8:60010113 r7:fffffffe r6:dc6cf000 r5:db5c2980 r4:00000000
[   19.052903] [<bf0cde00>] (usb_hcd_unlink_urb [usbcore]) from [<bf0cf174>] (usb_kill_urb.part.3+0x38/0xb0 [usbcore])
[   19.063810]  r9:00000001 r8:bf231a40 r7:00000000 r6:dc562d20 r5:db5c298c r4:db5c2980
[   19.072032] [<bf0cf13c>] (usb_kill_urb.part.3 [usbcore]) from [<bf0cf21c>] (usb_kill_urb+0x30/0x34 [usbcore])
[   19.082391]  r6:dc562d20 r5:db18ee78 r4:db18e348
[   19.087292] [<bf0cf1ec>] (usb_kill_urb [usbcore]) from [<bf231a78>] (rt2x00usb_flush_entry+0x38/0x60 [rt2x00usb])
[   19.098065] [<bf231a40>] (rt2x00usb_flush_entry [rt2x00usb]) from [<bf1f8cf8>] (rt2x00queue_for_each_entry+0x108/0x194 [rt2x00lib])
[   19.110424]  r7:00000000 r6:00000364 r5:db1f4e00 r4:0000001f
[   19.116362] [<bf1f8bf0>] (rt2x00queue_for_each_entry [rt2x00lib]) from [<bf231a3c>] (rt2x00usb_flush_queue+0xa4/0xa8 [rt2x00usb])
[   19.128539]  r9:00000000 r8:dc562a48 r7:db18c4e0 r6:00000001 r5:db1f4e00 r4:db1f4e00
[   19.136653] [<bf231998>] (rt2x00usb_flush_queue [rt2x00usb]) from [<bf1f91d4>] (rt2x00queue_flush_queue+0x4c/0xb0 [rt2x00lib])
[   19.148555]  r6:00000001 r5:dc562d20 r4:db1f4e00
[   19.153413] [<bf1f9188>] (rt2x00queue_flush_queue [rt2x00lib]) from [<bf1f92a4>] (rt2x00queue_flush_queues+0x6c/0x70 [rt2x00lib])
[   19.165585]  r4:db1f4f80
[   19.168263] [<bf1f9238>] (rt2x00queue_flush_queues [rt2x00lib]) from [<bf1f6354>] (rt2x00lib_disable_radio+0x50/0x9c [rt2x00lib])
[   19.180438]  r7:db18c4e0 r6:0000000c r5:dc562440 r4:dc562d20
[   19.186383] [<bf1f6304>] (rt2x00lib_disable_radio [rt2x00lib]) from [<bf1f6dc8>] (rt2x00lib_stop+0x34/0x48 [rt2x00lib])
[   19.197650]  r5:dc562440 r4:dc562d20
[   19.201417] [<bf1f6d94>] (rt2x00lib_stop [rt2x00lib]) from [<bf1f7404>] (rt2x00mac_stop+0x20/0x24 [rt2x00lib])
[   19.211866]  r5:dc562440 r4:dc562440
[   19.215770] [<bf1f73e4>] (rt2x00mac_stop [rt2x00lib]) from [<bf181fcc>] (drv_stop+0x28/0x7c [mac80211])
[   19.225761] [<bf181fa4>] (drv_stop [mac80211]) from [<bf1afc2c>] (ieee80211_stop_device+0x40/0x44 [mac80211])
[   19.236120]  r5:dc562440 r4:dc562440
[   19.240019] [<bf1afbec>] (ieee80211_stop_device [mac80211]) from [<bf19403c>] (ieee80211_do_stop+0x68c/0x914 [mac80211])
[   19.251376]  r5:dc562440 r4:db18c508
[   19.255274] [<bf1939b0>] (ieee80211_do_stop [mac80211]) from [<bf1942dc>] (ieee80211_stop+0x18/0x20 [mac80211])
[   19.265818]  r10:00000000 r9:00000000 r8:be93c9e0 r7:00009003 r6:00000000 r5:dc1c5e18
[   19.273996]  r4:db18c000
[   19.276726] [<bf1942c4>] (ieee80211_stop [mac80211]) from [<c05a7110>] (__dev_close_many+0x98/0xdc)
[   19.286185] [<c05a7078>] (__dev_close_many) from [<c05af744>] (__dev_change_flags+0xa8/0x178)
[   19.295090]  r5:ffff9002 r4:db18c000
[   19.298826] [<c05af69c>] (__dev_change_flags) from [<c05af834>] (dev_change_flags+0x20/0x50)
[   19.307643]  r8:be93c9e0 r7:db18c140 r6:00000000 r5:00009003 r4:db18c000
[   19.314653] [<c05af814>] (dev_change_flags) from [<c06251fc>] (devinet_ioctl+0x6ec/0x7ac)
[   19.323198]  r9:00000000 r8:be93c9e0 r7:db3f9a0c r6:00000000 r5:dc1c5e90 r4:00000000
[   19.331292] [<c0624b10>] (devinet_ioctl) from [<c0627458>] (inet_ioctl+0x19c/0x1c8)
[   19.339293]  r10:00000000 r9:dc1c4000 r8:be93c9e0 r7:c113d178 r6:be93c9e0 r5:dbb5e920
[   19.347472]  r4:00008914
[   19.350118] [<c06272bc>] (inet_ioctl) from [<c058d5c4>] (sock_ioctl+0x154/0x2f8)
[   19.357851] [<c058d470>] (sock_ioctl) from [<c02335c4>] (do_vfs_ioctl+0xa8/0x948)
[   19.365670]  r7:00000010 r6:db680540 r5:dbb5e920 r4:be93c9e0
[   19.371584] [<c023351c>] (do_vfs_ioctl) from [<c0233ea0>] (SyS_ioctl+0x3c/0x64)
[   19.379222]  r10:00000000 r9:dc1c4000 r8:be93c9e0 r7:00008914 r6:db680540 r5:00000010
[   19.387399]  r4:db680540
[   19.390049] [<c0233e64>] (SyS_ioctl) from [<c0107da0>] (ret_fast_syscall+0x0/0x48)
[   19.397960]  r9:dc1c4000 r8:c0107f64 r7:00000036 r6:00000003 r5:00000010 r4:be93c9e0
[   19.406054] Code: e594308c e3a00c01 e5942088 e3a01c02 (e5823004)
[   19.412422] ---[ end trace 2ef0867a0003cb98 ]---
[   19.417623] note: connmand[832] exited with preempt_count 2
Starting Inner Range LAURA main app: laura-app.
Starting Watchdog daemon: stopped /usr/sbin/watchdog (pid 872)
[   19.965484] watchdog: watchdog0: nowayout prevents watchdog being stopped!
[   19.972866] watchdog: watchdog0: watchdog did not stop!
watchdog.

Inner Range LAURA 0.6.2-r0 la000001

la000001 login:

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Craig,

The lsk- branches are those which the TI kernel releases are based on. They have more community patches than the non-lsk- branches do.

Checking on the kernel git commit history, the CPPI driver added runtime PM feature since kernel v4.4, and it seems to have bugs which cause the crash shown in your log. Can you please add "usbcore.autosuspend=-1" in the U-Boot bootargs to disable the runtime PM in USB subsystem, and let me know if the issue still happens?

If you boot the system without plugging in the WIFI dongle and then plugged it in after the system is booted, does the issue also happen?

Do you use the filesystem provided in the TI Processor Linux SDK package? Do you have any application or system setting to use the WIFI device during the system booting? I am trying to figure out how to replicate the issue and debug it.

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

Bin Liu said:
If you boot the system without plugging in the WIFI dongle and then plugged it in after the system is booted, does the issue also happen?

Yes, it also can happen if the Wi-Fi dongle is plugged in after the system is booted.

Bin Liu said:
Checking on the kernel git commit history, the CPPI driver added runtime PM feature since kernel v4.4, and it seems to have bugs which cause the crash shown in your log. Can you please add "usbcore.autosuspend=-1" in the U-Boot bootargs to disable the runtime PM in USB subsystem, and let me know if the issue still happens?

I've tried it, but it doesn't fix it. It can still crash, either at boot time or when the Wi-Fi dongle is plugged in later.

Bin Liu said:
Do you use the filesystem provided in the TI Processor Linux SDK package? Do you have any application or system setting to use the WIFI device during the system booting? I am trying to figure out how to replicate the issue and debug it.

I don't use the TI Processor Linux SDK package. I am using Yocto (currently the morty release) with the meta-ti layer to provide the TI Linux kernel.

I am using ConnMan for networking. The problem seems to occur when ConnMan enables the Wi-Fi in host access point mode.

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Craig,

It sounds like a kernel driver bug. I would have to replicate the failure and debug it. Can you please provide more info about the wifi dongle? module number? pictures? or link where to buy it? Thanks.

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

The USB Wi-Fi dongle is based on the RT2800 chipset, with RT chipset 5390, rev 0502, and with RF chipset 5370. An example of this is an Edup EP-N8537.

A very similar model that uses the same driver, but uses a slightly different chipset, is a D-Link DWA-140. It is also based on the RT2800 chipset, with RT chipset 5392, rev 0223, and with RF chipset 5372.

These devices use the rt2800usb driver. We're using the firmware file rt2870.bin version 0.36.

Note, we found it difficult to find a robust Wi-Fi dongle. Back in Sept 2015, we tested devices that use the drivers rt2800usb, rtl8192cu, ath9k_htc. We would have liked to try rtl8192eu but it was not supported on the kernel of Ubuntu 15.04 at the time. We found all somewhat unreliable, but rt2800usb was the best. But rt2800usb had a problem with a kernel lock-up when unplugging from a USB hub. It also caused a kernel lock-up if the device was disabled while in-use. I reported these in an earlier TI forum post [1]. I suspected these may be weaknesses of the TI USB driver, rather than the Wi-Fi driver itself, since the problem occurred on the BeagleBone Black based system, but not on an x86 Linux system.

I also found a problem with the Wi-Fi not starting up reliably in client mode, until the wireless network had been brought up, then down then up again. This only pertained to the combination of AM335x system, with rt2870.bin firmware version 0.36. I.e. it didn't happen on an x86 based system, and it didn't happen with the earlier 0.29 firmware (but 0.29 firmware had other reliability problems). I patched ConnMan to bring the wireless network up, then down then up again, to make it start reliably. (It's a most unpleasant hack, but I wasn't able to debug it with help from Linux rt2800 driver maintainers, and I thought it could take a lot of time for me to become familiar enough with Wi-Fi and/or USB Linux driver systems to be able to debug it properly myself.) Perhaps during this process, there is a race condition in which the USB driver ends up trying to access a NULL pointer.

[1]

0 Craig McQueen over 8 years ago in reply to Craig McQueen

Intellectual 850 points

I've removed the ConnMan patch to take the wireless network up/down/up, and the problem still occurs. So that rules out that particular detail.

I've retested with both Edup EP-N8537 and D-Link DWA-140. I found that the problem is more likely to occur if the CPU usage is higher. With low CPU usage, I could plug and unplug many times without problem. But when I logged in to serial console and ran:

while true; do true; done

then the kernel crash was much more likely after a Wi-Fi plug-in.

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Craig,

Thanks for the information. I have a DWA-125, I will first try to use it replicate the CPPI NULL pointer crash. If I cannot see it, I will order DWA-140.
I probably will start on this after the New Year, after the holiday break.
I won't be able to promise anything related to the link stability problems you described, but the NULL pointer issue should be solved, given that the issue doesn't exist in older kernels. I will keep you posted for the progress.

0 Bin Liu over 8 years ago in reply to Bin Liu

TI__Guru**** 173781 points

Craig,

I am unable to reproduce the issue with a DWA-125 dongle. After reviewed cppi41.c about how chan_busy[] is managed, I don't understand how the crash should happen.

Before I am going to order a DWA-140, can you please provide the descriptor log of two dongles you have which trigger the issue?
# lsusb -v -d <vid:pid>

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

Here is the Edup EP-N8537:

root@wt000043:~# lsusb -v -d 148f:5370

Bus 001 Device 004: ID 148f:5370
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x148f
  idProduct          0x5370
  bcdDevice            1.01
  iManufacturer           1 Ralink
  iProduct                2 802.11 n WLAN
  iSerial                 3 1.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           67
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              450mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           7
      bInterfaceClass       255
      bInterfaceSubClass    255
      bInterfaceProtocol    255
      iInterface              5 1.0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x05  EP 5 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x06  EP 6 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  bNumConfigurations      1
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

Here is the D-Link DWA-140:

root@wt000043:~# lsusb -v -d 2001:3c20

Bus 001 Device 005: ID 2001:3c20
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  idVendor           0x2001
  idProduct          0x3c20
  bcdDevice            1.01
  iManufacturer           1 Ralink
  iProduct                2 802.11 n WLAN
  iSerial                 3 1.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           53
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              450mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           5
      bInterfaceClass       255
      bInterfaceSubClass    255
      bInterfaceProtocol    255
      iInterface              5 1.0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0        64
  bNumConfigurations      1
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Thanks for the log. I was looking for any Interrupt endpoint, but same as DWA-125, none of the two dongles has one.

Can you please try to apply the following debug patch? I want to see if it gives any clue why the busy flag is set where it shouldn't.

diff --git a/drivers/dma/cppi41.c b/drivers/dma/cppi41.c
index f7e965f63274..2328e783f6b8 100644
--- a/drivers/dma/cppi41.c
+++ b/drivers/dma/cppi41.c
@@ -267,6 +267,7 @@ static struct cppi41_channel *desc_to_chan(struct cppi41_dd *cdd, u32 desc)
        BUG_ON(desc_num >= ALLOC_DECS_NUM);
        c = cdd->chan_busy[desc_num];
        cdd->chan_busy[desc_num] = NULL;
+       trace_printk("ch %d --> 0\n", desc_num);
 
        /* Usecount for chan_busy[], paired with push_desc_queue() */
        pm_runtime_put(cdd->ddev.dev);
@@ -465,6 +466,7 @@ static void push_desc_queue(struct cppi41_channel *c)
        desc_num = (desc_phys - cdd->descs_phys) / sizeof(struct cppi41_desc);
        WARN_ON(cdd->chan_busy[desc_num]);
        cdd->chan_busy[desc_num] = c;
+       trace_printk("ch %d --> 0x%p %pS\n", desc_num, c, __builtin_return_address(0));
 
        reg = (sizeof(struct cppi41_desc) - 24) / 4;
        reg |= desc_phys;
@@ -732,6 +734,7 @@ static int cppi41_stop_chan(struct dma_chan *chan)
 
        WARN_ON(!cdd->chan_busy[desc_num]);
        cdd->chan_busy[desc_num] = NULL;
+       trace_printk("ch %d --> 0\n", desc_num);
 
        /* Usecount for chan_busy[], paired with push_desc_queue() */
        pm_runtime_put(cdd->ddev.dev);

The steps to capture the log:

1. rebuild the kernel with FTRACE enabled. In menuconfig: Kernel hacking --> [*] Tracers

2. boot the board without plugging in the wifi dongle.

3. telnet to the board through Ethernet (dumping logs through uart is too slow.)

4. mount the debugfs: # mount -t debugfs nodev /sys/kernel/debug

5. dump the ftrace log in telnet window: # cat /sys/kernel/debug/tracing/trace_pipe

6. plug in the wifi dongle and try to trigger the problem, then save the ftrace log.

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

I've done up to step 4 (inclusive). But /sys/kernel/debug has no tracing sub-directory. Is there anything else I need to do to enable it?

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

I resolved the issue of the missing /sys/kernel/debug/tracing directory (I had to enable more kernel FTRACE options).

I followed the remainder of the instructions, and got the following:

kworker/0:3-63 [000] d..1 313.322770: cppi41_run_queue: ch 32 --> 0xdb2f5210 cppi41_runtime_resume+0x4c/0x84
connmand-788 [000] d.h2 313.505747: cppi41_stop_chan: ch 32 --> 0
kworker/0:3-63 [000] d..1 313.561664: cppi41_run_queue: ch 32 --> 0xdb2f5210 cppi41_runtime_resume+0x4c/0x84
kworker/0:3-63 [000] d..1 313.859613: cppi41_run_queue: ch 32 --> 0xdb2f5210 cppi41_runtime_resume+0x4c/0x84

I'm not sure if it captured the issue or not, because once there was a crash, the network connection locked up (I was using SSH). Anyway, that's the data.

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Craig,

Thanks for the log. It does tell something, the driver runtime resume got called twice more without stop the channel in between, which probably triggered the kernel crash. In my test, I don't see those two calls.

Can you please attach your rt2870.bin file for me to use? I am wondering if different version of the wifi dongle firmware could cause different behavior.

To get your project move forward before we solve this issue, can you please use the following patch as a workaround? It disables the runtime PM in the cppi41 driver, so the issue won't be triggered. This would be the same behavior as in kernel v4.1.

diff --git a/drivers/dma/cppi41.c b/drivers/dma/cppi41.c
index f7e965f63274..87150842fd0a 100644
--- a/drivers/dma/cppi41.c
+++ b/drivers/dma/cppi41.c
@@ -1059,7 +1062,7 @@ static int cppi41_dma_probe(struct platform_device *pdev)
        platform_set_drvdata(pdev, cdd);
 
        pm_runtime_enable(dev);
-       pm_runtime_set_autosuspend_delay(dev, 100);
+       pm_runtime_set_autosuspend_delay(dev, -1);
        pm_runtime_use_autosuspend(dev);
        ret = pm_runtime_get_sync(dev);
        if (ret < 0)

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

That's good news, thanks. I have tried the work-around patch, and it seems to fix the issue. Thank you!

I'm not sure what the plan is for follow-up work, but at this point, I'm very happy with this progress.

0 Craig McQueen over 8 years ago in reply to Bin Liu

Intellectual 850 points

Here is the rt2870.bin that I'm using — attached.

rt2870.zip

It's version 0.36 of rt2870.bin. It comes from linux-firmware git commit 42ad5367dd.

0 Bin Liu over 8 years ago in reply to Craig McQueen

TI__Guru**** 173781 points

Craig,

Glad you are happy with the workaround.

The next step is on my side to figure out how to reproduce the issue and find the root cause of the driver bug. But I cannot tell when I can reproduce it, no luck yet so far. The rt2870.bin you use is the same as what I have.

One thing to know that the value set by the workaround patch to disable the cppi driver runtime PM can be changed from userspace in sysfs entry "/sys/devices/platform/ocp/47400000.usb/47400000.dma-controller/power/autosuspend_delay_ms", so please ensure you don't have any userspace setting to modify the value.

0 Bin Liu over 7 years ago

TI__Guru**** 173781 points

Hi Craig,

Craig McQueen said:
Starting syslogd/klogd: [ 20.649326] ------------[ cut here ]------------ [ 20.654211] WARNING: CPU: 0 PID: 4 at /home/teamcity/buildAgent/work/ee889e486c4390d4/poky/build/tmp/work-shared/webtegriti/kernel-source/drivers/dma/cppi41.c:466 cppi41_run_queue+0x100/0x10c

I found a usecase which can trigger the similar failure, and created a patch to fix the issue.

Can you please test the kernel patch below to see if it solves the kernel crash?

marc.info/

Processors

Processors forum

Linux/PROCESSOR-SDK-AM335X: Kernel issue with USB WiFi dongle