Linux/AM3352: AM3352 Linux Panic on rare and random calls to tc

Gopher79

Part Number: AM3352

Tool/software: Linux

We rarely get this panic on our device,

[90006.155293] Internal error: Oops: 17 [#1] ARM
[90006.159874] Modules linked in: mdl_driver(O) rohc(O) fpga_driver(O)
[90006.166440] CPU: 0 PID: 42 Comm: kworker/u2:1 Tainted: G           O    4.14.67 #8
[90006.174342] Hardware name: Generic AM33XX (Flattened Device Tree)
[90006.180711] task: cc12a600 task.stack: cc23e000
[90006.185463] PC is at pwq_activate_delayed_work+0x14/0x68
[90006.191009] LR is at pwq_activate_delayed_work+0x14/0x68
[90006.196551] pc : [<c01340b8>]    lr : [<c01340b8>]    psr: 40010093
[90006.203091] sp : cc23fed8  ip : cc23fed8  fp : cc23feec
[90006.208541] r10: 00000088  r9 : 00000000  r8 : cc110700
[90006.213993] r7 : 00000000  r6 : 00000000  r5 : cb2741d0  r4 : cc110700
[90006.220803] r3 : 00000000  r2 : 00000001  r1 : cb2741d4  r0 : 00000000
[90006.227616] Flags: nZcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[90006.235155] Control: 10c5387d  Table: 8b1cc019  DAC: 00000051
[90006.241151] Process kworker/u2:1 (pid: 42, stack limit = 0xcc23e208)
[90006.247782] Stack: (0xcc23fed8 to 0xcc240000)
[90006.252329] fec0:                                                       cc110700 00000000
[90006.260872] fee0: cc23ff0c cc23fef0 c0135d1c c01340b0 cc1b8a80 c96f98cc cc005000 00000000
[90006.269415] ff00: cc23ff44 cc23ff10 c0136224 c0135cc4 c0915540 cc005014 cc1b8a98 cc1b8a80
[90006.277958] ff20: cc005000 cc005000 c0915540 cc005014 cc1b8a98 00000088 cc23ff74 cc23ff48
[90006.286500] ff40: c0136c74 c0135f5c 00000000 cc1ea440 cc1ea3c0 cc23e000 00000000 cc1b8a80
[90006.295043] ff60: c01369dc cc051e94 cc23ffac cc23ff78 c013b38c c01369e8 cc1ea458 cc1ea458
[90006.303585] ff80: cc23ffac cc1ea3c0 c013b24c 00000000 00000000 00000000 00000000 00000000
[90006.312127] ffa0: 00000000 cc23ffb0 c0107208 c013b258 00000000 00000000 00000000 00000000
[90006.320669] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[90006.329209] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[90006.337741] Backtrace:
[90006.340311] [<c01340a4>] (pwq_activate_delayed_work) from [<c0135d1c>] (pwq_dec_nr_in_flight+0x64                                                                                                                                         /0xcc)
[90006.350118]  r5:00000000 r4:cc110700
[90006.353854] [<c0135cb8>] (pwq_dec_nr_in_flight) from [<c0136224>] (process_one_work+0x2d4/0x2e8)
[90006.363029]  r7:00000000 r6:cc005000 r5:c96f98cc r4:cc1b8a80
[90006.368940] [<c0135f50>] (process_one_work) from [<c0136c74>] (worker_thread+0x298/0x3fc)
[90006.377482]  r10:00000088 r9:cc1b8a98 r8:cc005014 r7:c0915540 r6:cc005000 r5:cc005000
[90006.385655]  r4:cc1b8a80
[90006.388310] [<c01369dc>] (worker_thread) from [<c013b38c>] (kthread+0x140/0x158)
[90006.396037]  r10:cc051e94 r9:c01369dc r8:cc1b8a80 r7:00000000 r6:cc23e000 r5:cc1ea3c0
[90006.404211]  r4:cc1ea440 r3:00000000
[90006.407956] [<c013b24c>] (kthread) from [<c0107208>] (ret_from_fork+0x14/0x2c)
[90006.415501]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c013b24c
[90006.423674]  r4:cc1ea3c0
[90006.426324] Code: e92dd830 e24cb004 e1a05000 ebffffb5 (e5902000)
[90006.432687] ---[ end trace ea4998e60956d972 ]---
[90006.437508] Kernel panic - not syncing: Fatal exception
[90006.442964] Rebooting in 30 seconds..

Has anyone seen this and what could this point on?

over 6 years ago

0 Yordan Kovachev over 6 years ago

TI__Guru**** 161600 points

Hi,

I assume you use custom board, right?

Could you provide a little more info on this one? What are the conditions under which you see this panic? Which Processor SDK Linux version are you using?

Which modules from your board are you using when you see this happen? Do you have the full bootlog (dmesg) from when the kernel panic occurred?

Best Regards,
Yordan

0 Yordan Kovachev over 6 years ago in reply to Yordan Kovachev

TI__Guru**** 161600 points

Hi,

I haven't heard back. I am closing this for now, you can reopen the thread by providing the requested feedback.

Best Regards,
Yordan

0 Gopher79 over 6 years ago in reply to Yordan Kovachev

Expert 1125 points

Yordan,

This is a custom board. Here is a dmesg,

[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.14.67 (root@svin) (gcc version 7.2.1 20171011 (Linaro GCC 7.2-2017.11)) #10 Thu Mar 7 11:56:51 NZDT 2019
[ 0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: MMX Pyxis
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] cma: Reserved 48 MiB at 0x8d000000
[ 0.000000] On node 0 totalpages: 65536
[ 0.000000] free_area_init_node: node 0, pgdat c093d7a8, node_mem_map ccda1000
[ 0.000000] Normal zone: 576 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 65536 pages, LIFO batch:15
[ 0.000000] CPU: All CPU(s) started in SVC mode.
[ 0.000000] AM335X ES2.1 (neon)
[ 0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[ 0.000000] pcpu-alloc: [0] 0
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 64960
[ 0.000000] Kernel command line: console=tty0 consoleblank=0 console=ttyO0,115200n8 blkdevparts=mmcblk0:512K@0(boot),128K@0x80000(env1),128K@0xC0000(env2),2M@0x100000(uboot1),2M@0x300000(uboot2),8M(@0x500000linux1),8M@0xD00000(linux2),256M@0x1500000(rootfs1),256M@0x11500000(rootfs2),8M@0x21500000(dbf1),8M@0x21D00000(dbf2),64M@0x22500000(log),256M@0x26500000(dbf2),64M@0x80700000(vfat),3G@0x84700800(ext3) root=/dev/mmcblk0p8 rw rootwait
[ 0.000000] PID hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[ 0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[ 0.000000] Memory: 201464K/262144K available (5120K kernel code, 252K rwdata, 1640K rodata, 1024K init, 233K bss, 11528K reserved, 49152K cma-reserved, 0K highmem)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xd0800000 - 0xff800000 ( 752 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xd0000000 ( 256 MB)
[ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
[ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
[ 0.000000] .text : 0xc0008000 - 0xc0600000 (6112 kB)
[ 0.000000] .init : 0xc0800000 - 0xc0900000 (1024 kB)
[ 0.000000] .data : 0xc0900000 - 0xc093f098 ( 253 kB)
[ 0.000000] .bss : 0xc0943d60 - 0xc097e2e4 ( 234 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts
[ 0.000000] OMAP clockevent source: timer2 at 24000000 Hz
[ 0.000022] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
[ 0.000048] clocksource: timer1: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635851949 ns
[ 0.000063] OMAP clocksource: timer1 at 24000000 Hz
[ 0.000325] timer_probe: no matching timers found
[ 0.000702] Console: colour dummy device 80x30
[ 0.001394] console [tty0] enabled
[ 0.001453] Calibrating delay loop... 544.76 BogoMIPS (lpj=272384)
[ 0.010580] pid_max: default: 32768 minimum: 301
[ 0.010856] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.010898] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[ 0.012022] CPU: Testing write buffer coherency: ok
[ 0.012998] Setting up static identity map for 0x80100000 - 0x80100060
[ 0.015486] devtmpfs: initialized
[ 0.027297] random: get_random_u32 called from bucket_table_alloc+0x134/0x180 with crng_init=0
[ 0.027954] VFP support v0.3: implementor 41 architecture 3 part 30 variant c rev 3
[ 0.028328] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[ 0.028389] futex hash table entries: 256 (order: -1, 3072 bytes)
[ 0.033743] pinctrl core: initialized pinctrl subsystem
[ 0.035345] NET: Registered protocol family 16
[ 0.038694] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.057418] omap_hwmod: debugss: _wait_target_disable failed
[ 0.113786] cpuidle: using governor ladder
[ 0.113827] cpuidle: using governor menu
[ 0.120235] OMAP GPIO hardware version 0.1
[ 0.135217] hw-breakpoint: debug architecture 0x4 unsupported.
[ 0.150372] edma 49000000.edma: TI EDMA DMA engine driver
[ 0.153663] omap_i2c 44e0b000.i2c: could not find pctldev for node /ocp/l4_wkup@44c00000/scm@210000/pinmux@800/i2c0_pins, deferring probe
[ 0.153961] pps_core: LinuxPPS API ver. 1 registered
[ 0.153991] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.154050] PTP clock support registered
[ 0.154115] EDAC MC: Ver: 3.0.0
[ 0.155319] omap-mailbox 480c8000.mailbox: omap mailbox rev 0x400
[ 0.157039] clocksource: Switched to clocksource timer1
[ 0.169566] NET: Registered protocol family 2
[ 0.170667] TCP established hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.170747] TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.170805] TCP: Hash tables configured (established 2048 bind 2048)
[ 0.170964] UDP hash table entries: 256 (order: 0, 4096 bytes)
[ 0.171012] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[ 0.171302] NET: Registered protocol family 1
[ 0.172311] hw perfevents: no interrupt-affinity property for /pmu, guessing.
[ 0.172502] hw perfevents: enabled with armv7_cortex_a8 PMU driver, 5 counters available
[ 0.174760] workingset: timestamp_bits=14 max_order=16 bucket_order=2
[ 0.206955] io scheduler noop registered
[ 0.207018] io scheduler deadline registered
[ 0.207453] io scheduler cfq registered (default)
[ 0.207489] io scheduler mq-deadline registered
[ 0.207514] io scheduler kyber registered
[ 0.208930] pinctrl-single 44e10800.pinmux: 142 pins at pa f9e10800 size 568
[ 0.254068] random: fast init done
[ 0.284179] Serial: 8250/16550 driver, 10 ports, IRQ sharing disabled
[ 0.287761] omap_uart 44e09000.serial: no wakeirq for uart0
[ 0.287998] 44e09000.serial: ttyO0 at MMIO 0x44e09000 (irq = 30, base_baud = 3000000) is a OMAP UART0
[ 0.841254] console [ttyO0] enabled
[ 0.845674] omap_uart 48022000.serial: no wakeirq for uart1
[ 0.851804] 48022000.serial: ttyO1 at MMIO 0x48022000 (irq = 31, base_baud = 3000000) is a OMAP UART1
[ 0.862170] omap_uart 48024000.serial: no wakeirq for uart2
[ 0.868174] 48024000.serial: ttyO2 at MMIO 0x48024000 (irq = 32, base_baud = 3000000) is a OMAP UART2
[ 0.878476] omap_uart 481a6000.serial: no wakeirq for uart3
[ 0.884479] 481a6000.serial: ttyO3 at MMIO 0x481a6000 (irq = 33, base_baud = 3000000) is a OMAP UART3
[ 0.894747] omap_uart 481a8000.serial: no wakeirq for uart4
[ 0.900817] 481a8000.serial: ttyO4 at MMIO 0x481a8000 (irq = 34, base_baud = 3000000) is a OMAP UART4
[ 0.912092] omap_rng 48310000.rng: Random Number Generator ver. 20
[ 0.928728] loop: module loaded
[ 0.936899] libphy: Fixed MDIO Bus: probed
[ 0.991109] davinci_mdio 4a101000.mdio: davinci mdio revision 1.6, bus freq 1000000
[ 0.999191] davinci_mdio 4a101000.mdio: detected phy mask fffffff9
[ 1.007819] libphy: 4a101000.mdio: probed
[ 1.012100] davinci_mdio 4a101000.mdio: phy[1]: device 4a101000.mdio:01, driver TI DP83822 10/100 Mbps PHY
[ 1.022249] davinci_mdio 4a101000.mdio: phy[2]: device 4a101000.mdio:02, driver TI DP83822 10/100 Mbps PHY
[ 1.033328] cpsw 4a100000.ethernet: Detected MACID = 98:5d:ad:52:d2:eb
[ 1.040330] cpsw 4a100000.ethernet: initialized cpsw ale version 1.4
[ 1.047016] cpsw 4a100000.ethernet: ALE Table size 1024
[ 1.052554] cpsw 4a100000.ethernet: cpts: overflow check period 5000 (jiffies)
[ 1.061923] i2c /dev entries driver
[ 1.065749] IR NEC protocol handler initialized
[ 1.070521] IR RC5(x/sz) protocol handler initialized
[ 1.075825] IR RC6 protocol handler initialized
[ 1.080578] IR JVC protocol handler initialized
[ 1.085332] IR Sony protocol handler initialized
[ 1.090178] IR SANYO protocol handler initialized
[ 1.095113] IR Sharp protocol handler initialized
[ 1.100047] IR MCE Keyboard/mouse protocol handler initialized
[ 1.106162] IR XMP protocol handler initialized
[ 1.112475] cpuidle: enable-method property 'ti,am3352' found operations
[ 1.124944] GACT probability on
[ 1.128360] Mirror/redirect action on
[ 1.132288] u32 classifier
[ 1.135149] input device check on
[ 1.138998] Actions configured
[ 1.142599] Netfilter messages via NETLINK v0.30.
[ 1.147786] nf_conntrack version 0.5.0 (4096 buckets, 16384 max)
[ 1.154406] ctnetlink v0.93: registering with nfnetlink.
[ 1.160566] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 1.166412] Initializing XFRM netlink socket
[ 1.170981] NET: Registered protocol family 17
[ 1.175780] Bridge firewalling registered
[ 1.180017] Ebtables v2.0 registered
[ 1.183918] 8021q: 802.1Q VLAN Support v1.8
[ 1.188455] 9pnet: Installing 9P2000 support
[ 1.193180] omap_voltage_late_init: Voltage driver support not added
[ 1.203666] Key type encrypted registered
[ 1.232675] vrtc: supplied by vbat
[ 1.242422] vio: supplied by vbat
[ 1.249179] vdd1: supplied by vbat
[ 1.257101] vdd2: supplied by vbat
[ 1.268248] vdig1: supplied by vbat
[ 1.275366] vdig2: supplied by vbat
[ 1.282523] vpll: supplied by vbat
[ 1.289538] vdac: supplied by vbat
[ 1.296584] vaux1: supplied by vbat
[ 1.303695] vaux2: supplied by vbat
[ 1.310838] vaux33: supplied by vbat
[ 1.318009] vmmc: supplied by vbat
[ 1.325008] vbb: supplied by vbat
[ 1.331634] tps65910-rtc tps65910-rtc: Enabling rtc-tps65910.
[ 1.343027] tps65910-rtc tps65910-rtc: char device (253:0)
[ 1.343098] tps65910-rtc tps65910-rtc: registered as rtc0
[ 1.349159] omap_i2c 44e0b000.i2c: bus 0 rev0.11 at 100 kHz
[ 1.356411] cpufreq: cpufreq_online: CPU0: Running at unlisted freq: 550000 KHz
[ 1.364682] cpufreq: cpufreq_online: CPU0: Unlisted initial frequency changed to: 600000 KHz
[ 1.402600] tps65910-rtc tps65910-rtc: setting system clock to 2019-03-17 08:28:43 UTC (1552811323)
[ 1.413308] lis3_reg: disabling
[ 1.418854] Waiting for root device /dev/mmcblk0p8...
[ 1.469276] mmc0: new high speed MMC card at address 0001
[ 1.476453] mmcblk0: mmc0:0001 Q2J54A 3.64 GiB
[ 1.481878] mmcblk0boot0: mmc0:0001 Q2J54A partition 1 2.00 MiB
[ 1.488365] mmcblk0boot1: mmc0:0001 Q2J54A partition 2 2.00 MiB
[ 1.494923] mmcblk0rpmb: mmc0:0001 Q2J54A partition 3 512 KiB
[ 1.501853] mmcblk0: p1(boot) p2(env1) p3(env2) p4(uboot1) p5(uboot2) p6(@0x500000linux1) p7(linux2) p8(rootfs1) p9(rootfs2) p10(dbf1) p11(dbf2) p12(log) p13(dbf2) p14(vfat) p15(ext3)
[ 1.528372] EXT4-fs (mmcblk0p8): mounting ext3 file system using the ext4 subsystem
[ 1.542846] EXT4-fs (mmcblk0p8): mounted filesystem with ordered data mode. Opts: (null)
[ 1.551464] VFS: Mounted root (ext3 filesystem) on device 179:8.
[ 1.560587] devtmpfs: mounted
[ 1.566506] Freeing unused kernel memory: 1024K
[ 1.917333] EXT4-fs (mmcblk0p10): warning: mounting fs with errors, running e2fsck is recommended
[ 1.928409] EXT4-fs (mmcblk0p10): recovery complete
[ 1.934637] EXT4-fs (mmcblk0p10): mounted filesystem with ordered data mode. Opts: (null)
[ 2.098720] random: tr: uninitialized urandom read (8192 bytes read)
[ 2.107031] random: tr: uninitialized urandom read (8192 bytes read)
[ 2.491135] EXT4-fs (mmcblk0p12): recovery complete
[ 2.493608] EXT4-fs (mmcblk0p12): mounted filesystem with ordered data mode. Opts: (null)
[ 2.568810] EXT4-fs (mmcblk0p13): recovery complete
[ 2.568835] EXT4-fs (mmcblk0p13): mounted filesystem with ordered data mode. Opts: (null)
[ 3.218125] random: tr: uninitialized urandom read (8192 bytes read)
[ 3.218627] random: tr: uninitialized urandom read (8192 bytes read)
[ 3.372380] random: db_manager: uninitialized urandom read (256 bytes read)
[ 4.179636] EXT4-fs (mmcblk0p9): mounting ext3 file system using the ext4 subsystem
[ 4.298928] EXT4-fs (mmcblk0p9): recovery complete
[ 4.299951] EXT4-fs (mmcblk0p9): mounted filesystem with ordered data mode. Opts: (null)
[ 9.938027] fpga_driver: loading out-of-tree module taints kernel.
[ 9.940888] fpga_driver: unknown parameter 'samples' ignored
[ 9.941621] RF SYSFS nodes initialised
[ 9.941728] COMMON SYSFS nodes initialised.
[ 9.941798] RX PHY SYSFS nodes initialised
[ 9.941813] TX PHY SYSFS nodes initialised
[ 9.941866] MAC SYSFS nodes initialised
[ 9.941921] CPU IF and Misc SYSFS nodes initialised
[ 9.941942] REF PLL SYSFS nodes initialised
[ 9.941972] FPGA SPI nodes initialised
[ 9.941976] FPGA SPI driver V1.3 initialized.
[ 9.943623] mmx_fpgacmd spi0.0: FPGA SPI driver - registered on SPI bus.
[ 9.944656] spi_fpga_probe: SPI-FPGA driver loaded.
[ 12.940795] mdl:
[ 12.940811] Kernel Watchdog Timeout=120000[ms]
[ 12.940940] mdl:
[ 12.940944] Starting RoHC ...
[ 12.942903] mdl0:
[ 12.942914] Maximum packet concatenation set to 100.
[ 12.942922] mdl0:
[ 12.942926] MAC frame resend is set to Adaptive.
[ 12.943733] BCF max. allowed size (4-GFSK): 66[b]
[ 12.943740] BCF sizes (4-GFSK) used 65[b]
[ 12.943746] Link state counters UP: 21 DOWN: 35
[ 12.943830] PMAC | CYCLE | MAX(SLOTS) : 100
[ 12.943834] PMAC | CYCLE | MAX(MS) : 640
[ 12.943838] PMAC | CYCLE | MIN(SLOTS) : 23
[ 12.943842] PMAC | CYCLE | MIN(MS) : 147
[ 12.943845] PMAC | CYCLE | BCS SLOTS : 3
[ 12.943848] PMAC | CYCLE | ACS SLOTS : 10
[ 12.943852] PMAC | CYCLE | DATA SLOTS : 86
[ 12.943856] PMAC | EVENT | TS0 @ : 18
[ 12.943859] PMAC | BURST | BYTES : 244
[ 12.943862] PMAC | BURST | SLOTS : 9
[ 14.889523] net eth0: initializing cpsw version 1.12 (0)
[ 14.946727] TI DP83822 10/100 Mbps PHY 4a101000.mdio:01: attached PHY driver [TI DP83822 10/100 Mbps PHY] (mii_bus:phy_addr=4a101000.mdio:01, irq=POLL)
[ 14.946864] net eth0: could not add device link to 4a101000.mdio:02 err -17
[ 15.002719] TI DP83822 10/100 Mbps PHY 4a101000.mdio:02: attached PHY driver [TI DP83822 10/100 Mbps PHY] (mii_bus:phy_addr=4a101000.mdio:02, irq=POLL)
[ 15.007867] 8021q: adding VLAN 0 to HW filter on device eth0
[ 15.436817] mdl0: Setting MAC address to:
[ 15.436833] 98
[ 15.436840] 5d
[ 15.436843] ad
[ 15.436846] 52
[ 15.436849] d2
[ 15.436852] ea
[ 15.436855]
[ 15.505972] mdl0: eth_change_mtu is deprecated
[ 15.521850] br0: port 1(eth0) entered blocking state
[ 15.521864] br0: port 1(eth0) entered disabled state
[ 15.524706] device eth0 entered promiscuous mode
[ 15.532810] br0: port 2(mdl0) entered blocking state
[ 15.532824] br0: port 2(mdl0) entered disabled state
[ 15.533159] device mdl0 entered promiscuous mode
[ 15.550278] br0: port 2(mdl0) entered blocking state
[ 15.550293] br0: port 2(mdl0) entered forwarding state
[ 15.968353] cpsw 4a100000.ethernet eth0: Link is Up - Unsupported (update phy-core.c)/Half - flow control off

We run this device as a radio, part of a link, connecting over the air to another radio. Our radio is denoted as DUT and ... Host ---> (eth) DUT (BaseRadio) (air) ----> RemoteRadio.

The panic occurs after running a bunch of switch and IP reconfiguration commands, then pinging from Host to RemoteRadio.

If you like I can drill down to the exact commands we run. Unsure there is only one sequence that can cause this. Wired driver (eth) is cpsw.c.

As loadable modules, I am running only a bunch of my proprietary modules, which could both mean nothing or a lot...

Cheers

0 Yordan Kovachev over 6 years ago in reply to Gopher79

TI__Guru**** 161600 points

Hi,

Sorry for the delay. The kernel panic comes from the kernel workqueue, unfortunately since you use a custom board I won't be able to reproduce the issue.

What I can suggest to try is when running the commands in your sequence make sure that the command you run has taken effect before proceeding with the next one. If you use a script to run those commands try putting a delay of 1 or 2 seconds between the commands.

Another thing to test is the Debugging tips in the kernel documentation (Documentation/core-api/workqueue.rst):
Debugging
=========

Because the work functions are executed by generic worker threads
there are a few tricks needed to shed some light on misbehaving
workqueue users.

Worker threads show up in the process list as: ::

root 5671 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/0:1]
root 5672 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/1:2]
root 5673 0.0 0.0 0 0 ? S 12:12 0:00 [kworker/0:0]
root 5674 0.0 0.0 0 0 ? S 12:13 0:00 [kworker/1:0]

If kworkers are going crazy (using too much cpu), there are two types
of possible problems:

1. Something being scheduled in rapid succession
2. A single work item that consumes lots of cpu cycles

The first one can be tracked using tracing: ::

$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
^C

If something is busy looping on work queueing, it would be dominating
the output and the offender can be determined with the work item
function.

For the second type of problems it should be possible to just check
the stack trace of the offending worker thread. ::

$ cat /proc/THE_OFFENDING_KWORKER/stack

The work item's function should be trivially visible in the stack
trace.

Hope this helps.

Best Regards,
Yordan

0 Gopher79 over 6 years ago in reply to Yordan Kovachev

Expert 1125 points

Thanks a lot, will try to debug using the suggested.

Processors

Processors forum

Linux/AM3352: AM3352 Linux Panic on rare and random calls to tc