This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM3352: Spurius Can Interrupt

Part Number: AM3352

Tool/software:

When closing the CAN on Linux, approximately once every 30,000 times, CAN interruptions occur with the CTL register having the value 0x1 while using the Linux kernel version 4.9.206.

Steps to reproduce:
1.1) The bus under heavy activity is being read with the "read" interface of libc and, eventually, is being written with the "write" interface.

1.2) After a certain period, the application stops to read and write the bus.

1.3) After stopping the read and write operations, an ioctl call is made to deactivate the CAN interface (~IFF_UP).

1.4) When the interface is reactivated with the ioctl operation (IFF_UP), the Kernel sporadically freezes for a few seconds, and during the freeze:
1.4.1) The CAN initialization gets stuck in the "c_can_open" function, i.e., before the complete initialization of the device;
1.4.2) The C_CAN_CTRL_REG register has the TEST bit set high;
1.4.3) The Kernel attempts to handle the interrupt and, upon identifying that the interrupt was not handled after a large number of attempts due to the absence of a handler, the main interrupt of the CAN device is disabled.
1.4.4) After a few seconds, the Kernel proceeds with the initialization correctly.

2) Identified workaround:
2.1) We identified that if the interrupt handler is configured before starting the bus in "c_can_open", the Kernel can complete "c_can_open" without any failure.

3) Is there any suggestion to work around this scenario without needing to modify the Linux Kernel?

Kernel message:

2025-02-25 17:03:38.857]: b'[ 347.042461] irq 168: nobody cared (try booting with the "irqpoll" option)'

[2025-02-25 17:03:38.867]: b'[ 347.049533] CPU: 0 PID: 2949 Comm: UCC3TargetTests Not tainted 4.9.206.UCC3.R15-04 #1'

[2025-02-25 17:03:38.877]: b'[ 347.057660] Hardware name: Generic AM33XX (Flattened Device Tree)'
[2025-02-25 17:03:38.883]: b'[ 347.063983] Backtrace: '
[2025-02-25 17:03:38.883]: b'[ 347.066523] Function entered at [<c010c2d8>dump_backtrace(traps.c:209)] from [<c010c5f0>show_stack(traps.c:248)]'
[2025-02-25 17:03:38.896]: b'[ 347.072580] r7:000000a8 r6:ceb25d28 r5:00000000 r4:cf9aacc0'
[2025-02-25 17:03:38.907]: b'[ 347.078451] Function entered at [<c010c5d8>show_stack(traps.c:245)] from [<c02b8120>dump_stack(dump_stack.c:67)]'
[2025-02-25 17:03:38.907]: b'[ 347.084503] Function entered at [<c02b80fc>dump_stack(dump_stack.c:65)] from [<c0184654>__report_bad_irq(spurious.c:205)]'
[2025-02-25 17:03:38.907]: b'[ 347.090554] Function entered at [<c0184614>__report_bad_irq(spurious.c:192)] from [<c015a454>note_interrupt(spurious.c:418)]'
[2025-02-25 17:03:38.917]: b'[ 347.096609] r9:ceb24000 r8:cf806000 r7:000000a8 r6:ceb25d28 r5:00000000 r4:cf9aacc0'
[2025-02-25 17:03:38.923]: b'[ 347.104644] Function entered at [<c015a208>note_interrupt(spurious.c:272)] from [<c0157e50>handle_irq_event_percpu(handle.c:190)]'
[2025-02-25 17:03:38.927]: b'[ 347.110699] r10:00000036 r9:ceb24000 r8:cf806000 r7:00000001 r6:ceb25d28 r5:00000000'
[2025-02-25 17:03:38.943]: b'[ 347.118826] r4:cf9aacc0 r3:00000000'
[2025-02-25 17:03:38.943]: b'[ 347.122531] Function entered at [<c0157d4c>handle_irq_event(handle.c:195)] from [<c015ade8>irqd_irq_disabled(irq.h:279)]'
[2025-02-25 17:03:38.947]: b'[ 347.128582] r5:cf9aacd0 r4:cf9aacc0'
[2025-02-25 17:03:38.947]: b'[ 347.132287] Function entered at [<c015ad24>handle_level_irq(chip.c:499)] from [<c0157520>__handle_domain_irq(irqdesc.c:644)]'
[2025-02-25 17:03:38.963]: b'[ 347.138339] r5:c072df10 r4:00000000'
[2025-02-25 17:03:38.963]: b'[ 347.142045] Function entered at [<c01574b4>__handle_domain_irq(irqdesc.c:621)] from [<c01014b8>omap_intc_handle_irq(irq-omap-intc.c:365)]'
[2025-02-25 17:03:38.973]: b'[ 347.148099] r9:ceb24000 r8:cf806000 r7:ceb25c64 r6:ffffffff r5:60010113 r4:c075c574'
[2025-02-25 17:03:38.973]: b'[ 347.156134] Function entered at [<c010147c>omap_intc_handle_irq(irq-omap-intc.c:334)] from [<c010b078>__irq_svc(entry-armv.S:222)]'
[2025-02-25 17:03:38.986]: b'[ 347.162185] Exception stack(0xceb25c30 to 0xceb25c78)'
[2025-02-25 17:03:38.999]: b'[ 347.167428] 5c20: 00400140 c073cac0 c073ca80 fffffdfe'
[2025-02-25 17:03:38.999]: b'[ 347.175919] 5c40: 00000202 c072df10 00000000 ceb24000 cf806000 ceb24000 00000036 ceb25cd4'
[2025-02-25 17:03:39.003]: b'[ 347.184410] 5c60: ceb25cd8 ceb25c80 c0128ce8 c0128838 60010113 ffffffff'
[2025-02-25 17:03:39.013]: b'[ 347.191272] r5:60010113 r4:c0128838'
[2025-02-25 17:03:39.013]: b'[ 347.194977] Function entered at [<c01287b4>__do_softirq(softirq.c:246)] from [<c0128ce8>do_softirq_own_stack(interrupt.h:477)]'
[2025-02-25 17:03:39.025]: b'[ 347.201032] r10:00000036 r9:ceb24000 r8:cf806000 r7:00000001 r6:00000000 r5:c072df10'
[2025-02-25 17:03:39.025]: b'[ 347.209158] r4:00000000'
[2025-02-25 17:03:39.027]: b'[ 347.211780] Function entered at [<c0128bc4>irq_exit(softirq.c:399)] from [<c0157524>__handle_domain_irq(irqdesc.c:647)]'
[2025-02-25 17:03:39.042]: b'[ 347.217831] Function entered at [<c01574b4>__handle_domain_irq(irqdesc.c:621)] from [<c01014b8>omap_intc_handle_irq(irq-omap-intc.c:365)]'
[2025-02-25 17:03:39.043]: b'[ 347.223888] r9:ceb24000 r8:c051013c r7:ceb25d5c r6:ffffffff r5:60010013 r4:c075c574'
[2025-02-25 17:03:39.057]: b'[ 347.231924] Function entered at [<c010147c>omap_intc_handle_irq(irq-omap-intc.c:334)] from [<c010b078>__irq_svc(entry-armv.S:222)]'
[2025-02-25 17:03:39.057]: b'[ 347.237976] Exception stack(0xceb25d28 to 0xceb25d70)'
[2025-02-25 17:03:39.067]: b'[ 347.243219] 5d20: cf9c7440 00000000 00000001 c0386db0 cf9c7000 cf9c7440'
[2025-02-25 17:03:39.073]: b'[ 347.251709] 5d40: c076208c c0505b68 c051013c ceb24000 00000036 ceb25d8c c04c6f1c ceb25d78'
[2025-02-25 17:03:39.077]: b'[ 347.260196] 5d60: c03867f8 c0386800 60010013 ffffffff'
[2025-02-25 17:03:39.087]: b'[ 347.265436] r5:60010013 r4:c0386800'
[2025-02-25 17:03:39.087]: b'[ 347.269141] Function entered at [<c0386794>c_can_close(c_can.c:1186)] from [<c03f2a8c>__dev_close_many(dev.c:1424)]'
[2025-02-25 17:03:39.100]: b'[ 347.275192] r5:ceb25dc4 r4:cf9c7000'
[2025-02-25 17:03:39.100]: b'[ 347.278897] Function entered at [<c03f29c0>__dev_close_many(dev.c:1386)] from [<c03f2b54>list_del(list.h:107)]'
[2025-02-25 17:03:39.103]: b'[ 347.284953] r8:00000201 r7:00040081 r6:000000c0 r5:00000001 r4:cf9c7000'
[2025-02-25 17:03:39.103]: b'[ 347.291906] Function entered at [<c03f2b10>__dev_close(dev.c:1432)] from [<c03f72c8>__dev_change_flags(dev.c:6505)]'
[2025-02-25 17:03:39.103]: b'[ 347.297957] Function entered at [<c03f7204>__dev_change_flags(dev.c:6472)] from [<c03f73cc>dev_change_flags(dev.c:6571)]'
[2025-02-25 17:03:39.119]: b'[ 347.304013] r9:ceb24000 r8:00040081 r7:cf9c714c r6:00000000 r5:cf9c7000 r4:cf9c7000'
[2025-02-25 17:03:39.119]: b'[ 347.312047] Function entered at [<c03f73ac>dev_change_flags(dev.c:6566)] from [<c0410a84>dev_ifsioc(dev_ioctl.c:255)]'
[2025-02-25 17:03:39.134]: b'[ 347.318102] r9:ceb24000 r8:00000000 r7:c04c693c r6:ceb25e74 r5:cf9c7000 r4:00008914'
[2025-02-25 17:03:39.150]: b'[ 347.326138] Function entered at [<c041082c>dev_ifsioc(dev_ioctl.c:243)] from [<c0411084>dev_ioctl(dev_ioctl.c:533)]'
[2025-02-25 17:03:39.150]: b'[ 347.332191] r7:c0737a80 r6:b545bc50 r5:00000014 r4:00008914'
[2025-02-25 17:03:39.150]: b'[ 347.338061] Function entered at [<c0410bd8>dev_ioctl(dev_ioctl.c:395)] from [<c03d6908>sock_do_ioctl(socket.c:886)]'
[2025-02-25 17:03:39.166]: b'[ 347.344117] r10:00000036 r9:ceb24000 r8:cf39a3c0 r7:c0737a80 r6:b545bc50 r5:fffffdfd'
[2025-02-25 17:03:39.166]: b'[ 347.352243] r4:00008914'
[2025-02-25 17:03:39.166]: b'[ 347.354864] Function entered at [<c03d680c>sock_ioctl(socket.c:897)] from [<c01d83e0>vfs_ioctl(ioctl.c:44)]'
[2025-02-25 17:03:39.181]: b'[ 347.360918] r7:00000006 r6:ceceb620 r5:b545bc50 r4:cf39a3c0'
[2025-02-25 17:03:39.181]: b'[ 347.366789] Function entered at [<c01d8030>do_vfs_ioctl(ioctl.c:620)] from [<c01d8928>fdput(file.h:38)]'
[2025-02-25 17:03:39.181]: b'[ 347.372846] r10:00000036 r9:ceb24000 r8:cf39a3c0 r7:b545bc50 r6:00008914 r5:00000006'
[2025-02-25 17:03:39.197]: b'[ 347.380971] r4:cf39a3c1'
[2025-02-25 17:03:39.197]: b'[ 347.383593] Function entered at [<c01d88ec>SyS_ioctl(ioctl.c:685)] from [<c01076c0>ret_fast_syscall(entry-common.S:38)]'
[2025-02-25 17:03:39.213]: b'[ 347.389649] r9:ceb24000 r8:c01078a4 r7:00000036 r6:b545bc50 r5:00000000 r4:00000006'
[2025-02-25 17:03:39.213]: b'[ 347.397682] handlers:'
[2025-02-25 17:03:39.213]: b'[ 347.400037] [<c0386864>c_can_isr(c_can.c:1118)] 0xc0386864'
[2025-02-25 17:03:39.213]: b'[ 347.403653] Disabling IRQ #168'
[2025-02-25 17:03:39.213]: b'[RemoteCall] ActionId: 32.'
[2025-02-25 17:03:39.228]: b'[ 347.413906] omap_hwmod: d_can1: _wait_target_disable failed'
[2025-02-25 17:03:39.228]: b'[ 347.422236] omap_hwmod: d_can1: _wait_target_disable failed'
[2025-02-25 17:03:39.369]: b'[ 347.559034] irq 168: nobody cared (try booting with the "irqpoll" option)'
[2025-02-25 17:03:39.385]: b'[ 347.566101] CPU: 0 PID: 2949 Comm: UCC3TargetTests Not tainted 4.9.206.UCC3.R15-04 #1'
[2025-02-25 17:03:39.400]: b'[ 347.574228] Hardware name: Generic AM33XX (Flattened Device Tree)'
[2025-02-25 17:03:39.400]: b'[ 347.580550] Backtrace: '
[2025-02-25 17:03:39.400]: b'[ 347.583090] Function entered at [<c010c2d8>dump_backtrace(traps.c:209)] from [<c010c5f0>show_stack(traps.c:248)]'
[2025-02-25 17:03:39.400]: b'[ 347.589145] r7:000000a8 r6:ceb25ce8 r5:00000000 r4:cf9aacc0'
[2025-02-25 17:03:39.416]: b'[ 347.595017] Function entered at [<c010c5d8>show_stack(traps.c:245)] from [<c02b8120>dump_stack(dump_stack.c:67)]'
[2025-02-25 17:03:39.416]: b'[ 347.601068] Function entered at [<c02b80fc>dump_stack(dump_stack.c:65)] from [<c0184654>__report_bad_irq(spurious.c:205)]'
[2025-02-25 17:03:39.431]: b'[ 347.607119] Function entered at [<c0184614>__report_bad_irq(spurious.c:192)] from [<c015a454>note_interrupt(spurious.c:418)]'
[2025-02-25 17:03:39.431]: b'[ 347.613174] r9:ceb24000 r8:cf806000 r7:000000a8 r6:ceb25ce8 r5:00000000 r4:cf9aacc0'
[2025-02-25 17:03:39.447]: b'[ 347.621210] Function entered at [<c015a208>note_interrupt(spurious.c:272)] from [<c0157e50>handle_irq_event_percpu(handle.c:190)]'
[2025-02-25 17:03:39.447]: b'[ 347.627266] r10:00000000 r9:ceb24000 r8:cf806000 r7:00000001 r6:ceb25ce8 r5:00000000'
[2025-02-25 17:03:39.447]: b'[ 347.635393] r4:cf9aacc0 r3:00000000'
[2025-02-25 17:03:39.447]: b'[ 347.639098] Function entered at [<c0157d4c>handle_irq_event(handle.c:195)] from [<c015ade8>irqd_irq_disabled(irq.h:279)]'
[2025-02-25 17:03:39.463]: b'[ 347.645150] r5:cf9aacd0 r4:cf9aacc0'
[2025-02-25 17:03:39.463]: b'[ 347.648856] Function entered at [<c015ad24>handle_level_irq(chip.c:499)] from [<c0157520>__handle_domain_irq(irqdesc.c:644)]'
[2025-02-25 17:03:39.463]: b'[ 347.654909] r5:c072df10 r4:00000000'
[2025-02-25 17:03:39.478]: b'[ 347.658612] Function entered at [<c01574b4>__handle_domain_irq(irqdesc.c:621)] from [<c01014b8>omap_intc_handle_irq(irq-omap-intc.c:365)]'
[2025-02-25 17:03:39.478]: b'[ 347.664668] r9:ceb24000 r8:cf806000 r7:ceb25c24 r6:ffffffff r5:60070113 r4:c075c574'
[2025-02-25 17:03:39.494]: b'[ 347.672705] Function entered at [<c010147c>omap_intc_handle_irq(irq-omap-intc.c:334)] from [<c010b078>__irq_svc(entry-armv.S:222)]'
[2025-02-25 17:03:39.494]: b'[ 347.678756] Exception stack(0xceb25bf0 to 0xceb25c38)'
[2025-02-25 17:03:39.494]: b'[ 347.683999] 5be0: 00400140 c073cac0 c073ca80 fffffdfe'
[2025-02-25 17:03:39.510]: b'[ 347.692490] 5c00: 00000202 c072df10 00000000 ceb24000 cf806000 ceb24000 00000000 ceb25c94'
[2025-02-25 17:03:39.510]: b'[ 347.700981] 5c20: ceb25c98 ceb25c40 c0128ce8 c0128838 60070113 ffffffff'
[2025-02-25 17:03:39.510]: b'[ 347.707843] r5:60070113 r4:c0128838'
[2025-02-25 17:03:39.525]: b'[ 347.711548] Function entered at [<c01287b4>__do_softirq(softirq.c:246)] from [<c0128ce8>do_softirq_own_stack(interrupt.h:477)]'
[2025-02-25 17:03:39.541]: b'[ 347.717604] r10:00000000 r9:ceb24000 r8:cf806000 r7:00000001 r6:00000000 r5:c072df10'
[2025-02-25 17:03:39.541]: b'[ 347.725729] r4:00000000'
[2025-02-25 17:03:39.541]: b'[ 347.728351] Function entered at [<c0128bc4>irq_exit(softirq.c:399)] from [<c0157524>__handle_domain_irq(irqdesc.c:647)]'
[2025-02-25 17:03:39.556]: b'[ 347.734402] Function entered at [<c01574b4>__handle_domain_irq(irqdesc.c:621)] from [<c01014b8>omap_intc_handle_irq(irq-omap-intc.c:365)]'
[2025-02-25 17:03:39.556]: b'[ 347.740457] r9:ceb24000 r8:000000a8 r7:ceb25d1c r6:ffffffff r5:40070013 r4:c075c574'
[2025-02-25 17:03:39.556]: b'[ 347.748493] Function entered at [<c010147c>omap_intc_handle_irq(irq-omap-intc.c:334)] from [<c010b078>__irq_svc(entry-armv.S:222)]'
[2025-02-25 17:03:39.572]: b'[ 347.754545] Exception stack(0xceb25ce8 to 0xceb25d30)'
[2025-02-25 17:03:39.572]: b'[ 347.759786] 5ce0: 00000000 000000a8 00000100 00000000 cf9aacc0 cfbf65c0'
[2025-02-25 17:03:39.588]: b'[ 347.768276] 5d00: 000000a8 60070013 000000a8 cf9aacd0 00000000 ceb25d64 ceb25d18 ceb25d38'
[2025-02-25 17:03:39.588]: b'[ 347.776764] 5d20: c015b4ec c01597a0 40070013 ffffffff'
[2025-02-25 17:03:39.588]: b'[ 347.782002] r5:40070013 r4:c01597a0'
[2025-02-25 17:03:39.603]: b'[ 347.785707] Function entered at [<c0159460>__setup_irq(manage.c:1132)] from [<c0159c4c>chip_bus_sync_unlock(internals.h:128)]'
[2025-02-25 17:03:39.603]: b'[ 347.791762] r9:cf9c7000 r8:000000a8 r7:cf9aacd0 r6:00000000 r5:cf9aacc0 r4:cfbf65c0'
[2025-02-25 17:03:39.619]: b'[ 347.799798] Function entered at [<c0159b68>request_threaded_irq(manage.c:1652)] from [<c0386b10>c_can_open(c_can.c:1156)]'
[2025-02-25 17:03:39.619]: b'[ 347.805853] r10:00000036 r9:ceb24000 r8:00000201 r7:00040080 r6:cf9c7440 r5:00000000'
[2025-02-25 17:03:39.635]: b'[ 347.813979] r4:cf9c7000 r3:00000080'
[2025-02-25 17:03:39.635]: b'[ 347.817683] Function entered at [<c0386a9c>c_can_open(c_can.c:1139)] from [<c03f6ed4>__dev_open(dev.c:1343)]'
[2025-02-25 17:03:39.650]: b'[ 347.823737] r6:00000081 r5:c04c693c r4:cf9c7000'
[2025-02-25 17:03:39.650]: b'[ 347.828525] Function entered at [<c03f6de0>__dev_open(dev.c:1313)] from [<c03f72c8>__dev_change_flags(dev.c:6505)]'
[2025-02-25 17:03:39.650]: b'[ 347.834577] r5:00000001 r4:cf9c7000'
[2025-02-25 17:03:39.650]: b'[ 347.838282] Function entered at [<c03f7204>__dev_change_flags(dev.c:6472)] from [<c03f73cc>dev_change_flags(dev.c:6571)]'
[2025-02-25 17:03:39.666]: b'[ 347.844338] r9:ceb24000 r8:00040080 r7:cf9c714c r6:00000000 r5:cf9c7000 r4:cf9c7000'
[2025-02-25 17:03:39.666]: b'[ 347.852373] Function entered at [<c03f73ac>dev_change_flags(dev.c:6566)] from [<c0410a84>dev_ifsioc(dev_ioctl.c:255)]'
[2025-02-25 17:03:39.681]: b'[ 347.858429] r9:ceb24000 r8:00000000 r7:c04c693c r6:ceb25e74 r5:cf9c7000 r4:00008914'
[2025-02-25 17:03:39.681]: b'[ 347.866465] Function entered at [<c041082c>dev_ifsioc(dev_ioctl.c:243)] from [<c0411084>dev_ioctl(dev_ioctl.c:533)]'
[2025-02-25 17:03:39.681]: b'[ 347.872519] r7:c0737a80 r6:b545bbc0 r5:00000014 r4:00008914'
[2025-02-25 17:03:39.697]: b'[ 347.878389] Function entered at [<c0410bd8>dev_ioctl(dev_ioctl.c:395)] from [<c03d6908>sock_do_ioctl(socket.c:886)]'
[2025-02-25 17:03:39.697]: b'[ 347.884445] r10:00000036 r9:ceb24000 r8:cf39a300 r7:c0737a80 r6:b545bbc0 r5:fffffdfd'
[2025-02-25 17:03:39.697]: b'[ 347.892572] r4:00008914'
[2025-02-25 17:03:39.713]: b'[ 347.895195] Function entered at [<c03d680c>sock_ioctl(socket.c:897)] from [<c01d83e0>vfs_ioctl(ioctl.c:44)]'
[2025-02-25 17:03:39.728]: b'[ 347.901249] r7:00000006 r6:ceceb620 r5:b545bbc0 r4:cf39a300'
[2025-02-25 17:03:39.728]: b'[ 347.907120] Function entered at [<c01d8030>do_vfs_ioctl(ioctl.c:620)] from [<c01d8928>fdput(file.h:38)]'
[2025-02-25 17:03:39.728]: b'[ 347.913176] r10:00000036 r9:ceb24000 r8:cf39a300 r7:b545bbc0 r6:00008914 r5:00000006'
[2025-02-25 17:03:39.728]: b'[ 347.921301] r4:cf39a301'
[2025-02-25 17:03:39.744]: b'[ 347.923923] Function entered at [<c01d88ec>SyS_ioctl(ioctl.c:685)] from [<c01076c0>ret_fast_syscall(entry-common.S:38)]'
[2025-02-25 17:03:39.744]: b'[ 347.929977] r9:ceb24000 r8:c01078a4 r7:00000036 r6:b545bbc0 r5:00000001 r4:00000006'
[2025-02-25 17:03:39.744]: b'[ 347.938013] handlers:'
[2025-02-25 17:03:39.759]: b'[ 347.940367] [<c0386864>c_can_isr(c_can.c:1118)] 0xc0386864'
[2025-02-25 17:03:39.759]: b'[ 347.943984] Disabling IRQ #168'

  • Hi,

    Thank for the well documented problem re-creation. We will not be able to reproduce the issue that you have found though. Have you compared your workaround solution to a later kernel? 

    Best Regards,

    Schuyler

  • Hi,

    We have only tested with kernels from the 4.9.x tree up to the latest one. We tried using the 5.10 kernel, but due to other issues, we haven't been able to get this kernel to boot on our board yet. The problem here can be reproduced by placing our board on a CAN bus with two other external devices, which communicate with each other by sending multiple frames per millisecond at a rate of 250000 bps, meaning the bus was loaded. Additionally, with the bash script below on our board to continuously open and close the CAN, the problem occurred once after two hours.

    #!/bin/sh
    
    while true;
    do
    sleep 0.1
    ip link set can0 down
    sleep 0.1
    ip link set can0 up
    done

  • Hi,

    Since you have a potential driver solution to the issue I was wondering if you had compared the patch you created with later kernels. The CAN driver is a community driver, TI did not write it. The solution you described may been implemented already in a later kernel.  But you are saying the latest kernel has the same issue? Which version is that?

    Best Regards,

    Schuyler