AM623: open mcu_canx result crash in SDK10

Tony Tang

Part Number: AM623
Other Parts Discussed in Thread: SK-AM62B

Tool/software:

With latest SDK10 default wic file, on SK-AM62B board, just open mcu_can, system will crash in short. log as below

Iti is fine on SDK9. x

root@am62xx-evm:~# dmesg |grep can
[    0.674889] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.674897] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.674908] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.674915] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    6.902241] m_can_platform 4e08000.can: m_can device registered (irq=0, version=32)
[    6.933853] m_can_platform 4e18000.can: m_can device registered (irq=0, version=32)
[    6.973611] m_can_platform 20701000.can: m_can device registered (irq=422, version=32)
[    7.755492] m_can_platform 4e18000.can mcu_mcan1: renamed from can1
[    7.771815] m_can_platform 4e08000.can mcu_mcan0: renamed from can0
[    7.817488] m_can_platform 20701000.can main_mcan0: renamed from can2
root@am62xx-evm:~# ip link set mcu_mcan0 type can bitrate 1000000 dbitrate 5000000 fd on
root@am62xx-evm:~# ip link set mcu_mcan0 up
root@am62xx-evm:~# candump mcu_mcan0
[  139.017058] rcu: INFO: rcu_preempt self-detected stall on CPU
[  139.017081] rcu:     1-...!: (22332 ticks this GP) idle=86ec/1/0x4000000000000002 softirq=                    /0 fqs=1
[  139.017094] rcu:     (t=21000 jiffies g=9913 q=98 ncpus=4)
[  139.017099] rcu: rcu_preempt kthread timer wakeup didn't happen for 20993 jiffies! g9913 f                    x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[  139.017106] rcu:     Possible timer handling issue on cpu=1 timer-softirq=2324
[  139.017110] rcu: rcu_preempt kthread starved for 20996 jiffies! g9913 f0x0 RCU_GP_WAIT_FQS                    5) ->state=0x402 ->cpu=1
[  139.017116] rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expec                    ed behavior.
[  139.017119] rcu: RCU grace-period kthread stack dump:
[  139.017122] task:rcu_preempt     state:I stack:0     pid:16    ppid:2      flags:0x0000000
[  139.017136] Call trace:
[  139.017140]  0xffff8000809c7874
[  139.017144]  0xffff8000809c7b64
[  139.017147]  0xffff8000809c7f48
[  139.017150]  0xffff8000809cfad8
[  139.017153]  0xffff80008009ca78
[  139.017156]  0xffff80008009fc90
[  139.017159]  0xffff800080059878
[  139.017162]  0xffff800080014870
[  139.017166] rcu: Stack dump where RCU GP kthread last ran:
[  139.017174] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G           O       6.6.32-rt32-ti-rt-g                    4a9ad081f0f-dirty #1
[  139.017181] Hardware name: Texas Instruments AM625 SK (DT)
[  139.017185] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  139.017192] pc : 0xffff8000809c3c5c
[  139.017195] lr : 0xffff8000809c3c58
[  139.017198] sp : ffff0000001c7e30
[  139.017200] x29: ffff0000001c7e30 x28: 0000000000000000 x27: 0000000000000000
[  139.017210] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[  139.017218] x23: 0000000000000000 x22: ffff00000017e9c0 x21: ffff800080e4ba7c
[  139.017227] x20: ffff800080e4b980 x19: 0000000000000001 x18: 0000000000000000
[  139.017236] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[  139.017244] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[  139.017253] x11: 0000000000000000 x10: 00000000000008c0 x9 : ffff0000001c7d90
[  139.017261] x8 : ffff00000017f2e0 x7 : 00000000000000c0 x6 : ffff00007f66bd98
[  139.017270] x5 : ffff800080cb9db0 x4 : 0000000000000000 x3 : ffff7ffffe9b2000
[  139.017279] x2 : 0000000000000001 x1 : ffff0000001c7e10 x0 : 4000000000000000
[  139.017288] Call trace:
[  139.017290]  0xffff8000809c3c5c
[  139.017293]  0xffff800080073f84
[  139.017296]  0xffff800080074198
[  139.017299]  0xffff80008001cd84
[  139.017302]  0xffff800080022ac8
[  139.017310] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G           O       6.6.32-rt32-ti-rt-g                    4a9ad081f0f-dirty #1
[  139.017317] Hardware name: Texas Instruments AM625 SK (DT)
[  139.017319] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  139.017325] pc : 0xffff8000809c3c5c
[  139.017328] lr : 0xffff8000809c3c58
[  139.017331] sp : ffff0000001c7e30
[  139.017333] x29: ffff0000001c7e30 x28: 0000000000000000 x27: 0000000000000000
[  139.017342] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
[  139.017351] x23: 0000000000000000 x22: ffff00000017e9c0 x21: ffff800080e4ba7c
[  139.017360] x20: ffff800080e4b980 x19: 0000000000000001 x18: 0000000000000000
[  139.017368] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[  139.017377] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[  139.017385] x11: 0000000000000000 x10: 00000000000008c0 x9 : ffff0000001c7d90
[  139.017394] x8 : ffff00000017f2e0 x7 : 00000000000000c0 x6 : ffff00007f66bd98
[  139.017402] x5 : ffff800080cb9db0 x4 : 0000000000000000 x3 : ffff7ffffe9b2000
[  139.017411] x2 : 0000000000000001 x1 : ffff0000001c7e10 x0 : 4000000000000000
[  139.017420] Call trace:
[  139.017422]  0xffff8000809c3c5c
[  139.017425]  0xffff800080073f84
[  139.017428]  0xffff800080074198
[  139.017430]  0xffff80008001cd84
[  139.017433]  0xffff800080022ac8

root@am62xx-evm:~# dmesg |grep can
[    0.498062] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.498070] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.498077] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.498085] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    7.212447] m_can_platform 4e08000.can: m_can device registered (irq=0, version=32)
[    7.227433] m_can_platform 4e18000.can: m_can device registered (irq=0, version=32)
[    7.242693] m_can_platform 20701000.can: m_can device registered (irq=422, version=32)
[    8.013780] m_can_platform 4e18000.can mcu_mcan1: renamed from can1
[    8.028649] m_can_platform 4e08000.can mcu_mcan0: renamed from can0
[    8.049921] m_can_platform 20701000.can main_mcan0: renamed from can2
root@am62xx-evm:~#
root@am62xx-evm:~# ip link set mcu_mcan1 type can bitrate 1000000 dbitrate 5000000 fd on
root@am62xx-evm:~# ip link set mcu_mcan1 up
root@am62xx-evm:~#
[  292.334829] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  292.334851] rcu:     Tasks blocked on level-0 rcu_node (CPUs 0-3): P33/3:b..l
[  292.334871] rcu:     (detected by 1, t=21002 jiffies, g=10953, q=37 ncpus=4)
[  292.334879] task:ktimers/2       state:D stack:0     pid:33    ppid:2      flags:0x00000008
[  292.334889] Call trace:
[  292.334893]  0xffff8000809c7874
[  292.334898]  0xffff8000809c7b64
[  292.334901]  0xffff8000809c84b8
[  292.334904]  0xffff8000809cdf58
[  292.334907]  0xffff8000809ce810
[  292.334910]  0xffff8000800b2720
[  292.334913]  0xffff8000800b27ac
[  292.334916]  0xffff8000790dd604
[  292.334919]  0xffff8000790dd640
[  292.334922]  0xffff8000800b1928
[  292.334924]  0xffff8000800b1f00
[  292.334927]  0xffff8000800102c0
[  292.334930]  0xffff80008003c050
[  292.334933]  0xffff80008005efb4
[  292.334936]  0xffff800080059878
[  292.334939]  0xffff800080014870

It is ok with main_can

root@am62xx-evm:~# dmesg |grep can
[    0.674682] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page
[    0.674690] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page
[    0.674697] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    0.674710] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    7.251485] m_can_platform 4e08000.can: m_can device registered (irq=0, version=32)
[    7.303449] m_can_platform 4e18000.can: m_can device registered (irq=0, version=32)
[    7.325038] m_can_platform 20701000.can: m_can device registered (irq=422, version=32)
[    8.279292] m_can_platform 4e08000.can mcu_mcan0: renamed from can0
[    8.340323] m_can_platform 4e18000.can mcu_mcan1: renamed from can1
[    8.357951] m_can_platform 20701000.can main_mcan0: renamed from can2
root@am62xx-evm:~#
root@am62xx-evm:~# ip link set main_mcan0 type can bitrate 1000000 dbitrate 5000000 fd on
root@am62xx-evm:~# ip link set main_mcan0 up
root@am62xx-evm:~#
root@am62xx-evm:~#
root@am62xx-evm:~# uname -a
Linux am62xx-evm 6.6.32-rt32-ti-rt-g04a9ad081f0f-dirty #1 SMP PREEMPT_RT Fri Jul 26 14:42:37 UTC 2024 aarch64 GNU/Linux
root@am62xx-evm:~#

2 months ago

0 Aparna Patra 2 months ago

TI__Genius 12370 points

Hi Tony,

I will reproduce the MCAN tests today and let you know by EOD.

Regards,
Aparna

0 Aparna Patra 2 months ago in reply to Aparna Patra

TI__Genius 12370 points

Hi Tony,

I am able to reproduce this error at my setup. I will create a JIRA and further check with internal team if there is a fix available.

Regards,
Aparna

0 Tony Tang 2 months ago in reply to Aparna Patra

TI__Mastermind 27121 points

Hi Patra,

Is there a solution? we can't wait next release. must fix it on current SDK and soon.

0 Bin Liu 2 months ago in reply to Tony Tang

TI__Guru*** 147036 points

Hi Tony,

This usecase is broken in upstream kernel too, and a fix has been posted upstream. We will backport and validate the fix to SDK10.0 in a few days, and provide you the fix.

0 Tony Tang 1 month ago in reply to Bin Liu

TI__Mastermind 27121 points

Hi Bin,

Is there a target date?

0 Aparna Patra 1 month ago in reply to Tony Tang

TI__Genius 12370 points

Hi Tony,

The fix is shared by the developer, please refer the patch:

https://lore.kernel.org/linux-can/20240830215914.1610393-7-mkl@pengutronix.de/#r

Regards,
Aparna

0 Tony Tang 1 month ago in reply to Aparna Patra

TI__Mastermind 27121 points

Hi Aparma,

the patch can't apply to SDK10. there is not static enum hrtimer_restart m_can_coalescing_timer(struct hrtimer *timer) and static void m_can_coalescing_update(struct m_can_classdev *cdev, u32 ir) function is the SDK10 m_can.c. it is a new version again?

0 Aparna Patra 1 month ago in reply to Tony Tang

TI__Genius 12370 points

Hi Tony,

Please apply the whole series from the earlier shared link.

This series has not been merged to ti-linux kernel yet.

Regards,
Aparna

0 Tony Tang 1 month ago in reply to Aparna Patra

TI__Mastermind 27121 points

Do you mean all of this? can you help to make a package for customer?

2024-08-30 21:53 [PATCH net 0/n] pull-request: can 2024-08-30 Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 01/13] can: bcm: Remove proc entry when dev is unregistered Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 02/13] can: m_can: Release irq on error in m_can_open Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 03/13] can: m_can: Reset coalescing during suspend/resume Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 04/13] can: m_can: Remove coalesing disable in isr during suspend Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 05/13] can: m_can: Remove m_can_rx_peripheral indirection Marc Kleine-Budde
2024-08-30 21:53 ` Marc Kleine-Budde [this message]
2024-08-30 21:53 ` [PATCH net 07/13] can: m_can: disable_all_interrupts, not clear active_interrupts Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 08/13] can: m_can: Reset cached active_interrupts on start Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 09/13] can: m_can: Limit coalescing to peripheral instances Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 10/13] can: mcp251xfd: fix ring configuration when switching from CAN-CC to CAN-FD mode Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 11/13] can: mcp251xfd: mcp251xfd_ring_init(): check TX-coalescing configuration Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 12/13] can: mcp251x: fix deadlock if an interrupt occurs during mcp251x_open Marc Kleine-Budde
2024-08-30 21:53 ` [PATCH net 13/13] can: kvaser_pciefd: Use a single write when releasing RX buffers Marc Kleine-Budde
2024-09-03  2:08 ` [PATCH net 0/n] pull-request: can 2024-08-30 Jakub Kicinski
  -- strict thread matches above, loose matches on Subject: below --
2024-08-29 19:20 [PATCH net 0/13] pull-request: can 2024-08-29 Marc Kleine-Budde
2024-08-29 19:20 ` [PATCH net 06/13] can: m_can: Do not cancel timer from within timer Marc Kleine-Budde

0 Aparna Patra 1 month ago in reply to Tony Tang

TI__Genius 12370 points

Hi Tony,

Sorry, I cannot help package these patches. I am out of office for this week.

Please ask the customer to apply the changes one by one.

Regards,
Aparna

0 Tony Tang 1 month ago in reply to Aparna Patra

TI__Mastermind 27121 points

Hi Apama

Were you back?

We can't patch these patches, some functions are not exist, please verify it on your side, then provide us a correct soluiton.

0 Aparna Patra 1 month ago in reply to Tony Tang

TI__Genius 12370 points

Hi Tony,

I will verify it today and let you know by EOD.

Thanks,
Aparna

0 Tony Tang 1 month ago in reply to Aparna Patra

TI__Mastermind 27121 points

Hi Aparna,

What is going on? I guess you encounter problem also.

0 Mukul Bhatnagar 1 month ago in reply to Tony Tang

TI__Guru* 81755 points

Hello Tony

Yes, we will likely need some more time. Comments from the development team.

>> We are waiting for the mentioned fixes to make it to linux-stable and then merge the stable-sync since the patches are not on linux-master yet, current plan is to start backporting them .

This is tracked to be fixed in SDK 10.1 , but Aparna will work with dev team to see if we can enable you sooner.

0 Jim Mrowca 1 month ago in reply to Aparna Patra

Expert 2257 points

Aparna & Mukul:

Does this AM62x issue with mcu_can also apply to AM64x on the Linux side??

concerned,

Jim

0 Mukul Bhatnagar 1 month ago in reply to Jim Mrowca

TI__Guru* 81755 points

No it does not.

Are you using LInux or RTOS and are you using all CANs on AM64 ? If you are seeing any issues , please feel free to open another E2E thread.

The issue here is specific to AM62, where some of the CANs are in MCU domain and have to enabled in Linux , for customers to use all 3 CANs in Linux. The regression happened due to some additional low power management features. So no impact on AM64, where CAN domains are all in Main domain and there is no support for low power modes in AM64.

0 Aparna Patra 29 days ago in reply to Tony Tang

TI__Genius 12370 points

Hi Tony,

PFA the fix to the MCU mcan timing issue in polling mode. The two commit that caused the issue are reverted and patches from upstream have been backported. Please ask the customer to apply these and test. I have also merged all the commits into one diff file as you had requested previously. You can chose to apply the patches either ways on top of 10.0 ti-linux-kernel.

I am in the process of testing it out myself. I will notify the thread once it is verified. However, the developer has already verified it via automated test farm.

AM62x-mcu-mcan-fix-patches.zip
AM62x-mcu-mcan-fix-merged.patch

Regards,
Aparna

0 Aparna Patra 27 days ago in reply to Aparna Patra

TI__Genius 12370 points

Hi Tony,

I have verified by applying the above fix on my setup and MCU_MCANs work without any hang. Please refer the logs below:

root@am62xx-evm:~# dmesg | grep can                                             
[    0.401130] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page          
[    0.414480] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page          
[    0.427830] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page          
[    0.441178] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page          
[    8.090799] m_can_platform 4e08000.can: m_can device registered (irq=0, vers)
[    8.111684] m_can_platform 4e18000.can: m_can device registered (irq=0, vers)
[    8.132176] m_can_platform 20701000.can: m_can device registered (irq=422, v)
[    8.766718] m_can_platform 20701000.can main_mcan0: renamed from can2        
[    8.837050] m_can_platform 4e08000.can mcu_mcan0: renamed from can0          
[    8.908008] m_can_platform 4e18000.can mcu_mcan1: renamed from can1          
00 fd on2xx-evm:~# ip link set mcu_mcan0 type can bitrate 1000000 dbitrate 50000
root@am62xx-evm:~# ip link set mcu_mcan0 up                                     
root@am62xx-evm:~# candump mcu_mcan0                                            
[  106.781987] can: controller area network core                                
[  106.786564] NET: Registered PF_CAN protocol family                           
[  106.805194] can: raw protocol

Regards,
Aparna

Processors

Processors forum

AM623: open mcu_canx result crash in SDK10