TDA4VM J721e runtime kernel panic

He Li

Hi there,

The runtime kernel panic issues have happened, and the details log as follow:

1. Master chip kernel panic log:

nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w)
<1>[165713.192060] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010
<1>[165713.192063] Mem abort info:
<1>[165713.192065] ESR = 0x96000007
<1>[165713.192067] EC = 0x25: DABT (current EL), IL = 32 bits
<1>[165713.192069] SET = 0, FnV = 0
<1>[165713.192070] EA = 0, S1PTW = 0
<1>[165713.192071] Data abort info:
<1>[165713.192073] ISV = 0, ISS = 0x00000007
<1>[165713.192074] CM = 0, WnR = 0
<1>[165713.192077] user pgtable: 64k pages, 48-bit VAs, pgdp=00000008ceeb2a00
<1>[165713.192078] [0000000000000010] pgd=00000008d4d90003, pud=00000008d4d90003, pmd=00000008d3980003, pte=0000000000000000
<0>[165713.192084] Internal error: Oops: 96000007 [#1] PREEMPT SMP
<4>[165713.192087] Modules linked in: xhci_plat_hcd xhci_hcd usbcore omap_rng cdns3 rng_core roles pru_rproc udc_core usb_common irq_pruss_intc icss_iep crct10dif_ce ti_j721e_cpsw_virt_mac tipwm_driver pruss sa2ul sha512_generic authenc cdns3_ti sch_fq_codel ip_tables x_tables ipv6 pvrsrvkm(O) rpmsg_kdrv_switch rpmsg_pru virtio_rpmsg_bus rpmsg_char ti_k3_dsp_remoteproc ti_k3_r5_remoteproc
<4>[165713.192113] CPU: 0 PID: 380 Comm: iperf3 Tainted: G O 5.4.74-ACU-3.0.1.2 #4
<4>[165713.192114] Hardware name: Texas Instruments K3 J721E SoC (DT)
<4>[165713.192117] pstate: 20000005 (nzCv daif -PAN -UAO)
<4>[165713.192126] pc : virt_cpsw_nuss_tx_poll+0xb4/0x280 [ti_j721e_cpsw_virt_mac]
<4>[165713.192130] lr : virt_cpsw_nuss_tx_poll+0xa4/0x280 [ti_j721e_cpsw_virt_mac]
<4>[165713.192131] sp : ffff80001632f690
<4>[165713.192132] x29: ffff80001632f690 x28: ffff00084b9f0000
<4>[165713.192135] x27: ffff00084ea420b0 x26: ffff00084076cc10
<4>[165713.192137] x25: ffff00084ea420a8 x24: 00000000000005ea
<4>[165713.192139] x23: 000000000000003e x22: ffff00084c1a6200
<4>[165713.192141] x21: 0000000000000002 x20: 0000000000000040
<4>[165713.192143] x19: ffff00084ea420d0 x18: 0000000000000000
<4>[165713.192145] x17: 0000000000000000 x16: 0000000000000000
<4>[165713.192147] x15: 0000000000000000 x14: 6c42f5c088b441bd
<4>[165713.192150] x13: ffff800012860000 x12: 0000000000000001
<4>[165713.192152] x11: 1e865ed87b7579ac x10: 0000000000000000
<4>[165713.192155] x9 : 0000000000000003 x8 : ffff800012860038
<4>[165713.192157] x7 : 0000000000000001 x6 : 0000000000000000
<4>[165713.192160] x5 : fffffffffffffff7 x4 : ffff80086eb00000
<4>[165713.192162] x3 : 0000000000000000 x2 : 000000000000ff00
<4>[165713.192164] x1 : 000000000000003e x0 : 0000000000000000
<4>[165713.192167] Call trace:
<4>[165713.192173] virt_cpsw_nuss_tx_poll+0xb4/0x280 [ti_j721e_cpsw_virt_mac]
<4>[165713.192193] net_rx_action+0x114/0x340
<4>[165713.192202] __do_softirq+0x120/0x25c
<4>[165713.192206] do_softirq.part.0+0x60/0x68
<4>[165713.192209] __local_bh_enable_ip+0x8c/0x98
<4>[165713.192218] ip_finish_output2+0x17c/0x580
<4>[165713.192220] __ip_finish_output+0xc8/0x200
<4>[165713.192223] ip_output+0xf4/0x1a8
<4>[165713.192225] ip_local_out+0x44/0x58
<4>[165713.192228] __ip_queue_xmit+0x124/0x390
<4>[165713.192255] ip_queue_xmit+0x10/0x18 [ipv6]
<4>[165713.192259] __tcp_transmit_skb+0x448/0xa88
<4>[165713.192261] __tcp_send_ack.part.0+0xc8/0x118
<4>[165713.192263] tcp_send_ack+0x20/0x30
<4>[165713.192265] tcp_cleanup_rbuf+0x64/0x100
<4>[165713.192267] tcp_recvmsg+0x1cc/0x950
<4>[165713.192282] inet6_recvmsg+0x4c/0x118 [ipv6]
<4>[165713.192286] sock_read_iter+0x90/0xe8
<4>[165713.192292] new_sync_read+0xe4/0x178
<4>[165713.192294] __vfs_read+0x2c/0x40
<4>[165713.192298] vfs_read+0xc8/0x1a0
<4>[165713.192301] ksys_read+0x68/0xf0
<4>[165713.192303] __arm64_sys_read+0x18/0x20
<4>[165713.192308] el0_svc_common.constprop.0+0x68/0x160
<4>[165713.192311] el0_svc_handler+0x20/0x80
<4>[165713.192313] el0_svc+0x8/0x204
<0>[165713.192320] Code: f94037e3 d538d084 2a1703e1 aa0303e0 (f940087c)
<4>[165713.192325] ---[ end trace fb0da6d660b201f8 ]---
<0>[165713.193109] Kernel panic - not syncing: Fatal exception in interrupt
<2>[165713.193112] SMP: stopping secondary CPUs
<0>[165713.193123] Kernel Offset: disabled

<0>[165713.193128] CPU features: 0x0002,20006008

<0>[165713.193129] Memory Limit: none

2. Slave chip kernel panic log:

<4>[24663.834031] ------------[ cut here ]------------
<2>[24663.834039] kernel BUG at lib/genalloc.c:506!
<0>[24663.834042] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
<4>[24663.834046] Modules linked in: xhci_plat_hcd xhci_hcd usbcore omap_rng rng_core cdns3 roles udc_core usb_common pru_rproc irq_pruss_intc icss_iep ti_j721e_cpsw_virt_mac crct10dif_ce tipwm_driver pruss sa2ul cdns3_ti sha512_generic authenc sch_fq_codel ip_tables x_tables ipv6 pvrsrvkm(O) rpmsg_kdrv_switch rpmsg_pru virtio_rpmsg_bus rpmsg_char ti_k3_dsp_remoteproc ti_k3_r5_remoteproc
<4>[24663.834075] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 5.4.74-ACU-3.0.1.2 #4
<4>[24663.834077] Hardware name: Texas Instruments K3 J721E SoC (DT)
<4>[24663.834079] pstate: 00000005 (nzcv daif -PAN -UAO)
<4>[24663.834091] pc : gen_pool_free_owner+0xf0/0x100
<4>[24663.834093] lr : gen_pool_free_owner+0xa4/0x100
<4>[24663.834095] sp : ffff80001000fd40
<4>[24663.834096] x29: ffff80001000fd40 x28: 0000000000000040
<4>[24663.834099] x27: ffff000846dbc0b0 x26: ffff00084076cc10
<4>[24663.834101] x25: ffff000846dbc0a8 x24: 0000000000000007
<4>[24663.834104] x23: ffff00007d8800ff x22: 0000000000000000
<4>[24663.834106] x21: 0000000000000001 x20: ffff00084b1d1c00
<4>[24663.834108] x19: ffff00007d880080 x18: 0000000000000000
<4>[24663.834110] x17: 0000000000000000 x16: 0000000000000000
<4>[24663.834112] x15: 0000000000000000 x14: 0000000000000000
<4>[24663.834114] x13: ffff800011f40000 x12: 0000000000000001
<4>[24663.834116] x11: 0000000000000000 x10: 0000000000000001
<4>[24663.834119] x9 : 0000000000000001 x8 : ffff800011f40038
<4>[24663.834122] x7 : 0000000000000001 x6 : 0000000000000000
<4>[24663.834124] x5 : fffffffffffffffd x4 : 0000000000000002
<4>[24663.834126] x3 : ffff800011f40038 x2 : 0000000000000003
<4>[24663.834128] x1 : 00000000fffff000 x0 : 0000000000000001
<4>[24663.834133] Call trace:
<4>[24663.834137] gen_pool_free_owner+0xf0/0x100
<4>[24663.834168] k3_cppi_desc_pool_free

<4>[24663.834171] virt_cpsw_nuss_xmit_free.isra.0+0x60/0xf8 [ti_j721e_cpsw_virt_mac]
<4>[24663.834175] virt_cpsw_nuss_tx_poll+0xa4/0x280 [ti_j721e_cpsw_virt_mac]
<4>[24663.834188] net_rx_action+0x114/0x340
<4>[24663.834200] __do_softirq+0x120/0x25c
<4>[24663.834207] irq_exit+0xb8/0xd8
<4>[24663.834216] __handle_domain_irq+0x64/0xb8
<4>[24663.834219] gic_handle_irq+0x5c/0x148
<4>[24663.834220] el1_irq+0xb8/0x180
<4>[24663.834223] arch_cpu_idle+0x10/0x18
<4>[24663.834235] do_idle+0xc0/0x140
<4>[24663.834237] cpu_startup_entry+0x24/0x68
<4>[24663.834247] rest_init+0xd0/0xdc
<4>[24663.834252] arch_call_rest_init+0xc/0x14
<4>[24663.834254] start_kernel+0x408/0x43c
<0>[24663.834258] Code: d65f03c0 14000044 17fffff6 d4210000 (d4210000)
<4>[24663.834262] ---[ end trace 71655ce054378e36 ]---
<0>[24663.834974] Kernel panic - not syncing: Fatal exception in interrupt
<2>[24663.834977] SMP: stopping secondary CPUs
<0>[24663.834986] Kernel Offset: disabled
<0>[24663.834989] CPU features: 0x0002,20006008
<0>[24663.834990] Memory Limit: none

over 3 years ago

0 He Li over 3 years ago

Prodigy 90 points

master_panic_log.rar slave_panic_log.rar

0 Keerthy J over 3 years ago in reply to He Li

TI__Guru**** 166510 points

Hi,

Can you add more details on the SDK version being used and also the use case that is running?

Best Regards,
Keerthy

0 He Li over 3 years ago

Prodigy 90 points

SDK 7.2， just runtime without special use case

0 Karthik Ramanan over 3 years ago in reply to He Li

TI__Guru** 113810 points

He Li,

Have you tried to reproduce this issue on the TI EVM and/or can you share more details on the conditions that trigger this issue?

We would like to reproduce this issue and debug further.

Regards

Karthik

0 Stanley Liu over 3 years ago in reply to Karthik Ramanan

TI__Guru 50045 points

Hi,

Could you provide the latest update on this issue?

Regards,

Stanley

0 Kangjia Dong over 3 years ago

TI__Intellectual 2285 points

After modify DTS to delete dma-coherent, it can avoid kernel panic isuue. The reason show as below:

The “dma-coherent” subnode defined in the virtual-mac driver is probed by the virt-mac driver, where it tells the driver to allocate coherent memory. Below line is the from the virt-mac driver dt documentation:

“- dma-coherent : indicates that device operates with coherent memory.”

From the symptom, it feels like some memory contention happened under race conditions. When they remove the option, Linux may have treated the DMA buffers as non-coherent memory, that either changed the timing or avoided the contention. This is just a high-level guess. This should not be the permanent solution as Linux shall use the packet buffers DMA’ed by the ethfw in coherent manner.

0 Keerthy J over 3 years ago in reply to Kangjia Dong

TI__Guru**** 166510 points

Hi,

Ran memtester overnight with:

c7x application running in parallel.
msmc shared between A72 and C7x.
No Crashes observed.

This seems to be related with cpsw usage. More tests will be done.
In the meantime removing dma-coherent can be used for now.

- Keerthy

0 Keerthy J over 3 years ago in reply to Keerthy J

TI__Guru**** 166510 points

Hello,

Firewalled DMSC region of MSMC & executed the test case.

No firewall exceptions before the crash. Ruled out any malicious write on the DMSC region of MSMC.
CPSW2G driver debug on going.

0 Keerthy J over 3 years ago in reply to Keerthy J

TI__Guru**** 166510 points

Hello,

We have eliminated the need to to C7x firmware to reproduce the crash. Just with PSDK Linux with CPSW2G & CPSW9G running
iperf3 for 6 hours the same crash was reproduced. Also we have added code instrumentation & observed that alloc/free buffers when it crashes
is not in sync. We have raised the priority with internal team & we hope to root cause on CPSW driver side.

- Keerthy

0 Karthik Ramanan over 3 years ago in reply to Keerthy J

TI__Guru** 113810 points

For TI internal reference this is tracked by : https://jira.itg.ti.com/browse/LCPD-28247

0 He Li over 3 years ago

Prodigy 90 points

Any Update ?

0 Keerthy J over 3 years ago in reply to He Li

TI__Guru**** 166510 points

Hi,

Our development team is debugging this. We suspected a race in alloc/dealloc sequence in CPSW driver. We protected them by using spinlock.
Still the issue occurs. So that was eliminated as well. Trying to look at UDMA buffers & checking if there are issues there. We will keep you
posted.

Thanks,
Keerthy

0 He Li over 3 years ago in reply to Keerthy J

Prodigy 90 points

Any update about this issue?

0 Keerthy J over 3 years ago in reply to He Li

TI__Guru**** 166510 points

Hi,

Here is the latest feedback from our ethernet expert.

The issue looks to be related to either submission of descriptor to the DMA or reading back completed descriptor provided by DMA.

Two suspicions:

1) SW writes the DMA descriptor to the ring and rings doorbell. However because of the DDR load, before the DMA descriptor hits the DDR, DMA starts to do transfers based on old descriptor. This is on the submission side.

2) DMA provides the completed descriptor. However when CPU reads the descriptor, it reads stale data from the cache. (ideally should not happen since it is cache coherent).

There are a few experiments which succeeded like reducing the total number of descriptors that can be submitted from CPSW, but no concrete root cause done. It is still in progress.

- Keerthy

0 Keerthy J over 3 years ago in reply to Keerthy J

TI__Guru**** 166510 points

Patch shared. Testing in progress. We will wait for your feedback.

- Keerthy

0 He Li over 3 years ago in reply to Keerthy J

Prodigy 90 points

We're reproducing the problem. Did you test the validity in advance ？

0 Keerthy J over 3 years ago in reply to He Li

TI__Guru**** 166510 points

Hi,

With the latest patch that we shared we had not seen the issue with overnight testing. Can you share the logs?

- Keerthy

0 He Li over 3 years ago in reply to Keerthy J

Prodigy 90 points

We are still stress testing.

0 He Li over 3 years ago in reply to He Li

Prodigy 90 points

I have continue stressed almost 12 hours, and it works well. May be good news for us.

And one more questions, former we have got another patch from TI side as follow and it also works well.

We have a question : what is the different between these two patches ? Can we use both two patches at the same time ?

--- a/board-support/linux-5.4.74+gitAUTOINC+9574bba32a-g9574bba32a/arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dts
+++ b/board-support/linux-5.4.74+gitAUTOINC+9574bba32a-g9574bba32a/arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dts
@@ -158,7 +158,6 @@

cpsw9g_virt_mac: main_r5fss_cpsw9g_virt_mac0 {
compatible = "ti,j721e-cpsw-virt-mac";
- dma-coherent;
ti,psil-base = <0x4a00>;
ti,remote-name = "mpu_1_0_ethswitch-device-0";

diff --git a/board-support/linux-5.4.74+gitAUTOINC+9574bba32a-g9574bba32a/arch/arm64/boot/dts/ti/k3-j721e-main.dtsi b/board-support/linux-5.4.74+gitAUTOINC+9574bba32a-g9574bba32a/arch/arm64/boot/dts/ti/k3-j721e-main.dtsi
index 1d5e264bc..538e7f2bf 100644
--- a/board-support/linux-5.4.74+gitAUTOINC+9574bba32a-g9574bba32a/arch/arm64/boot/dts/ti/k3-j721e-main.dtsi
+++ b/board-support/linux-5.4.74+gitAUTOINC+9574bba32a-g9574bba32a/arch/arm64/boot/dts/ti/k3-j721e-main.dtsi

@@ -112,7 +112,6 @@
#address-cells = <2>;
#size-cells = <2>;
ranges;
- dma-coherent;
dma-ranges;

ti,sci-dev-id = <199>;

0 Keerthy J over 3 years ago in reply to He Li

TI__Guru**** 166510 points

He Li said:
We have a question : what is the different between these two patches ? Can we use both two patches at the same time ?

Please use the latest patch alone that you are testing. We have the analysis for that patch & hence that is the correct workaround.
No need to add the earlier patch.

Regards,
Keerthy

0 He Li over 3 years ago in reply to He Li

Prodigy 90 points

In addition, the operation of "while (*elem_ptr == 0);" in your last patch may cause the system stuck, Whether this operation is risky ?

0 Keerthy J over 3 years ago in reply to He Li

TI__Guru**** 166510 points

Hi,

In case you enter that path without the patch Linux will crash. That path should only occur before crash.
Hence the check was added.

- Keerthy

Processors

Processors forum

TDA4VM J721e runtime kernel panic