TDA4VH-Q1: The Linux system crashed after running for 8 hours.

Hello,

From the crash log it seems like the origin is kind of related with ext4_create. So where are you hosting the rootfs?
Is this a custom eMMC? 

hongyao.jin said:

However, about 8 hours later, all the apps received an illegal instruction signal, causing them to crash. Immediately after that, the Linux kernel also crashed.

What use cases were running? Was the Linux Idle or some rootfs is being accessed contnuously?

- Keerthy

hi

Keerthy J said:

So where are you hosting the rootfs?

Our storage medium is EMMC, so the rootfs is stored on the EMMC.

Keerthy J said:

Is this a custom eMMC? 

yes

Keerthy J said:

What use cases were running?

Our app mainly involves the processing of camera images, the transmission and reception of Ethernet packets, as well as the execution of some models.

Keerthy J said:

Was the Linux Idle or some rootfs is being accessed contnuously?

Our app will continuously store logs to the emmc via syslog.

Hi Hongyao,

This is happening because the eMMC EXT4 partition ran out of memory as you are continuously logging information for 8 hours

Regards

Gokul

hi

Gokul Praveen said:

This is happening because the eMMC EXT4 partition ran out of memory as you are continuously logging information for 8 hours

We have a 480M log partition. However, after we encountered this issue, we restarted the board and by using the "df -h" command, we could see that the log partition still had free space and was not completely filled.

Hi Hongyao,

I will look into this and get back to you.

Regards

Gokul

hi

We have disabled the logs for all our apps. Currently, the apps are generating very little log output, except for some error messages.

But this problem still occurred.

[25997.236028] audit: type=1334 audit(1693552402.146:100): prog-id=58 op=UNLOAD
[26619.816668] audit: type=1334 audit(1693553024.742:101): prog-id=59 op=LOAD
[26619.974840] audit: type=1334 audit(1693553024.902:102): prog-id=59 op=UNLOAD
[27239.808831] audit: type=1334 audit(1693553644.753:103): prog-id=60 op=LOAD
[27239.971333] audit: type=1334 audit(1693553644.915:104): prog-id=60 op=UNLOAD
[27846.982006] audit: type=1334 audit(1693554251.944:105): prog-id=61 op=LOAD
[27847.133728] audit: type=1334 audit(1693554252.096:106): prog-id=61 op=UNLOAD
[28469.786978] audit: type=1334 audit(1693554874.766:107): prog-id=62 op=LOAD
[28469.958280] audit: type=1334 audit(1693554874.937:108): prog-id=62 op=UNLOAD
[29072.648838] audit: type=1334 audit(1693555477.643:109): prog-id=63 op=LOAD
[29072.771445] audit: type=1334 audit(1693555477.767:110): prog-id=63 op=UNLOAD
[29203.562560] audit: type=1701 audit(1693555608.560:111): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=173076 comm="sleep" exe="/usr/bin/sleep.coreutils" sig=4 res=1
[29203.578501] audit: type=1701 audit(1693555608.560:112): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=173077 comm="sh" exe="/usr/bin/bash.bash" sig=4 res=1
[29203.593604] audit: type=1701 audit(1693555608.564:113): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=173078 comm="sh" exe="/usr/bin/bash.bash" sig=4 res=1
[29203.608696] audit: type=1701 audit(1693555608.564:114): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=173079 comm="sh" exe="/usr/bin/bash.bash" sig=4 res=1
[29203.623770] audit: type=1701 audit(1693555608.568:115): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=746 comm="dds.ev.0" exe="/hirain/appact/hi_6vpercep_app/bin/hi_6vpercep_app.out" sig=4 res=1
[29203.642196] audit: type=1701 audit(1693555608.568:116): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=173080 comm="sh" exe="/usr/bin/bash.bash" sig=4 res=1
[29203.657230] audit: type=1701 audit(1693555608.568:117): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=957 comm="dds.ev.0" exe="/hirain/appact/hi_park_percep/bin/hi_park_percep.out" sig=4 res=1
[29203.675475] audit: type=1701 audit(1693555608.568:118): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=173081 comm="sh" exe="/usr/bin/bash.bash" sig=4 res=1
[29203.690510] audit: type=1701 audit(1693555608.572:119): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=173082 comm="sh" exe="/usr/bin/bash.bash" sig=4 res=1
[29203.705547] audit: type=1701 audit(1693555608.572:120): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=724 comm="dds.ev.0" exe="/hirain/appact/hi_alg_avp/bin/hi_alg_avp.out" sig=4 res=1
[29206.103140] Unable to handle kernel NULL pointer dereference at virtual address 00000000000001b4
[29206.111957] Mem abort info:
[29206.114771]   ESR = 0x0000000096000004
[29206.118537]   EC = 0x25: DABT (current EL), IL = 32 bits
[29206.123869]   SET = 0, FnV = 0
[29206.126925]   EA = 0, S1PTW = 0
[29206.130079]   FSC = 0x04: level 0 translation fault
[29206.134962] Data abort info:
[29206.137841]   ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
[29206.143331]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[29206.148389]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[29206.153706] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000943933000
[29206.160154] [00000000000001b4] pgd=0000000000000000, p4d=0000000000000000
[29206.166950] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
[29206.173206] Modules linked in: 8021q garp mrp stp llc ti_cpsw_proxy_client rpmsg_ctrl rpmsg_char wave5 videobuf2_dma_contig v4l2_mem2mem crct10dif_ce omap_mailbox videobuf2_v4l2 omap_hwspinlock phy_can_transceiver videobuf2_memops videobuf2_common ti_k3_r5_remoteproc k3_j72xx_bandgap ti_k3_dsp_remoteproc videodev ti_k3_common pvrsrvkm(O) sa2ul mc rti_wdt overlay cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[29206.211001] CPU: 7 UID: 0 PID: 90 Comm: kworker/7:1 Tainted: G           O       6.12.17-tda4vhmiddleclassda-00773-gcdcaeac783e3-dirty #1
[29206.223329] Tainted: [O]=OOT_MODULE
[29206.226807] Hardware name: Texas Instruments J784S4 EVM (DT)
[29206.232457] Workqueue: events kernfs_notify_workfn
[29206.237251] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[29206.244198] pc : find_inode_fast+0x40/0x11c
[29206.248379] lr : find_inode_fast+0x2c/0x11c
[29206.252554] sp : ffff800082813c60
[29206.255856] x29: ffff800082813c60 x28: ffff0008cc51de58 x27: 0000000000000000
[29206.262979] x26: 0000000008000002 x25: 0000000000000002 x24: 0000000000000000
[29206.270102] x23: ffff800081333418 x22: ffff000f598d78b8 x21: ffff0008bcebe800
[29206.277225] x20: 00000000000009d4 x19: 0000000000000174 x18: ffff80008bf63a80
[29206.284346] x17: 0000000000000000 x16: ffff800080d3bd68 x15: 00326f6ccf64105c
[29206.291469] x14: 00000d4f634ea924 x13: 0000000000000128 x12: 0000000000000001
[29206.298591] x11: 0000000000000000 x10: 00000000000009e0 x9 : ffff800082813d20
[29206.305714] x8 : 0101010101010101 x7 : ffff7f7f7f7f7f7f x6 : f62c55e0be842000
[29206.312835] x5 : 0000000000000015 x4 : 000004b31f8df921 x3 : 0000000000000000
[29206.319957] x2 : 00000000000009d4 x1 : ffff0008bf2d5280 x0 : 0000000000000001
[29206.327082] Call trace:
[29206.329519]  find_inode_fast+0x40/0x11c
[29206.333353]  ilookup+0x88/0x160
[29206.336487]  kernfs_notify_workfn+0x160/0x228
[29206.340834]  process_one_work+0x148/0x28c
[29206.344841]  worker_thread+0x2d0/0x3d8
[29206.348586]  kthread+0x110/0x114
[29206.351807]  ret_from_fork+0x10/0x20
[29206.355379] Code: b40001b3 d1034273 d503201f b4000153 (f9402263)
[29206.361458] ---[ end trace 0000000000000000 ]---
[29208.918218] kauditd_printk_skb: 40 callbacks suppressed
[29208.918231] audit: type=1701 audit(1693555613.919:161): auid=0 uid=0 gid=0 ses=1 subj=kernel pid=851 comm="(sd-pam)" exe="/usr/lib/systemd/systemd-executor" sig=4 res=1
[29223.314375] Unable to handle kernel paging request at virtual address 0000000000001a70
[29223.317345] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[29223.322292] Mem abort info:
[29223.331052] Mem abort info:
[29223.333830]   ESR = 0x0000000096000006
[29223.336609]   ESR = 0x0000000096000046
[29223.340343]   EC = 0x25: DABT (current EL), IL = 32 bits
[29223.344075]   EC = 0x25: DABT (current EL), IL = 32 bits
[29223.349367]   SET = 0, FnV = 0
[29223.354658]   SET = 0, FnV = 0
[29223.357697]   EA = 0, S1PTW = 0
[29223.360735]   EA = 0, S1PTW = 0
[29223.363861]   FSC = 0x06: level 2 translation fault
[29223.366987]   FSC = 0x06: level 2 translation fault
[29223.371845] Data abort info:
[29223.376704] Data abort info:
[29223.379569]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[29223.382434]   ISV = 0, ISS = 0x00000046, ISS2 = 0x00000000
[29223.387900]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[29223.393365]   CM = 0, WnR = 1, TnD = 0, TagAccess = 0
[29223.398397]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[29223.403429]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[29223.408720] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000943485000
[29223.414013] user pgtable: 4k pages, 48-bit VAs, pgdp=000000093f6bf000
[29223.420431] [0000000000001a70] pgd=0800000941d77003
[29223.426850] [0000000000000000] pgd=080000093f6e6003
[29223.426850] , p4d=0800000941d77003, pud=0800000944634003
[29223.431712] , p4d=080000093f6e6003
[29223.436570] , pmd=0000000000000000
[29223.441861] , pud=080000093f6e7003
[29223.445246]
[29223.448630] , pmd=0000000000000000
[29223.452017] Internal error: Oops: 0000000096000006 [#2] PREEMPT SMP
[29223.453494]
[29223.456880] Modules linked in: 8021q garp mrp stp llc ti_cpsw_proxy_client rpmsg_ctrl rpmsg_char wave5 videobuf2_dma_contig v4l2_mem2mem crct10dif_ce omap_mailbox videobuf2_v4l2 omap_hwspinlock phy_can_transceiver videobuf2_memops videobuf2_common ti_k3_r5_remoteproc k3_j72xx_bandgap ti_k3_dsp_remoteproc videodev ti_k3_common pvrsrvkm(O) sa2ul mc rti_wdt overlay cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[29223.502387] CPU: 3 UID: 0 PID: 832 Comm: dds.ev.0 Tainted: G      D    O       6.12.17-tda4vhmiddleclassda-00773-gcdcaeac783e3-dirty #1
[29223.514533] Tainted: [D]=DIE, [O]=OOT_MODULE
[29223.518788] Hardware name: Texas Instruments J784S4 EVM (DT)
[29223.524429] pstate: 000000c5 (nzcv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[29223.531373] pc : mod_objcg_state.constprop.0+0x1ec/0x268
[29223.536681] lr : mod_objcg_state.constprop.0+0x1e0/0x268
[29223.541977] sp : ffff800084e43380
[29223.545278] x29: ffff800084e43380 x28: 000000000000001f x27: 0000000000000000
[29223.552398] x26: 0000000000000620 x25: 0000000000000000 x24: ffff8000813a96c0
[29223.559517] x23: 0000000000000000 x22: 0000000000000005 x21: 0000000000000070
[29223.566636] x20: ffff0008ca1eac80 x19: ffff000f5c39a8e0 x18: 0000000000000001
[29223.573754] x17: 0000000000000000 x16: 0000000000000001 x15: 0000000000000001
[29223.580874] x14: 0000000000000000 x13: 0000000000000003 x12: 000000000046d9e9
[29223.587992] x11: 0000000000000040 x10: ffff0008bf259378 x9 : 0000000000000000
[29223.595111] x8 : 000000003b13b13c x7 : 0000000000000001 x6 : 0000000000000006
[29223.602229] x5 : 0000000000040d23 x4 : 0000000000001abb x3 : 0000000000000000
[29223.609348] x2 : ffff80008129d000 x1 : ffff0008cbab6300 x0 : ffff0008c87c5800
[29223.616467] Call trace:
[29223.618901]  mod_objcg_state.constprop.0+0x1ec/0x268
[29223.623851]  __memcg_slab_post_alloc_hook+0x1a8/0x360
[29223.628888]  kmem_cache_alloc_noprof+0x1f0/0x214
[29223.633493]  alloc_buffer_head+0x20/0xd0
[29223.637405]  folio_alloc_buffers+0xa4/0x218
[29223.641575]  create_empty_buffers+0x28/0x170
[29223.645832]  ext4_block_write_begin+0x360/0x390
[29223.650350]  ext4_da_write_begin+0xc0/0x240
[29223.654520]  generic_perform_write+0xd0/0x25c
[29223.658865]  ext4_buffered_write_iter+0x6c/0x124
[29223.663471]  ext4_file_write_iter+0x54/0x720
[29223.667728]  __kernel_write_iter+0xfc/0x288
[29223.671900]  dump_user_range+0xd4/0x1dc
[29223.675723]  elf_core_dump+0xb64/0xd74
[29223.679461]  do_coredump+0x1084/0x1580
[29223.683196]  get_signal+0x7c8/0x8c8
[29223.686674]  do_signal+0x118/0xeb4
[29223.690067]  do_notify_resume+0xc8/0x140
[29223.693977]  el0_undef+0x84/0x98
[29223.697196]  el0t_64_sync_handler+0xd8/0x12c
[29223.701452]  el0t_64_sync+0x190/0x194
[29223.705103] Code: 97f92cbb f9400a80 d503201f f00080a2 (b99a7321)
[29223.711179] ---[ end trace 0000000000000000 ]---
[29223.715781] note: dds.ev.0[832] exited with irqs disabled
[29223.715781] Internal error: Oops: 0000000096000046 [#3] PREEMPT SMP
[29223.721209] ------------[ cut here ]------------
[29223.727409] Modules linked in: 8021q garp mrp stp llc ti_cpsw_proxy_client rpmsg_ctrl rpmsg_char wave5
[29223.732011] WARNING: CPU: 3 PID: 832 at /kernel/time/timer.c:1832 __run_timers+0x248/0x26c
[29223.741306]  videobuf2_dma_contig
[29223.749544] Modules linked in:
[29223.749545]  v4l2_mem2mem crct10dif_ce
[29223.752845]  8021q
[29223.755884]  omap_mailbox
[29223.759616]  garp
[29223.761615]  videobuf2_v4l2
[29223.764221]  mrp
[29223.766134]  omap_hwspinlock
[29223.768911]  stp
[29223.770736]  phy_can_transceiver
[29223.773600]  llc
[29223.775426]  videobuf2_memops
[29223.778637]  ti_cpsw_proxy_client
[29223.780463]  videobuf2_common
[29223.783415]  rpmsg_ctrl
[29223.786715]  ti_k3_r5_remoteproc
[29223.789667]  rpmsg_char
[29223.792098]  k3_j72xx_bandgap
[29223.795310]  wave5
[29223.797742]  ti_k3_dsp_remoteproc
[29223.800692]  videobuf2_dma_contig
[29223.802691]  videodev
[29223.805990]  v4l2_mem2mem
[29223.809289]  ti_k3_common
[29223.811547]  crct10dif_ce
[29223.814153]  pvrsrvkm(O)
[29223.816757]  omap_mailbox
[29223.819363]  sa2ul
[29223.821882]  videobuf2_v4l2
[29223.824487]  mc
[29223.826487]  omap_hwspinlock
[29223.829265]  rti_wdt
[29223.831003]  phy_can_transceiver
[29223.833868]  overlay
[29223.836040]  videobuf2_memops
[29223.839253]  cryptodev(O)
[29223.841424]  videobuf2_common
[29223.844376]  fuse
[29223.846981]  ti_k3_r5_remoteproc
[29223.849932]  drm
[29223.851843]  k3_j72xx_bandgap
[29223.855056]  drm_panel_orientation_quirks
[29223.856882]  ti_k3_dsp_remoteproc
[29223.859833]  backlight
[29223.863824]  videodev
[29223.867123]  ipv6
[29223.869468]  ti_k3_common
[29223.871727]
[29223.873640]  pvrsrvkm(O)
[29223.876247] CPU: 0 UID: 0 PID: 1 Comm: systemd Tainted: G      D    O       6.12.17-tda4vhmiddleclassda-00773-gcdcaeac783e3-dirty #1
[29223.877724]  sa2ul
[29223.880244] Tainted: [D]=DIE, [O]=OOT_MODULE
[29223.892122]  mc
[29223.894121] Hardware name: Texas Instruments J784S4 EVM (DT)
[29223.898373]  rti_wdt
[29223.900112] pstate: a00000c5 (NzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[29223.905750]  overlay
[29223.907922] pc : __queue_work+0x260/0x42c
[29223.914862]  cryptodev(O)
[29223.917034] lr : __queue_work+0x220/0x42c
[29223.921026]  fuse
[29223.923630] sp : ffff8000814aadb0
[29223.927622]  drm
[29223.929534] x29: ffff8000814aadb0
[29223.932833]  drm_panel_orientation_quirks
[29223.934658]  x28: ffff0008bf47d800
[29223.937957]  backlight
[29223.941949]  x27: 00000000ffffffff
[29223.945335]  ipv6
[29223.947680]
[29223.951065]
[29223.952977] x26: ffff000f5c370f18
[29223.954457] CPU: 3 UID: 0 PID: 832 Comm: dds.ev.0 Tainted: G      D    O       6.12.17-tda4vhmiddleclassda-00773-gcdcaeac783e3-dirty #1
[29223.955934]  x25: 0000000000000000
[29223.959234] Tainted: [D]=DIE, [O]=OOT_MODULE
[29223.971373]  x24: ffff0008bc2d3e68
[29223.974758] Hardware name: Texas Instruments J784S4 EVM (DT)
[29223.979010]
[29223.982395] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[29223.988034] x23: ffff0008bf47d848
[29223.989512] pc : __run_timers+0x248/0x26c
[29223.996450]  x22: ffff0008bc2d3e00
[29223.999749] lr : __run_timers+0x1a4/0x26c
[29224.003741]  x21: ffff000f5c35af18
[29224.007125] sp : ffff800081483e50
[29224.011118]
[29224.014502] x29: ffff800081483eb0
[29224.017801] x20: 0000000000000000
[29224.019279]  x28: 0000000000000100
[29224.022578]  x19: ffff0008bf47d840
[29224.025878]  x27: ffff800081483e60
[29224.029263]  x18: 0000000000000000
[29224.032648]
[29224.036033]
[29224.039418] x26: ffff8000812979c0
[29224.040896] x17: 0000000000000000
[29224.042374]  x25: ffff000f5c396470
[29224.045674]  x16: 0000000000000001
[29224.048972]  x24: ffff800081483e60
[29224.052357]  x15: 0000000000000000
[29224.055743]
[29224.059128]
[29224.062513] x23: 0000000000000000
[29224.063991] x14: 0000000000000800
[29224.065469]  x22: dead000000000122
[29224.068768]  x13: 0000000000000000
[29224.072067]  x21: 0000000000000000
[29224.075451]  x12: fffffdffed70c9c0
[29224.078836]
[29224.082221]
[29224.085606] x20: ffff000f5c396400
[29224.087084] x11: 0000000000000040
[29224.088561]  x19: ffff000f5c39a160
[29224.091860]  x10: 0000000000000000
[29224.095159]  x18: 0000000000000006
[29224.098543]  x9 : 0000000000000001
[29224.101928]
[29224.105313]
[29224.108698] x17: ffff800edb2c9000
[29224.110176] x8 : 0000000000000000
[29224.111654]  x16: ffff800081480000
[29224.114953]  x7 : 0000000000040000
[29224.118251]  x15: ffff800084e42a00
[29224.121636]  x6 : 0000000000000000
[29224.125021]
[29224.128406]
[29224.131791] x14: 0000000000000000
[29224.133269] x5 : ffff000f5c35c080
[29224.134747]  x13: 00000000000002d6
[29224.138046]  x4 : 0000000000000000
[29224.141345]  x12: 0000000000000001
[29224.144730]  x3 : 0000000000000005
[29224.148115]
[29224.151500]
[29224.154885] x11: 0000000000000000
[29224.156364] x2 : ffff000f5c35af40
[29224.157842]  x10: 0000000000000001
[29224.161141]  x1 : 0000000000000000
[29224.164439]  x9 : 0000000000000001
[29224.167824]  x0 : ffff0008bc2d3e00
[29224.171209]
[29224.174594]
[29224.177979] x8 : ffff800081483e68
[29224.179457] Call trace:
[29224.180935]  x7 : 000000000000007d
[29224.184234]  __queue_work+0x260/0x42c
[29224.186666]  x6 : 0000000000000000
[29224.190051]  __queue_delayed_work+0xcc/0x128
[29224.193696]
[29224.197082]  mod_delayed_work_on+0xc4/0xcc
[29224.201332] x5 : ffff000f5c396428
[29224.202811]  kblockd_mod_delayed_work_on+0x20/0x30
[29224.206889]  x4 : 0000000000000240
[29224.210188]  blk_mq_delay_run_hw_queue+0x98/0x15c
[29224.214961]  x3 : 0000000000000009
[29224.218346]  blk_mq_run_hw_queue+0x1c8/0x2c4
[29224.223032]
[29224.226417]  blk_mq_dispatch_rq_list+0x294/0x6f0
[29224.230668] x2 : 0000000000000000
[29224.232146]  __blk_mq_sched_dispatch_requests+0x45c/0x564
[29224.236745]  x1 : ffff800081483e60
[29224.240044]  blk_mq_sched_dispatch_requests+0x30/0x78
[29224.245423]  x0 : 0000000000000000
[29224.248808]  blk_mq_run_hw_queue+0x12c/0x2c4
[29224.253840]
[29224.257225]  blk_mq_flush_plug_list.part.0+0x194/0x4bc
[29224.261478] Call trace:
[29224.262956]  blk_mq_flush_plug_list+0x1c/0x2c
[29224.268075]  __run_timers+0x248/0x26c
[29224.270507]  __blk_flush_plug+0xe8/0x148
[29224.274846]  run_timer_softirq+0xcc/0xfc
[29224.278492]  blk_finish_plug+0x38/0x4c
[29224.282398]  handle_softirqs+0x104/0x248
[29224.286302]  ext4_do_writepages+0x340/0x9a0
[29224.290035]  __do_softirq+0x14/0x20
[29224.293941]  ext4_writepages+0x78/0xf8
[29224.298107]  ____do_softirq+0x10/0x1c
[29224.301579]  do_writepages+0x84/0x2b0
[29224.305311]  call_on_irq_stack+0x24/0x4c
[29224.308955]  filemap_fdatawrite_wbc+0x80/0xb4
[29224.312602]  do_softirq_own_stack+0x1c/0x28
[29224.316508]  __filemap_fdatawrite_range+0x64/0x98
[29224.320846]  irq_exit_rcu+0x8c/0xc4
[29224.325011]  file_write_and_wait_range+0x68/0xd4
[29224.329696]  el1_interrupt+0x38/0x68
[29224.333169]  ext4_sync_file+0x54/0x2f4
[29224.337768]  el1h_64_irq_handler+0x18/0x24
[29224.341326]  vfs_fsync_range+0x34/0x80
[29224.345058]  el1h_64_irq+0x64/0x68
[29224.349137]  ext4_buffered_write_iter+0xa4/0x124
[29224.352869]  make_task_dead+0x104/0x17c
[29224.356253]  ext4_file_write_iter+0x54/0x720
[29224.360852]  die+0x248/0x270
[29224.364670]  __kernel_write_iter+0xfc/0x288
[29224.368923]  die_kernel_fault+0x22c/0x4c8
[29224.371788]  dump_user_range+0xd4/0x1dc
[29224.375953]  fault_from_pkey+0x0/0x98
[29224.379945]  elf_core_dump+0xb64/0xd74
[29224.383764]  do_page_fault+0x1e8/0x4f8
[29224.387410]  do_coredump+0x1084/0x1580
[29224.391142]  do_translation_fault+0xa8/0xb8
[29224.394874]  get_signal+0x7c8/0x8c8
[29224.398607]  do_mem_abort+0x44/0x94
[29224.402772]  do_signal+0x118/0xeb4
[29224.406244]  el1_abort+0x40/0x64
[29224.409716]  do_notify_resume+0xc8/0x140
[29224.413101]  el1h_64_sync_handler+0xa4/0xe4
[29224.416313]  el0_ia+0xbc/0xf8
[29224.420217]  el1h_64_sync+0x64/0x68
[29224.424382]  el0t_64_sync_handler+0xcc/0x12c
[29224.427333]  mod_objcg_state.constprop.0+0x1ec/0x268
[29224.430806]  el0t_64_sync+0x190/0x194
[29224.435058]  __memcg_slab_post_alloc_hook+0x1a8/0x360
[29224.440006] Code: aa1603e0 f9401aa1 f9001ab7 a9008662 (f9000037)
[29224.443649]  kmem_cache_alloc_noprof+0x1f0/0x214
[29224.448682] ---[ end trace 0000000000000000 ]---
[29224.454755]  alloc_buffer_head+0x20/0xd0
[29224.459354] note: systemd[1] exited with irqs disabled
[29224.463952]  folio_alloc_buffers+0xa4/0x218
[29224.467887] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
[29224.472975]  create_empty_buffers+0x28/0x170
[29224.477141] Mem abort info:
[29224.485900]  ext4_block_write_begin+0x360/0x390
[29224.490151]   ESR = 0x0000000096000006
[29224.492930]  ext4_da_write_begin+0xc0/0x240
[29224.497442]   EC = 0x25: DABT (current EL), IL = 32 bits
[29224.501174]  generic_perform_write+0xd0/0x25c
[29224.505341]   SET = 0, FnV = 0
[29224.510633]  ext4_buffered_write_iter+0x6c/0x124
[29224.514971]   EA = 0, S1PTW = 0
[29224.518010]  ext4_file_write_iter+0x54/0x720
[29224.522607]   FSC = 0x06: level 2 translation fault
[29224.525733]  __kernel_write_iter+0xfc/0x288
[29224.529986] Data abort info:
[29224.534843]  dump_user_range+0xd4/0x1dc
[29224.539009]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[29224.541875]  elf_core_dump+0xb64/0xd74
[29224.545694]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[29224.551159]  do_coredump+0x1084/0x1580
[29224.554890]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[29224.559923]  get_signal+0x7c8/0x8c8
[29224.563654] user pgtable: 4k pages, 48-bit VAs, pgdp=000000093f6bf000
[29224.568945]  do_signal+0x118/0xeb4
[29224.572417] [0000000000000008] pgd=080000093f6e6003
[29224.578835]  do_notify_resume+0xc8/0x140
[29224.582221] , p4d=080000093f6e6003
[29224.587080]  el0_undef+0x84/0x98
[29224.590986] , pud=080000093f6e7003
[29224.594371]  el0t_64_sync_handler+0xd8/0x12c
[29224.597583] , pmd=0000000000000000
[29224.600967]  el0t_64_sync+0x190/0x194
[29224.605219]
[29224.608604] ---[ end trace 0000000000000000 ]---
[29224.612250] Internal error: Oops: 0000000096000006 [#4] PREEMPT SMP
[29224.613774] ------------[ cut here ]------------
[29224.618329] Modules linked in: 8021q
[29224.624577] Voluntary context switch within RCU read-side critical section!
[29224.629176]  garp
[29224.632745] WARNING: CPU: 3 PID: 832 at /kernel/rcu/tree_plugin.h:331 rcu_note_context_switch+0x3ac/0x404
[29224.639674]  mrp
[29224.641585] Modules linked in:
[29224.651124]  stp
[29224.652950]  8021q
[29224.655988]  llc
[29224.657814]  garp
[29224.659813]  ti_cpsw_proxy_client
[29224.661639]  mrp
[29224.663550]  rpmsg_ctrl
[29224.666849]  stp
[29224.668675]  rpmsg_char
[29224.671106]  llc
[29224.672931]  wave5
[29224.675363]  ti_cpsw_proxy_client
[29224.677187]  videobuf2_dma_contig
[29224.679186]  rpmsg_ctrl
[29224.682484]  v4l2_mem2mem
[29224.685783]  rpmsg_char
[29224.688214]  crct10dif_ce
[29224.690820]  wave5
[29224.693251]  omap_mailbox
[29224.695856]  videobuf2_dma_contig
[29224.697855]  videobuf2_v4l2
[29224.700460]  v4l2_mem2mem
[29224.703759]  omap_hwspinlock
[29224.706537]  crct10dif_ce
[29224.709142]  phy_can_transceiver
[29224.712008]  omap_mailbox
[29224.714613]  videobuf2_memops
[29224.717824]  videobuf2_v4l2
[29224.720430]  videobuf2_common
[29224.723381]  omap_hwspinlock
[29224.726159]  ti_k3_r5_remoteproc
[29224.729111]  phy_can_transceiver
[29224.731976]  k3_j72xx_bandgap
[29224.735188]  videobuf2_memops
[29224.738400]  ti_k3_dsp_remoteproc
[29224.741352]  videobuf2_common
[29224.744303]  videodev
[29224.747602]  ti_k3_r5_remoteproc
[29224.750554]  ti_k3_common
[29224.752813]  k3_j72xx_bandgap
[29224.756025]  pvrsrvkm(O)
[29224.758629]  ti_k3_dsp_remoteproc
[29224.761582]  sa2ul
[29224.764101]  videodev
[29224.767400]  mc
[29224.769398]  ti_k3_common
[29224.771658]  rti_wdt
[29224.773396]  pvrsrvkm(O)
[29224.776002]  overlay
[29224.778174]  sa2ul
[29224.780692]  cryptodev(O)
[29224.782865]  mc
[29224.784862]  fuse
[29224.787467]  rti_wdt
[29224.789206]  drm
[29224.791117]  overlay
[29224.793289]  drm_panel_orientation_quirks
[29224.795114]  cryptodev(O)
[29224.797287]  backlight
[29224.801279]  fuse
[29224.803884]  ipv6
[29224.806229]  drm
[29224.808141]
[29224.810052]  drm_panel_orientation_quirks
[29224.811879] CPU: 0 UID: 0 PID: 1 Comm: systemd Tainted: G      D W  O       6.12.17-tda4vhmiddleclassda-00773-gcdcaeac783e3-dirty #1
[29224.813356]  backlight
[29224.817350] Tainted: [D]=DIE, [W]=WARN, [O]=OOT_MODULE
[29224.829226]  ipv6
[29224.831572] Hardware name: Texas Instruments J784S4 EVM (DT)
[29224.836690]
[29224.838602] pstate: 200000c5 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[29224.844242] CPU: 3 UID: 0 PID: 832 Comm: dds.ev.0 Tainted: G      D W  O       6.12.17-tda4vhmiddleclassda-00773-gcdcaeac783e3-dirty #1
[29224.845719] pc : rcu_sched_clock_irq+0x304/0xee0
[29224.852659] Tainted: [D]=DIE, [W]=WARN, [O]=OOT_MODULE
[29224.864796] lr : rcu_sched_clock_irq+0x5b0/0xee0
[29224.869395] Hardware name: Texas Instruments J784S4 EVM (DT)
[29224.874513] sp : ffff800080003ab0
[29224.879112] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[29224.884749] x29: ffff800080003b90
[29224.888048] pc : rcu_note_context_switch+0x3ac/0x404
[29224.894987]  x28: 0000000000000000
[29224.898286] lr : rcu_note_context_switch+0x3ac/0x404
[29224.903232]  x27: ffff000f5c35c0c0
[29224.906616] sp : ffff800084e42e50
[29224.911561]
[29224.914947] x29: ffff800084e42e50
[29224.918245] x26: ffff800081320380
[29224.919724]  x28: ffffffffffffffff
[29224.923022]  x25: ffff800081299000
[29224.926321]  x27: ffff800080f8c000
[29224.929706]  x24: 00001a945f3001b8
[29224.933091]
[29224.936475]
[29224.939861] x26: ffff800080eb6078
[29224.941339] x23: ffff000f5c35c138
[29224.942817]  x25: 0000000000000000
[29224.946116]  x22: ffff800081297000
[29224.949415]  x24: ffff800080bbcb78
[29224.952799]  x21: 0000000000000000
[29224.956184]
[29224.959569]
[29224.962953] x23: 0000000000000000
[29224.964432] x20: 0000000000000000
[29224.965910]  x22: ffff0008cbab6300
[29224.969209]  x19: ffff000f5c353180
[29224.972507]  x21: ffff0008cbab6300
[29224.975892]  x18: 0000000000000006
[29224.979278]
[29224.982663]
[29224.986047] x20: ffff8000810d50c0
[29224.987525] x17: ffff800edb287000
[29224.989003]  x19: ffff000f5c39e0c0
[29224.992302]  x16: ffff800080000000
[29224.995600]  x18: 0000000000000006
[29224.998985]  x15: ffff8000814aa430
[29225.002371]
[29225.005755]
[29225.009140] x17: 3038356337386338
[29225.010618] x14: 0000000000000000
[29225.012096]  x16: 3030306666666620
[29225.015394]  x13: 306478302f303278
[29225.018692]  x15: ffff800084e42870
[29225.022077]  x12: 302b646165685f72
[29225.025462]
[29225.028847]
[29225.032233] x14: 0000000000000000
[29225.033711] x11: 0000000000000040
[29225.035189]  x13: 7131323038203a6e
[29225.038488]  x10: ffff0008bc009170
[29225.041787]  x12: 692064656b6e696c
[29225.045172]  x9 : ffff0008bc009168
[29225.048556]
[29225.051941]
[29225.055326] x11: 000000000000045e
[29225.056804] x8 : ffff800081320000
[29225.058283]  x10: 000000000000005d
[29225.061582]  x7 : ffff8000810d50c0
[29225.064880]  x9 : 0000000000096f59
[29225.068265]  x6 : ffff800edb287000
[29225.071650]
[29225.075035]
[29225.078421] x8 : 7f7f7f7f7f7f7f7f
[29225.079900] x5 : ffff800080003ab0
[29225.081378]  x7 : 205d393233383136
[29225.084676]  x4 : 0000000000000001
[29225.087975]  x6 : 362e34323239325b
[29225.091360]  x3 : ffff800080003ab0
[29225.094745]
[29225.098129]
[29225.101514] x5 : ffff000f5c393808
[29225.102992] x2 : 0000000000000000
[29225.104471]  x4 : 0000000000000000
[29225.107770]  x1 : 0000000000000000
[29225.111069]  x3 : 0000000000000027
[29225.114454]  x0 : 0000000000000000
[29225.117839]
[29225.121224]
[29225.124608] x2 : 0000000000000000
[29225.126087] Call trace:
[29225.127565]  x1 : 0000000000000000
[29225.130865]  rcu_sched_clock_irq+0x304/0xee0
[29225.133296]  x0 : ffff0008cbab6300
[29225.136683]  update_process_times+0x7c/0xcc
[29225.140934]
[29225.144319]  tick_nohz_handler+0xbc/0x150
[29225.148485] Call trace:
[29225.149963]  __hrtimer_run_queues+0x130/0x1b4
[29225.153956]  rcu_note_context_switch+0x3ac/0x404
[29225.156387]  hrtimer_interrupt+0xe8/0x244
[29225.160726]  __schedule+0x98/0xa84
[29225.165325]  arch_timer_handler_phys+0x2c/0x44
[29225.169317]  schedule+0x34/0x104
[29225.172702]  handle_percpu_devid_irq+0x80/0x120
[29225.177128]  do_exit+0xe4/0x8fc
[29225.180340]  handle_irq_desc+0x40/0x58
[29225.184852]  make_task_dead+0x84/0x17c
[29225.187977]  generic_handle_domain_irq+0x1c/0x28
[29225.191709]  die+0x248/0x270
[29225.195442]  gic_handle_irq+0x54/0xd0
[29225.200041]  die_kernel_fault+0x22c/0x4c8
[29225.202906]  do_interrupt_handler+0x50/0x8c
[29225.206552]  fault_from_pkey+0x0/0x98
[29225.210544]  el1_interrupt+0x34/0x68
[29225.214708]  do_page_fault+0x1e8/0x4f8
[29225.218354]  el1h_64_irq_handler+0x18/0x24
[29225.221912]  do_translation_fault+0xa8/0xb8
[29225.225644]  el1h_64_irq+0x64/0x68
[29225.229722]  do_mem_abort+0x44/0x94
[29225.233887]  handle_softirqs+0xa8/0x248
[29225.237272]  el1_abort+0x40/0x64
[29225.240744]  __do_softirq+0x14/0x20
[29225.244563]  el1h_64_sync_handler+0xa4/0xe4
[29225.247776]  ____do_softirq+0x10/0x1c
[29225.251247]  el1h_64_sync+0x64/0x68
[29225.255412]  call_on_irq_stack+0x24/0x4c
[29225.259058]  mod_objcg_state.constprop.0+0x1ec/0x268
[29225.262531]  do_softirq_own_stack+0x1c/0x28
[29225.266436]  __memcg_slab_post_alloc_hook+0x1a8/0x360
[29225.271381]  irq_exit_rcu+0x8c/0xc4
[29225.275547]  kmem_cache_alloc_noprof+0x1f0/0x214
[29225.280579]  el1_interrupt+0x38/0x68
[29225.284050]  alloc_buffer_head+0x20/0xd0
[29225.288649]  el1h_64_irq_handler+0x18/0x24
[29225.292208]  folio_alloc_buffers+0xa4/0x218
[29225.296114]  el1h_64_irq+0x64/0x68
[29225.300192]  create_empty_buffers+0x28/0x170
[29225.304357]  make_task_dead+0x104/0x17c
[29225.307743]  ext4_block_write_begin+0x360/0x390
[29225.311995]  die+0x248/0x270
[29225.315813]  ext4_da_write_begin+0xc0/0x240
[29225.320324]  die_kernel_fault+0x22c/0x4c8
[29225.323190]  generic_perform_write+0xd0/0x25c
[29225.327355]  fault_from_pkey+0x0/0x98
[29225.331346]  ext4_buffered_write_iter+0x6c/0x124
[29225.335684]  do_page_fault+0x1e8/0x4f8
[29225.339329]  ext4_file_write_iter+0x54/0x720
[29225.343928]  do_translation_fault+0xa8/0xb8
[29225.347660]  __kernel_write_iter+0xfc/0x288
[29225.351913]  do_mem_abort+0x44/0x94
[29225.356079]  dump_user_range+0xd4/0x1dc
[29225.360244]  el1_abort+0x40/0x64
[29225.363716]  elf_core_dump+0xb64/0xd74
[29225.367535]  el1h_64_sync_handler+0xa4/0xe4
[29225.370746]  do_coredump+0x1084/0x1580
[29225.374478]  el1h_64_sync+0x64/0x68
[29225.378644]  get_signal+0x7c8/0x8c8
[29225.382376]  __queue_work+0x260/0x42c
[29225.385849]  do_signal+0x118/0xeb4
[29225.389320]  __queue_delayed_work+0xcc/0x128
[29225.392965]  do_notify_resume+0xc8/0x140
[29225.396351]  mod_delayed_work_on+0xc4/0xcc
[29225.400602]  el0_undef+0x84/0x98
[29225.404506]  kblockd_mod_delayed_work_on+0x20/0x30
[29225.408585]  el0t_64_sync_handler+0xd8/0x12c
[29225.411797]  blk_mq_delay_run_hw_queue+0x98/0x15c
[29225.416569]  el0t_64_sync+0x190/0x194
[29225.420821]  blk_mq_run_hw_queue+0x1c8/0x2c4
[29225.425506] ---[ end trace 0000000000000000 ]---
[29225.429151]  blk_mq_dispatch_rq_list+0x294/0x6f0
[29225.433433] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000038
[29225.438003]  __blk_mq_sched_dispatch_requests+0x45c/0x564
[29225.442603] Mem abort info:
[29225.451362]  blk_mq_sched_dispatch_requests+0x30/0x78
[29225.456740]   ESR = 0x0000000096000006
[29225.459519]  blk_mq_run_hw_queue+0x12c/0x2c4
[29225.464552]   EC = 0x25: DABT (current EL), IL = 32 bits
[29225.468283]  blk_mq_flush_plug_list.part.0+0x194/0x4bc
[29225.472534]   SET = 0, FnV = 0
[29225.477826]  blk_mq_flush_plug_list+0x1c/0x2c
[29225.482944]   EA = 0, S1PTW = 0
[29225.485982]  __blk_flush_plug+0xe8/0x148
[29225.490321]   FSC = 0x06: level 2 translation fault
[29225.493447]  blk_finish_plug+0x38/0x4c
[29225.497352] Data abort info:
[29225.502210]  ext4_do_writepages+0x340/0x9a0
[29225.505943]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
[29225.508808]  ext4_writepages+0x78/0xf8
[29225.512973]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[29225.518438]  do_writepages+0x84/0x2b0
[29225.522171]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[29225.527204]  filemap_fdatawrite_wbc+0x80/0xb4
[29225.530849] user pgtable: 4k pages, 48-bit VAs, pgdp=0000000943485000
[29225.536141]  __filemap_fdatawrite_range+0x64/0x98
[29225.540480] [0000000000000038] pgd=0800000941d77003
[29225.546898]  file_write_and_wait_range+0x68/0xd4
[29225.551583] , p4d=0800000941d77003
[29225.556441]  ext4_sync_file+0x54/0x2f4
[29225.561040] , pud=0800000944634003
[29225.564426]  vfs_fsync_range+0x34/0x80
[29225.568159] , pmd=0000000000000000
[29225.571543]  ext4_buffered_write_iter+0xa4/0x124
[29225.575275]
[29225.578660]  ext4_file_write_iter+0x54/0x720
[29225.588989]  __kernel_write_iter+0xfc/0x288
[29225.593161]  dump_user_range+0xd4/0x1dc
[29225.596984]  elf_core_dump+0xb64/0xd74
[29225.600723]  do_coredump+0x1084/0x1580
[29225.604458]  get_signal+0x7c8/0x8c8
[29225.607935]  do_signal+0x118/0xeb4
[29225.611325]  do_notify_resume+0xc8/0x140
[29225.615235]  el0_ia+0xbc/0xf8
[29225.618192]  el0t_64_sync_handler+0xcc/0x12c
[29225.622447]  el0t_64_sync+0x190/0x194
[29225.626098] Code: f9400be7 34fff6f5 a94003e1 f86068e0 (f9400421)
[29225.632173] ---[ end trace 0000000000000000 ]---
[29225.636775] Kernel panic - not syncing: Oops: Fatal exception in interrupt
[29225.643631] SMP: stopping secondary CPUs
[29226.699540] SMP: failed to stop secondary CPUs 3,6
[29226.704326] Kernel Offset: disabled
[29226.707801] CPU features: 0x08,00002002,80200000,4200420b
[29226.713183] Memory Limit: none
[29226.716226] ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

Hi Hongyao,

The error coming up now is very different from the earlier error as now the errors are not related to memory running out.

Can you try storing the logs in a different boot media like SD card, as I don't think the storage media is the issue here. 

Regards

Gokul