• State
  • Locked Locked
  • Replies 4 replies
  • Subscribers 98 subscribers
  • Views 366 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

J784S4XEVM: Kernel Crash with Complex GStreamer Pipeline

Bartosz Tomczyk
Prodigy 120 points

Part Number: J784S4XEVM
Other Parts Discussed in Thread: SN65DSI86, PCM3168A

Tool/software:

We are encountering intermittent kernel crashes on our J784S4XEVM board, using the ti-edgeai-image with SDK 11.

The crash occurs when running a complex GStreamer pipeline that utilizes several TI-specific hardware-accelerated elements, including tiovxmultiscaler, tiovxldc, tiovxmosaic, v4l2h265enc, and v4l2h265dec.

The issue is not deterministic but occurs consistently, failing approximately 1 in every 10 pipeline runs. The result is a kernel panic related to a paging request, which requires a hard reboot to recover the system.

Below is the full kernel crash log for your analysis:

[437427.293830] Unable to handle kernel paging request at virtual address fffeffff80000000
[437427.301879] Mem abort info:
[437427.304837]   ESR = 0x0000000096000144
[437427.308759]   EC = 0x25: DABT (current EL), IL = 32 bits
[437427.314194]   SET = 0, FnV = 0
[437427.317355]   EA = 0, S1PTW = 0
[437427.320608]   FSC = 0x04: level 0 translation fault
[437427.325598] Data abort info:
[437427.328585]   ISV = 0, ISS = 0x00000144, ISS2 = 0x00000000
[437427.334193]   CM = 1, WnR = 1, TnD = 0, TagAccess = 0
[437427.339353]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[437427.344782] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000083145000
[437427.351602] [fffeffff80000000] pgd=0000000000000000, p4d=000000008353b003, pud=0000000000000000
[437427.360480] Internal error: Oops: 0000000096000144 [#1] PREEMPT SMP
[437427.366837] Modules linked in: xt_conntrack xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c xt_addrtype iptable_filter ip_tables x_tables br_netfilter bridge stp llc overlay bluetooth ecdh_generic ecc cfg80211 rfkill imx490 rpmsg_ctrl ti_am335x_adc rpmsg_char ds90ub953 panel_edp cdns3 kfifo_buf cdns_usb_common cdns_pltfrm j721e_csi2rx crct10dif_ce wave5 snd_soc_j721e_evm mux_gpio cdns_csi2rx display_connector phy_can_transceiver videobuf2_dma_contig v4l2_mem2mem ds90ub960 videobuf2_v4l2 omap_mailbox ti_k3_r5_remoteproc tidss v4l2_fwnode omap_hwspinlock videobuf2_memops at24 videobuf2_common v4l2_async m_can_platform drm_dma_helper ti_sn65dsi86 cdns_mhdp8546 cdns_dsi m_can pvrsrvkm(O) snd_soc_davinci_mcasp videodev drm_display_helper ti_k3_dsp_remoteproc can_dev ti_am335x_tscadc i2c_atr snd_soc_ti_udma snd_soc_pcm3168a_i2c k3_j72xx_bandgap ti_k3_common sa2ul drm_dp_aux_bus mc snd_soc_ti_edma snd_soc_pcm3168a ti_j721e_ufs cdns_dphy cdns_dphy_rx drm_kms_helper snd_soc_ti_sdma
[437427.367011]  cdns3_ti rti_wdt cryptodev(O) fuse drm drm_panel_orientation_quirks backlight ipv6
[437427.465162] CPU: 1 UID: 0 PID: 26908 Comm: queue_reference Tainted: G           O       6.12.17-ti-00773-gcdcaeac783e3-dirty #1
[437427.476704] Tainted: [O]=OOT_MODULE
[437427.480266] Hardware name: Texas Instruments J784S4 EVM (DT)
[437427.485995] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[437427.493026] pc : dcache_clean_poc+0x20/0x38
[437427.497294] lr : arch_sync_dma_for_device+0x24/0x30
[437427.502243] sp : ffff800089b5bcc0
[437427.505630] x29: ffff800089b5bcc0 x28: ffff00099971b180 x27: 0000000000000000
[437427.512837] x26: 0000000000000000 x25: 0000000000000000 x24: ffff0008bce58810
[437427.520043] x23: 0000000000000002 x22: 0000000000000001 x21: 0000000000000000
[437427.527248] x20: ffff00098b5de500 x19: 0000000000000000 x18: 0000000000000000
[437427.534453] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffff7f5ac4c0
[437427.541659] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[437427.548864] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000
[437427.556070] x8 : 0000000000000000 x7 : ffff0009ad3626c0 x6 : 0000000000000000
[437427.563275] x5 : ffff800089b5bd68 x4 : ffff800081566388 x3 : 000000000000003f
[437427.570480] x2 : 0000000000000040 x1 : fffeffff802f8000 x0 : fffeffff80000000
[437427.577687] Call trace:
[437427.580209]  dcache_clean_poc+0x20/0x38
[437427.584121]  dma_direct_sync_sg_for_device+0xf8/0x100
[437427.589244]  __dma_sync_sg_for_device+0x20/0x40
[437427.593851]  dma_heap_dma_buf_end_cpu_access+0x9c/0xcc
[437427.599062]  dma_buf_end_cpu_access+0x2c/0x44
[437427.603494]  dma_buf_ioctl+0x278/0x508
[437427.607316]  __arm64_sys_ioctl+0xac/0xf0
[437427.611315]  invoke_syscall+0x48/0x10c
[437427.615139]  el0_svc_common.constprop.0+0xc0/0xe0
[437427.619916]  do_el0_svc+0x1c/0x28
[437427.623305]  el0_svc+0x28/0x98
[437427.626438]  el0t_64_sync_handler+0x120/0x12c
[437427.630867]  el0t_64_sync+0x190/0x194
[437427.634607] Code: d2800082 9ac32042 d1000443 8a230000 (d50b7a20) 
[437427.640770] ---[ end trace 0000000000000000 ]---
|


Could you please look into this and help fix the issue?

  • 0
    TI__Guru*** 132401 points

    Hi,

    The TI assigned resource is currently out of office.  Please expect a possible delay in response.

    Regards,

    kb

  • 0 in reply to KB
    TI__Mastermind 27500 points

    Hi ,

    Can you send your GStreamer pipeline? Are you running any other applications when running the pipeline?

    Did you generate the EdgeAI image with Yocto or did you obtain it somewhere else?

    Best,
    Jared

  • 0 in reply to Jared McArthur
    Prodigy 120 points

      

    Yes, we build the EdgeAI image ourselves using Yocto:

    ./oe-layertool-setup.sh -f configs/processor-sdk-analytics/processor-sdk-analytics-11.00.00-config.txt

cd build
. conf/setenv

# Add meta-vay-camera layer
bitbake-layers add-layer $DIR/sources/meta-vay-camera/

echo 'ARAGO_BRAND = "edgeai"' >> conf/local.conf
echo 'MACHINE = "j784s4-evm"' >> conf/local.conf

bitbake -c cleanall tisdk-edgeai-image || true
bitbake -k tisdk-edgeai-image

    Regarding the GStreamer pipeline, that's where things get tricky. We don't have a simple gst-launch-1.0 command to share because our pipeline is built on the fly by our own C++ middleware. This middleware is the only application we run alongside the pipeline.

    Our application setups and change parameters while it runs. For instance, it setups the tiovxldc block size and LUT, change the tiovxmultiscaler scaling settings, and changes the v4l2h265enc bitrate in real-time. Because it's not a fixed pipeline, the issue pops up randomly, which makes it incredibly hard to pin down.

    We've been trying to create a simpler test case, but it's tough. The pipeline elements all depend on each other. For example, if we remove tiovxldc to see if it's the cause, the rest of the pipeline breaks because it expects a specific image size.If we swap out the hardware-accelerated tiovx/encoder/decoder plugins software ones, the problem goes away. But the performance drops so much that it could just hide the problem(it might be timing related)

    So, we're a bit stuck. Have you or your team ever seen anything like this with dynamic tiovx pipelines? Any ideas or suggestions on what we could try next would be a huge help.

  • 0 in reply to Bartosz Tomczyk
    TI__Mastermind 27500 points

    Hi ,

    I'm don't think this has been observed with dynamic GStreamer pipelines. The most similar issue I've seen is this:  PROCESSOR-SDK-AM68A: DMA memory problem on HS-SE device 

    Are you running these tests on the EVM or a custom board? If it's a custom board, is the RAM a different size?

    Are you able to sub out each of the hardware accelerated plugins one at a time to see if it's a specific plugin that breaks the pipeline?

    Are you able to see what pipeline is when the kernel panic occurs, or does it kernel panic when dynamically changing the pipeline?

    Best,
    Jared