This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM62A7-Q1: wave5: H.265 encoder: unreferenced kernel paging request while encoding an RTP stream

Part Number: AM62A7-Q1

Hi,

I'm currently running a kernel 6.12.35. wave5 code on tag 11.01.04. I see the following kernel error, resulting in the system hanging.

[  609.727744] Unable to handle kernel paging request at virtual address dead000000000108
[  609.735693] Mem abort info:
[  609.738490]   ESR = 0x0000000096000044
[  609.742241]   EC = 0x25: DABT (current EL), IL = 32 bits
[  609.747553]   SET = 0, FnV = 0
[  609.750608]   EA = 0, S1PTW = 0
[  609.753744]   FSC = 0x04: level 0 translation fault
[  609.758611] Data abort info:
[  609.761484]   ISV = 0, ISS = 0x00000044, ISS2 = 0x00000000
[  609.766958]   CM = 0, WnR = 1, TnD = 0, TagAccess = 0
[  609.771998]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[  609.777298] [dead000000000108] address between user and kernel address ranges
[  609.784419] Internal error: Oops: 0000000096000044 [#1] PREEMPT SMP
[  609.790675] Modules linked in: rpmsg_ctrl rpmsg_char tpm_tis_spi inv_icm42600_spi tpm_tis_core panel_semtech_gs2962 inv_icm42600 inv_sensors_timestamp kfifo_buf
 snd_soc_simple_card crct10dif_ce snd_soc_simple_card_utils snd_soc_adau7002 e5010_jpeg_enc v4l2_jpeg rtc_ti_k3 k3_j72xx_bandgap ti_k3_dsp_remoteproc ti_k3_r5_remo
teproc ti_k3_common j721e_csi2rx wave5 snd_soc_davinci_mcasp cdns_csi2rx snd_soc_ti_udma videobuf2_dma_contig mcrc64 v4l2_mem2mem sa2ul videobuf2_v4l2 ar0823 snd_s
oc_ti_edma v4l2_fwnode videobuf2_memops omap_hwspinlock v4l2_async videobuf2_common snd_soc_ti_sdma videodev mc omap_mailbox at24 ina238 tps65219_pwrbutton spi_oma
p2_mcspi cfg80211 rfkill fuse ipv6
[  609.850492] CPU: 1 UID: 0 PID: 25 Comm: kworker/1:0 Not tainted 6.12.35-g16f5a07f-01131-g2a4b06c4e29a #1
[  609.859958] Hardware name: Benchmark Electronics 4K Camera Proto B (DT)
[  609.866558] Workqueue: events v4l2_m2m_device_run_work [v4l2_mem2mem]
[  609.873013] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  609.879961] pc : v4l2_m2m_buf_remove_by_idx+0x84/0xe8 [v4l2_mem2mem]
[  609.886313] lr : v4l2_m2m_buf_remove_by_idx+0x28/0xe8 [v4l2_mem2mem]
[  609.892663] sp : ffff800081a4bbd0
[  609.895968] x29: ffff800081a4bbd0 x28: 0000000000000000 x27: 000000000007f800
[  609.903096] x26: 00000000aefd2800 x25: 0000000000000780 x24: ffff0000014efdc8
[  609.910224] x23: ffff0000014efc28 x22: ffff0000014efc00 x21: ffff0000014eff60
[  609.917351] x20: ffff0000014efdc8 x19: ffff000001daf800 x18: 0000000000000000
[  609.924478] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000002
[  609.931605] x14: 0000000000000800 x13: 00000000000001e6 x12: 0000000000000000
[  609.938732] x11: 0000000000000000 x10: 00000000000009d0 x9 : ffff800081a4bcc0
[  609.945859] x8 : ffff800081a4bc88 x7 : 0000000000000000 x6 : ffff0000014eff50
[  609.952986] x5 : dead000000000100 x4 : dead000000000100 x3 : dead000000000122
[  609.960113] x2 : 0000000000000100 x1 : 0000000000000000 x0 : ffff000001dafc00
[  609.967242] Call trace:
[  609.969678]  v4l2_m2m_buf_remove_by_idx+0x84/0xe8 [v4l2_mem2mem]
[  609.975685]  start_encode+0x28c/0x554 [wave5]
[  609.980063]  wave5_vpu_enc_device_run+0x10c/0x230 [wave5]
[  609.985466]  v4l2_m2m_try_run+0x84/0x140 [v4l2_mem2mem]
[  609.990692]  v4l2_m2m_device_run_work+0x14/0x20 [v4l2_mem2mem]
[  609.996522]  process_one_work+0x148/0x28c
[  610.000532]  worker_thread+0x2d0/0x3d8
[  610.004277]  kthread+0x110/0x114
[  610.007500]  ret_from_fork+0x10/0x20
[  610.011074] Code: 91100060 aa0303f3 d2802002 f9424863 (f9000483)
[  610.017154] ---[ end trace 0000000000000000 ]---
[  610.021760] note: kworker/1:0[25] exited with irqs disabled
[  610.027441] note: kworker/1:0[25] exited with preempt_count 1
[  612.029229] ti-sci 44043000.system-controller: Mbox timedout in resp(caller: ti_sci_cmd_get_device_exclusive+0x18/0x24)
[  612.040102] ti-sci 44043000.system-controller: Mbox send fail -110

Any idea how to fix this?

Regards,

Bas Vermeulen

  • Hi Bas, 

    Can you provide us a way to reproduce the issue on our EVM? Also, does this package have the updated Gstreamer version that you are testing with(1.24.13)?

    Best Regards

    Suren

  • Hi Suren,

    Yes. this is with the updated gstreamer 1.24.13 running on there, and the modified VPAC/TIDL firmware. I currently don't have a way to reproduce it on the EVM, I just saw it in a longer test on our custom hardware. I'll see if I can find a way to reproduce on EVM, but am not sure when I'll have enough time to do so.

    We are encoding two streams, one 4K stream and a 1920x1088 stream, at 30 frames per second. The camera is pointed at the ceiling, so nothing much happening there.

    I'll try to find some time to reproduce on EVM once my SDK update is done.

    Regards,

    Bas Vermeulen

  • Thanks Bas.

    I will wait for your inputs on reproducing the issue, In the meantime, also look at the below commits and see if its already part of the SDK you are validating with or needs to be pulled into your setup.

    https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/drivers/media/platform/chips-media/wave5?h=ti-linux-6.12.y&id=462e6c68194dc4696ebc3f73754013641903c642

    https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/drivers/media/platform/chips-media/wave5?h=ti-linux-6.12.y&id=40d7adcd233be37e8db917a9898144d5fe50930f

    Best Regards

    Suren

  • They weren't part of my SDK yet, but I integrated all patches for the wave5. The problem still persists.

    I'll see if I can find a way to reproduce on the evm tomorrow.

    Regards,

    Bas Vermeulen

  • Hi Bas, 

    Until you figure out to reproduce, couple of questions.

    So you just keep it running  two encodes - one 4K stream and a 1920x1088 stream, at 30 frames per second, assuming they are streamed to network and long running test? Are you trying to do any other stress tests in the background or power cycle tests to reproduce the issue?

    Best Regards

    Suren

  • The application is two streams - one 4K and one 1920x1088 running through multiple multiscalers to scale things down. We also have two custom plugins running (a custom text overlay that works directly on dmabuffers, and a custom plugin that signs the h26x output stream with the media signing framework) in both pipelines. Next to those primary pipelines we also have one branch running motioncells, and a branch where we can run tidlinferer if it is necessary (configured). Everything is running with dmabuf-import as much as possible (including the v4l2h265encs)

    The whole pipeline runs at 30 fps, except when the inference is active, then we drop down to 27-28 fps.

  •  ,

    I think I know what happens, just not sure why it happens. Is there any chance of being able to discuss this with an engineer with detailed knowledge of the wave5 driver?

    In drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c in the function start_encode, once wave5_vpu_enc_start_one_frame returns success, we call v4l2_m2m_src_buf_remove_by_idx() on the src_buf in there. This goes through the m2m rdy_queue to find the buffer in question and then removes it from the rdy_queue with list_del. The problem occurs when trying to remove the entry from the rdy_queue, because when we remove it, the next member of list_head is already NULL (or at least poisoned).

    The rdy_queue is protected by a spinlock, taken with irqsave, so I don't understand why this changes sometimes.

    Regards,

    Bas Vermeulen

  • Hi Bas,

    My colleague Brandon has shared a patch for you to verify via email. Please have that validated and update us on the results, If it works and fixes the problem, then we might need to upstream the patch.

    Best Regards

    Suren

  • Hi Suren,

    The patch is currently in test. Previously the crash occurred in about 2-3 hours, so I should be able to give an indication around end of day my time. I'll keep the setup running for the weekend after that.

    Regards,

    Bas Vermeulen

  • Thanks Bas. 

    Let us know how it goes with your weekend run. 

    Best Regards

    Suren

  • The latest patch from Brandon fixes the race condition we found. I've been running a slightly modified (older) version of the patch for 3 days without any occurrences of the problem.

    From 66dbdc5b5d2dd37a405bdd9111a780ece399550c Mon Sep 17 00:00:00 2001
    From: Brandon Brnich <b-brnich@ti.com>
    Date: Wed, 11 Mar 2026 12:05:18 -0500
    Subject: [PATCH] media: chips-media: wave5: Move src_buf Removal to
     finish_encode
    
    During encoder processing, there is a case where the IRQ response could
    return the buffer back to userspace via v4l2_m2m_buf_done call. In this
    time, userspace could queue up this same buffer before start_encode removes
    the index from the ready queue. This would then lead to a case where the
    buffer in the ready queue could be a self loop due to the
    WRITE_ONCE(prev->next, new) call in __list_add.
    
    When __list_del is finally called, the loop is already made so nothing
    points back to ready queue list head and pointers are poisoned.
    
    A buffer should not be marked as DONE before the buffer is removed from
    m2m ready queue. Move removal entirely to finish_encode.
    
    Signed-off-by: Brandon Brnich <b-brnich@ti.com>
    ---
    
    This bug is very hard to reproduce in simple encode environments. It
    primarily occurs during long run cases where CPU is strained doing other
    forms of computation. Crash log shared below.
    
    I see other drivers removing buffer in their device_run process, but
    they do it before any chance of DONE state transition. Wave5 should do
    the same.
    
    [ 609.879961] pc : v4l2_m2m_buf_remove_by_idx+0x84/0xe8 [v4l2_mem2mem]
    [ 609.886313] lr : v4l2_m2m_buf_remove_by_idx+0x28/0xe8 [v4l2_mem2mem]
    [ 609.892663] sp : ffff800081a4bbd0
    [ 609.895968] x29: ffff800081a4bbd0 x28: 0000000000000000 x27: 000000000007f800
    [ 609.903096] x26: 00000000aefd2800 x25: 0000000000000780 x24: ffff0000014efdc8
    [ 609.910224] x23: ffff0000014efc28 x22: ffff0000014efc00 x21: ffff0000014eff60
    [ 609.917351] x20: ffff0000014efdc8 x19: ffff000001daf800 x18: 0000000000000000
    [ 609.924478] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000002
    [ 609.931605] x14: 0000000000000800 x13: 00000000000001e6 x12: 0000000000000000
    [ 609.938732] x11: 0000000000000000 x10: 00000000000009d0 x9 : ffff800081a4bcc0
    [ 609.945859] x8 : ffff800081a4bc88 x7 : 0000000000000000 x6 : ffff0000014eff50
    [ 609.952986] x5 : dead000000000100 x4 : dead000000000100 x3 : dead000000000122
    [ 609.960113] x2 : 0000000000000100 x1 : 0000000000000000 x0 : ffff000001dafc00
    [ 609.967242] Call trace:
    [ 609.969678] v4l2_m2m_buf_remove_by_idx+0x84/0xe8 [v4l2_mem2mem]
    [ 609.975685] start_encode+0x28c/0x554 [wave5]
    [ 609.980063] wave5_vpu_enc_device_run+0x10c/0x230 [wave5]
    [ 609.985466] v4l2_m2m_try_run+0x84/0x140 [v4l2_mem2mem]
    [ 609.990692] v4l2_m2m_device_run_work+0x14/0x20 [v4l2_mem2mem]
    [ 609.996522] process_one_work+0x148/0x28c
    [ 610.000532] worker_thread+0x2d0/0x3d8
    [ 610.004277] kthread+0x110/0x114
    [ 610.007500] ret_from_fork+0x10/0x20
    
     .../chips-media/wave5/wave5-vpu-enc.c         | 32 +++----------------
     1 file changed, 5 insertions(+), 27 deletions(-)
    
    diff --git a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
    index 7613fcdbafed..3e198a7cefb1 100644
    --- a/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
    +++ b/drivers/media/platform/chips-media/wave5/wave5-vpu-enc.c
    @@ -223,17 +223,9 @@ static int start_encode(struct vpu_instance *inst, u32 *fail_res)
     		dst_buf->vb2_buf.timestamp = src_buf->vb2_buf.timestamp;
     		v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_ERROR);
     		v4l2_m2m_buf_done(dst_buf, VB2_BUF_STATE_ERROR);
    -	} else {
    +	} else
     		dev_dbg(inst->dev->dev, "%s: wave5_vpu_enc_start_one_frame success\n",
     			__func__);
    -		/*
    -		 * Remove the source buffer from the ready-queue now and finish
    -		 * it in the videobuf2 framework once the index is returned by the
    -		 * firmware in finish_encode
    -		 */
    -		if (src_buf)
    -			v4l2_m2m_src_buf_remove_by_idx(m2m_ctx, src_buf->vb2_buf.index);
    -	}
     
     	return 0;
     }
    @@ -259,27 +251,13 @@ static void wave5_vpu_enc_finish_encode(struct vpu_instance *inst)
     		__func__,  enc_output_info.pic_type, enc_output_info.recon_frame_index,
     		enc_output_info.enc_src_idx, enc_output_info.enc_pic_byte, enc_output_info.pts);
     
    -	/*
    -	 * The source buffer will not be found in the ready-queue as it has been
    -	 * dropped after sending of the encode firmware command, locate it in
    -	 * the videobuf2 queue directly
    -	 */
     	if (enc_output_info.enc_src_idx >= 0) {
    -		struct vb2_buffer *vb = vb2_get_buffer(v4l2_m2m_get_src_vq(m2m_ctx),
    -						       enc_output_info.enc_src_idx);
    -		if (vb->state != VB2_BUF_STATE_ACTIVE)
    -			dev_warn(inst->dev->dev,
    -				 "%s: encoded buffer (%d) was not in ready queue %i.",
    -				 __func__, enc_output_info.enc_src_idx, vb->state);
    -		else
    -			src_buf = to_vb2_v4l2_buffer(vb);
    -
    -		if (src_buf) {
    +		src_buf = v4l2_m2m_src_buf_remove(m2m_ctx);
    +		if (!src_buf)
    +			dev_warn(inst->dev->dev, "%s: no source buffer found\n", __func__);
    +		else {
     			inst->timestamp = src_buf->vb2_buf.timestamp;
     			v4l2_m2m_buf_done(src_buf, VB2_BUF_STATE_DONE);
    -		} else {
    -			dev_warn(inst->dev->dev, "%s: no source buffer with index: %d found\n",
    -				 __func__, enc_output_info.enc_src_idx);
     		}
     	}
     
    -- 
    2.43.0
    
    

    With this patch my problem is fixed. Many thanks for the support Brandon and  !

    Regards,

    Bas Vermeulen

  • Hi Bas,

    Since the patch fixed the problem. I am going to close this thread for now. Feel free to open newer threads for any further assistance.

    Best Regards

    Suren