TDA4VH-Q1: Kernel panic - not syncing: Asynchronous SError Interrupt - Wave5 driver

Hi, 

What HLOS SDK version are you using? Is there a particular use-case that you are seeing this happen with? I have not noticed this kernel panic when starting a decoder pipeline yet.

Thanks,
Sarabesh S.

Hi,

I am currently on the 6.15.8 linux kernel.  The panic occurs when I start a single gstreamer decode pipeline with the v4l2h264dec plugin.  The panic seems to always occur in  wave5_dec_clr_disp_flag().  I have been decoding a live h264 video stream.  Today, I sourced video from an MP4 file and the panic occurred at the same point. The panic occurs suspiciously close to wave5_vpu_dec_start_streaming(), but I am not sure how to isolate the source of the async error.

Thanks,

Jeff

Could you share the gstreamer pipeline?

Thanks,
Sarabesh S.

Hi Sarabesh,

Below is a pipeline which induces the Serror every time.  The stack trace always indicates the issue is encountered when wave5_dec_clr_disp_flag() is executing.  I have built a debug kernel and hit the breakpoint in wave5_dec_clr_disp_flag() using KGDB.  As I step into the code, the SError is triggered immediately.  I wanted to read the CFSR register at 0xE000ED28, but the address was not valid (I think because there are multiple cores).  Not sure of the next steps for diagnosing the cause of the fault...

Note $1 is just the path to an MP4 file..

/opt/GStreamer/bin/gst-launch-1.0 \
filesrc location=${1} \
! qtdemux name=demux demux.video_0 \
! h264parse config-interval=0 \
! v4l2h264dec capture-io-mode=4 \
! videoconvert \
! videorate ! video/x-raw,max-rate=30/1 \
! queue \
! rtpvrawpay \
! 'application/x-rtp, media=(string)video, encoding-name=(string)RAW' \
! udpsink host=127.0.0.1 port=5000 sync=false async=false

Thanks again,
Jeff

Hi Jeff, 

Thanks for the information. I'll review this and get back to you. 

Regards,
Sarabesh S.

Thanks Sarabesh. 

I did a bit more digging.  I see that the ESR is 0x00000000bf000000.  This decodes to a SError with IDS==1.  The Arm A-profile Architecture Reference Manual indicates that in this case, ESR_EL1[23:0] are IMPLEMENTATION DEFINED.  I am not sure where to find the implementation specific definition for ISS==24'b0.  Please let me know what you find.  I am stuck.

Thanks, Jeff.

Thanks Jeff, 

Currently discussing with the team. I'll follow up soon.

Regards,
Sarabesh S.

Hi Sarabesh,

Do you have any updates following discussion with the team? This is a high priority issue and your guidance is much appreciated.

Best,

Luke

Hi Luke, 

Have you been able to reproduce this on a TI EVM at all?

Hello,

Could you please share the input stream. I will try to replicate this on my setup with the latest SDK. 

Thank you,
Sarabesh S.

Hi.  Attached is the source video used in the above pipeline.  Note that the pipeline was working with the v6.12.17 linux kernel.  When we stepped forward to the v6.15.8 kernel, we started seeing the issue.  The wave5 codec driver requires a 70MB allocation of CMA memory.  The CONFIG_ARCH_FORCE_MAX_ORDER kernel parameter was set to 15 (assuming a 4k page size) in order to support this allocation.

Thanks,
Jeff

Hi Jeffrey, 

Thanks for the stream. A few clarifying questions below:

1. Did you already set the CMA allocation w/ CONFIG_ARCH_FORCE_MAX_ORDER=15 in the 6.12 kernel or is it new? What was the default?
2. Can you confirm what CMA is being passed in u-boot CMA parameter? 
3. Does the kernel crash happen if you change the CMA in the u-boot cmdline as shown here:
   https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-j784s4/11_01_00_03/exports/docs/linux/How_to_Guides/Target/How_To_Carve_Out_CMA.html#how-to-configure-the-cma-size

Thank you,
Sarabesh S.

Hi Sarabesh,

I have been working with Jeffrey on this issue, and we have a lot more detail now.

First, the CMA configuration and CONFIG_ARCH_FORCE_MAX_ORDER=15 was required on both the upstream 6.12.17 kernel as well as the 6.16.9 kernel (or the 6.15.7 kernel, both behave the same).  Without this, the pipeline will not even start.  So, this is simply a requirement for now for us to make this start at all.

Second, I was able to set up a minimal reproducer based on the gstreamer command line given above and the Big Buck Bunny video Jeff attached previously.  I was able to reproduce the crash on the TI J784S4XEVM board, so I am presently doing my testing on it.

I bisected the issue with the mainline kernel versions on the J784S4XEVM board from 6.12.17 to 6.16.9 and found the commit which caused the issue to be:

https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit?id=2092b3833487e5ce138f4303f98e46ba0f87f1d0

Note that this went into the 6.13 kernel series, but is currently being backported by TI's SDK version 11 into TI's 6.12 vendor kernel.

I also learned that if I simply revert this commit in the 6.16.9 mainline kernel the problem goes away.

I found this patch series was just posted by the codec IP block vendor:

https://lore.kernel.org/linux-media/20250922055255.116-1-jackson.lee@chipsnmedia.com/

I tried applying those patches, and the problem continued to occur.  So, at this point I believe this to be an issue that still exists upstream.

In order to make this crash happen, we have learned that having debugging turned up when playing the big buck bunny video causes the crash to occur reliably.  This is not required in our production-intent use of this, which is interesting.  Based on what is causing this (the codec suspend support being added), I assume this might just be related to the workload being pushed to the codec.

Here is the command I am using to reproduce this issue:

export GST_DEBUG=6
gst-launch-1.0 \
    filesrc location=Big_Buck_Bunny_720_10s_30MB.mp4 \
    ! qtdemux name=demux demux.video_0 \
    ! h264parse config-interval=0 \
    ! v4l2h264dec capture-io-mode=4 output-io-mode=4 \
    ! videoconvert \
    ! videorate ! video/x-raw,max-rate=30/1 \
    ! queue \
    ! rtpvrawpay \
    ! 'application/x-rtp, media=(string)video, encoding-name=(string)RAW' \
    ! udpsink host=127.0.0.1 port=5000 sync=false async=false

I plan to attempt this test with the latest tagged TI vendor kernel to see if it is happening there, but my guess based on what we are seeing is that it is.

I will add an update to this ticket when I have those details.

Hello, 

Glad you were able to find the fix in the 11.01.14 tag. I agree that this is related to the runtime pm support commit . The resolution is likely due to increasing the autosuspend_delay so the VPU is not powered down while still having frames in use. I noticed the CNM patch series you shared is setting the delay to 500ms but in our tree you can see it is set to 5000ms (https://git.ti.com/cgit/ti-linux-kernel/ti-linux-kernel/commit/drivers/media/platform/chips-media/wave5/wave5-vpu.c?h=11.01.14&id=b4249050fcdb302f50ca047f4606dc8025cf27bb) GST_DEBUG=6 probably slowed things down enough for auto-suspend to fire mid-operation which caused the crash.

Regards,
Sarabesh S.

Glad to hear . Let us know if you have any further questions.

Regards,
Sarabesh S.