AM62A7: CMA usage is increasing and kernel crash is seen when using video encoder in long run

Vinay S

Part Number: AM62A7

Hi,

We are testing 4k@60fps h265 video encoding in am62a7 EVK board with below gstreamer pipeline

gst-launch-1.0 -v videotestsrc num-buffers=300 ! video/x-raw, width=3480, height=2160, framerate=60/1 ! v4l2h265enc ! filesink location=test.mp4

but after the pipeline is completed CMA memory reduces slowly (approx 20MB each time)

if the same pipeline is run continuously with sleep of 5seconds in between after some time below error is shown and kernel crash is seen.

alloc_contig_range: [fb900, fc02c) PFNs busy

cma: cma_alloc: alloc failed, req-size: 1836 pages, ret: -12

vdec 30210000.video-codec: dma_alloc_coherent of size 7520256 failed

below is while loop used for continuous video encoding

while [ 1 ]
do
gst-launch-1.0 -v videotestsrc num-buffers=300 ! video/x-raw, width=3480, height=2160, framerate=60/1 ! v4l2h265enc ! filesink location=test.mp4; sleep 5;
done

please find the attached log when pipeline is run with GST_DEBUG=2

gst-debug.log

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
gst-launch-1.0 -v videotestsrc num-buffers=300 ! video/x-raw, width=3480, height=2160, framerate=60/1 ! v4l2h265enc capture-io-mode=dmabuf ! fillesink location=test.mp4
Setting pipeline to PAUSED ...
0:00:00.133124160  1789     0x36240270 WARN                    v4l2 gstv4l2object.c:4344:gst_v4l2_object_probe_caps:<v4l2h265enc0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
Pipeline is P[  995.993533] alloc_contig_range: 21 callbacks suppressed
REROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotests[  995.993542] alloc_contig_range: [dc800, dc8eb) PFNs busy
rc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=([  996.011812] alloc_contig_range: [dc900, dc9eb) PFNs busy
int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/v4l2h265enc:v4l2h265enc0.GstPad:src: caps = video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)main, level=(string)1, width=(int)3480, height=(int)2160, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-mode=(string)progressive, colorimetry=(string)bt601, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)main, level=(string)1, width=(int)3480, height=(int)2160, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-mode=(string)progressive, colorimetry=(string)bt601, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
Redistribute latency...
/GstPipeline:pipeline0/v4l2h265enc:v4l2h265enc0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
0:00:00.217001085  1789     0x3624e770 WARN          v4l2bufferpool gstv4l2bufferpool.c:820:gst_v4l2_buffer_pool_start:<v4l2h265enc0:pool:sink> Uncertain or not enough buffers, enabling copy threshold
0:00:00.406862185  1789     0x3624e770 WARN          v4l2bufferpool gstv4l2bufferpool.c:820:gst_v4l2_buffer_pool_start:<v4l2h265enc0:pool:src> Uncertain or not enough buffers, enabling copy threshold
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

gst-launch-1.0 -v videotestsrc num-buffers=300 ! video/x-raw, width=3480, height=2160, framerate=60/1 ! v4l2h265enc capture-io-mode=dmabuf ! fillesink location=test.mp4
Setting pipeline to PAUSED ...
0:00:00.133124160  1789     0x36240270 WARN                    v4l2 gstv4l2object.c:4344:gst_v4l2_object_probe_caps:<v4l2h265enc0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
Pipeline is P[  995.993533] alloc_contig_range: 21 callbacks suppressed
REROLLING ...
/GstPipeline:pipeline0/GstVideoTestSrc:videotests[  995.993542] alloc_contig_range: [dc800, dc8eb) PFNs busy
rc0.GstPad:src: caps = video/x-raw, format=(string)I420, width=([  996.011812] alloc_contig_range: [dc900, dc9eb) PFNs busy
int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/v4l2h265enc:v4l2h265enc0.GstPad:src: caps = video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)main, level=(string)1, width=(int)3480, height=(int)2160, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-mode=(string)progressive, colorimetry=(string)bt601, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = video/x-h265, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)main, level=(string)1, width=(int)3480, height=(int)2160, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1, interlace-mode=(string)progressive, colorimetry=(string)bt601, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono
Redistribute latency...
/GstPipeline:pipeline0/v4l2h265enc:v4l2h265enc0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)3480, height=(int)2160, framerate=(fraction)60/1, multiview-mode=(string)mono, interlace-mode=(string)progressive, colorimetry=(string)bt601, pixel-aspect-ratio=(fraction)1/1
0:00:00.217001085  1789     0x3624e770 WARN          v4l2bufferpool gstv4l2bufferpool.c:820:gst_v4l2_buffer_pool_start:<v4l2h265enc0:pool:sink> Uncertain or not enough buffers, enabling copy threshold
0:00:00.406862185  1789     0x3624e770 WARN          v4l2bufferpool gstv4l2bufferpool.c:820:gst_v4l2_buffer_pool_start:<v4l2h265enc0:pool:src> Uncertain or not enough buffers, enabling copy threshold
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:35.212552355
Setting pipeline to NULL ...
0:00:35.658976810  1789     0x36240270 WARN              bufferpool gstbufferpool.c:1408:gst_buffer_pool_set_flushing:<v4l2h265enc0:pool:sink> can't change flushing state of inactive pool
0:00:35.767793230  1789     0x36240270 WARN              bufferpool gstbufferpool.c:1408:gst_buffer_pool_set_flushing:<v4l2h265enc0:pool:sink> can't change flushing state of inactive pool
Freeing pipeline ...

similar observation is seen during h264 encoding also with v4l2h264enc

We are using Linux SDK version 08.06.00.45 and no additional SW changes are done to BSP.

Is this behavior expected?

over 2 years ago

0 Suren Porwar over 2 years ago

TI__Mastermind 23235 points

Hi Vinay,

Can you increase the CMA size at boot up and try and see if you are able to solve the issue:

To change CMA size stop at u-boot prompt during bootup and update cma as below and then boot :

target# setenv args_all $args_all cma=1000M
target# boot

Let me know if this helps.

Best Regards,
Suren

0 Vinay S over 2 years ago in reply to Suren Porwar

Intellectual 365 points

Hi Suren,

I have tried increasing CMA but the issue will be delayed but after sometime same error will occur once CMA gets decreased.

Each time the encoder pipeline runs CMA used is not getting freed properly and CMA reduces by 20MB so the issue occurs

0 Suren Porwar over 2 years ago in reply to Vinay S

TI__Mastermind 23235 points

Hi Vinay,

Do you see same behavior with 4K @ 30fps?

According to TRM:

Best Regards,

Suren

0 Vinay S over 2 years ago in reply to Suren Porwar

Intellectual 365 points

Hi Suren,

Yes same behavior is seen when continuously encoding at 4k@30fps also

Please find the attached log explaining the issue.

video-encoding-long-run-evk.zip

Here after each time pipeline is run free CMA memory is reducing and at 1 point of time it crashes the kernel when it fails to acquire required CMA

0 Suren Porwar over 2 years ago in reply to Vinay S

TI__Mastermind 23235 points

Hi Vinay,

I would wait for the SDK 9.0 release to verify if this is fixed. This release is planned to release in the next two weeks.

Best Regards,

Suren

Processors

Processors forum

AM62A7: CMA usage is increasing and kernel crash is seen when using video encoder in long run