Linux/PROCESSOR-SDK-AM57X: Big latency of H.264 GStreamer pipeline

Ondrej Bilka

Tool/software: Linux

I am using latest prebuild sdk(am57xx-evm-linux-sdk-bin-04.02.00.09.tar.xz) on evaluation board. We have problem that hardware h264 decoder causes unacceptably big latency. Sofware decoder displays same stream with latency under 100ms(but 80% cpu usage). Hardware accelerated video has delay of around three second if we don't use latency parameter in rtsp source (with or without jitterbuffer)

If we use latency parameter it needs 400ms for camera resolution 1024x768 to work properly. But smaller resolutions are worse, decoding 320x200 or 640x480 or 800x600 take minimum of 600ms latency.

With smaller latency gstreamer renders only one frame per second and displays following warning

WARNING: from element /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: A lot of buffers are being dropped.
Additional debug info:
../../../../gstreamer-1.8.3/libs/gst/base/gstbasesink.c(2854): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstWaylandSink:waylandsink0:
There may be a timestamping problem, or this computer is too slow.

Pipeline for software decoding that works properly is following:

CX=1024
CY=768
X=1024
Y=768
LATENCY=100
gst-launch-1.0 rtspsrc latency=$LATENCY location="rtsp://192.168.33.46/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=${CX}x${CY}&fps=25" ! \
rtpjitterbuffer ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! fpsdisplaysink video-sink=kmssink

Hardware accelerated one is following pipeline (problem is same with kmssink and waylandsink):

LATENCY=500
CX=320
CY=200
X=320
Y=200
gst-launch-1.0 rtspsrc latency=$LATENCY location="rtsp://192.168.33.46/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=${CX}x${CY}&fps=25" !\
rtpjitterbuffer ! rtph264depay ! h264parse ! ducatih264dec ! vpe ! 'video/x-raw, width='${X}', height='${Y} ! waylandsink

over 6 years ago

0 Biser Gatchev-XID over 6 years ago

TI__Guru**** 393215 points

The software team have been notified. They will respond here.

0 Margarita Gashova over 6 years ago

TI__Guru* 80190 points

Hello,

Please add few queue elements in the pipeline.
For example you could add these elements here:

gst-launch-1.0 rtspsrc latency=$LATENCY location="rtsp://192.168.33.46/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=${CX}x${CY}&fps=25" ! queue !
rtpjitterbuffer ! rtph264depay ! queue ! h264parse ! queue ! ducatih264decvpe ! 'video/x-raw, width='${X}', height='${Y} ! queue ! waylandsink

Let me know the result.

BR
Margarita

0 Ondrej Bilka over 6 years ago in reply to Margarita Gashova

Intellectual 390 points

Yes, that helps. But it looks unrelated to smaller resolution being slower. It reduced latency by around 200ms everywhere, I could play 1024x768 with 200ms latency but 320x200,640x480,800x600 needs 400ms (problem in original post now happens at latency 100 and 300)

0 Margarita Gashova over 6 years ago in reply to Ondrej Bilka

TI__Guru* 80190 points

Hello,

Could you try this:
...! ducatih264decvpe ! 'video/x-raw, width='${X}', height='${Y} ! queue max-size-buffers=2 ! waylandsink
If you still has a problem with smaller resolution you could try to set rtspsrc property drop-on-latency=true. The default value is false.
Let me know the result.

BR
Margarita

0 Ondrej Bilka over 6 years ago in reply to Margarita Gashova

Intellectual 390 points

Hi, there is no change with using these. Also tried set do-retransmission=false to eliminate that possibility

0 Ondrej Bilka over 6 years ago in reply to Ondrej Bilka

Intellectual 390 points

I could also reproduce this with example rtsp server from gst-rtsp-server, namely this one
github.com/.../test-video.c

0 Margarita Gashova over 6 years ago in reply to Ondrej Bilka

TI__Guru* 80190 points

Hello,

You could try to set these rtpjitterbuffer properties latency=100 and drop-on-latency=true.

I am not sure what is the pipeline that you tried but you could add queue between decoder and vpe.
You could set vpe's num-input-buffers=8 property.
The rtspsrc element is not a TI element. I would recommend you to search for more information about it and the latency property.

BR
Margarita

0 Ondrej Bilka over 6 years ago in reply to Margarita Gashova

Intellectual 390 points

It didn't work. Is there any known restriction on resolution for ducati plugin? It look like problem in decoder and its timestamp handling as replacing it with avdec_h264 it works lot better

0 Margarita Gashova over 6 years ago in reply to Ondrej Bilka

TI__Guru* 80190 points

Hello,

You could check this file gstducatih264dec.c for what are the resolutions. Please post the latest pipeline that you are using.
If in your pipeline sync=false is not set please set it (...! waylandsink sync=false or ....! kmssink sync=false).

gst-launch-1.0 rtspsrc latency=200 location="rtsp://......videocodec=h264&h264profile=main&resolution=1024x768&fps=25" ! queue ! rtph264depay ! queue ! h264parse ! ! queue ! ducatih264dec ! vpe ! 'video/x-raw, width=1024, height=768' ! waylandsink sync=false

BR
Margarita

0 Ondrej Bilka over 6 years ago in reply to Margarita Gashova

Intellectual 390 points

My question was more about hardware limitations on resolution. gstducatih264dec.c is just wrapper on libdce, it accepts any resolution with parameter from 16 to 2048 and only does some padding. It references libdce ti/sdo/codecs/h264dec/ih264vdec.h but I didn't find lot about resolution there.

As latency and sync=false it is problematic because of our application as a cctv system. Risk that pipeline may accumulate several seconds of delay(or in worst case gets stuck on one frame) and users takes wrong action because he assumes predictable delay/doesn't see change in time is lot worse than predictable latency.

I tried lot of pipelines without noticable effect, last one is
... ! rtpjitterbuffer latency=200 drop-on-latency=true ! rtph264depay ! queue ! h264parse ! queue ! ducatih264decvpe num-input-buffers=8 ! 'video/x-raw, width='${X}', height='${Y} ! queue ! waylandsink

0 Margarita Gashova over 6 years ago in reply to Ondrej Bilka

TI__Guru* 80190 points

Hello,

You could check the decoder user guide for more information about resolution and what the decoder itself supports.
git.ti.com/.../docs
Have you checked the pipeline in my previous post where rtpjitterbuffer element is removed.

BR
Margarita

0 Ondrej Bilka over 6 years ago in reply to Margarita Gashova

Intellectual 390 points

Thanks, manual helped me to understand issues here.

Still for resolution documentation just says that arbitrary resolution within 16-2048 is supported.

From documention I found out as possible cause that decoder keep several frames for reordering. It is useless for rtp streams as source and rtpjitterbuffer already handle retransmission and reordering. There is max-reorder-frames property in ducatih264dec but it isn't used when it could be infered from stream.

It looks accessible in decoder api but not gstreamer plugin one. See section decreasing DDR usage. Could that be patched? It should also help with our another problem that if we run multiple pipelines after one exhausts TILER memory random pipeline will crash(happens for 4 1024x768 decoders on wayland).

Back to pipelines there is no difference with or without using jitterbuffer, also your last example didn't include one, everything need latency 200ms for 1024x768 and 400ms for 320x200 and other resolution.

With sync=false on sink it is easy to create latency of several seconds and increasing. I could do it by adding videoconvert that causes software transformation.

0 Margarita Gashova over 6 years ago in reply to Ondrej Bilka

TI__Guru* 80190 points

Hello,

Gst ducatih264dec has property max-reorder-frames the default value is 16. The range is [0-16], 0 means no reordering. Also display_delay and dpbSizeInFrames are set to auto(check gstducatih264dec.c).

BR
Margarita

0 Ondrej Bilka over 6 years ago in reply to Margarita Gashova

Intellectual 390 points

It looks that max-reorder-frames is ignored and value from stream is used instead.

0 Ondrej Bilka over 6 years ago in reply to Ondrej Bilka

Intellectual 390 points

As I written in another thread I am now sure that this issue is caused by too large buffer in decoder.

It just uses value derived from GOP size.
That makes no sense when camera uses baseline profile which means no B frames and documentation says that this parameter is meaningful only when B frames are present. Or if we know correct reordering parameter from camera documentation. In gstreamer network errors/reordering are handled by jitterbuffer. Frames will come with same ordering as from source, if that isn't possible due to time limit frame will be marked as lost.

In our case camera uses GOP=32 as default which causes these problems. Changing camera GOP to lower value removes this problem.

What probably happens as it is consistent with observed one frame/second is following:

First frame in GOP is I frame which is displayed instantly.
No other frame is displayed because before next 16 frames(which is about 500ms) to fill decoder buffer arrive sink detects that frame is too late and sends signal to drop frame.

0 Margarita Gashova over 6 years ago in reply to Ondrej Bilka

TI__Guru* 80190 points

Hello,

I ran few tests on my side here is the observations:

-default libgstducati.so

I decoded a video with this pipeline:

gst-launch-1.0 filesrc location=b_test.avi ! avidemux ! h264parse ! video/x-h264, stream-format=byte-stream, num-reorder-frames=0, ! ducatih264dec ! vpe ! video/x-raw, format=NV12, width=1280, height=720, framerate=30/1 ! waylandsink

In the log I observed this:

0:00:00.151752383 1343 0x159030 WARN ducati gstducatih264dec.c:410:gst_ducati_h264dec_set_sink_caps:<ducatih264dec0> Using 0 frames for reordering
/GstPipeline:pipeline0/GstDucatiH264Dec:ducatih264dec0.GstPad:sink: caps = "video/x-h264\,\ stream-format\=$string$byte-stream\,\ num-reorder-frames\=$int$0\,\ variant\=$string$itu\,\ framerate\=$fraction$30/1\,\ width\=$int$1280\,\ height\=$int$720\,\ parsed\=$boolean$true\,\ alignment\=$string$au\,\ profile\=$string$high\,\ level\=$string$4"

0:00:00.498532665 1343 0x159030 WARN ducati gstducatividdec.c:590:codec_process:<ducatih264dec0> changing max-ref-frames in caps to 11

max-ref=frames is set to 11 in my case. It is depending of the stream.The decoder decides how many frames to be locked based on the stream properties.

Also these params are set to AUTO in gstducatih264dec.c file:

self->params->displayDelay = IVIDDEC3_DISPLAY_DELAY_AUTO;
params->dpbSizeInFrames = IH264VDEC_DPB_NUMFRAMES_AUTO;

I changed both params to IVIDDEC3_DISPLAY_DELAY_1 and IH264VDEC_DPB_NUMFRAMES_1. Here is the result for this case:

0:00:01.087428941 1099 0x212490 WARN ducati gstducatividdec.c:594:codec_process:<ducatih264dec0> changing max-ref-frames in caps to 3
/GstPipeline:pipeline0/GstDucatiH264Dec:ducatih264dec0.GstPad:src: caps = "video/x-raw\,\ format\=$string$NV12\,\ width\=$int$1408\,\ height\=$int$816\,\ framerate\=$fraction$30/1\,\ drm_mem\=$boolean$true\,\ max-ref-frames\=$int$3"

You could try to change these params(check the h264dec user guide for more information).
Hope this helps.

BR
Margarita

Processors

Processors forum

Linux/PROCESSOR-SDK-AM57X: Big latency of H.264 GStreamer pipeline