AM5728: Videomixer timestamping problem

Hello,

I would recommend you to add queue elements in the pipeline, set sync=false and set caps filters with height, width,  etc.

Vishal said:

I was not able to successfully change decodebin to use ducatih264dec, not sure why.

Did you add parser element between dec and enc?

BR
Margarita

Hi Margarita,

Thank you for the quick reply. I just ran this updated pipeline. The parser element in between enc and dec was missing. I added queues before the videomixer inputs, but it doesn't seem to make a difference in the low framerate seen on the waylandsink.

gst-launch-1.0 -v  videotestsrc ! video/x-raw, width=800, height=480, framerate=30/1 ! \
ducatih264enc ! h264parse ! ducatih264dec ! \
video/x-raw, format=NV12, framerate=30/1 ! vpe ! video/x-raw, format=RGB, framerate=30/1 ! \
queue ! videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! \
waylandsink sync=false \
multifilesrc location="img.%04d.png" index=0 stop-index=1 loop=true caps="image/png,framerate=\(fraction\)30/1" ! pngdec ! videorate ! queue ! mix.

I also noticed that the ducatih264dec seems to resize the height and width:

/GstPipeline:pipeline0/GstH264Parse:h264parse0.GstPad:src: caps = "video/x-h264\,\ width\=\(int\)800\,\ height\=\(int\)480\,\ alignment\=\(string\)au\,\ stream-format\=\(string\)byte-stream\,....
/GstPipeline:pipeline0/GstDucatiH264Dec:ducatih264dec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)896\,\ height\=\(int\)576\,\ framerate\=\(fraction\)30/1\,....

If you could recommend specifically where to put the queue elements and caps that would be very helpful.

Thanks,

Vishal

Hello,

Please add  height, width properties in the caps filters before and after vpe.

You could add queues before the enc and dec.

...

queue ! ducatih264enc ! h264parse ! queue ! ducatih264dec ! vpe ! video/x-raw, width=800, height=480, format=RGB, framerate=30/1 !

...

It seems like you have made some changes in vpe element, right?

I would recommend you to run the pipeline without the second branch (png decoding) and check the behavior on your side.

BR

Margarita

Hi Margarita,

Yes I am using a VPE patch to enable RGB output that I obtained from another forum post.

I ran the following pipeline but the output is still a low framerate:

gst-launch-1.0 -v  videotestsrc ! video/x-raw, width=800, height=480, framerate=30/1 ! \
queue ! ducatih264enc ! h264parse ! queue ! ducatih264dec ! \
video/x-raw, width=896, height=576 ! vpe ! video/x-raw, width=800, height=480, format=RGB, framerate=30/1 ! \
queue ! videomixer name=mix sink_0::alpha=1 ! \
waylandsink sync=false

I ran this with and without the height & width caps before the vpe - same result. The ducatih264dec didn't like 800x480 caps, so I had to set it to 896x576 as I mentioned before (not sure why this is the case).

Is it possible to smoothly run videomixer for my usecase without the vpe colorspace conversion?

Hi Margarita,

Yes that runs, but it does not satisfy my latency requirements. When I try with my own encoded source from RTP (as opposed to the encoded videotestsrc), I get an increasing latency over time, on the order of 10s of seconds to even minutes. Is there a way to reduce this latency or at least keep it at a low fixed latency?

Also, I noticed you mentioned in the other thread that glvideomixer is not available on AM5728, but is there another way to utilize the GPU for video mixing?

I appreciate any resources or advice.

Thanks,

Vishal

Hello,

Have you tried with the compositor element instead of videomixer?

Could you try to increase num-buffers properties of vpe element also?

Please set(increase the value of) the queue properties also.

BR
Margarita

Hi Margarita,

Thanks for your reply.

I have tried all your suggestions above, but the output video still has an increasing latency over time.

Hello,

Vishal said:

gst-launch-1.0 -v videotestsrc ! video/x-raw, width=800, height=480, framerate=30/1 ! \ queue ! ducatih264enc ! h264parse ! queue ! ducatih264dec ! \ video/x-raw, width=896, height=576 ! vpe ! video/x-raw, width=800, height=480, format=RGB, framerate=30/1 ! \ queue ! videomixer name=mix sink_0::alpha=1 ! \ waylandsink sync=false

What will be your use case?

Because in pipeline above you could remove encoder, parse, and decoder element. Something like:

gst-launch-1.0 videomixer name=mixer ! videoconvert ! waylandsink videotestsrc pattern=snow ! "video/x-raw,format=AYUV,width=800,height=600,framerate=(fraction)30/1" ! queue ! mixer. videotestsrc pattern=13 ! "video/x-raw,format=AYUV,width=320,height=240,framerate=(fraction)10/1" ! queue ! mixer.

BR
Margarita

My use case involves an RTP video stream encoded in MJPEG (30 fps) which needs to be decoded and I need to overlay a buffer of PNGs on top of it at (~10 fps). The pipeline I've got working with a lot of latency looks like:

gst-launch-1.0 -v videomixer name=mixer ! videoconvert ! waylandsink sync=false \
udpsrc port=1234 caps="application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000, ssrc=(guint)2823054885" ! rtpjitterbuffer ! rtpjpegdepay ! queue max-size-buffers=5000 ! decodebin ! vpe ! video/x-raw, width=800, height=480, format=NV12, framerate=30/1 ! queue max-size-buffers=5000 ! mixer. \
videotestsrc pattern=1 ! video/x-raw, format=AYUV, width=320, height=240, framerate=10/1 ! queue max-size-buffers=5000 ! mixer.

Except I would replace the second video branch with multifilesrc with PNGs which I've confirmed is working. This pipeline runs, but the latency seems to increase with time, which is not acceptable for my use case. In my use case I would also transcode the output of videomixer to h264 and then filesink, not waylandsink; I'm just using wayland sink so I can see the latency on the display. My final pipeline will look more like this (but even in this I am getting variable latency) :

gst-launch-1.0 -v udpsrc port=1234 caps="application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000, ssrc=(guint)2823054885" ! rtpjitterbuffer ! rtpjpegdepay ! decodebin ! vpe ! \
video/x-raw, format=RGB, width=800, height=480, framerate=30/1, interlace-mode=progressive, pixel-aspect-ratio=1/1 ! \
videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! videoconvert ! \
video/x-raw, format=NV12, width=800, height=480, framerate=30/1 ! \
ducatih264enc ! filesink location=test_transcode.h264 sync=true \
multifilesrc location="img.%04d.png" index=0 stop-index=1 loop=true caps="image/png,framerate=\(fraction\)10/1" ! pngdec ! videorate ! mix.

Thanks

Hello,

I would recommend you:
1. add delay in the filesink element (ts-offset property). You could check the range of this property with gst-inspect command.
2. disable synchronisation (sync=false).
3. before the filesink please add ! queue max-size-bytes=65536 max-size-buffers=1000 max-size-time=0 leaky=downstream !...again with gst-inspect you could check the queue properties.
4. add few more queues in the pipeline.
5.add h264parse element after ducatih264enc (in other case the .h264 file could be corrupted).


BR
Margarita

Hi Margarita,

Thanks for the suggestions. I tried the following pipeline:

gst-launch-1.0 -v udpsrc port=1234 caps="application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000, ssrc=(guint)2823054885" ! rtpjitterbuffer ! rtpjpegdepay ! queue ! decodebin ! vpe ! \
video/x-raw, format=RGB, width=800, height=480, framerate=30/1, interlace-mode=progressive, pixel-aspect-ratio=1/1 ! \
videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! videoconvert ! \
video/x-raw, format=NV12, width=800, height=480, framerate=30/1 ! \
queue ! ducatih264enc ! h264parse ! queue max-size-bytes=65536 max-size-buffers=1000 max-size-time=0 leaky=downstream ! \
filesink location=test_transcode.h264 ts-offset=-30000000000 sync=false \
multifilesrc location="img.%04d.png" index=0 stop-index=1 loop=true caps="image/png,framerate=\(fraction\)2/1" ! pngdec ! videorate ! mix.

It runs but the recorded video file still has latencies greater than 30-60 seconds, and it doesn't seem to be constant. I tried playing with the values of ts-offset (negative and positive) but it doesn't change much in the end result.

To give more background context, the incoming RTP stream contains 1920x1080 MPEG video and the VPE is scaling down to 800x480. When I run this pipeline, the CPU usage is close to 100% between the two cores. I am assuming videomixer is utilizing the CPU for software rendering. Is this behavior typical for this use-case or should videomixer be able to handle with a lower fixed latency?

Thank you,

Vishal

Hello,

Vishal said:

It runs but the recorded video file still has latencies greater than 30-60 seconds, and it doesn't seem to be constant.

It could be a network issue. Could you try to stream lower resolution

Could you increase payload=96 and set encoding-name(recommended) ? 

Vishal said:

the incoming RTP stream contains 1920x1080 MPEG video

I am confused what you are streaming mpeg or mjpeg or else?

You could check with the top command the CPU load if it is higher the fps drop could be observe.

Videomixer element is not a TI element, it is a open source element so I would recommend you to search over the internet about the usage etc.

Vishal said:

decodebin ! vpe ! \ video/x-raw, format=RGB, width=800, height=480, framerate=30/1, interlace-mode=progressive, pixel-aspect-ratio=1/1 ! \

The format=RGB is it should be NV12 since the encoder requires NV12 format. I see that there is a videoconvert element which converts the output of the pngdec there is no point to convert the output of the decodebin to RGB and then to NV12.It should work if you set NV12 directly. This are additional mem copies.

Per my understanding your usecase is video + png->mix->encode the both branches and save in file right?

BR
Margarita

Hi Margarita,

1. Sorry about the confusion, the encoded video stream is in MJPEG. Also your understanding of the usecase is correct.

2. I don't think it's a network issue because pipelines to wayland and filesink without videomixer work perfectly.

3. The CPU usage as I mentioned earlier goes up to 100% between the 2 cores.

4. I changed payload=96, but that doesn't work because JPEG encoding corresponds to 26.

5. Originally I was using RGB because I wanted to avoid using videoconvert all-together to minimize CPU software rendering. I changed it back to NV12 as you suggested. The latency seems to be more reproducible now. However now I am noticing that after approx 40-50 seconds of recording, the PNG alternating still continues to record but the background camera feed is completely frozen. It normally stays frozen, but once I have seen it come back live. I tried adding queues before the videomixer sinks, but no effect. Any idea as to what is going on there? The pipeline I used is:

gst-launch-1.0 -v udpsrc port=1234 caps="application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000, ssrc=(guint)2823054885" ! rtpjitterbuffer ! rtpjpegdepay ! queue ! decodebin ! vpe ! \
video/x-raw, format=NV12, width=800, height=480, framerate=30/1 ! \
videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! videoconvert ! \
video/x-raw, format=NV12, width=800, height=480, framerate=30/1 ! \
queue ! ducatih264enc ! h264parse ! queue max-size-bytes=65536 max-size-buffers=1000 max-size-time=0 leaky=downstream ! \
filesink location=test_transcode.h264 ts-offset=30000000000 sync=false \
multifilesrc location="img.%04d.png" index=0 stop-index=1 loop=true caps="image/png,framerate=\(fraction\)5/1" ! pngdec ! videorate ! mix.

Thanks,

Vishal

Hello,

Vishal said:

4. I changed payload=96, but that doesn't work because JPEG encoding corresponds to 26.

For payload value >=96,  encoding-name should be set also. Here is example encoding-name=(string)JPEG.

I see in the pipeline after the videorate element I do not see the caps filter. Here is example of videorate usage ...! videorate ! video/x-raw,framerate=5/1 ! ...

Vishal said:

, the PNG alternating still continues to record but the background camera feed is completely frozen. It normally stays frozen, but once I have seen it come back live

I am not sure that I understand this.

If you remove the png branch do you see this?

Since the latency is reduced could you give a try when ts-offset=30000000000 is removed?

BR
Margarita

Hi Margarita,

Apologies for the late reply. I tried your latest suggestions , but the issue I mentioned before still remains. What I meant last time is, the final output (h264 encoded mixed video stream) has some latency, but has a valid recording up until about 40-50 seconds, after which the camera stream is frozen but the png branch continues to swap through the buffer of pngs. So basically the rest of the video recording after 40-50 seconds, is a frozen camera stream with a swapping PNG overlay.

Any suggestions as to why this may be happening? This freezing behaviour does not exist when I save the camera stream to h264 encoded to filesink directly (with no videomixer). Here is the pipeline I ran:

gst-launch-1.0 -v udpsrc port=1234 caps="application/x-rtp, media=(string)video, payload=(int)26, encoding-name=(string)JPEG, clock-rate=(int)90000, ssrc=(guint)2823054885" ! rtpjitterbuffer ! rtpjpegdepay ! queue ! decodebin ! vpe ! \
video/x-raw, format=NV12, width=800, height=480, framerate=30/1 ! \
videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! videoconvert ! \
video/x-raw, format=NV12, width=800, height=480, framerate=30/1 ! \
queue ! ducatih264enc ! h264parse ! queue max-size-bytes=65536 max-size-buffers=1000 max-size-time=0 leaky=downstream ! \
filesink location=test_transcode.h264 ts-offset=30000000000 sync=false \
multifilesrc location="img.%04d.png" index=0 stop-index=1 loop=true caps="image/png,framerate=\(fraction\)5/1" ! pngdec ! videorate ! video/x-raw,framerate=5/1 ! mix.

Thanks,

1. When I try that pipeline, the final output video is all jumbled up with random colours, and while it runs I periodically get the message:

invalid video buffer received
WARNING: from element /GstPipeline:pipeline0/GstVideoConvert:videoconvert0: Internal GStreamer error: code not implemented.  Please file a bug at bugzilla.gnome.org/enter_bug.cgi.
Additional debug info:
../../../../gst-plugins-base-1.8.3/gst-libs/gst/video/gstvideofilter.c(292): gst_video_filter_transform (): /GstPipeline:pipeline0/GstVideoConvert:videoconvert0:

2. The pipeline does not work when I replace udpsrc with filesrc

Hi Margarita,

Thank you for the quick reply.

I ran your updated pipeline, but the framerate was at 5 fps so I added a 30 fps cap at the videomixer sink (for camera feed). Again I am seeing the same issue of camera feed freezing as I described several times. However, by curiosity I scaled down the camera feed with the vpe to 400x240, and found that the latency was minimal and I did not see any freezing even after 2 minutes of recording. Do you suspect any thing that may cause this issue to be happening at the larger resolution and not at lower res? Could it have something to do with the limited CMA / CMEM buffer pools size?

gst-launch-1.0 -v udpsrc port=1234 caps="application/x-rtp, payload=(int)26, encoding-name=(string)JPEG" ! rtpjitterbuffer ! queue ! rtpjpegdepay ! queue ! jpegparse ! queue ! ducatijpegdecvpe ! video/x-raw, format=NV12, width=400, height=240, framerate=30/1 ! videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! videoconvert ! video/x-raw, format=NV12, width=400, height=240 ! queue ! ducatih264enc ! h264parse ! queue ! filesink location=test_transcode.h264 sync=false multifilesrc location="img.%04d.png" index=0 stop-index=1 loop=true caps="image/png, framerate=\(fraction\)5/1" ! pngdec ! mix.

Thanks again,

Hi Margarita,

There seem to be some caps missing in the pipeline you just sent. I'm getting a not-negotiated error. It shows from udpsrc, but I believe the issue is with ducatijpegdec, as I've encountered this before but was not sure what caps were missing:

Setting pipeline to PAUSED ...
/GstPipeline:pipeline0/GstMultiFileSrc:multifilesrc0.GstPad:src: caps = "image/png\,\ width\=\(int\)1\,\ height\=\(int\)1\,\ framerate\=\(fraction\)5/1\,\ pixel-aspect-ratio\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstPngDec:pngdec0.GstPad:sink: caps = "image/png\,\ width\=\(int\)1\,\ height\=\(int\)1\,\ framerate\=\(fraction\)5/1\,\ pixel-aspect-ratio\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstPngDec:pngdec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)300\,\ height\=\(int\)300\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)sRGB\,\ framerate\=\(fraction\)5/1"
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstVideoMixer2Pad:sink_1: caps = "video/x-raw\,\ format\=\(string\)RGBA\,\ width\=\(int\)300\,\ height\=\(int\)300\,\ interlace-mode\=\(string\)progressive\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ colorimetry\=\(string\)sRGB\,\ framerate\=\(fraction\)5/1"
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps = "application/x-rtp\,\ payload\=\(int\)26\,\ encoding-name\=\(string\)JPEG\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"
/GstPipeline:pipeline0/GstRtpJitterBuffer:rtpjitterbuffer0.GstPad:sink: caps = "application/x-rtp\,\ payload\=\(int\)26\,\ encoding-name\=\(string\)JPEG\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"
/GstPipeline:pipeline0/GstRtpJitterBuffer:rtpjitterbuffer0.GstPad:sink: caps = "application/x-rtp\,\ payload\=\(int\)26\,\ encoding-name\=\(string\)JPEG\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = "application/x-rtp\,\ payload\=\(int\)26\,\ encoding-name\=\(string\)JPEG\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = "application/x-rtp\,\ payload\=\(int\)26\,\ encoding-name\=\(string\)JPEG\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"
/GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0.GstPad:sink: caps = "application/x-rtp\,\ payload\=\(int\)26\,\ encoding-name\=\(string\)JPEG\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"
/GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0.GstPad:src: caps = "image/jpeg\,\ framerate\=\(fraction\)0/1\,\ width\=\(int\)1920\,\ height\=\(int\)1080"
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = "image/jpeg\,\ framerate\=\(fraction\)0/1\,\ width\=\(int\)1920\,\ height\=\(int\)1080"
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:src: caps = "image/jpeg\,\ framerate\=\(fraction\)0/1\,\ width\=\(int\)1920\,\ height\=\(int\)1080"
/GstPipeline:pipeline0/GstJpegParse:jpegparse0.GstPad:sink: caps = "image/jpeg\,\ framerate\=\(fraction\)0/1\,\ width\=\(int\)1920\,\ height\=\(int\)1080"
/GstPipeline:pipeline0/GstJpegParse:jpegparse0.GstPad:src: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstQueue:queue2.GstPad:sink: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstQueue:queue2.GstPad:sink: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstDucatiJpegDec:ducatijpegdec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)0/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true"
/GstPipeline:pipeline0/GstDucatiJpegDec:ducatijpegdec0.GstPad:sink: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)UYVY\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstDucatiJpegDec:ducatijpegdec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)0/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)1"
ERROR: from element /GstPipeline:pipeline0/GstUDPSrc:udpsrc0: Internal data flow error.
Additional debug info:
../../../../gstreamer-1.8.3/libs/gst/base/gstbasesrc.c(2948): gst_base_src_loop (): /GstPipeline:pipeline0/GstUDPSrc:udpsrc0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 0:00:00.297842901

Thanks,

Vishal

Hi Margarita,

Thank you for the suggestions.

Adding the queue leaky=downstream before the videomixer seemed to have increased the time it records before the freeze. Now it tends to freeze after about 80 seconds of recording.

I tried the pipeline you provided without the png branch and that works fine without freezing.

I also tried a pipeline with videomixer and instead of png branch I mixed in videotestsrc and that also gives similar results of freezing:

gst-launch-1.0 -v udpsrc port=1234 caps="application/x-rtp, payload=(int)26, encoding-name=(string)JPEG" ! rtpjitterbuffer ! queue ! rtpjpegdepay ! queue ! jpegparse ! queue ! ducatijpegdecvpe ! video/x-raw, format=NV12, width=800, height=480 ! queue leaky=downstream ! videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! queue ! videoconvert ! video/x-raw, format=NV12, width=800, height=480 ! queue ! ducatih264enc ! h264parse ! queue ! filesink location=test_transcode.h264 sync=false \
videotestsrc pattern=1 ! \
video/x-raw,format=ARGB,framerate=\(fraction\)30/1,width=200,height=200 ! mix.

Is there an alternative option to obtain this video mixing, ultilizing the hardware blocks or GPU acceleration on AM5728?

Thanks again,

Vishal

Hi Margarita,

I ran the pipeline you sent and it works fine on my side as well. In order to adapt to my use case, I changed it to the following and tried:

gst-launch-1.0 compositor name=mix ! queue ! videoconvert ! video/x-raw, format=NV12, width=800, height=480 ! queue ! ducatih264enc ! h264parse ! queue ! filesink location=test_transcode.h264 sync=false udpsrc port=1234 caps="application/x-rtp, payload=(int)26, encoding-name=(string)JPEG" ! rtpjitterbuffer ! queue ! rtpjpegdepay ! queue ! jpegparse ! queue ! ducatijpegdecvpe ! 'video/x-raw, format=(string)NV12, width=(int)800, height=(int)480' ! mix. videotestsrc pattern=18 ! 'video/x-raw, format=(string)AYUV,framerate=(fraction)30/1, width=(int)200, height=(int)200' ! mix.

Again, it results in the same problem of freezing after 40 seconds or so. I also tried a waylandsink variant (instead of ducatih264enc --> filesink) of the above pipeline, and it also freezes after a minute or so of recording. Notice that I had to increase the framerate of the videotestsrc to 30/1. When I left it at 5/1, the camera feed was also recorded at 5/1. I tried to add caps after the camera feed to fix it to 30/1, but I was not successful. I tried adding framerate caps in the udpsrc caps as well as after ducatijpegdecvpe. Example:

gst-launch-1.0 compositor name=mix ! queue ! videoconvert ! video/x-raw, format=NV12, width=800, height=480 ! queue ! ducatih264enc ! h264parse ! queue ! filesink location=test_transcode.h264 sync=false udpsrc port=1234 caps="application/x-rtp, payload=(int)26, encoding-name=(string)JPEG" ! rtpjitterbuffer ! queue ! rtpjpegdepay ! queue ! jpegparse ! queue ! ducatijpegdecvpe ! 'video/x-raw, format=(string)NV12, width=(int)800, height=(int)480, framerate=(fraction)30/1' ! mix. videotestsrc pattern=18 ! 'video/x-raw, format=(string)AYUV,framerate=(fraction)5/1, width=(int)200, height=(int)200' ! mix.

Results in :

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
Redistribute latency...
**
ERROR:gstducatividenc.c:426:gst_ducati_videnc_configure: assertion failed: (self->output_pool == NULL)
Aborted (core dumped)

What is the proper way of setting the framerate of the udpsrc camera stream, so that videomixer/compositor has a consistent 30 fps at it's sink?

Thanks,

Vishal

Hello Vishal,

Could you  remove rtpjitterbuffer element in the pipeline with videomixer element?

BR
Margarita

Hi Margarita,

I tried the following pipeline without rtpjitterbuffer

gst-launch-1.0 -v udpsrc port=1234 caps="application/x-rtp, payload=(int)26, encoding-name=(string)JPEG" ! queue ! rtpjpegdepay ! queue ! jpegparse ! queue ! ducatijpegdecvpe ! video/x-raw, format=NV12, width=800, height=480 ! queue leaky=downstream ! videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! queue ! videoconvert ! video/x-raw, format=NV12, width=800, height=480 ! queue ! ducatih264enc ! h264parse ! queue ! filesink location=test_transcode.h264 sync=false \
videotestsrc pattern=1 ! \
video/x-raw,format=ARGB,framerate=\(fraction\)30/1,width=200,height=200 ! mix.

Same freezing problem.

Thanks,

Hi Margarita,

Thanks for the follow-up.

The pipeline you provided runs successfully without freezing. When I change the videotestsrc branch to my udpsrc branch the same problem of freezing after 1 minute of recording occurs. I tried increasing the buffer-size property of udpsrc as well like:

gst-launch-1.0 udpsrc buffer-size=2147483647 port=1234 caps="application/x-rtp, payload=(int)26, encoding-name=(string)JPEG" ! rtpjitterbuffer ! queue ! rtpjpegdepay ! queue ! jpegparse ! queue ! ducatijpegdec ! vpe num-input-buffers=8 ! video/x-raw, format=NV12, width=800, height=480 ! queue leaky=downstream ! videomixer name=mix sink_0::alpha=1 sink_1::alpha=0.7 ! queue ! videoconvert ! video/x-raw, format=NV12, width=800, height=480 ! queue ! vpe ! ducatih264enc ! h264parse ! queue ! filesink location=test_transcode.h264 sync=false videotestsrc pattern=1 ! video/x-raw,format=ARGB,framerate=\(fraction\)30/1,width=200,height=200 ! mix.

While its running the CPU usage is typically around 130 %, and I've seen memory usage start around 200 MiB and increase above 360 MiB as the recording continues. I have yet to try replacing udpsrc to rtpsrc.

It seems to me that using videomixer for my application is probably not the best way to go considering it does not utilize the GPU for video mixing. My end goal is to get QT generated content overlayed on-top of a live camera stream within gstreamer so that it can be recorded (encoded and saved to file). I'm currently looking into other options revolving around EGLImage. If you have any other suggestions, please point me in the right direction.

Thanks again for your support,

Vishal