Linux/AM5728: Ducatijpegdec GStreamer Plugin adds 8 extra lines of decoded image, produces incompatibility issue with VPE

Tom W

Part Number: AM5728

Tool/software: Linux

Hello,

I've been using the ducatijpegdec GStreamer plugin for some time, and I've noticed what I believe is a compatibility issue with the VPE driver. When decoding a 1080p image (jpeg), the decoder will frequently add 8 extra lines of image, where GStreamer, and the VPE driver recognize it as a 1088-line image. Seen below are the caps negotiation process in GStreamer, as well as some kernel debug messages from the VPE driver for the following GStreamer pipelines:

Sender Pipeline:
gst-launch-1.0 videotestsrc ! video/x-raw, format=I420, width=1920, height=1080, framerate=30/1 ! jpegenc ! rtpjpegpay mtu=1472 ! udpsink host=192.168.1.233 multicast-iface=enp0s31f6 port=1234

Receiver Pipeline (AM572x EVM):
gst-launch-1.0 udpsrc multicast-iface=eth0 port=1234 ! application/x-rtp, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegparse ! ducatijpegdec ! vpe ! video/x-raw, format=NV12, width=1920, height=1080 ! appsink

GStreamer Caps Negotiation:
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "application/x-rtp\,\ encoding-name\=\(string\)JPEG\,\ payload\=\(int\)26\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"

/GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0.GstPad:sink: caps = "application/x-rtp\,\ encoding-name\=\(string\)JPEG\,\ payload\=\(int\)26\,\ media\=\(string\)video\,\ clock-rate\=\(int\)90000"

(gst-launch-1.0:1594): GStreamer-WARNING **: ../../gstreamer-1.6.3/gst/gstpad.c:4943:store_sticky_event:<rtpjpegdepay0:src> Sticky event misordering, got 'segment' before 'caps'
/GstPipeline:pipeline0/GstRtpJPEGDepay:rtpjpegdepay0.GstPad:src: caps = "image/jpeg\,\ framerate\=\(fraction\)0/1\,\ width\=\(int\)1920\,\ height\=\(int\)1080"

(gst-launch-1.0:1594): GStreamer-WARNING **: ../../gstreamer-1.6.3/gst/gstpad.c:4943:store_sticky_event:<jpegparse0:sink> Sticky event misordering, got 'segment' before 'caps'
/GstPipeline:pipeline0/GstJpegParse:jpegparse0.GstPad:sink: caps = "image/jpeg\,\ framerate\=\(fraction\)0/1\,\ width\=\(int\)1920\,\ height\=\(int\)1080"
/GstPipeline:pipeline0/GstJpegParse:jpegparse0.GstPad:src: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstDucatiJpegDec:ducatijpegdec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)0/1\,\ drm_mem\=\(boolean\)true"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstAppSink:appsink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)0/1\,\ drm_mem\=\(boolean\)true"
/GstPipeline:pipeline0/GstDucatiJpegDec:ducatijpegdec0.GstPad:sink: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstDucatiJpegDec:ducatijpegdec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)0/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)1"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)0/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)1"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)0/1"

VPE Kernel Debug Messages

[ 5451.226486] vpe 489d0000.vpe: Setting format for type 10, wxh: 1920x1088, fmt: 842094158 bpl_y 1920
[ 5451.226497] vpe 489d0000.vpe: bpl_uv 1920
[ 5451.226512] vpe 489d0000.vpe: hs config: src_w = 1920, dst_w = 1920, decimation = none, lin_acc_inc = 01000000
[ 5451.226523] vpe 489d0000.vpe: vs config(POLY): src_h = 1088, dst_h = 1080,row_acc_inc = 000101e5
[ 5451.226547] vpe 489d0000.vpe: Setting format for type 9, wxh: 1920x1080, fmt: 842094158 bpl_y 1920
[ 5451.226556] vpe 489d0000.vpe: bpl_uv 1920
[ 5451.226606] vpe 489d0000.vpe: get 32 buffer(s) of size 2088960
[ 5451.226615] vpe 489d0000.vpe: and 1044480
[ 5451.227248] vpe 489d0000.vpe: get 6 buffer(s) of size 2073600
[ 5451.227259] vpe 489d0000.vpe: and 1036800

The receiver pipeline immediately segfaults while trying to handle these buffers in the VPE driver. According to the VPE Adjusting the Sender Pipeline to output at 1280*720 remedies this issue. My guess is that this is due to the VPE not being able to accept larger than a 1080p image, which fits the specs that are listed in the AM572x Technical Reference Manual (SPRUHZ64 section 10.3.4.1, page 2364).

Here is a debug log from the Receiver GStreamer pipeline.
5751.jul13_nv12_appsink1080.log

Of note is that this is the commit that is used in the 03.03 version of the PSDK: e797c1d832cc8ee1dd66d1683991cb6d7316ed63.

EDIT 1:

I have been going through the gst-plugin-ducati repo, and I believe that the reason the 1088p image is created is due to the buffer alignment required to communicate with the IVA-HD. 1080 is not divisible by 16 or 32, and the buffer size, as well as the GStreamer caps, are negotiated by the ALIGN2 macro. 1088 is divisible by 16 and 32, so is 1920.

/* align x to next highest multiple of 2^n */
#define ALIGN2(x,n)   (((x) + ((1 << (n)) - 1)) & ~((1 << (n)) - 1))

over 6 years ago

0 Margarita Gashova over 6 years ago

TI__Guru* 80190 points

Hello,

In the log it seems that you have height =1088 on the decoder output (src) so on the sink (input) of the vpe you will also have 1088.
You could check this file:
psdk/board-support/linux.../drivers/media/platform/ti-vpe/vpdma.h for the limitation.

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

Yes, that is definitely the source of the problem. Unfortunately, the header file you mentioned did not contain a definite reference to a maximum input size, but the VPE driver has been recorded to crash if an image is above 1920x1080 (in the following link it was 1928x1080)

e2e.ti.com/.../1751584

Since GStreamer records the size of the image up into the decoder sink pad as 1920x1080, and 1088 on the src pad, it must be the decoder adding 8 extra lines somehow.

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

Hello, just wondering if there is anything that can be done in the VPE or Ducati code that would provide a quick fix for this issue, as it is a significant blocker for our system.

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

Could you try pipeline like:
gst-launch-1.0 udpsrc multicast-iface=eth0 port=1234 ! application/x-rtp, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegparse ! ducatijpegdec ! videoscale ! video/x-raw, format=NV12, width=1920, height=1080 ! appsink

BR
Margarita

0 Margarita Gashova over 6 years ago in reply to Margarita Gashova

TI__Guru* 80190 points

Hello,

You could also try:

gst-launch-1.0 udpsrc multicast-iface=eth0 port=1234 ! application/x-rtp, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegparse ! ducatijpegdecvpe ! appsink

gst-launch-1.0 udpsrc multicast-iface=eth0 port=1234 ! application/x-rtp, encoding-name=JPEG, payload=26 ! rtpjpegdepay ! jpegparse ! ducatijpegdecvpe ! 'video/x-raw, format=NV12, width=1920, height=1080' ! appsink

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

Pipeline 1: Works, however there is something to note here: Hari (the VPE plugin developer) and I have created a revision on the VPE plugin that allows it to output RGB, and unfortunately due to a compatibility issue with Qt5, I need RGB. (Unless you know a way to display NV12-formatted images from gstreamer in Qt5 that I'm unaware of).

Pipeline 2: No crash, however unless an output size or format is specified, ducatijpegdecvpe actually triggers the VPE 'passthrough mode' and the VPE is not used. If a size is specified in the caps proceeding after ducatijpegdecvpe, the VPE driver activates, receives a 1088p image, and crashes.

Pipeline 3: segfault, see pipeline 2.

All three pipelines are noted to have the 1088p images coming out of the decoder.

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

Tom Wallis said:
Pipeline 1: Works, however there is something to note here: Hari (the VPE plugin developer) and I have created a revision on the VPE plugin that allows it to output RGB, and unfortunately due to a compatibility issue with Qt5, I need RGB. (Unless you know a way to display NV12-formatted images from gstreamer in Qt5 that I'm unaware of).

Unfortunately I am not aware with qt5. You could post in the qt forum https://forum.qt.io regarding NV12 display.

You could check videoconvert element also. It could accept nv12 on the input convert it to rgb on the output.

Hope this helps.

BR
Margarita

0 Margarita Gashova over 6 years ago in reply to Margarita Gashova

TI__Guru* 80190 points

Hello,

I did some tests on the EVM with latest PSDK.

I generated mjpeg file with this pipeline:

gst-launch-1.0 videotestsrc num-buffers=100 ! 'video/x-raw,format=(string)NV12,width=1920,height=1080' ! jpegenc ! jpegparse ! filesink location=1.mjpeg -v

The important caps are in green:

/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstJpegEnc:jpegenc0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)30/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ interlace-mode\=\(string\)progressive"
/GstPipeline:pipeline0/GstJpegEnc:jpegenc0.GstPad:src: caps = "image/jpeg\,\ sof-marker\=\(int\)0\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstJpegParse:jpegparse0.GstPad:sink: caps = "image/jpeg\,\ sof-marker\=\(int\)0\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstJpegParse:jpegparse0.GstPad:src: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)30/1"
/GstPipeline:pipeline0/GstFileSink:filesink0.GstPad:sink: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)30/1"

For decoding I used this pipeline:

gst-launch-1.0 -v filesrc location=1.mjpeg ! jpegparse ! ducatijpegdecvpe ! 'video/x-raw,format=(string)NV12,width=1920,height=1080' ! kmssink

Here is the output:

Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:02.225191177
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
root@am57xx-evm:~#

Here is the more detailed log:

/GstPipeline:pipeline0/GstJpegParse:jpegparse0.GstPad:src: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstDucatiJpegDec:decoder.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstVpe:vpe.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstKMSSink:kmssink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:src.GstProxyPad:proxypad1: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstVpe:vpe.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstDucatiJpegDec:decoder.GstPad:sink: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:sink: caps = "image/jpeg\,\ parsed\=\(boolean\)true\,\ format\=\(string\)I420\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstDucatiJpegDec:decoder.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstVpe:vpe.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:src.GstProxyPad:proxypad1: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstVpe:vpe.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0/GstVpe:vpe.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"
/GstPipeline:pipeline0/GstDucatiJpegdecVpe:ducatijpegdecvpe0.GstGhostPad:src.GstProxyPad:proxypad1: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)1/1"

I tried this pipeline also:

gst-launch-1.0 filesrc location=1.mjpeg ! jpegparse ! ducatijpegdec ! vpe ! 'video/x-raw,format=(string)NV12,width=1920,height=1080' ! kmssink

The result is the same as for previous pipeline.

The file is playable. I do not observe any segmentation fault.

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

I have tried videoconvert on my end, however I have run into the issue that doing a NV12->RGB convert with it was putting too much strain on the CPU, hence the patch to make VPE output RGB. I have attached the patch. (note, E2E has a .patch filter, simply replace the .txt with .patch and it'll work)4214.gst_vpe_rgb_support.txt

These pipelines would work on my end as well, but what happens in this use case is that the VPE goes into passthrough mode when the caps from the jpeg decoder are the same as the caps after the VPE. So unless an image size transformation, or a colourspace transformation is specified in the caps proceeding the VPE element, the VPE is not actually used.

I have attached the kernel logs from the VPE driver, the pertinent section is below:

[ 1149.325353] vpe 489d0000.vpe: CSC00 00000000
[ 1149.325361] vpe 489d0000.vpe: CSC01 00000000
[ 1149.325370] vpe 489d0000.vpe: CSC02 00000000
[ 1149.325378] vpe 489d0000.vpe: CSC03 00000000
[ 1149.325386] vpe 489d0000.vpe: CSC04 00000000
[ 1149.325395] vpe 489d0000.vpe: CSC05 10000000 <-- Colourspace Bypass

Unfortunately Image Size Bypass (SC registers) do not print unless there is a segfault. However, here is also the passthrough definition from the gstvpe.c source code (for the VPE plugin element, ducatijpegdecvpe is seen in code as just the ducatijpegdec and VPE plugins as one plugin)

self->passthrough = !(self->interlaced || self->output_width != self->input_width || self->output_height != self->input_height || self->output_fourcc != self->input_fourcc);

So you can see that if the stream is NV12, 1920x1080 in and out, passthrough mode is enabled, which turns off the VPE.

To reiterate, this is not an issue with the VPE driver, or the integrity of my pipeline, this is an issue with the ducatijpegdec plugin adding 8 extra lines of image data to a decoded image.

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

It seems like the ducati adds this 8 extra lines. I tried with ducatih264dec I see the same in the log file as for ducatijpegdec.
It is not clear to me when you are observing this seg.fault. I tried with pipeline :
gst-launch-1.0 filesrc location=1.mjpeg ! jpegparse ! ducatijpegdec ! vpe ! 'video/x-raw,format=(string)NV12,width=1280,height=720' ! kmssink
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1920\,\ height\=\(int\)1088\,\ framerate\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)1"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1280\,\ height\=\(int\)720\,\ framerate\=\(fraction\)1/1"

In this case the vpe perform scaling and I do not observed any problem on my side.
1.Please confirm "scaling" use case is working on your side.

2.Do you observe seg fault in use case NV12 -> YUY2 for example.
As you could see the low level vpe driver supports more formats compared to gst-vpe which supports only:
video/x-raw
format: NV12
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw
format: YUYV
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw
format: YUY2
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
on the sink and src.

3. Do you observe the error only in case NV12->RGB(the patch) that you attached?

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

All are tested without RGB patch and with RGB patch, on the newest processor SDK.

1. 'Scaling' as you specified ends in a segfault for me, with or without the RGB patch.

2. I see segfault in YUY2, interlacing issues with YUYV without RGB patch as well as with the RGB patch

3. Errors with and without patch as well, as specified above.

My guess is that the segfaults are coming from the interface with between VPE and the Ducati.

Backtrace log is attached. It's from PSDK 3.04, but I haven't made a PSDK 4.0 build with all the -dbg packages in yocto yet. I ran a GDB backtrace on the PSDK 4 pipeline though and it appears the same, but without the full trace.

3343.gstreamer_backtrace_jul11.txt

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

Hmm, I just tried it again with a new '1.mjpg' and it worked....

1. scaling works with NV12
2. YUY2 works, YUYV causes a compatibility issue with waylandsink and kmssink, but I'm planning on using appsink in my final application so that's no big issue.
3. RGB patch and non-RGB patch pipelines tested.

This may appear to be only on 'live' sources. In our final application we're going to be working with udpsrc, perhaps there is an issue with how ducati handles that type of source.

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

I am glad that is working on your side.

Tom Wallis said:
2. YUY2 works, YUYV causes a compatibility issue with waylandsink and kmssink, but I'm planning on using appsink in my final application so that's no big issue.

I asked only to try with yuy2 yuyv only for the tests. In this case you could connect the vpe to the fakesink element.

Tom Wallis said:
This may appear to be only on 'live' sources. In our final application we're going to be working with udpsrc, perhaps there is an issue with how ducati handles that type of source.

Could you try to remove multicast-iface property and give a try?

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

I asked only to try with yuy2 yuyv only for the tests. In this case you could connect the vpe to the fakesink element.

Yep, this is what I've been doing. Good suggestion

Could you try to remove multicast-iface property and give a try?

Unfortunately this had no effect.

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

Tom Wallis said:
This may appear to be only on 'live' sources. In our final application we're going to be working with udpsrc, perhaps there is an issue with how ducati handles that type of source.

You could replace udpsrc with videotestsrc and check the behavior.

I also would recommend the sync=false to be set in the videosink element(no matter is it kmssink/waylandsink etc).

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

I tried two different pipelines:

gst-launch-1.0 -v videotestsrc ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! vpe ! video/x-raw, format=YUY2, width=600, height=480 ! waylandsink sync=false

and

gst-launch-1.0 -v videotestsrc ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! ducatijpegenc ! jpegparse ! ducatijpegdec ! vpe ! video/x-raw, format=YUY2, width=600, height=480 ! waylandsink sync=false

the first pipeline displays poorly deinterlaced video on the screen, and the second pipeline segfaults. Changing the VPE output format to NV12 causes both pipelines to not segfault, but not display any video on the screen.

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

Let me check it.

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

Thank you for staying on this, Margarita. It's been driving me up the wall for weeks.

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

Here is the result on my side with this pipelines I do not observe seg.fault.

1. use case software jpegenc->decode->display via kmssink:

gst-launch-1.0 -v -e videotestsrc ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! queue ! jpegenc ! jpegparse ! queue ! ducatijpegdec ! vpe ! 'video/x-raw,format=(string)NV12,width=800,height=480' ! queue ! kmssink scale=true

2. Same usecase but the display sink is waylandsink:

gst-launch-1.0 -v -e videotestsrc ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! queue ! jpegenc ! jpegparse ! queue ! ducatijpegdec ! vpe ! 'video/x-raw,format=(string)NV12,width=800,height=480' ! queue ! waylandsink

3. use case ducati encode-> ducati decode->waylandsink. Same is working with kmssink also. The resolution that I tried after vpe are 800x480 and 640x480.

gst-launch-1.0 -e videotestsrc ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! queue ! ducatijpegenc ! jpegparse ! queue ! ducatijpegdec ! vpe ! 'video/x-raw,format=(string)NV12,width=800,height=480' ! queue ! waylandsink

4.use case videotest ->tee -> encode-> save in file. Second branch is videotestsrc->tee->display. ducatih264dec element is missing in this case since the pipeline is split to 2 branches after the videosrc.

gst-launch-1.0 -e videotestsrc ! 'video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, framerate=(fraction)30/1' ! vpe ! tee name=t ! queue ! ducatijpegenc ! queue ! jpegparse ! filesink location=1.mjpeg t. ! queue ! waylandsink use-drm=true

5. Same use case as previous but the videosink resolution is set in the capsfilter.

gst-launch-1.0 -e videotestsrc ! 'video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080, framerate=(fraction)30/1' ! vpe ! tee name=t ! queue ! ducatijpegenc ! queue ! jpegparse ! filesink location=1.mjpeg t. ! 'video/x-raw,format=(string)NV12,width=800,height=480' ! queue ! waylandsink use-drm=true

6. Same use case as previous but the ducati decoder element is added in the pipeline. As you could see in this case the pipeline is split after the parser.

gst-launch-1.0 -e videotestsrc ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! queue ! ducatijpegenc ! jpegparse ! tee name=t ! queue ! filesink location=1.mjpeg t. ! queue ! ducatijpegdec ! vpe ! 'video/x-raw,format=(string)NV12,width=800,height=480' ! queue ! kmssink

The pipelines are tested for waylandsink and kmssink.
Hope this helps.

BR
Margarita

0 Tom W over 6 years ago in reply to Margarita Gashova

Expert 1280 points

That looks like it did it! I added our hardware and it doesn't appear to segfault any longer. Issues still persist with Qt but that's out of scope with this post. Thank you Margarita for sticking it out

0 Margarita Gashova over 6 years ago in reply to Tom W

TI__Guru* 80190 points

Hello,

Great!
But for QT issue I would recommend you to open a new thread.

BR
Margarita

Processors

Processors forum

Linux/AM5728: Ducatijpegdec GStreamer Plugin adds 8 extra lines of decoded image, produces incompatibility issue with VPE