Linux/AM5728: Cameras and SDK error

The software team have been notified.

We expect to dig into this further after mid September.

Are you trying to run 9 instances of the decoder? The decoder memory comes from TILER pool and it is limited to 128 MBytes (hardware restrictions) . It seems like the pipeline is running out of the tiler memory.

From customer:

We use PROCESSOR-SDK-LINUX-AM57X 04_00_00_04 which is suitable for this module. There is a file named am57xx-evm-linux-sdk-bin-04.00.00.04.tar.xz which consist of AM57xx Linux SDK prebuilt BSP binaries and root filesystem. When we use this reference root filesystem we observe strange behaviour while using Gstreamer for playing H264 video streams from IP camera.

Our steps were as follows:
1) We stop the "Matrix" application after the system boots up.
2) We launch weston-terminal and we run several Gstreamer pipelines in this format:

gst-launch-1.0 playbin uri="rtsp://192.168.35.212/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=1024x768&fps=25" video-sink=waylandsink

3) In random interval from the begining one or more streams stop working (freeze on one frame). Three or four streams launched by pipeline above are enough for reproducing the problem. I can sometimes even force this stuck by simply getting focus of a window with particular camera stream (this usually stucks).

Our observations:
- There are no Gstreamer error messages when a video get stuck.
- The problem is similar when using different resolution and/or fps.
- Camera is AXIS P3905 and it seems there is no problem with connection and camera itself.
- Based on analysis of tcpdump utility output it seems just "image" is freezed because network data stream is still active.
- I also tried to compile the newest kernel and SGX driver from TI GIT repo (4.9.47) but the problem remains.
git://git.ti.com/processor-sdk/processor-sdk-linux.git : ti-lsk-linux-4.9.y
- So far I did not observe this problem when I use H264 video file instead of camera live stream (i.e. uri=file://<file> in pipeline)

Dear Support Team

any feedback on this?

Bartosz,

I apologize for the delay. Our key SW engineer for this question, Manisha, is currently out of the office. Manisha returns to the office next week and we can work on this issue with you then.

Does this issue only occur when running multiple streams and not for single instance of  decoding and displaying using playbin?

Did you try creating the gstreamer pipeline by using individual elements and not using playbin? What happens if you use fakesink or use h264avdec plugin rather using waylandsink or ducatih264dec?

What is meant by getting focus of a window here " I can sometimes even force this stuck by simply getting focus of a window with particular camera stream (this usually stucks)." ?

Manisha,

thank you for your support. Firstly, please see answers to your questions with additional comments from customer. Secondly we have a new question regarding H264 video decoding.

1) Regarding your questions

a)      Does this issue only occur when running multiple streams and not for a single instance of decoding and displaying using playbin?
No, it does not. I observed this issue for a single instance as well.

b)      Did you try creating the gstreamer pipeline by using individual elements and not using playbin?
Yes I did, but I found out some new information within gstreamer elements. Please see below.
 
c)      What happens if you use fakesink or use h264avdec plugin rather using waylandsink or ducatih264dec?
Fakesink does not help, because there are no gstreamer errors as I said. Just an image get stuck. avdec_h264 works properly as far as I can tell, but ther is a big CPU overhead.

d)      Can you indicate what do you mean by getting focus of a window, as per "I can sometimes even force this stuck by simply getting focus of a window with particular camera stream (this usually stucks)." ?
Just clicking on a video window or dragging this window to a different position on a screen.

Regarding b):
I found out that the result is the same in case of simple connection of ducatih264dec source to waylandsink sink. However, when I use VPE postprocessing with the same resolution as in the request for camera stream (1024x768) then it works properly as far as I can tell. Further, I used verbose mode within the same gstreamer pipelines. I found out that in first case it passes a "wrong" size/resolution to waylandsink (1152x864) although final video window seems to have right dimensions (1024x768). Example:
 
Common variable for a shorter form> GST_COMMON='gst-launch-1.0 rtspsrc location="rtsp://192.168.35.212/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=1024x768&fps=25" ! rtph264depay ! h264parse'

********************************
Wrong behaviour:
$GST_COMMON ! ducatih264dec ! waylandsink

/GstPipeline:pipeline0/GstDucatiH264Dec:ducatih264dec0.GstPad:sink: caps = "video/x-h264\,\ stream-format\=\(string\)byte-stream\,\ alignment\=\(string\)au\,\ level\=\(string\)4.1\,\ profile\=\(string\)main\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ width\=\(int\)1024\,\ height\=\(int\)768\,\ framerate\=\(fraction\)0/1\,\ parsed\=\(boolean\)true"
/GstPipeline:pipeline0/GstDucatiH264Dec:ducatih264dec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1152\,\ height\=\(int\)864\,\ framerate\=\(fraction\)0/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)12"
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1152\,\ height\=\(int\)864\,\ framerate\=\(fraction\)0/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true" 

********************************
"Correct" behaviour:
$GST_COMMON ! ducatih264dec ! vpe ! 'video/x-raw, width=1024, height=768' ! waylandsink

/GstPipeline:pipeline0/GstDucatiH264Dec:ducatih264dec0.GstPad:sink: caps = "video/x-h264\,\ stream-format\=\(string\)byte-stream\,\ alignment\=\(string\)au\,\ level\=\(string\)4.1\,\ profile\=\(string\)main\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ width\=\(int\)1024\,\ height\=\(int\)768\,\ framerate\=\(fraction\)0/1\,\ parsed\=\(boolean\)true"
/GstPipeline:pipeline0/GstDucatiH264Dec:ducatih264dec0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1152\,\ height\=\(int\)864\,\ framerate\=\(fraction\)0/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true\,\ max-ref-frames\=\(int\)12"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1152\,\ height\=\(int\)864\,\ framerate\=\(fraction\)0/1\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ drm_mem\=\(boolean\)true"
/GstPipeline:pipeline0/GstVpe:vpe0.GstPad:src: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1024\,\ height\=\(int\)768\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)0/1"
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1024\,\ height\=\(int\)768\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)0/1"
********************************

Customer questions - Why would I ever need to call VPE postprocessing when there is no need to scale video? And in case it is necessary, what are limitations? Now it looks more like a bug.


2) Regarding H264 decoding

We use Debian "jessie" armhf and I want to compile gst-ducati with software from repositories by myself. I installed others gst-plugins from standard repositories. I downloaded source code of ipcdev, libdce2 and gst-plugin-ducati from TI git repositories. After several hacks I was able to compile, link and finally launch gstreamer with gst-ducati-plugin. I used this pipeline: 

gst-launch-1.0 rtspsrc location="rtsp://192.168.35.212/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=640x480&fps=24" ! rtph264depay ! h264parse ! ducatih264dec ! vpe ! 'video/x-raw, format=YUY2, width=640, height=480' ! xvimagesink

The problem is that this pipeline seems to work just once per 10 or 20 launching approximately. Usually it fails with these messages (set GST_DEBUG variable):
--------------------------------- start ----------------------------------------
Setting pipeline to PAUSED ...
 >> dce_init
Open omapdrm device
 >> dce_ipc_init
open(/dev/rpmsg-dce) -> 0x150780
>> Engine_open Params::name = ivahd_vidsvr size = 12
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.35.212/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=640x480&fps=24
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
0:00:00.613931977   499 0xb500ed80 WARN         rtpjitterbuffer rtpjitterbuffer.c:185:rtp_jitter_buffer_set_clock_rate: Clock rate changed from 0 to 90000
0:00:00.625204614   499 0xb5032e60 WARN                  ducati gstducatih264dec.c:402:gst_ducati_h264dec_set_sink_caps:<ducatih264dec0> num-reorder-frames not found on caps, calculation from stream parameters gives 16
0:00:00.625272609   499 0xb5032e60 WARN                  ducati gstducatih264dec.c:410:gst_ducati_h264dec_set_sink_caps:<ducatih264dec0> Using 16 frames for reordering
0:00:00.632206112   499 0xb5032e60 WARN                  ducati gstducatividdec.c:552:codec_process:<ducatih264dec0> err=-1, extendedError=00208001
0:00:00.632252960   499 0xb5032e60 ERROR                 ducati gstducati.c:63:gst_ducati_log_extended_error_info: Bit 0 (00000001): no error-free slice
0:00:00.632283379   499 0xb5032e60 ERROR                 ducati gstducati.c:63:gst_ducati_log_extended_error_info: Bit 15 (00008000): fatal
0:00:00.632310219   499 0xb5032e60 ERROR                 ducati gstducati.c:63:gst_ducati_log_extended_error_info: Bit 21 (00200000): unsupported resolution
0:00:00.632718838   499 0xb5032e60 WARN                  ducati gstducatividdec.c:572:codec_process:<ducatih264dec0> changing max-ref-frames in caps to 18
---------------------------------- end -----------------------------------------

In case the pipeline is launched successfully (rarely) then a video stream is correctly displayed. Behaviour is the same while changing fakesink instead of xvimagesink. Decoder and postprocessing (VPE) seems to work properly in case the pipeline is launched successfully - CPU usage is about 4% while using the pipeline with fakesink and almost static scene for a camera.  Do you know where the problem might be?

Thank you.

Hi Bartosz,,

Thank you for the answers to my questions. We shall look into it and get back around 1st week of Oct.

Can you please open another e2e ticket for H.264 decoding issue? Let's not clutter one ticket with multiple issues. Search become difficult for other customers having same questions.

Regards,
Manisha

Hi Bartosz,

Meanwhile can you ask customer to include queue elements in the pipeline. Some queues could be full  or src element start dropping frames causing the deadlock.

As well include the gst-debug. Also can they generate the stream with base profile or other profiles with no B frames and test it?

Common variable for a shorter form> GST_COMMON='gst-launch-1.0 -v --gst-debug=2 rtspsrc location="rtsp://192.168.35.212/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=1024x768&fps=25" ! queue ! rtph264depay ! queue ! h264parse ! queue '

********************************

$GST_COMMON ! queue ! ducatih264dec ! queue ! waylandsink

Manisha,

customer adjusted the pipeline upon your request and tested it again both with Main profile and with Baseline profile. Unfortunately, the result is the same. You can find log files with output messages below:

/cfs-file/__key/communityserver-discussions-components-files/791/gst_5F00_h264_5F00_stuck_5F00_main.log

/cfs-file/__key/communityserver-discussions-components-files/791/gst_5F00_h264_5F00_stuck_5F00_baseline.log

Hi Manisha and Team

please see the latest feedback from customer:

"What our aim was:
- To have standard general purpose distribution (Debian 8) with Xserver.
- We have several applications tunned for this configuration for another HW platform and we supposed to use them with minimal changes on the AM572X if it is possible.
- We use mainly OpenGL + GLU + GLX; SFML, GTK+.

Finding about AM572x:
- There is no Xserver support for SGX544 by TI/PowerVR.
- Xserver - We used GC320 - FOSS kernel driver Etnaviv + xserver video driver which implements Xv extension. Thus, it is possible to use xvimagesink in Gstreamer.
- Mesa/OpenGL DRI (GC320) - imposibble to use with libGL, just SW rendering.
- gst-plugin-ducati partially functional with some hacks (ducatih264dec + VPE). The problem is that gstreamer pipeline seems to work just once per 10 or 20 launching approximately. In case the pipeline (ducatih264dec + VPE + xvimagesink) is launched successfully (rarely) then a video stream is correctly displayed, but there is still quite high CPU overhead. I suppose it is because of buffer copying between decoder+VPE => xvimagesink.
- Testing Wayland/Weston from debian repositories and PVR libraries was not successful because of libraries incompatibility, apparently.
- Overall, this approach seems to be dead-end.

Recommended solution?
- Linux distribution - now Debian 8 (armhf) => move to Arago project?
- 2D graphics app - now OpenGL + GLU => re-write to OpenGLES?
- graphics backend - now SFML and GTK+ => re-write to Qt5?
- video image manipulation (scaling, rotation, flip, flop) - now opengl textures (glimagesink) manipulation => waylandsink + something or "glesimagesink"?
"

Hi Manisha,

thanks for your great support and for the hint with IVI shell. We have a couple of questions regarding your information:

1) Does the Qt5 from the PLSDK rootfs have ivi-application protocol support?

2) Is it possible to render gstreamer video (waylandsink) into Qt widget?

We found it might be possible with similar snippet:

QPlatformNativeInterface *native = QGuiApplication::platformNativeInterface();

struct wl_surface *surface = static_cast<struct wl_surface *>(native->nativeResourceForWindow("surface", this->windowHandle()));

 

gst_video_overlay_set_window_handle(

            GST_VIDEO_OVERLAY (GST_MESSAGE_SRC (msg)),

            (guintptr) surface

        );

gst_video_overlay_set_render_rectangle(

            GST_VIDEO_OVERLAY (GST_MESSAGE_SRC (msg)),

            rect.x(), rect.y(),

            rect.width(), rect.height()

        );

 

Do you have any experience with this functionality?

Thank you in advance.

Regards,

Bartosz

Hi Bartoz,
We are not able to rootcause yet why introducing vpe will note reproduce the issue.
Can you check if the issue is reproducible when you don't add any capsfilter after vpe element in the pipeline?

Ram

Bartosz Marcinkowski said:

1) Does the Qt5 from the PLSDK rootfs have ivi-application protocol support?

QT application can also run on ivi shell. We have not tested it though using PLSDK. Please try at your end and let us know how it goes.

Bartosz Marcinkowski said:

2) Is it possible to render gstreamer video (waylandsink) into Qt widget?

We don't have any experience with this functionality.

"I noticed there was released a new SDK (PROCESSOR-SDK-LINUX-AM57X  04_01_00_06). I downloaded a new refecence root filesystem. It should include patch solving Waylandsink issue which you mentioned. Unfortunatelly, bad behaviour has remained.

Bad behaviour is also observed while using pipeline with VPE without capsfilter:
gst-launch-1.0 -v rtspsrc latency=1000 location="rtsp://192.168.35.212/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=1024x768&fps=25" ! rtph264depay ! h264parse ! ducatih264dec ! vpe ! waylandsink

You can observe that Waylandsink sink has wrong width and height again:
/GstPipeline:pipeline0/GstWaylandSink:waylandsink0.GstPad:sink: caps = "video/x-raw\,\ format\=\(string\)NV12\,\ width\=\(int\)1152\,\ height\=\(int\)864\,\ pixel-aspect-ratio\=\(fraction\)1/1\,\ framerate\=\(fraction\)0/1"

Are we able to reproduce this problem with the IP camera on our end?

Regards,

Bartosz

Bartosz Marcinkowski said:

I downloaded a new refecence root filesystem. It should include patch solving Waylandsink issue which you mentioned. Unfortunatelly, bad behaviour has remained.

Processor SDK Linux picks kernel release for integration that has happen 6 weeks before. So any kernel patch that gets applied in six week duration before that quarter end misses integrating into PLSDK release. Waylandsink issue fix patch missed the integration cycle. Please apply it offline and continue with the testing.

Bartosz Marcinkowski said:

Bad behaviour is also observed while using pipeline with VPE without capsfilter:
gst-launch-1.0 -v rtspsrc latency=1000 location="rtsp://192.168.35.212/axis-media/media.amp?videocodec=h264&h264profile=main&resolution=1024x768&fps=25" ! rtph264depay ! h264parse ! ducatih264dec ! vpe ! waylandsink

With capsfilter it works well??

Bartosz Marcinkowski said:

Are we able to reproduce this problem with the IP camera on our end?

We don't have setup to test with IP camera.

Hi Manisha, Team,

we are still stuck with AM57XX, because positioning of video windows is crucial feature in this application and our customer is not able to do that in any way.

Please see below feedback and provide us any guidelines to move forward.

Thanks.

"In Weston with default desktop shell backend, it is not possible to position windows from script at all. Despite the fact that with IVI shell it is possible to achieve this feature, launching Gstreamer with Waylandsink ends with Segmentation fault. Do you have any progress with this issue on your end?

In case we would not be able to position windows with HW accelerated video, we cannot use AM57XX. If you have any suggestion, please send it. "

On top, regarding previous questions:

"Our setup consists of TI HW (EVM) + TI SW (SDK) within this issue (Gstreamer Waylandsink + IVI shell). So could you reproduce it?

"Waylandsink issue fix patch missed the integration cycle. Please apply it offline and continue with the testing."  

As far as I know Waylandsink patch you provided is just "update" of userspace gstreamer plugin and has nothing to do with Linux kernel. Is that correct?

Software manifest for the latest SDK release reveals, that gstreamer1.0-plugins-bad-waylandsink is installed in version 1.8.3-r5.0 where 1.8.3 is version of particular SW and r5.0 is version of TI recipe which apply additional patches and so on. According to Arago GIT repository the patch should be already applied in the gstreamer1.0-plugins-bad-waylandsink which is installed in the latest version of SDK. But maybe I am wrong?

Hi Bartosz,

I lost your previous post and not able to get much history with waylandsink issue. It's no longer the part of this e2e thread. There was an attempt to split the post to create a new thread as the issue was different then what this thread was originally created. Can you either please open a new thread with subject as gstreamer with wayland sink and ivi shell. Please post previous question and above one in the newly created e2e post.

Hi Manisha,

posted another thread as per your guidelines:
e2e.ti.com/.../635755

Please help addressing the concern as soon as possible.

Thank you.