SK-AM62A-LP: apps_python framerate question

Hi Sajan,

The assigned topic expert is currently out of office and will return on September 2nd. Please expect a delay in response. Thank you for your patience.

Regards,
Johnson

Hi Sajan,

Using apps_python on the AM62A should imply that you are running analytics on the board. In that case, the inference would limit the framerate of the output. In case of the app crashing, could you share the yaml config?

Best Regards,
Jay

Hello Jay,

Jay Goyal said:

could you share the yaml config

title: "Preview"
inputs:
    input0:
        source: /dev/video-imx219-cam0
        subdev-id: /dev/v4l-imx219-subdev0
        format: rggb
        width: 1920
        height: 1080
        framerate: 15
models:
    model0:
        model_path: /opt/model_zoo/samplemodel
        topN: 1
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
flows:
    flow0: [input0,model0,output0]

Jay Goyal said:

would limit the framerate of the output

But In my finding, seems no change affects when I modify the input and output frame rate. It will record the video in 20fps.

Warm Regards,
Sajan

Hi Sajan,

It seems that the issue is with the pipeline that apps_python constructs. The gstreamer nodes used in those don't respect the framerate and send data at the maximum possible framerate. If limiting the framerate is important, a videorate element can be added. This will create the issue of duplication of frames if the framerate dips. For example, if you have set the framerate to 30 and it dips to 26, videorate will force the pipeline to 30fps by duplicating some frames.

If this satisfies your requirements, here is the corresponding patch (SDK 11.01):

--- gst_wrapper.py
+++ gst_wrapper.py
@@ -1198,6 +1198,8 @@

     element = make_element("appsrc", property=property)
     post_proc_elements += element
+    element = make_element("videorate", property={"max-rate": int(eval(flow.input.fps))})
+    post_proc_elements += element

     caps = "video/x-raw, format=NV12, width=%d, height=%d" % (
         flow.sensor_width,

Nevertheless, I have raised a bug report and will keep an eye on its progress.

Best Regards,
Jay

Hello Jay,

I tried updating the gst_wrapper.py with the patch you gave. But It doesn't works for me. When the app starts green screen cames and seems the below error.

libtidl_onnxrt_EP loaded 0x43b6680 
Final number of subgraphs created are : 1, - Offloaded Nodes - 124, Total Nodes - 124 
APP: Init ... !!!
   697.510398 s: MEM: Init ... !!!
   697.510479 s: MEM: Initialized DMA HEAP (fd=5) !!!
   697.510672 s: MEM: Init ... Done !!!
   697.510704 s: IPC: Init ... !!!
   697.528775 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   697.533507 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   697.533663 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
   697.533679 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
   697.533690 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
   697.535168 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-0 
   697.535467 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-1 
   697.535760 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-2 
   697.536063 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-3 
   697.536100 s:  VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
   697.536131 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

v4l2src device=/dev/video-imx219-cam0 io-mode=5 pixel-aspect-ratio=None ! queue leaky=2 ! capsfilter caps="video/x-bayer, width=(int)1920, height=(int)1080, format=(string)rggb;" ! tiovxisp dcc-isp1
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1920, height=(int)1080;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffer0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1188, height=(int)668;" ! tiovxmultiscaler target=1 ! capsfilter caps="video/x-raw, width=(int)454, height=(int)256;" ! tiovxdlcolorconv0


==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=True format=3 block=True name=post_0 ! videorate max-rate=20 ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! kms6

^C[ERROR] GStreamer error: negotiation problem.
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!

Hi Sajan,

It seems that you need to set the appsrc property 'is-live' to true. I seem to have missed the same in the patch. Apologies for the same. Here is the updated patch:

--- gst_wrapper.py
+++ gst_wrapper.py
@@ -1188,6 +1188,7 @@
         "block": True,
         "do-timestamp": True,
         "name": flow.gst_post_sink_name,
+        "is-live": True,
     }

     for o in flow.outputs:
@@ -1197,6 +1198,8 @@
             break

     element = make_element("appsrc", property=property)
+    post_proc_elements += element
+    element = make_element("videorate", property={"max-rate": int(eval(flow.input.fps))})
     post_proc_elements += element

     caps = "video/x-raw, format=NV12, width=%d, height=%d" % (

If the issue still persists, please share your SDK version as well.

For reference, here is some info on live sources from gstreamer wiki: https://gstreamer.freedesktop.org/documentation/additional/design/live-source.html?gi-language=c.

Best Regards,
Jay

Hi Jay,

It is now working but seems framerate issue. When I make the input and output frame rate to 30, the frame rate becomes 3 to 5 fps. I got the almost correct frame rate at 20 and 24 fps.

Warm Regards,
Sajan

Hi Sajan,

I do see this issue. As a workaround, you can set a mosaic at 0,0 with 1920x1080 dimensions. This seems to make the latency better. The resulting flow(in yaml config) should look like this:

flow0: [input0, model0, output0, [0, 0, 1920, 1080]]

On my end, with this, I get around 22fps when framerate is set to 30 in test config. I would still suggest going with the base SDK code for getting the maximum performance because videorate does seem to have performance implications in general.

In the meantime, I will try to figure out where this issue actually comes from and figure out a more complete fix.

Best Regards,
Jay