AM62A7: How to apply the self-developed gstreamer plugin before the "tiovxisp" plugin

peng qiang

Intellectual 310 points

Part Number: AM62A7

Tool/software:

Hi experts

We want to develop our own plugins to implement the application of some simple algorithms.

The specific process is as follows：

1.According to official information from gstreamer（gitlab.freedesktop.org/.../lv2）,built a self-developed plugin.

2.Then apply the plugin to the normal pipeline；

3.An error occurred.Please refer to the attachment for details；

Fullscreen pipeline.txt Download

1.normal pipeline

gst-launch-1.0 \
v4l2src device=/dev/video3 io-mode=5 ! \
video/x-bayer, width=1920, height=1280, format=bggr12, framerate=60/1 ! \
plugin ! \
tiovxisp sensor-name=SENSOR_OX3C \
dcc-isp-file=/opt/imaging/ox03c/dcc_viss_wdr.bin \
sink_0::dcc_2a_file=/opt/imaging/ox03c/dcc_2a_wdr.bin sink_0::device=/dev/v4l-subdev2 format-msb=11 \
sink_0::pool-size=2 src::pool-size=2 ! \
tiovxldc dcc-file=/opt/imaging/ox03c/dcc_ldc_wdr.bin sensor-name=SENSOR_OX3C \
sink::pool-size=2 src::pool-size=2 ! \
video/x-raw, format=NV12, width=1920, height=1280,framerate=60/1 ! queue ! \
kmssink driver-name=tidss async=false sync=false src-x=390 src-y=130 src-w=1290 src-h=770 can-scale=false

2.add self-developed gstreamer plugin

gst-launch-1.0 \
v4l2src device=/dev/video3 io-mode=5 ! \
video/x-bayer, width=1920, height=1280, format=bggr12, framerate=60/1 ! \
ISPAlgorithm ! \
tiovxisp sensor-name=SENSOR_OX3C \
dcc-isp-file=/opt/imaging/ox03c/dcc_viss_wdr.bin \
sink_0::dcc_2a_file=/opt/imaging/ox03c/dcc_2a_wdr.bin sink_0::device=/dev/v4l-subdev2 format-msb=11 \
sink_0::pool-size=2 src::pool-size=2 ! \
tiovxldc dcc-file=/opt/imaging/ox03c/dcc_ldc_wdr.bin sensor-name=SENSOR_OX3C \
sink::pool-size=2 src::pool-size=2 ! \
video/x-raw, format=NV12, width=1920, height=1280,framerate=60/1 ! queue ! \
kmssink driver-name=tidss async=false sync=false src-x=390 src-y=130 src-w=1290 src-h=770 can-scale=false

3.Error message

root@am62axx-evm:/usr/local/apd/scripts# gst-launch-1.0 \
> v4l2src device=/dev/video3 io-mode=5 ! \
> video/x-bayer, width=1920, height=1280, format=bggr12, framerate=60/1 ! \
> ISPAlgorithm ! \
> tiovxisp sensor-name=SENSOR_OX3C \
> dcc-isp-file=/opt/imaging/ox03c/dcc_viss_wdr.bin \
> sink_0::dcc_2a_file=/opt/imaging/ox03c/dcc_2a_wdr.bin sink_0::device=/dev/v4l-subdev2 format-msb=11 \
> sink_0::pool-size=2 src::pool-size=2 ! \
> tiovxldc dcc-file=/opt/imaging/ox03c/dcc_ldc_wdr.bin sensor-name=SENSOR_OX3C \
> sink::pool-size=2 src::pool-size=2 ! \
> video/x-raw, format=NV12, width=1920, height=1280,framerate=60/1 ! queue ! \
> kmssink driver-name=tidss async=false sync=false src-x=390 src-y=130 src-w=1290 src-h=770 can-scale=false
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   894.402447 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   894.402588 s:  VX_ZONE_INIT:Enabled
   894.402615 s:  VX_ZONE_ERROR:Enabled
   894.402672 s:  VX_ZONE_WARNING:Enabled
   894.403578 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0 
   894.404138 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1 
   894.404609 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2 
   894.405213 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3 
   894.405296 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
   894.406151 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: No downstream pool to import from.
Additional debug info:
../gst-plugins-good-1.20.7/sys/v4l2/gstv4l2object.c(5305): gst_v4l2_object_decide_allocation (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
When importing DMABUF or USERPTR, we need a pool to import from
Execution ended after 0:00:00.001277535
Setting pipeline to NULL ...
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../gstreamer-1.20.7/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)

(gst-launch-1.0:1397): GStreamer-Video-CRITICAL **: 18:11:34.918: gst_video_info_from_caps: assertion 'caps != NULL' failed
ERROR: from element /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0: Unable to init TIOVX module
Additional debug info:
../gst-libs/gst/tiovx/gsttiovxmiso.c(1512): gst_tiovx_miso_negotiated_src_caps (): /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0
Freeing pipeline ...
   894.455631 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
   894.459304 s:  VX_ZONE_INIT:[tivxDeInitLocal:204] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 0 alloc's of 0 bytes 
DDR_SHARED_MEM: Free's : 0 free's  of 0 bytes 
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

4.When we set io-mode to 2 and 4, it can run normally, but there will be frame drops.

May I ask if there are any other solutions for me to apply my self-developed plugin before the tivoxisp plugin when io mode=5.

over 1 year ago

0 Jianzhong Xu over 1 year ago

TI__Guru 59285 points

Hello Peng Qiang,

Using io-mode=5 (dmabuf-import) avoids buffer copy and therefore can achieve higher frame rate. Can you implement your plugin similar to tiovxisp: https://github.com/TexasInstruments/edgeai-gst-plugins/blob/main/ext/tiovx/gsttiovxisp.c.

Regards,

Jianzhong

0 peng qiang over 1 year ago in reply to Jianzhong Xu

Intellectual 310 points

Hi jianzhong

I don't quite understand what you're trying to express.

My understanding is as follows: I need to set the properties of my self-developed plugin according to the tiovlisp plugin.

If so, which parameters are necessary?

+1 Jianzhong Xu over 1 year ago in reply to peng qiang

TI__Guru 59285 points

Hi Peng Qiang,

Sorry for not being clear. Let me reach out to our dev team for more instructions.

Meanwhile, please take a look at how this plugin is implemented: https://github.com/TexasInstruments/edgeai-gst-plugins/blob/main/ext/ti/gsttidlinferer.cpp.

I'll get back to you in a day or so.

Regards,

Jianzhong

0 peng qiang over 1 year ago in reply to Jianzhong Xu

Intellectual 310 points

Hi jianzhong

May I ask when there will be a response.

0 Rahul Ravikumar over 1 year ago in reply to peng qiang

TI__Expert 4238 points

Hi Peng Qiang,

For io-mode=5 to work, your plugin needs to implement a buffer pool, so that v4l2src can import and use the buffers
Which base class are you using in your plugin?

You can refer to below code to add a bufferpool in your plugin

https://github.com/TexasInstruments/edgeai-gst-plugins/blob/2717228b0a409a966b51530abbe7fe4b8d929f9d/ext/ti/gsttidlinferer.cpp#L772
https://github.com/TexasInstruments/edgeai-gst-plugins/blob/2717228b0a409a966b51530abbe7fe4b8d929f9d/ext/ti/gsttidlinferer.cpp#L847

Regards
Rahul T R

0 peng qiang over 1 year ago in reply to Rahul Ravikumar

Intellectual 310 points

Rahul Ravikumar said:
Which base class are you using in your plugin?

Hi Rahul

base class:GSTElement

I have reviewed the code and found that the GST_TIOVX_SBUFFER-POOL function is used.

Do I need to change the parent class in the plugin?

What source files do I need to add if I change the parent class?

0 Rahul Ravikumar over 1 year ago in reply to peng qiang

TI__Expert 4238 points

Hi Peng Qiang,

If your plugin is 1 input, 1 output I suggest using BaseTransform similar to this plugin
https://github.com/TexasInstruments/edgeai-gst-plugins/blob/main/ext/ti/gsttidlinferer.cpp.

Regards
Rahul T R

0 peng qiang over 1 year ago in reply to Rahul Ravikumar

Intellectual 310 points

Hi Rahul

This plugin is related to deep learning, my plugin is not deep learning.

Let me explain my needs：

1. the plug-in should be inserted before tiovlisp;

2. how should I access the data stream pool when io mode=5.

0 Rahul Ravikumar over 1 year ago in reply to peng qiang

TI__Expert 4238 points

Hi Peng Qiang,

What I meant is use the gsttidlinferer.cpp. as template

Can you please let me know what is the input and output caps (data types)
of your plugin, I can provide a easier template to use

Regards
Rahul T R

0 peng qiang over 1 year ago in reply to Rahul Ravikumar

Intellectual 310 points

Rahul Ravikumar said:
What I meant is use the gsttidlinferer.cpp. as template

This cpp is related to TIDL, and I did not use deep learning. This template contains too much redundant content for me.

Rahul Ravikumar said:
Can you please let me know what is the input and output caps (data types)
of your plugin, I can provide a easier template to use

Both input and output are raw data types

0 peng qiang over 1 year ago in reply to Rahul Ravikumar

Intellectual 310 points

Hi Rahul

May I ask if there is a more intuitive solution?

0 Jianzhong Xu over 1 year ago in reply to peng qiang

TI__Guru 59285 points

Hi Peng Qiang,

We're implementing a reference plugin with raw input and raw output which can support dmabuf-import. It should be ready in a couple of days.

Regards,

Jianzhong

0 Rahul Ravikumar over 1 year ago in reply to Jianzhong Xu

TI__Expert 4238 points

Hi Peng Qiang,

Here is the reference plugin you can use
ext/ti: Add a reference plugin for ISP preproc · TexasInstruments/edgeai-gst-plugins@492caa4 (github.com)

This takes a raw image input and just copies same to output

Regards
Rahul T R

0 peng qiang 11 months ago in reply to Rahul Ravikumar

Intellectual 310 points

Hi Rahul Ravikumar

May I ask where “GstTIISPreprocClass” is defined?

0 Rahul Ravikumar 11 months ago in reply to peng qiang

TI__Expert 4238 points

Hi Peng Qiang,

It is defined using these macros
https://github.com/TexasInstruments/edgeai-gst-plugins/blob/492caa40d5167303cb89f76d3292dc80b530e880/ext/ti/gsttidlpreproc.h#L82
https://github.com/TexasInstruments/edgeai-gst-plugins/blob/492caa40d5167303cb89f76d3292dc80b530e880/ext/ti/gsttiisppreproc.c#L142

You can read about these macros in Gstreamer docs

Regards
Rahul T R

0 peng qiang 11 months ago in reply to Rahul Ravikumar

Intellectual 310 points

Hi Rahul Ravikumar

We want to transmit a parameter to the upstream element through the sink pad of tiovlisp element for processing in the next frame.

I found that tiovlisp element is a multi input element.

May I ask how to obtain the sink pad (GstPad structure) in tiovx element?

0 Rahul Ravikumar 11 months ago in reply to peng qiang

TI__Expert 4238 points

Hi Peng Qiang,

If I understand correctly, you want to pass some information from tiovxisp element to tipreproc element right?

Regards
Rahul T R

0 peng qiang 11 months ago in reply to Rahul Ravikumar

Intellectual 310 points

Hi Rahul T R

Yes,you are right.

I want to pass parameters from tiovx to tipreproc.

Tiovx is a multi input element. I cannot confirm which sink is receiving the data stream from the upstream element.

0 peng qiang 11 months ago in reply to Rahul Ravikumar

Intellectual 310 points

Hi Rahul T R

May I ask if there is any way to determine which sink the tiovlisp plugin is using ？

0 Rahul Ravikumar 11 months ago in reply to peng qiang

TI__Expert 4238 points

Hi Peng Qiang,

It is not possible to send data back via sink pad
You need to add a source pad to tiovxisp to send the custom
data and add another sink pad to tiovxisppreproc to receive data

Regards
Rahul T R

0 peng qiang 11 months ago in reply to Rahul Ravikumar

Intellectual 310 points

Hi Rahul T R

The gstreamer I learned allows the use of pads for transmitting data to transmit events.

Did you impose any restrictions?

Rahul Ravikumar said:
source pad to tiovxisp

tiispsproc is the upstream element；

tiovlisp is the downstream element;

How can events be transmitted from the src pad of downstream elements to the sink pad of upstream elements.

0 Rahul Ravikumar 11 months ago in reply to peng qiang

TI__Expert 4238 points

Hi Peng Qiang,

Events are used for specific purposes, like caps negotiation, bufferpool negotiation, state change etc..
It can not be used to transmit custom data

Regards
Rahul T R

Processors

Processors forum

AM62A7: How to apply the self-developed gstreamer plugin before the "tiovxisp" plugin