Reading video or feed from imx 219 using gstreamer and opencv in a python code

Hi Sourabh,

Could you please share which SDK are you using?

Thank you,

Fabiana Jaimes

Hi Fabiana,

I am using Linux SDK for edge AI applications on TDA4VM Jacinto. Following is the documentation that I am following:

https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-edgeai/TDA4VM/09_00_00/exports/edgeai_docs/common/sdk_overview.html

I could run the pipeline for cv2 by making some changes. The pipeline is as follows:

Hi Sourabh,

I recommend taking a look at our GStreamer pipeline examples for imx219: https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-edgeai/TDA4VM/09_00_00/exports/edgeai_docs/common/edgeai_dataflows.html#id2 

I also recommend taking a look at the post-processing module as it uses OpenCV.

You are correct that tiovxcolorconvert does not currently support BGR so you will have to convert RGB to BGR for your use case. I will put in a requirement request for BGR support.

Thank you,

Fabiana Jaimes

Hi Fabiana,

I am using the pipeline from the link shared by you for imx 219 camera. I have been following that document for using the gstreamer pipeline with opencv. From that link only I could run gstreamer pipeline for reading h264 video in cv2. But for imx 219 pipeline, it is failing. Following is the pipeline from the documentation, which gives out 2 streams, one for model and other for drawing predicted results

v4l2src device=/dev/video-rpi-cam0 io-mode=5 pixel-aspect-ratio=None ! queue leaky=2 ! capsfilter caps="video/x-bayer, width=(int)1920, height=(int)1080, format=(string)rggb;" ! tiovxisp dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin sensor-name=SENSOR_SONY_IMX219_RPI ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler name=split_01
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers=2 drop=True name=sen_0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1168, height=(int)748;" ! tiovxmultiscaler target=1 ! capsfilter caps="video/x-raw, width=(int)416, height=(int)416;" ! tiovxdlpreproc out-pool-size=4 data-type=3 tensor-format=1 ! capsfilter caps="application/x-tensor-tiovx;" ! appsink max-buffers=2 drop=True name=pre_0

I am modifying it as I need only one stream to test my compiled model on the board. The edited pipeline is as follows:

v4l2src device=/dev/video-rpi-cam0 io-mode=5 pixel-aspect-ratio=None ! capsfilter caps="video/x-bayer, width=(int)1920, height=(int)1080, format=(string)rggb;" ! tiovxisp dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin  sensor-name=SENSOR_SONY_IMX219_RPI ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler !\
capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! videoconvert ! capsfilter caps="video/x-raw, format=(string)BGR;" ! appsink max-buffers=2 drop=True

The error is regarding the ISP I think, following is the error message

0:00:00.273941699  1598     0x370c2860 ERROR               tiovxisp gsttiovxisp.c:879:gst_tiovx_isp_read_2a_config_file:<tiovxisp0:sink_0> Unable to open 2A config file: (null)
0:00:00.273975626  1598     0x370c2860 ERROR               tiovxisp gsttiovxisp.c:1123:gst_tiovx_isp_init_module:<tiovxisp0> Unable to read 2a config file
0:00:00.273994578  1598     0x370c2860 ERROR              tiovxmiso gsttiovxmiso.c:1215:gst_tiovx_miso_modules_init:<tiovxisp0> Subclass init module failed

Kindly let me know, in case I am missing something or making some minor mistake.

Thanks in advance.

Hi Sourabh,

Is the imx219 camera being detected upon booting? Are you able to run simple OOB demo with imx219 as input?

Thank you,

Fabiana Jaimes

Hi Fabiana,

Yes, my IMX219 camera gets detected. Also, I can use it to run the edgeai-gst-demos provided by TI.

Hi Sourabh,

Here is a single IMX219 example stream: 

gst-launch-1.0 v4l2src device=/dev/video-rpi-cam0 ! video/x-bayer, width=1920, height=1080, format=rggb ! tiovxisp sink_0::device=/dev/v4l-subdev2 dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin \
sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin sensor-name=SENSOR_SONY_IMX219_RPI ! video/x-raw,format=NV12, width=1920, height=1080 ! kmssink sync=false

I hope this helps.

Thank you,

Fabiana Jaimes

Hi Fabiana,

The pipeline you shared is not working. Following is the error message.

APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   140.633889 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   140.633962 s:  VX_ZONE_INIT:Enabled
   140.633971 s:  VX_ZONE_ERROR:Enabled
   140.633980 s:  VX_ZONE_WARNING:Enabled
   140.634743 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
   140.635181 s:  VX_ZONE_INIT:[tivxHostInitLocal:96] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
ERROR: from element /GstPipeline:pipeline0/GstKMSSink:kmssink0: Could not open DRM module (NULL)
Additional debug info:
../gst-plugins-bad-1.20.6/sys/kms/gstkmssink.c(850): gst_kms_sink_start (): /GstPipeline:pipeline0/GstKMSSink:kmssink0:
reason: No such file or directory (2)
ERROR: pipeline doesn't want to preroll.
ERROR: from element /GstPipeline:pipeline0/GstKMSSink:kmssink0: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.
Additional debug info:
../gstreamer-1.20.6/libs/gst/base/gstbasesink.c(5878): gst_base_sink_change_state (): /GstPipeline:pipeline0/GstKMSSink:kmssink0:
Failed to start
ERROR: pipeline doesn't want to preroll.
Failed to set pipeline to PAUSED.
Setting pipeline to NULL ...
Freeing pipeline ...
   140.915418 s:  VX_ZONE_ERROR:[vxGetStatus:700] Reference is NULL
   140.915438 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:110] De-Initialization Done for HOST !!!
   140.919816 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 0 alloc's of 0 bytes
DDR_SHARED_MEM: Free's : 0 free's  of 0 bytes
DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

Hello,

Thanks for the query. The expert looking into this issue is current out-of-office for a  week. So there will be a delay in responding to your query. Request your patience till then.

Thanks.

Hi Praveen,

Thanks for the response. Awaiting for the solution from expert.

Regards,
Sourabh

Hello Sourabh,

Thank you for understanding. The expert is back this week and must be able to respond to your query.

Thanks.

Hi Sourabh,

Do you have a display connected to your board when you run the pipeline?

Thank you,

Fabiana Jaimes

Hi Fabiana,

Yes, I do have a display connected to the board. I run the demos given in the document (that we talked about in previous conversations), through edgeai_gst_apps python scripts by using sample IMX219 configuration file (which selects input stream, model and output steam). My goal was to run the gstreamer pipeline using opencv so that I can try out my custom models using IMX219. I could run the gstreamer pipeline for h264 video using opencv (given in sample videos). I have shared both the pipelines in previous replies. 

Thank you,
Sourabh

Hi Fabiana,

The above shared pipeline works. I can see the live stream on the display. Now, I will try to consume the same pipeline using cv2. Will update you on the same.

Thank you,

Sourabh

Hi Fabiana,

I modified the pipeline shared by you so that I can work with my custom models using cv2. Following is the working pipeline which can be consumed in python cv2.

gst_pipeline = 'v4l2src device=/dev/video-rpi-cam0 io-mode=5 ! queue ! video/x-bayer, width=1920, height=1080, format=rggb ! tiovxisp sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin format-msb=7 sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a.bin sink_0::device=/dev/v4l-rpi-subdev0 ! \
    video/x-raw,format=NV12, width=1920, height=1080 !  tiovxdlcolorconvert ! video/x-raw, format=RGB ! videoconvert ! video/x-raw, format=BGR ! queue ! appsink'

Thank you for the support, much appreciated!

Regards,
Sourabh