SK-AM62A-LP: tiovxisp getting failed to output GRAY8 data

Jaimin Patel

Part Number: SK-AM62A-LP

Tool/software:

Hello Team,

We are using below gstreamer pipeline for GRAY8 data. camera sensor provided monochrome data and that we are trying to pass in ISP using bayer format but ISP getting failed to output GRAY8 data.

Below pipeline failed for GRAY8 output in tiovxisp:

gst-launch-1.0 -v v4l2src device=/dev/video-rpi-cam0 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1200, framerate=60/1, format=rggb ! tiovxisp sink_0::device=/dev/v4l-rpi-subdev0 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/ar0235/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ar0235/dcc_2a.bin format-msb=9 ! video/x-raw, format=GRAY8, wid
th=1920, height=1200 ! fakesink

Working pipeline for NV12 output in tiovxisp:

gst-launch-1.0 -v v4l2src device=/dev/video-rpi-cam0 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1200, framerate=60/1, format=rggb ! tiovxisp sink_0::device=/dev/v4l-rpi-subdev0 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/ar0235/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ar0235/dcc_2a.bin format-msb=9 ! video/x-raw, format=NV12, wid
th=1920, height=1200 ! fakesink

Error log:

root@am62axx-evm:/usr/sbin#
root@am62axx-evm:/usr/sbin# gst-launch-1.0 -v v4l2src device=/dev/video-rpi-cam0 io-mode=dmabuf-import ! video/x-bayer, width=1920, height=1200, framerate=60/1, format=rggb ! tiovxisp sink_0::device=/dev/v4l-rpi-subdev0 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/ar0235/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/ar0235/dcc_2a.bin format-msb=9 ! video/x-raw, format=GRAY8, wid
th=1920, height=1200 ! fakesink
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=6) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
8935.369247 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
8935.369359 s: VX_ZONE_INIT:Enabled
8935.369374 s: VX_ZONE_ERROR:Enabled
8935.369389 s: VX_ZONE_WARNING:Enabled
8935.370185 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
8935.370316 s: VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-bayer, width=(int)1920, height=(int)1200, framerate=(fraction)60/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-bayer, width=(int)1920, height=(int)1200, framerate=(fraction)60/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB
/GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0.GstTIOVXIspPad:sink_0: caps = video/x-bayer, width=(int)1920, height=(int)1200, framerate=(fraction)60/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-bayer, width=(int)1920, height=(int)1200, framerate=(fraction)60/1, format=(string)rggb, interlace-mode=(string)progressive, colorimetry=(string)sRGB
/GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0.GstTIOVXMisoPad:src: caps = video/x-raw, format=(string)GRAY8, width=(int)1920, height=(int)1200
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, format=(string)GRAY8, width=(int)1920, height=(int)1200
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)1920, height=(int)1200
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, format=(string)GRAY8, width=(int)1920, height=(int)1200
8935.749007 s: VX_ZONE_ERROR:[ownReleaseReferenceInt:601] Invalid reference
ERROR: from element /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0: Unable to init TIOVX module
Additional debug info:
../git/gst-libs/gst/tiovx/gsttiovxmiso.c(1508): gst_tiovx_miso_negotiated_src_caps (): /GstPipeline:pipeline0/GstTIOVXISP:tiovxisp0
Execution ended after 0:00:00.370444909
Setting pipeline to NULL ...
Freeing pipeline ...
8936.051869 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ee95350 of type 00000816 at external count 1, internal count 0, releasing it
8936.051920 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=user_data_object_108) now as a part of garbage collection
8936.052065 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ee95570 of type 00000816 at external count 1, internal count 0, releasing it
8936.052092 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=user_data_object_109) now as a part of garbage collection
8936.052176 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ef352f8 of type 00000813 at external count 1, internal count 0, releasing it
8936.052202 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=object_array_111) now as a part of garbage collection
8936.052249 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ee959b0 of type 00000816 at external count 1, internal count 0, releasing it
8936.052272 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=user_data_object_112) now as a part of garbage collection
8936.052307 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ef35498 of type 00000813 at external count 1, internal count 0, releasing it
8936.052330 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=object_array_113) now as a part of garbage collection
8936.052364 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9eea3700 of type 00000817 at external count 1, internal count 0, releasing it
8936.052386 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=raw_image_114) now as a part of garbage collection
8936.052420 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ef35638 of type 00000813 at external count 1, internal count 0, releasing it
8936.052443 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=object_array_115) now as a part of garbage collection
8936.052478 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ee95bd0 of type 00000816 at external count 1, internal count 0, releasing it
8936.052500 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=user_data_object_116) now as a part of garbage collection
8936.052534 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9ef357d8 of type 00000813 at external count 1, internal count 0, releasing it
8936.052556 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=object_array_117) now as a part of garbage collection
8936.052590 s: VX_ZONE_WARNING:[vxReleaseContext:1144] Found a reference 0xffff9eec7708 of type 0000080f at external count 1, internal count 0, releasing it
8936.052613 s: VX_ZONE_WARNING:[vxReleaseContext:1146] Releasing reference (name=image_118) now as a part of garbage collection
8936.052768 s: VX_ZONE_INIT:[tivxHostDeInitLocal:115] De-Initialization Done for HOST !!!
8936.057370 s: VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
DDR_SHARED_MEM: Alloc's: 20 alloc's of 18823304 bytes
DDR_SHARED_MEM: Free's : 20 free's of 18823304 bytes
DDR_SHARED_MEM: Open's : 0 allocs of 0 bytes
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!
root@am62axx-evm:/usr/sbin#
root@am62axx-evm:/usr/sbin#
root@am62axx-evm:/usr/sbin#

Any idea why ISP failed to output GRAY8 ?

Thanks,
Jaimin

over 1 year ago

0 Jianzhong Xu over 1 year ago

TI__Guru 58955 points

Hello Jaimin,

GRAY8 output format is meant for IR processing by the ISP. Please refer to https://github.com/TexasInstruments/edgeai-gst-plugins/blob/main/ext/tiovx/gsttiovxisp.c#L1157.I believe that's why your pipeline with GRAY8 failed.

Regards,

Jianzhong

0 Jaimin Patel over 1 year ago in reply to Jianzhong Xu

Intellectual 650 points

Thanks for update Jianzhong.

0 Jaimin Patel 11 months ago in reply to Jaimin Patel

Intellectual 650 points

Hi Jianzhong,

ISP is accepting input video format as bayer. So we are passing monochrome data in ISP through RGGB format. However we are not able to get GRAY8 data from ISP. Is there any way available to fetch GRAY8 data from ISP?

If GRAY8 data is not possible from ISP then can we achieve below through any way?

Here received NV12(YUV) data contains image information in all three components (Y, U and V). However monochrome image should only have Y data. Is there any why such that ISP provide only Y data and U/V marked as 0?

Thanks,

0 Jaimin Patel 11 months ago in reply to Jaimin Patel

Intellectual 650 points

In addition to above comment we tried to dump ISP processed NV12 image from ISP and we observed that U and V components of YUV contains 0x80 in all bytes. Does these U and V component required to re-create image if we convert NV12 in to GRAY8 format?

0 Jianzhong Xu 11 months ago in reply to Jaimin Patel

TI__Guru 58955 points

Hi Jaimin,

Let me investigate if and how we can get GRAY8 output for monochrome input. Please ping me if you don't hear me in a couple of days.

Thanks,

Jianzhong

0 Jaimin Patel 10 months ago in reply to Jianzhong Xu

Intellectual 650 points

Hi Jianzhong,

Did you get a chance to check the way to get GRAY8 output from ISP? Additionally I found in below link indicates we can get GRAY8 and GRAY16 both data from tiovxisp plugin.

https://github.com/TexasInstruments/edgeai-gst-plugins/wiki/tiovxisp

But not sure why we are unable to get image data in GRAY8 and GRAY16 format. Please let us know.

Thanks

0 Jianzhong Xu 10 months ago in reply to Jaimin Patel

TI__Guru 58955 points

Hi Jamin,

We don't have the API to output Y data only without U&V. You can try to use the gst plugin "videoconvert" to convert NV12 to GRAY8.

Jaimin Patel said:
Additionally I found in below link indicates we can get GRAY8 and GRAY16 both data from tiovxisp plugin.

That is meant for RGB-IR data processing.

Regards,

Jianzhong

0 Jaimin Patel 10 months ago in reply to Jianzhong Xu

Intellectual 650 points

Hi Jianzhong,

Jaimin Patel said:
In addition to above comment we tried to dump ISP processed NV12 image from ISP and we observed that U and V components of YUV contains 0x80 in all bytes. Does these U and V component required to re-create image if we convert NV12 in to GRAY8 format?

All data bytes in U and V components contains 0x80 value when we pass monochrome input to ISP. Is this image data? Or it is dummy and we can ignore them?

Jianzhong Xu said:
That is meant for RGB-IR data processing.

Does that mean we can not fetch GRAY8/GRAY16 data from ISP while providing input format in monochrome?

Thanks

0 Jianzhong Xu 10 months ago in reply to Jaimin Patel

TI__Guru 58955 points

Jaimin Patel said:
All data bytes in U and V components contains 0x80 value when we pass monochrome input to ISP. Is this image data? Or it is dummy and we can ignore them?

What did you configure the ISP to process for the monochrome data?

Jaimin Patel said:
Does that mean we can not fetch GRAY8/GRAY16 data from ISP while providing input format in monochrome?

Yes, GRAY8 output is only available for RGB-IR input.

Regards,

Jianzhong

0 Jianzhong Xu 10 months ago in reply to Jianzhong Xu

TI__Guru 58955 points

Hello Jaimin,

Jaimin Patel said:
All data bytes in U and V components contains 0x80 value when we pass monochrome input to ISP. Is this image data? Or it is dummy and we can ignore them?

U and V components being 0x80 means this is mono chrome data. To confirm, you can capture a raw input image and YUV output image. They should be very close to each other.

To get GRAY8 output for mono chrome input, the SoC is capable, but the SW is not ready yet. We'll need to update the Gstreamer plugin gsttiovxisp and the underlining TIOVX. We currently don't have a plan to do that. In your application, I hope you can read Y data only from the YUV output buffer.

Regards,

Jianzhong

Processors

Processors forum

SK-AM62A-LP: tiovxisp getting failed to output GRAY8 data