SK-AM62A-LP: Monochrome image processing capabilities

Hello,

The subject matter expert is currently out office. Please expect a couple days delay in response time.

-Daolin

Hi Jaimin,

Jaimin Patel said:

We are planning to use Auto white balance and Auto exposure functionality of ISP with monochrome image sensor.

What do you expect auto white balance to do for monochrome images which have no color? 

Regards,

Jianzhong

Hello,

As per further discussion Auto white balance will not impact on monochrome images so this feature is not required. Could you please provide information in remaining points as below:

1. Will Auto exposure functionality work with ISP on monochrome image sensor? 

2. If monochrome images (8/10 bit luma data) are provided to ISP then what will be the format of output processed image?

3. Can we convert monochrome image(8/10 bit Luma Data) in to JPEG using hardware JPEG encoder? As per datasheet it only supports YUV images at input. Doesn't it will work for monochrome images?

Thanks

Hello,

On execution of "gst-inspect-1.0 tiovxisp" we can see that ISP plugin only accepts bayer data in sink pad. Is there any patch available to pass GRAY8 or GRAY16 data to ISP?

Jianzhong Xu said:

No. Monochrome input is not supported by the JPEG HW accelerator.

We want to perform monochrome video streaming using MJPEG encoder. As per datasheet MJPEG support "8b 420 / 422 support". Does that mean we can't encode monochrome video using hardware encoder?  Let is know if we can do it through some conversion.

Thanks

Jaimin Patel said:

Is there any patch available to pass GRAY8 or GRAY16 data to ISP?

There is no patch supporting GRAY8/16. However, you can just use bayer format. The ISP processes four channels (RGGB, BGGR, etc) separately except for demosaicing. So if you pass a monochrome input, the pixels will be processed in four channels (groups) instead of a single one. That is fine.

Jaimin Patel said:

Let is know if we can do it through some conversion.

There is a GST plugin, tiovxdlcolorconvert, which can convert GRAY8 into NV12. For GRAY12, what you could do is to take the Y12 output from the ISP and provide dummy UV (all 0's) to the MJPEG encoder. Please note that the ISP outputs either Y8 or Y12, but no Y16.

Regards,

Jianzhong

Jianzhong Xu said:

u

As per comment I have run below pipeline in that, videotestsrc output: GRAY8 that pass in to input of tiovxisp.
in that I am getting this error and failed.

gst-launch-1.0 -v videotestsrc ! video/x-raw, width=1920, height=1080, format=GRAY8, framerate=30/1 ! tiovxisp ! video/x-raw, width=1920, height=1080, format=Y12 , framerate=30/1 ! fakesink

Error:
WARNING: erroneous pipeline: could not link videotestsrc0 to tiovxisp0, tiovxisp0 can't handle caps video/x-raw, width=(int)1920, height=(int)1080, format=(string)GRAY8, framerate=(fraction)30/1

Can you please suggest pipeline ?

Thanks.

Hi Jaimin,

Instead of using "GRAY8", you can just use "RGGB" or "BGGR". The ISP doesn't care about the color format until it does color processing. For you monochrome input, there is no color processing, so ISP will essentially just process the raw input pixels.

Regards,

Jianzhong

Hello Jianzhong,

Thank you for providing above information.

In our product we planned to use monochrome image sensor which will provide 8 or 10 bit RAW data. Our requirement is to stream MJPEG (Hardware Encoder) encoded monochrome video after ISP and LCD processing. Currently we don't have monochrome sensor and want to make sure that this functionality will work.

To confirm above thing we are using videotestsrc gstreamer plugin which provide data in GRAY8 format

Case 1: We have tried to stream MJPEG monochrome video from videotestsrc plugin using below command:

gst-launch-1.0 -v videotestsrc ! video/x-raw, width=1920,height=1080, format=GRAY8, framerate=30/1 ! tiovxdlcolorconvert ! video/x-raw, width=1920,height=1080, format=NV12,framerate=30/1 ! v4l2jpegenc ! fakesink

In this case pipeline is crashing with below error:
../gstreamer-1.20.7/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason not-negotiated (-4)

Case 2: Further we have tried to dump one raw video using below command:
gst-launch-1.0 -v videotestsrc ! video/x-raw, width=1920,height=1080, format=GRAY8, framerate=30/1 ! filesink location=gray8_video.raw

Now we are trying to execute below pipeline using recorded video and it is not working:
gst-launch-1.0 -v filesrc location=gray8_video.raw ! video/x-bayer, width=1920, height=1080, framerate=30/1, format=bggr ! tiovxisp sink_0::device=/dev/v4l-rpi-subdev0 sensor-name="SENSOR_SONY_IMX219_RPI" dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/imx219/linear/dcc_2a_10b.bin format-msb=9 ! video/x-raw, format=GRAY8, width=1920, height=1080, framerate=30/1 ! tiovxdlcolorconvert ! video/x-raw, format=NV12, width=1920, height=1080, framerate=30/1 ! v4l2jpegenc ! fakesink

Case 3: Also as per your input in above comment we tried to use below command and it is not working.

gst-launch-1.0 -v videotestsrc ! video/x-bayer width=1920, height=1080, format=bggr, framerate=30/1 ! tiovxisp ! video/x-raw, width=1920, height=1080, format=GRAY8 , framerate=30/1 ! fakesink

Could you please let us know what is problem with above pipelines?

Is tiovxdlcolorconvert plugin hardware accelerated? Will it perform color conversion on ISP?

Attached log file for  case 1 ,case 2 and case 3.



1030.Log.zip

Is MJPEG video streaming validated using "v4l2jpegenc" for any monochrome image sensor with AM62A34 processor? (Where image sensor provide 8/10 bit RAW data) Please provide used pipeline so we can order the image sensor and validate the same.

Thanks

Hello Jaimin,

I would recommend you to follow the AM6xA ISP Tuning Guide, section 6.2 Enable Camera Streaming With Initial VPAC Configuration, to generate a baseline ISP configuration for monochrome sensor. You can choose MONO as the color pattern.

You'll need to add a fake sensor to Gstreamer plugin as described in sections 5.3 Updating TIOVX Modules and 5.4 Update GStreamer Plug-in for VISS. Then you can run a pipeline similar to the one with IMX219.

You can also get an OV9281 camera (not expensive) and do the experiment.

Regards,

Jianzhong 

Hi Jianzhong,

Thank you for providing above information. We will create dummy config file for image sensor and try to execute pipeline. Also we are ordering OV9281 camera for testing.

We need to close our project architecture phase in couple of days and actual validation on OV9281 will take some time as we dont have this sensor currently. We need just one confirmation from your side so we can close our architecture phase.

Need Confirmation:
We can encode 1080p@60 stream from monochrome sensor using MJPEG hardware encoder ("v4l2jpegenc") and stream it with AM62A34 processor. (Where image sensor provide 8/10 bit RAW data and it is processed through ISP before passing to v4l2jpegenc)

Could you please provide confirmation on above point ? Also it would be helpful if you can send us validated gstreamer pipeline for this case with OV9281.

Is tiovxdlcolorconvert plugin hardware accelerated? Will it perform color conversion on ISP?

Thanks

Jaimin Patel said:

Need Confirmation:
We can encode 1080p@60 stream from monochrome sensor using MJPEG hardware encoder ("v4l2jpegenc") and stream it with AM62A34 processor. (Where image sensor provide 8/10 bit RAW data and it is processed through ISP before passing to v4l2jpegenc)

As I said earlier, monochrome input is not supported by the MJPEG HW accelerator. You'll to have use tiovxdlcolorconvert to convert GRAY8 into NV12. Alternatively, you can take the Y8/Y12 output from the ISP and provide dummy UV (all 0's) to the MJPEG encoder. This requires some custom work, but it should be doable.

Jaimin Patel said:

Is tiovxdlcolorconvert plugin hardware accelerated? Will it perform color conversion on ISP?

No, this plugin runs on A53 core.

Thanks Jianzhong Xu 

Jianzhong Xu said:

Alternatively, you can take the Y8/Y12 output from the ISP and provide dummy UV (all 0's) to the MJPEG encoder. This requires some custom work, but it should be doable.

For performance reason we are avoiding to use tiovxdlcolorconvert plugin which will run on the CPU. So we are trying to implement above approach.

Could you please provide some information how we can set UV values to 0 in Y8 output from ISP? Is there any provision in ISP itself to achieve this through some ISP configuration?

Jaimin Patel said:

Is there any provision in ISP itself to achieve this through some ISP configuration?

No, ISP won't do it.

Jaimin Patel said:

Could you please provide some information how we can set UV values to 0 in Y8 output from ISP?

When you provide the YUV input to MJPEG encoder, the YUV data must be in a certain format (which I'm not very familiar with). Just set U & V data to 0. 

Regards,

Jianzhong

Hi Jianzhong,

We have SK-AM62A-LP EVM and want to test OV9281 sensor with it as suggested in above conversation. Could you please provide purchase link for OV9281 module which will work with SK-AM62A-LP EVM without any hardware changes?

Does it require any code change on EVM? as I can not see OV9281 sensor support in firmware.

Thanks

Hi Jaimin,

I've tested this OV9281 camera on AM62x EVM, and believe it should work with AM62A EVM as well.

Jaimin Patel said:

I can not see OV9281 sensor support in firmware.

I used an OV9281 driver from another customer and won't be able to share it with you. The upstream Linux has a driver for OV9282 which should be compatible with OV9281. 

Regards,

Jianzhong

Hello,

We have checked below link and not able to place order as delivery is not possible in our region.

https://www.amazon.com/Raspberry-External-Monochrome-Bullseye-libcamera/dp/B09WTP5GZH/ref=sr_1_4?crid=2VKA2GBTXP0QI&keywords=ov9281&qid=1667241628&qu=eyJxc2MiOiIzLjA3IiwicXNhIjoiMi4xNyIsInFzcCI6IjEuNzgifQ%3D%3D&sprefix=ov9281%2Caps%2C103&sr=8-4&th=1

Alternatively we found the sensor on manufacture site https://www.inno-maker.com/product/cam-mipi9281raw-v2/

Will this image sensor work with AM62A EVM?

Thanks

That one should work.