This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM62A7-Q1: Question about adjusting parameters of RGB-IR sensor OV ox05B

Part Number: AM62A7-Q1

Dear Ti staff,

We currently have a valid driver for OV ox05B sensor, but only the RGB channel can be adjusted. Our goal is to add the function of configuring the exposure and gain of the IR channel.

After analyzing the existing driver code, our thought are as follows:

1. struct ox05b should be expanded with more v4l2_ctrl member, such as *ir_exposure.

struct ox05b {
	struct device *dev;

	struct clk *clk;
	u32 clk_rate;

	struct i2c_client *client;
	struct regmap *regmap;
	struct gpio_desc *reset_gpio;

	struct v4l2_subdev sd;

	struct media_pad pad;

	struct v4l2_mbus_framefmt format;
	struct v4l2_async_notifier notifier;

	struct v4l2_ctrl_handler ctrls;
	struct v4l2_ctrl *exposure;
	struct v4l2_ctrl *again;
	struct v4l2_ctrl *dgain;
	struct v4l2_ctrl *h_flip;
	struct v4l2_ctrl *v_flip;
	struct v4l2_ctrl *pixel_rate;

	u32 fps;

	/*
	 * Mutex for serialized access:
	 * Protect sensor module set pad format and start/stop streaming safely.
	 */
	struct mutex lock;

	bool streaming;
};

2. adding corresponding controls for ir_exposure etc.

/* Add new controls */
	ox05b->exposure = v4l2_ctrl_new_std(ctrl_hdr, &ox05b_ctrl_ops,
					     V4L2_CID_EXPOSURE, 1,
					     OX05B_EXPOSURE_MAX,
					     1, OX05B_EXPOSURE_DEFAULT);

	ox05b->again = v4l2_ctrl_new_std(ctrl_hdr, &ox05b_ctrl_ops,
					  V4L2_CID_ANALOGUE_GAIN, 0,
					  OX05B_AGAIN_MAX, 1,
					  OX05B_AGAIN_DEFAULT);

	ox05b->dgain = v4l2_ctrl_new_std(ctrl_hdr, &ox05b_ctrl_ops,
					  V4L2_CID_DIGITAL_GAIN, 0,
					  OX05B_DGAIN_MAX, 1,
					  OX05B_DGAIN_DEFAULT);

3. implement lower level driver, passing the values into corresponding registers and utilizing the group hold and AB mode switching mechaism of the sensor to control its exposure and etc in both RGB and IR frame.

The main question here is that the available control ops needed in v4l2_ctrl_new_std() seems to be insufficient, for example for exposure there is only V4L2_CID_EXPOSURE defined in v4l2_controls.h. Should we try to add more control op definitions to fulfill our requirement? 

As we are not familiar with the V4L2 framework, please provide us with some advice for this question.

By the way, we have also read your driver code for ov2312. The structure and methodology are very similiar to ox05B's , yet the driver for ov2312 can only configure the parameters for RGB frames.

Thank you for the support in advance and looking forward to your reply.

 

Huang Jingjie

  • Hello Jingjie,

    Our goal is to add the function of configuring the exposure and gain of the IR channel.

    This would require auto exposure algorithm for the IR channel, in addition to the driver changes that you proposed. I will need to check internally about both of these.

    Would you mind sharing why you need to change the exposure and gain of the IR channel?

    Regards,

    Jianzhong

  • Hello Jingjie,

    The main question here is that the available control ops needed in v4l2_ctrl_new_std() seems to be insufficient, for example for exposure there is only V4L2_CID_EXPOSURE defined in v4l2_controls.h. Should we try to add more control op definitions to fulfill our requirement? 

    That is correct. You would need to add a new control like "V4L2_CID_IR_EXPOSURE". Current V4L2 framework has a single ioctl for setting exposure/gain for each v4l-subdev. Our ov2312 driver uses a single v4l-subdev, and thus only supports a single ioctl for exposure/gain setting, which is used for RGB capture only. 

    In addition, you would need to update two additional modules:

    1. TI imaging module: you'll need to add an auto exposure algorithm for IR channel, which is not currently available. Current code: git.ti.com/.../TI_aaa_ae.c

    2. GStreamer plugin for tiovxisp: you'll need to update the wrapper which passes the exposure/gain settings provided by the imaging module to the sensor driver. Current code: github.com/.../gsttiovxisp.c

     Regards,

    Jianzhong

  • Hello Jianzhong,

    I just consulted my colleague about this question and here is the explaination:

    1. this RGB-IR sensor belongs to DMS. According to their experience, drivers would vary in postures and body sizes, causing troubles for the cognition algorithm as fixed exposure sensor in such condition may produce overexposure or underexposure images. If auto exposure can be achieved, the captured image would be more adequate.

    2. Normally sensors chosen for DMS don't have HDR feature, and in this case fixed exposure can easily lead to local overexposure.

    As for driver changes, recently I've read some materials about V4L2 framework, and had some questions:

    1. it seems that current framework didn't provide the ability to trace back to the corresponding v4l2_device when a v4l2_ctrl_handler is called. My initial assumption is that, since we are accessing RGB frames and IR frames via /dev/video2 and /dev/video3, we might be able to differ the source of the updated exposure and gain values by their source v4l2_device.

    2. My second thought is that we can assign different v4l2_device with different v4l2_ctrl_handler to distinguish them. But in Chapter 1.12.12 (https://www.kernel.org/doc/html/v4.10/media/kapi/v4l2-controls.html), it says we should not make two identical controls for two handlers. Thus the question circled back to the fact that v4l2_controls.h only provides limited controls definitions.

    Hope above explaination answered your question, and please help me with mine. 

    Regards,

    Huang Jingjie

  • Hi Jingjie,

    Thank you for providing the information. I've created a request to support auto-exposure for IR-dominant streams. I'll let you know once I know when that support will be available.

    TI's SDK SW architecture uses GStreamer and TIOVX to pass auto-exposure setting from ISP to sensor driver. Under this architecture, adding a new control is probably the best way. Inside TIOVXISP plugin, the ISP output format can be used to distinguish RGB streams and IR streams: https://github.com/TexasInstruments/edgeai-gst-plugins/blob/main/ext/tiovx/gsttiovxisp.c#L1716.

    Regards,

    Jianzhong

  • Hi Jianzhong,

     

    Thank you very much for the reply. I'll update the info with my colleagues and have a look at the TIOVXISP plugin.

     

    Regards,

    Huang Jingjie

  • Hi Jianzhong,

    Regarding the codes inside gst_tiovx_isp_postprocess() I have some more questions:

    1. It seems that what the function do here is to simply assign corresponding control id & value and send it to ioctl, right? Is there anything else requiring modification?

    2. You mentioned ISP output format, do you mean sink_pad->aewb_config.sensor_img_format? I failed to find the definitions for most of the structs, and this seems to be the most related one. 

    3. Our current driver has the function get_ox05b_ae_dyn_params() implemented. Is this function strongly dependent on sensor itself and are RGB and IR frame share the same parameters?

    Thank you for the support in advance and looking forward to your reply.

    Regards,

    Huang Jingjie

  • Hello Jingjie,

    Since I'll be on vacation for a few weeks, my colleague Suren will take over the support from now on.

    3. Our current driver has the function get_ox05b_ae_dyn_params() implemented. Is this function strongly dependent on sensor itself and are RGB and IR frame share the same parameters?

    Yes, this function depends on the sensor and TI's 2A algorithm. I believe the parameters for RGB and IR will be different.

    Regards,

    Jianzhong

  • Hi Jingjie,

    Give me a day's time to respond, as I am checking internally with our experts on the query you had on dynamic parameters. 

    Best Regards,

    Suren

  • Hi Jingjie,

    Since the parameters will be different, did you look at the sensor registers and configure them accordingly?

    Also Gang, has replied on another thread on AE issues that your colleague has reported.

    Best Regards,

    Suren

  • Hi Suren,

    I've modified the V4L2 to add new controls and we are testing if these controls are running as expected. I'll reply once they are verified.

    As for the AE issue you mentioned, would you mind provide me with a link to the question, or a name so that I can contact him or her?

    Regards,

    Huang Jingjie

  • Hi Suren,

    Thanks for the link. I've contact the colleague asking that question and aligned the status with him.

    We've run into some trouble when trying to verify the modifications, here are some details:

    1. when using v4l2-ctl, we now can see our controls added to the sub device:

    root@am62axx-evm:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/video2 -l
    root@am62axx-evm:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/video3 -l
    root@am62axx-evm:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/v4l-subdev2 -l
    
    User Controls
    
                           exposure 0x00980911 (int)    : min=1 max=2116 step=1 default=1025 value=1025
                    horizontal_flip 0x00980914 (bool)   : default=0 value=0
                      vertical_flip 0x00980915 (bool)   : default=0 value=0
                        exposure_ir 0x0098092c (int)    : min=1 max=2116 step=1 default=257 value=257
                   analogue_gain_ir 0x0098092d (int)    : min=0 max=248 step=1 default=16 value=16
                    digital_gain_ir 0x0098092e (int)    : min=0 max=4095 step=1 default=256 value=256
    
    Image Source Controls
    
                      analogue_gain 0x009e0903 (int)    : min=0 max=248 step=1 default=16 value=16
    
    Image Processing Controls
    
                         pixel_rate 0x009f0902 (int64)  : min=280000000 max=280000000 step=1 default=280000000 value=280000000 flags=read-only
                       digital_gain 0x009f0905 (int)    : min=0 max=4095 step=1 default=256 value=256
    

    2. but the problem is that these controls are only valid for v4l-subdev2, while we used video2/3 to capture image for verification. The following commands are very likely to be incorrect, as we cannot sense any difference after changing the exposure_ir value.

    root@am62axx-evm:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/v4l-subdev2 -c exposure_ir=350
    [  573.652818] ox05b 4-0036: Device is not powered up : ctrl->id 9963820
    root@am62axx-evm:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/v4l-subdev2 -l
    
    User Controls
    
                           exposure 0x00980911 (int)    : min=1 max=2116 step=1 default=1025 value=1025
                    horizontal_flip 0x00980914 (bool)   : default=0 value=0
                      vertical_flip 0x00980915 (bool)   : default=0 value=0
                        exposure_ir 0x0098092c (int)    : min=1 max=2116 step=1 default=257 value=350
                   analogue_gain_ir 0x0098092d (int)    : min=0 max=248 step=1 default=16 value=16
                    digital_gain_ir 0x0098092e (int)    : min=0 max=4095 step=1 default=256 value=256
    
    Image Source Controls
    
                      analogue_gain 0x009e0903 (int)    : min=0 max=248 step=1 default=16 value=16
    
    Image Processing Controls
    
                         pixel_rate 0x009f0902 (int64)  : min=280000000 max=280000000 step=1 default=280000000 value=280000000 flags=read-only
                       digital_gain 0x009f0905 (int)    : min=0 max=4095 step=1 default=256 value=256
    
    
    root@am62axx-evm:/opt/edgeai-gst-apps# v4l2-ctl -d /dev/video3 --set-fmt-video=width=2592,height=1944,pixelformat=BGI0  --stream-mmap --stream-count=1 --stream-to=test00.raw
    [  695.002806] j721e-csi2rx 30102000.ticsi2rx: remote_pad->stream_count 1
    [  695.009478] j721e-csi2rx 30102000.ticsi2rx: routing->num_routes 2
    [  695.015837] j721e-csi2rx 30102000.ticsi2rx: r->source_pad 1 remote_pad->index 2
    [  695.023195] j721e-csi2rx 30102000.ticsi2rx: r->source_pad 2 remote_pad->index 2
    [  695.030653] ox05b 4-0036:  fd->num_entries 2
    [  695.035111] j721e-csi2rx 30102000.ticsi2rx: Ignore format 0x30494742 ctx->idx 1 df 0x2b vc 1 size 1
    [  695.044603] ox05b 4-0036:  fd->num_entries 2
    [  695.048999] cdns-csi2rx 30101000.csi-bridge: bpp 10 has_fd 1
    [  695.049066] cdns-csi2rx 30101000.csi-bridge:  Dump DPHY Configure link_freq 350000000
    [  695.049125] cdns-csi2rx 30101000.csi-bridge:  Dump DPHY Configure hs_clk_rate 350000000
    [  695.070836] cdns-csi2rx 30101000.csi-bridge:  Dump DPHY Configure lanes 4
    [  695.078152] cdns-mipi-dphy 30110000.phy: DPHY wait for lane 4 ready
    [  695.084555] cdns-csi2rx 30101000.csi-bridge:  csi2rx_configure_external_dphy ret 0
    [  695.092457] ox05b 4-0036: ox05b_set_stream: enable 1
    [  695.289085] ox05b 4-0036: ox05b_start_stream: ox05b_2592x1944_60fps_AB ret 0
    [  695.332916] ox05b 4-0036: ox05b_start_stream: stream on end
    [  695.338598] j721e-csi2rx 30102000.ticsi2rx: Start DMA: 0 sizeimage 10077696
    <[  695.478377] ox05b 4-0036: ox05b_set_stream: enable 0
    

    3. on the contrary, the RGB side works just fine.At least we can see the exposure for RGB channel adjusted when using gstreamer under current setup:

    root@am62axx-evm:/opt/edgeai-gst-apps# gst-launch-1.0 \
    > v4l2src device=/dev/video2 io-mode=5 ! \
    > video/x-bayer, width=2592, height=1944, format=bggi10 ! queue leaky=2 ! \
    > tiovxisp sensor-name=SENSOR_OX05B_RGBIR \
    > dcc-isp-file=/opt/imaging/ox05b/dcc_viss.bin \
    > sink_0::dcc-2a-file=/opt/imaging/ox05b/dcc_2a.bin sink_0::device=/dev/v4l-subdev2 format-msb=9 ! \
    > video/x-raw, format=NV12 ! tiovxldc dcc-file=/opt/imaging/ox05b/dcc_ldc.bin sensor-name=SENSOR_OX05B_RGBIR ! \
    > video/x-raw, format=NV12, width=1920, height=1080 ! \
    > kmssink driver-name=tidss sync=false
    clearAPP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=5) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
       944.019514 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
       944.024779 s:  VX_ZONE_INIT:Enabled
       944.024818 s:  VX_ZONE_ERROR:Enabled
       944.024827 s:  VX_ZONE_WARNING:Enabled
       944.025743 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
       944.026184 s:  VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
    Setting pipeline to PAUSED ...
    Pipeline is live and does not need PREROLL ...
    Setting pipeline to PLAYING ...
    New clock: GstSystemClock
    [  938.100173] j721e-csi2rx 30102000.ticsi2rx: remote_pad->stream_count 1
    [  938.106837] j721e-csi2rx 30102000.ticsi2rx: routing->num_routes 2
    [  938.113192] j721e-csi2rx 30102000.ticsi2rx: r->source_pad 1 remote_pad->index 1
    [  938.120549] ox05b 4-0036:  fd->num_entries 2
    [  938.125059] j721e-csi2rx 30102000.ticsi2rx: Ignore format 0x30494742 ctx->idx 0 df 0x2b vc 0 size 1
    [  938.134429] ox05b 4-0036:  fd->num_entries 2
    [  938.138995] cdns-csi2rx 30101000.csi-bridge: bpp 10 has_fd 1
    [  938.144790] cdns-csi2rx 30101000.csi-bridge:  Dump DPHY Configure link_freq 350000000
    [  938.152756] cdns-csi2rx 30101000.csi-bridge:  Dump DPHY Configure hs_clk_rate 350000000
    [  938.160877] cdns-csi2rx 30101000.csi-bridge:  Dump DPHY Configure lanes 4
    [  938.167968] cdns-mipi-dphy 30110000.phy: DPHY wait for lane 4 ready
    [  938.174349] cdns-csi2rx 30101000.csi-bridge:  csi2rx_configure_external_dphy ret 0
    [  938.182264] ox05b 4-0036: ox05b_set_stream: enable 1
    [  938.377223] ox05b 4-0036: ox05b_start_stream: ox05b_2592x1944_60fps_AB ret 0
    [  938.420931] ox05b 4-0036: ox05b_start_stream: stream on end
    [  938.426618] j721e-csi2rx 30102000.ticsi2rx: Start DMA: 0 sizeimage 10077696
    [  938.588751] ox05b 4-0036: exposure 0x844 again 0x10 dgain 0x100
    [  939.221652] ox05b 4-0036: exposure 0x844 again 0x11 dgain 0x100
    [  939.538433] ox05b 4-0036: exposure 0x844 again 0x46 dgain 0x100
    [  939.854900] ox05b 4-0036: exposure 0x844 again 0x80 dgain 0x100
    

    I think if we have valid AE algorithm for IR channel and certain modification for the tiovxisp plugin, we could quickly test our codes using similiar commands as above. Given the fact that these are not available at the moment, we want to find another way to verify these new controls.

    We found some online resources utilizing videox to change settings and capture frames at the same time: 

    (https://forums.developer.nvidia.com/t/how-to-set-exposure-and-gain-parameters-for-imx219-with-v4l2-ctl/149092)

    v4l2-ctl -d /dev/video0 --set-fmt-video=width=3280,height=2464,pixelformat=RG10 --set-ctrl exposure=33300,gain=110,bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test_3280x2464.raw
    Somehow, after reading related codes, I still have no idea how to link the v4l2_control_handler from v4l-subdev2 to video2/3. Would you please help me with this issue, or provide us with some other workarounds?

    Regards,

    Huang Jingjie

  • Hi Jingjie,

    Print the topology of camera subsystem using -p media-ctl option

    media-ctl -p 

    This gives the current links between camera subdevices, and each subdevice current pads configuration such as, format, resolution, framerate.

    This topology can also be displayed in a human readable graphical form thanks to --print-dot option:

    media-ctl --print-dot > graph.dot

    Retrieve then this .dot file into your host PC in order to convert it to .png:
    dot -Tpng -Nfontname=Roboto -Nfontsize=10 -Efontname=Roboto -Efontsize=10 graph.dot > graph.png

    In order to control camera sensor, its corresponding entity device node must be found in the topology. To do so, the first entity of the graph must be found, ie the entity having no sink pad:

    In your case v4l-subdev2 has just the source pad and so the user-controls are shown only on v4l-subdev2.

    Hope this helps.

    Best Regards,

    Suren

  • Hi Suren,

    Thanks for the tips. Actually I was wondering the relationship among these v4l2 devices & subdevs, and this graph makes it clear.

    Also I just tried to add '&' in my streaming command and use v4l2-ctl to manually adjust the exposure_ir while streaming. The video was changing according to my configurations. This means that new controls are valid.

    I think the next steps for us are to contact omnivision for ir channel dynamic parameters and to modify the tiovxisp plugin. I'll try to post our progress here, and please provide us with some updates on your AE algorithm when possible.

    Regards,

    Huang Jingjie

  • Hi Jingie,

    I am glad the changes are working on your setup.

    Would you be able to share your changes as a diff/patches, so I can share it with the SW team to comment.

    I am following up with SW team internally on AE algorithm updates for IR. Will keep you posted as I hear anything.

    Best Regards,

    Suren

  • Hi Suran,

    Would you mind telling me what exactly need to be shared and in which form? I‘ll also have to ask our supervisor about what can be shared.

    Regards,

    Huang Jingjie

  • Hi Jingjie,

    Appreciate if you can share the changes that you guys have made as a zip file so our SW team can understand and provide comments. 

    Best Regards,

    Suren

  • Hi Suren,

    Please check out the package, parts of these files are modified to add new controls.
    modification.gz

    By the way, previously I asked Jianzhong some questions about gst_tiovx_isp_postprocess(). Regarding this function we still have trouble finding a valid data to differ the source, thus unable to invoke the corresponding control. If possible, please provide some advice on this issue.

    Regards,

    Huang Jingjie

  • There is no activity for more than one year. So, close the ticket first. Please submit a new tickets if there is still question/issue.