SK-AM62A-LP: Get embedded metadata from image sensor having same datatype

Hi Jaimin,

I have assigned your query to our expert. He is currently out of office, so expect a response when he will be back next week.

Apologies for the delay.

Best Regards,

Suren

Hello Jaimin,

Jaimin Patel said:

We are using image sensor which provides metadata in first 2 raws and last 2 raws.

Do these 4 rows of metadata have 1920 width as well?

Jaimin Patel said:

gst-launch-1.0 -v v4l2src device=/dev/video-rpi-cam0 ! video/x-bayer, width=1920, height=1200, framerate=120/1, format=rggb10 ! multifilesink location=/data/frame%d.rggb

Did you mean "height=1200" or "height=1204"?

Regards,

Jianzhong

Hi Jianzhong,

Do these 4 rows of metadata have 1920 width as well?

=> Yes width is 1920 for all metadata raws. Where 2 raws are before image and 2 raws are after image.

gst-launch-1.0 -v v4l2src device=/dev/video-rpi-cam0 ! video/x-bayer, width=1920, height=1200, framerate=120/1, format=rggb10 ! multifilesink location=/data/frame%d.rggb

Jaimin Patel said:

1. Does CSI interface discard metadata itself?

No, the CSI-2 RX interface sends the data to the video device (/dev/videoX) associated to the virtual channel defined in the packet header. 

Jaimin Patel said:

2. Do we need to perform any additional configuration in CSI to capture metadata in the frame?

I don't think so, unless the metadata and image data have different virtual channels.

Jaimin Patel said:

3. As per our understanding here we dont need to define separate channel to receive metadata as format for metadata is same as image pixel data. Please confirm.

This is correct assuming the metadata and image data have same virtual channel number.

Hi Jianzhong,

Image sensor is sending metadata in same virtual channel on which it sends image data still we are not able to receive it on application layer. Is there any special configuration required at any driver for this?

1. Is there any way to debug this through registers whether CSI interface receives metadata raws or not?

2. Can we debug that on which virtual channels we are receiving data through registers? This will help us to double check that the embedded metadata is not receiving on other virtual channels.

3. Please let us know how we can debug this further to get metadata on application layer.

Thanks

Hello Jaimin,

Are you receiving valid image data? If you remove the 4 rows of metadata, does the remaining data give you the correct image?

Regards,

Jianzhong

Yes, I am receiving valid image data. I have tested it using test pattern and stream from image sensor. I have tested test pattern image with hexdump and also confirmed that the hexcode is proper in image.

If I use 1920x1200 resolution then only receiving the image and if I use 1920x1204 resolution to receive metadata then bottom 4 rows are displaying with black pixels. (Our image sensor sends metadata in top and bottom 2 raws which we are not receiving.)

Thanks

Jaimin Patel said:

If I use 1920x1200 resolution then only receiving the image

This is fine.

Jaimin Patel said:

if I use 1920x1204 resolution to receive metadata then bottom 4 rows are displaying with black pixels.

What if you manually remove the top 2 rows and bottom 2 rows of metadata? Is the remaining data the correct image?

Hi Jianzhong,

Here initial 1200 raws are showing correct image data and the bottom 4 raws are indicating black pixels. If I remove top 2 rows then it will removes raws from actual image. Similarly if I remove bottom 2 raws then it will discard last 2 raws showing black pixels.

Is there any way to add debug logs in driver where we can confirm whether CSI or PHY discarding embedded metadata due to any reason?

Thanks,

Hitesh

Hello Hitesh,

Since the metadata has the same virtual channel and data type as the image data, the D-PHY and CSI2RX treat the metadata and image data the same. Therefore the metadata shouldn't be discarded.

Jaimin Patel said:

Here initial 1200 raws are showing correct image data and the bottom 4 raws are indicating black pixels. If I remove top 2 rows then it will removes raws from actual image. Similarly if I remove bottom 2 raws then it will discard last 2 raws showing black pixels.

I'm a little confused here. I thought you said earlier that the frame format was:

2 rows of metadata

1200 rows of image

2 rows of metadata

Also, just to clarify, when you said "raw", did you actually mean "row"?

Regards,

Jianzhong

Correct, Our image format along with metadata is as below:

2 rows of metadata
1200 rows of image
2 rows of metadata

Considering above format we are configuring 1204 as image height in driver and other places. But with this configuration we are receiving image where initial 1200 rows contains image data and bottom 4 rows displaying black pixels.

Is there any way to dump all csi receive data such that we can check whether it receiving embedded metadata or not? Or can we check which virtual channels are receiving data on CSI?

Yes, by mistake I typed rows as raws.

Thanks

Looks like you're not receiving the metadata. The last 4 rows are just garbage data.

I would recommend you to check with the sensor vendor to make sure you're sending the metadata correctly.

Hi Jianzhong,

I discussed this further with image sensor vendor regarding virtual channel number and as per them there is no virtual channel, there is only one main MIPI channel for the video with embedded data.

Is this possible to receive data from CSI without virtual channel?

Thanks

Hi Jaimin,

You've already proved that you can receive image data from the sensor without using a specific virtual channel number. When you program the sensor to send metadata together with the image data, the CSI Rx has no knowledge about that except the frame size is different. So you should receive metadata in the frame if the metadata is sent out from the sensor.

Regards,

Jianzhong

Hello,

We have image sensor EVM where we are able to get embedded metadata successfully. We are configuring same registers on image sensor through AM62A EVM but still not able to receive it. Also discussed same with image sensor vendor and cross checked that we are configuring proper registers on image sensor to get embedded metadata.

As a next step we want to debug what all data we are receiving from CSI and for that we want to add debug log in driver. Could you please help us where we can add debug to get this information?

CSI driver is directly receiving DMA buffers and not able to confirm whether metadata is discarded from lower lever due to any reason and same want to confirm.

Please help us to identify this.

Thanks

Hi Jamin,

Let me check internally and get back to you as soon as I can.

Regards,

Jianzhong

Jaimin Patel said:

We have image sensor EVM where we are able to get embedded metadata successfully

Does this EVM have the same sensor driver as used with AM62A?

Jaimin Patel said:

As a next step we want to debug what all data we are receiving from CSI and for that we want to add debug log in driver.

You already have access to the data. All CSI data filtered by the virtual channel id and data type goes straight to the DMA channel. 

Regards,

Jianzhong

Jianzhong Xu said:

Does this EVM have the same sensor driver as used with AM62A?

EVM is from image sensor vendor and we have no access for them source.  Linux Driver is developed by us as it is not available.

Jianzhong Xu said:

You already have access to the data. All CSI data filtered by the virtual channel id and data type goes straight to the DMA channel. 

We have tried to check the value of STREAM0_DATA_CFG (0x30101108) register and reading value as 0x00000000 when stream is starting through image sensor. As per datasheet bit number 16 to 31 indicate virtual channel number and it's value is 0 as means it will process all virtual channel. Does that mean all channel data will be added in to single frame?

Below is reference of some major functions we implemented. Please let us know if we are missing something.

Probe Function:

static int ar0235_probe(struct i2c_client *client)
{
	int ret;
	struct ar0235 *ar0235;
    struct device *dev = &client->dev;

	ar0235 = devm_kzalloc(&client->dev, sizeof(struct ar0235), GFP_KERNEL);
	if (!ar0235)
		return -ENOMEM;
 	
	v4l2_i2c_subdev_init(&ar0235->sd, client, &ar0235_subdev_ops);

    /* Check the hardware configuration in device tree */
    if (ar0235_check_hwcfg(dev))
        return -EINVAL;

    /* Get system clock (xclk) */
    ar0235->xclk = devm_clk_get(dev, NULL);
    if (IS_ERR(ar0235->xclk)) {
        dev_err(dev, "failed to get xclk\n");
        return PTR_ERR(ar0235->xclk);
    }

    ar0235->xclk_freq = clk_get_rate(ar0235->xclk);
    if (ar0235->xclk_freq != AR0235_XCLK_FREQ) {
        dev_err(dev, "xclk frequency not supported: %d Hz\n",
                        ar0235->xclk_freq);
        return -EINVAL;
    }

    ret = ar0235_get_regulators(ar0235);
    if (ret) {
        dev_err(dev, "failed to get regulators\n");
        return ret;
    }

    /* Request optional enable pin */
    ar0235->reset_gpio = devm_gpiod_get_optional(dev, "reset",
                             GPIOD_OUT_LOW);

#ifdef HAS_EXT_SUPPLY_CTRL // 3 gpio to enable regulators of camera
    ar0235->cam_1v25_gpio = devm_gpiod_get_optional(dev, "cam-1v25",
                             GPIOD_OUT_LOW);
    ar0235->cam_1v8_gpio = devm_gpiod_get_optional(dev, "cam-1v8",
                             GPIOD_OUT_LOW);
    ar0235->cam_2v8_gpio = devm_gpiod_get_optional(dev, "cam-2v8",
                             GPIOD_OUT_LOW);
     ar0235->ext_clk_gpio = devm_gpiod_get_optional(dev, "ext-clk",
                             GPIOD_OUT_LOW);
#endif

    /*
     * The sensor must be powered for ar0235_identify_module()
     * to be able to read the CHIP_ID register
     */
    ret = ar0235_power_on(dev);
    if (ret)
        return ret;

    ret = ar0235_identify_module(ar0235);
    if (ret)
        goto error_power_off;

	/* 1980 * 1200 by default */
	ar0235->mode = &supported_modes[0];

	ret = ar0235_ctrls_init(&ar0235->sd);
	if (ret < 0)
		goto error_power_off;

	ar0235->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE | V4L2_SUBDEV_FL_HAS_EVENTS;
	ar0235->pad.flags = MEDIA_PAD_FL_SOURCE;
	ar0235->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;

	/* Initialize default format */
	ar0235_set_default_format(ar0235);

	ret = media_entity_pads_init(&ar0235->sd.entity, 1, &ar0235->pad);
	if (ret < 0)
		goto error_handler_free;

	ret = v4l2_async_register_subdev_sensor(&ar0235->sd);
	if (ret < 0)
		goto error_media_entity;

    	/* Enable runtime PM and turn off the device */
    	pm_runtime_set_active(dev);
    	pm_runtime_enable(dev);
    	pm_runtime_idle(dev);

    return 0;

error_media_entity:
    media_entity_cleanup(&ar0235->sd.entity);

error_handler_free:
    ar0235_free_controls(ar0235);

error_power_off:
    ar0235_power_off(dev);

    return ret;
}

Callback Registration:

/* Various V4L2 operations tables */
static struct v4l2_subdev_video_ops ar0235_subdev_video_ops = {
	.s_stream = ar0235_set_stream,
};

static struct v4l2_subdev_core_ops ar0235_subdev_core_ops = {
    .subscribe_event = v4l2_ctrl_subdev_subscribe_event,
    .unsubscribe_event = v4l2_event_subdev_unsubscribe,
};

static const struct v4l2_subdev_pad_ops ar0235_subdev_pad_ops = {
	.enum_mbus_code = ar0235_enum_mbus_code,
	.set_fmt = ar0235_set_fmt,
	.get_fmt = ar0235_get_fmt,
};

static struct v4l2_subdev_ops ar0235_subdev_ops = {
	.core = &ar0235_subdev_core_ops,
	.video = &ar0235_subdev_video_ops,
	.pad = &ar0235_subdev_pad_ops,
};

static const struct v4l2_ctrl_ops ar0235_ctrl_ops = {
	.s_ctrl = ar0235_s_ctrl,
};

Call Back Implementation:

static int ar0235_enum_mbus_code(struct v4l2_subdev *sd,
		struct v4l2_subdev_state *sd_state,
		struct v4l2_subdev_mbus_code_enum *code)
{
	struct i2c_client *client = v4l2_get_subdevdata(sd);
	struct ar0235 *priv = to_ar0235(client);
	const struct ar0235_mode *mode = priv->mode;

	dev_err(&client->dev, "****** In %s ******\n", __func__); 
	if (code->index !=0)
		return -EINVAL;

	if(mode->sensor_depth==8)
		code->code = MEDIA_BUS_FMT_Y8_1X8;

	if(mode->sensor_depth==10) {
		//code->code = MEDIA_BUS_FMT_Y10_1X10;
		code->code = MEDIA_BUS_FMT_SRGGB10_1X10;
	}

	dev_err(&client->dev, "****** Out %s ******\n", __func__); 
	return 0;
}


static void ar0235_set_default_format(struct ar0235 *ar0235)
{
	struct v4l2_mbus_framefmt *fmt;

	fmt = &ar0235->fmt;
	fmt->width = supported_modes[0].width;
	fmt->height = supported_modes[0].height;
	//fmt->code = MEDIA_BUS_FMT_Y10_1X10;
	fmt->code = MEDIA_BUS_FMT_SRGGB10_1X10;
	fmt->colorspace = V4L2_COLORSPACE_RAW;
    fmt->ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
    fmt->quantization = V4L2_QUANTIZATION_DEFAULT;
    fmt->xfer_func = V4L2_XFER_FUNC_NONE;
	fmt->field = V4L2_FIELD_NONE;
}

static int ar0235_set_fmt(struct v4l2_subdev *sd,
			  struct v4l2_subdev_state *sd_state,
			  struct v4l2_subdev_format *fmt)
{
	const struct ar0235_mode *mode;
	struct i2c_client *client = v4l2_get_subdevdata(sd);
	struct ar0235 *priv = to_ar0235(client);

	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
		return 0;

	mode = ar0235_find_best_fit(fmt);

	if(mode->sensor_depth==8)
		fmt->format.code = MEDIA_BUS_FMT_Y8_1X8;
	if(mode->sensor_depth==10) {
		//fmt->format.code = MEDIA_BUS_FMT_Y10_1X10;
		fmt->format.code = MEDIA_BUS_FMT_SRGGB10_1X10;
	}
	fmt->format.width = mode->width;
	fmt->format.height = mode->height;
	fmt->format.field = V4L2_FIELD_NONE;
	priv->mode = mode;

	return 0;
}

static int ar0235_get_fmt(struct v4l2_subdev *sd,
			  struct v4l2_subdev_state *sd_state,
			  struct v4l2_subdev_format *fmt)
{
	s64 pixel_rate;
	struct i2c_client *client = v4l2_get_subdevdata(sd);
	struct ar0235 *priv = to_ar0235(client);
	const struct ar0235_mode *mode = priv->mode;

	if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
		return 0;

	fmt->format.width = mode->width;
	fmt->format.height = mode->height;
	if(mode->sensor_depth==8)
		fmt->format.code = MEDIA_BUS_FMT_Y8_1X8;
	if(mode->sensor_depth==10) {
		//fmt->format.code = MEDIA_BUS_FMT_Y10_1X10;
		fmt->format.code = MEDIA_BUS_FMT_SRGGB10_1X10;
	}
	fmt->format.field = V4L2_FIELD_NONE;
	fmt->format.colorspace = V4L2_COLORSPACE_RAW;
    	fmt->format.ycbcr_enc = V4L2_YCBCR_ENC_DEFAULT;
    	fmt->format.quantization = V4L2_QUANTIZATION_DEFAULT;
    	fmt->format.xfer_func = V4L2_XFER_FUNC_NONE;

	pixel_rate = mode->vts_def * mode->hts_def * mode->max_fps;
	//pixel_rate = 340000000;
	dev_err(&client->dev, "****** Pixel Rate = %lld ******\n", pixel_rate); 
	__v4l2_ctrl_modify_range(priv->pixel_rate, pixel_rate,
					pixel_rate, 1, pixel_rate);

	dev_err(&client->dev, "****** Out %s ******\n", __func__); 
	return 0;
}

Jaimin Patel said:

As per datasheet bit number 16 to 31 indicate virtual channel number and it's value is 0 as means it will process all virtual channel. Does that mean all channel data will be added in to single frame?

But you have only 1 virtual channel.