Image capture using Parallel ISP Interface

Hi Kris,

 Did you get this working ? I am trying a similar thing and having some issues ?

Regards,

Kiran

Hi Kiran,

I am having the same problem with an Aptina imager (MT9D131).  If i use the V4L2_PIX_FMT_SGRBG10 format, I can get an image, but the color mapping is wrong.  Since my imager outputs 8 bit data, not 10, I have tried many different pixel formats but none give me an image.

Have you found any solutions?  Also, it seems as though many of the pixel formats in the videodev2.h file are not supported...when selected, I get an error at run time.   Below are the choices that I have, any thoughts? 

Thanks,

Matt

/*      Pixel format         FOURCC                        depth  Description 
#define V4L2_PIX_FMT_RGB332  v4l2_fourcc('R', 'G', 'B', '1') /*  8  RGB-3-3-2     */
#define V4L2_PIX_FMT_RGB444  v4l2_fourcc('R', '4', '4', '4') /* 16  xxxxrrrr ggggbbbb */
#define V4L2_PIX_FMT_RGB555  v4l2_fourcc('R', 'G', 'B', 'O') /* 16  RGB-5-5-5     */
#define V4L2_PIX_FMT_RGB565  v4l2_fourcc('R', 'G', 'B', 'P') /* 16  RGB-5-6-5     */
#define V4L2_PIX_FMT_RGB555X v4l2_fourcc('R', 'G', 'B', 'Q') /* 16  RGB-5-5-5 BE  */
#define V4L2_PIX_FMT_RGB565X v4l2_fourcc('R', 'G', 'B', 'R') /* 16  RGB-5-6-5 BE  */
#define V4L2_PIX_FMT_BGR24   v4l2_fourcc('B', 'G', 'R', '3') /* 24  BGR-8-8-8     */
#define V4L2_PIX_FMT_RGB24   v4l2_fourcc('R', 'G', 'B', '3') /* 24  RGB-8-8-8     */
#define V4L2_PIX_FMT_BGR32   v4l2_fourcc('B', 'G', 'R', '4') /* 32  BGR-8-8-8-8   */
#define V4L2_PIX_FMT_RGB32   v4l2_fourcc('R', 'G', 'B', '4') /* 32  RGB-8-8-8-8   */
#define V4L2_PIX_FMT_GREY    v4l2_fourcc('G', 'R', 'E', 'Y') /*  8  Greyscale     */
#define V4L2_PIX_FMT_Y16     v4l2_fourcc('Y', '1', '6', ' ') /* 16  Greyscale     */
#define V4L2_PIX_FMT_PAL8    v4l2_fourcc('P', 'A', 'L', '8') /*  8  8-bit palette */
#define V4L2_PIX_FMT_YVU410  v4l2_fourcc('Y', 'V', 'U', '9') /*  9  YVU 4:1:0     */
#define V4L2_PIX_FMT_YVU420  v4l2_fourcc('Y', 'V', '1', '2') /* 12  YVU 4:2:0     */
#define V4L2_PIX_FMT_YUYV    v4l2_fourcc('Y', 'U', 'Y', 'V') /* 16  YUV 4:2:2     */
#define V4L2_PIX_FMT_UYVY    v4l2_fourcc('U', 'Y', 'V', 'Y') /* 16  YUV 4:2:2     */
#define V4L2_PIX_FMT_VYUY    v4l2_fourcc('V', 'Y', 'U', 'Y') /* 16  YUV 4:2:2     */
#define V4L2_PIX_FMT_YUV422P v4l2_fourcc('4', '2', '2', 'P') /* 16  YVU422 planar */
#define V4L2_PIX_FMT_YUV411P v4l2_fourcc('4', '1', '1', 'P') /* 16  YVU411 planar */
#define V4L2_PIX_FMT_Y41P    v4l2_fourcc('Y', '4', '1', 'P') /* 12  YUV 4:1:1     */
#define V4L2_PIX_FMT_YUV444  v4l2_fourcc('Y', '4', '4', '4') /* 16  xxxxyyyy uuuuvvvv */
#define V4L2_PIX_FMT_YUV555  v4l2_fourcc('Y', 'U', 'V', 'O') /* 16  YUV-5-5-5     */
#define V4L2_PIX_FMT_YUV565  v4l2_fourcc('Y', 'U', 'V', 'P') /* 16  YUV-5-6-5     */
#define V4L2_PIX_FMT_YUV32   v4l2_fourcc('Y', 'U', 'V', '4') /* 32  YUV-8-8-8-8   */

/* two planes -- one Y, one Cr + Cb interleaved  */
#define V4L2_PIX_FMT_NV12    v4l2_fourcc('N', 'V', '1', '2') /* 12  Y/CbCr 4:2:0  */
#define V4L2_PIX_FMT_NV21    v4l2_fourcc('N', 'V', '2', '1') /* 12  Y/CrCb 4:2:0  */
#define V4L2_PIX_FMT_NV16    v4l2_fourcc('N', 'V', '1', '6') /* 16  Y/CbCr 4:2:2  */
#define V4L2_PIX_FMT_NV61    v4l2_fourcc('N', 'V', '6', '1') /* 16  Y/CrCb 4:2:2  */

/*  The following formats are not defined in the V4L2 specification */
#define V4L2_PIX_FMT_YUV410  v4l2_fourcc('Y', 'U', 'V', '9') /*  9  YUV 4:1:0     */
#define V4L2_PIX_FMT_YUV420  v4l2_fourcc('Y', 'U', '1', '2') /* 12  YUV 4:2:0     */
#define V4L2_PIX_FMT_YYUV    v4l2_fourcc('Y', 'Y', 'U', 'V') /* 16  YUV 4:2:2     */
#define V4L2_PIX_FMT_HI240   v4l2_fourcc('H', 'I', '2', '4') /*  8  8-bit color   */
#define V4L2_PIX_FMT_HM12    v4l2_fourcc('H', 'M', '1', '2') /*  8  YUV 4:2:0 16x16 macroblocks */

/* see http://www.siliconimaging.com/RGB%20Bayer.htm */
#define V4L2_PIX_FMT_SBGGR8  v4l2_fourcc('B', 'A', '8', '1') /*  8  BGBG.. GRGR.. */
#define V4L2_PIX_FMT_SGBRG8  v4l2_fourcc('G', 'B', 'R', 'G') /*  8  GBGB.. RGRG.. */
/*
 * 10bit raw bayer, expanded to 16 bits
 * xxxxrrrrrrrrrrxxxxgggggggggg xxxxggggggggggxxxxbbbbbbbbbb...
 */
#define V4L2_PIX_FMT_SGRBG10 v4l2_fourcc('B', 'A', '1', '0')
/* 10bit raw bayer DPCM compressed to 8 bits */
#define V4L2_PIX_FMT_SGRBG10DPCM8 v4l2_fourcc('B', 'D', '1', '0')
#define V4L2_PIX_FMT_SBGGR16 v4l2_fourcc('B', 'Y', 'R', '2') /* 16  BGBG.. GRGR.. */

/* compressed formats */
#define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
#define V4L2_PIX_FMT_JPEG     v4l2_fourcc('J', 'P', 'E', 'G') /* JFIF JPEG     */
#define V4L2_PIX_FMT_DV       v4l2_fourcc('d', 'v', 's', 'd') /* 1394          */
#define V4L2_PIX_FMT_MPEG     v4l2_fourcc('M', 'P', 'E', 'G') /* MPEG-1/2/4    */

/*  Vendor-specific formats   */
#define V4L2_PIX_FMT_WNVA     v4l2_fourcc('W', 'N', 'V', 'A') /* Winnov hw compress */
#define V4L2_PIX_FMT_SN9C10X  v4l2_fourcc('S', '9', '1', '0') /* SN9C10x compression */
#define V4L2_PIX_FMT_PWC1     v4l2_fourcc('P', 'W', 'C', '1') /* pwc older webcam */
#define V4L2_PIX_FMT_PWC2     v4l2_fourcc('P', 'W', 'C', '2') /* pwc newer webcam */
#define V4L2_PIX_FMT_ET61X251 v4l2_fourcc('E', '6', '2', '5') /* ET61X251 compression */
#define V4L2_PIX_FMT_SPCA501  v4l2_fourcc('S', '5', '0', '1') /* YUYV per line */
#define V4L2_PIX_FMT_SPCA505  v4l2_fourcc('S', '5', '0', '5') /* YYUV per line */
#define V4L2_PIX_FMT_SPCA508  v4l2_fourcc('S', '5', '0', '8') /* YUVY per line */
#define V4L2_PIX_FMT_SPCA561  v4l2_fourcc('S', '5', '6', '1') /* compressed GBRG bayer */
#define V4L2_PIX_FMT_PAC207   v4l2_fourcc('P', '2', '0', '7') /* compressed BGGR bayer */
#define V4L2_PIX_FMT_PJPG     v4l2_fourcc('P', 'J', 'P', 'G') /* Pixart 73xx JPEG */
#define V4L2_PIX_FMT_YVYU    v4l2_fourcc('Y', 'V', 'Y', 'U') /* 16  YVU 4:2:2     */

Hi Mat,

 I got it working on my platform. Which platform are you trying on ? Your intention is to capture RAW data from the camera sensor connected through parallel interface ?

Regards,

Kiran

I'm using the Dev kit from Mistral Solutions with their multimedia daughter card.  I am trying to capture raw data from the camera sensor.  If I use the V4L2_PIX_FMT_SGRBG10 for the input capture, and V4L2_PIX_FMT_UYVY for the display, I can get an image, but the color mapping is way off...black is being displayed as blue.  My camera sensor is outputting Cb-Y-Cr-Y.

Any suggestions?  Thanks, Matt

Whcih kernel source are you using. is this under linux or windows mobile. I ma y be able to help you with some settings if you are using Linux. Your intention is to capture

RAW data from sensor->configure ISP CCDC to convert to YUV-> save it to memory

or

RAW data from sensor->configure ISP CCDC and then convert to RGB to display on screen

I'm using Linux kernel 2.6.29.  My intention is to:

capture RAW data from sensor->configure ISP CCDC and then convert to RGB to display on screen

can you share your driver sourec file, omap34xxcam.c, the board configuration file in arch/arm/mach-omap2/board-<>.c , isp files ? can you send it directly to kiranc@ti.com from your email. I can then reply to your email with modified files to try.By default the confguration in baseport is to capture RAW data from sensor and then configure ISP/CCDC accordingly.

Hi Kris, Did you get this working ? I am trying a similar thing and having some issues ?what a different is i use ov2655 instead of  ov3640.

Regards,

luofc

As along as you configure the isp properly, you should not have any problem using a different sensor.I suggest you check the VSYNC, PCLK polarities. What problem you are seeing with ov2655?

Cheers,
Krishna

Hi Kris,
Thanks for you reply.
I configure the ov2655 to output 800x600 YUV422(UVYV order)data,and configure the OMAP3530 like below:

========================================================================
static struct omap34xxcam_sensor_config ov3640_hwc = {
    .sensor_isp = 0,
    .xclk = OMAP34XXCAM_XCLK_B,
    .capture_mem = PAGE_ALIGN(2048 * 1536 * 2) * 2,
};

static struct isp_interface_config ov3640_if_config = {
    .ccdc_par_ser = ISP_PARLL,
    .dataline_shift = 0x0,
    .hsvs_syncdetect = ISPCTRL_SYNC_DETECT_VSRISE,
    .strobe = 0x0,
    .prestrobe = 0x0,
    .shutter = 0x0,
//    .prev_sph = 2,
//    .prev_slv = 1,
//    .wenlog = ISPCCDC_CFG_WENLOG_AND,
    .wait_hs_vs = 2,
    .u.par.par_bridge = 0x02,
    .u.par.par_clk_pol = 0x0,
};

static int ov3640_sensor_set_prv_data(void *priv)
{

    struct omap34xxcam_hw_config *hwc = priv;

    hwc->u.sensor.xclk = ov3640_hwc.xclk;
    hwc->u.sensor.sensor_isp = ov3640_hwc.sensor_isp;
    hwc->u.sensor.capture_mem = ov3640_hwc.capture_mem;
    hwc->dev_index = 0;
    hwc->dev_minor = 0;
    hwc->dev_type = OMAP34XXCAM_SLAVE_SENSOR;
    return 0;
}

static int ov3640_sensor_power_set(enum v4l2_power power)
{
    static enum v4l2_power previous_power = V4L2_POWER_OFF;
    switch (power) {
    case V4L2_POWER_ON:
        isp_configure_interface(&ov3640_if_config);
        break;
    case V4L2_POWER_OFF:
        break;
    case V4L2_POWER_STANDBY:
        break;
    }
    previous_power = power;

    return 0;
}

static struct ov3640_platform_data ov3640_platform_data = {
    .power_set     = ov3640_sensor_power_set,
    .priv_data_set     = ov3640_sensor_set_prv_data,
    .default_regs     = ov3640_common,
};

static struct i2c_board_info __initdata ov3640_i2c_board_info= {
    I2C_BOARD_INFO("ov3640", OV3640_I2C_ADDR),
    .platform_data = &ov3640_platform_data,
};
========================================================================

When I run the video capture program given in the v4l2 documentation, I get the following error:

omap3isp omap3isp: isp_wait: wait is too much

omap3isp omap3isp:ccdc 1 won't become idle!

it keeps printing these messages on the minicom and I do not get any image at all.

don't let the driver to check "isp_wait" status .

maybe your isp idle time is too little,and your interrupt code need more time than the idle time.

Hi,

We are also facing similar problem. We are facing problem while capturing image.

Platform details:

OMAP3503

Aptina MT9V113 image sensor

Linux kernel 2.6.29

The MT9V113 driver was integrated to the kernel. Initially we had the problem of registering the Camera ISP as master and image as slave with V4L2. We resolved that. We find that the I2C operation (both read and write) on the sensor is failing now. Also, While probing the I2C bus we get "UU" on the slave address of Camera.

The xclka supplied to the camera is coming up and then vanishing during the kernel boot.

We are using the below structures initializations

static struct isp_interface_config mt9v113_if_config = {
        .ccdc_par_ser           = ISP_PARLL_YUV_BT,
        .dataline_shift         = 0x1,
        .hsvs_syncdetect        = ISPCTRL_SYNC_DETECT_VSRISE,
        .strobe                 = 0x0,
        .prestrobe              = 0x0,
        .shutter                = 0x0,
        .u.par.par_bridge       = 0x0,
        .u.par.par_clk_pol      = 0x0,
};

static int mt9v113_set_prv_data(void *priv)
{
        struct omap34xxcam_hw_config *hwc;
        hwc = (struct omap34xxcam_hw_config *)priv;

        hwc->dev_index = (int)0;
        hwc->dev_minor = (int)0;
        hwc->dev_type = (int)OMAP34XXCAM_SLAVE_SENSOR;
        /* ABHI Dec26 */
        hwc->u.sensor.sensor_isp = 1;
        hwc->u.sensor.xclk = OMAP34XXCAM_XCLK_A;
        return 0;
}

Hi .

      We are also facing the same problem in ov3640 interfaced with omap 3530 8bit interface.

I think the isp driver is developed for RAW 10bit interface

By configuring the isp interface configuration as parallel interface.

we are also able to get the image from the sensor but color mapping is wrong.

We are not able to find any callback handler on isp interrupt section omap34xx_isp_isr ,

when sensor and isp is configured as "data comming out of sensor as YUYV"

sensor ----> ccdc ---> memory

(configured as yuv in sensor and ccdc)

We are facing problems in configuring the isp driver as 8bit raw data as input from the sensor.

1. ispccdc.c

driver/media/video/isp/ispccdc.c

function ispccdc_config_datapath is configured as 10bit for RAW input from the sensor.

Thanks and regards,

Ananthapadmanaban

Hi,

We could overcome the problem of I2C read/write. Basically if there is no xclk given to the image sensor the I2C operations fail. Before any operation the xclk should be supplied by OMAP to sensor. The supplying of clocks is handled by ISP_put function of isp.c.But still there is no image capture happening. All the interrupts of ISP was enabled and the ISP_IRQ0STATUS seems to indicate there is not error condition occuring. We are getting the HS_VS, VD0 and VD1 interrupts promptly.

Thanks and Regards,

Srikanth

Does anyone have a resolution for this?

I'm writing an Aptina MT9V032 driver. The camera has a test mode that outputs a gray gradient that appears purple. I've tried different Bayer Image formats for the input but I get the same result regardless.

Thanks,

Neil MacMunn
Gumstix Research Canada

Hi Neil,

  If you are using omap isp to get the yuv data from bayer data. Try this

In drivers/media/video/isp/isppreview.c

    params->features = PREV_CFA | PREV_DEFECT_COR | PREV_NOISE_FILTER |PREV_GAMMA_BYPASS;
    params->features &= ~(PREV_AVERAGER | PREV_INVERSE_ALAW |
                        PREV_HORZ_MEDIAN_FILTER | 
                        PREV_DARK_FRAME_SUBTRACT |
                        PREV_LENS_SHADING |
                        PREV_DARK_FRAME_CAPTURE |
                        PREV_CHROMA_SUPPRESS | PREV_LUMA_ENHANCE);

And try with reset value of white balance co-efficient drivers/media/video/isp/isppreview.c

#define FLR_WBAL_COEF0        0x20
#define FLR_WBAL_COEF1        0x20
#define FLR_WBAL_COEF2        0x20
#define FLR_WBAL_COEF3        0x20

#define FLR_WBAL_COEF0_ES1    0x20
#define FLR_WBAL_COEF1_ES1    0x20
#define FLR_WBAL_COEF2_ES1    0x20
#define FLR_WBAL_COEF3_ES1    0x20


Thanks and regards,
Ananth.

Hi Kris,

This could be due to wrong configuration of ISP, this means that SBL is not becoming idle. what all the changes have you done to ISP code base? Which PSP release are based on?

Thanks,

Vaibhav

Thanks Ananth.

I'll give this a try but I'm not sure that isppreview is being used. This is the structure I've declared in board-overo.c

static struct isp_interface_config mt9v032_interface_config = {
    .ccdc_par_ser            = ISP_NONE,
    .dataline_shift            = 0x0,
    .hsvs_syncdetect        = ISPCTRL_SYNC_DETECT_VSRISE,
    .strobe                = 0x0,
    .prestrobe            = 0x0,
    .shutter            = 0x0,
    .prev_sph            = 0,
    .prev_slv            = 0,
    .wenlog                = ISPCCDC_CFG_WENLOG_OR,
    .wait_hs_vs            = 2,
    .u.par.par_bridge        = 0x0,
    .u.par.par_clk_pol        = 0x0,                    /* non inverted */
};

My understanding is that the selection of ccdc_par_ser dictates whether isppreview is used or not. When I load mt9v032.ko and omap34xxcam.ko isppreview.ko doesn't get loaded. Am I missing something?

Thanks,
Neil

Yeah that worked. My grayscale pattern is actually gray now. The colors in a captured image look pretty strange though. I think that might require some driver side adjustment. Thanks!

Hi Neil,

  Following are the steps i followed to get better quality from the sensor.

By-pass flourescent Light for RGBtoRGB Blending available in drivers/media/video/isp/isppreview.c

/* Default values in Office Flourescent Light for RGBtoRGB Blending */
static struct ispprev_rgbtorgb flr_rgb2rgb = {
        {       /* RGB-RGB Matrix */
#if defined(CONFIG_USE_DEFAULT)
                {0x01E2, 0x0F30, 0x0FEE},
                {0x0F9B, 0x01AC, 0x0FB9},
                {0x0FE0, 0x0EC0, 0x0260}
#else
                {0x100,0 ,0 },
                {0, 0x100,0 },
                {0,0 , 0x100}
#endif
        },      /* RGB Offset */
        {0x0000, 0x0000, 0x0000}
};

Then Fine tune the Tables based on comparision of color chart output values.

1. Adjustment of Gamma table.

2. Adjustment of color correction matrix.

Apart from this, if you have any success stories to get better image. post your procedure in this thread.

Thanks and regards,

Ananth.

That did improve my image quality very much. Thank you.

I've realized that my CMOS sensor is very sensitive to IR light so I've ordered an IR cut filter to see if I can improve the image more. I'm seeing strange colouring and haven't had any luck with white balance in GIMP so I expect that this will improve things considerably.

Is there any way to modify the tables you mentioned from user space? At the moment I'm recompiling the kernel whenever I make the slightest change. As I'm sure you can imagine this is quite tedious.

Thanks.

--

Neil

Hi Neil,

   Make use of private ioctls available in the driver.

1. VIDIOC_PRIVATE_ISP_CCDC_CFG

2. VIDIOC_PRIVATE_ISP_PRV_CFG

3. VIDIOC_PRIVATE_ISP_AEWB_CFG

4. VIDIOC_PRIVATE_ISP_AEWB_REQ

5. VIDIOC_PRIVATE_ISP_HIST_CFG

6. VIDIOC_PRIVATE_ISP_HIST_REQ

7. VIDIOC_PRIVATE_ISP_AF_CFG

8. VIDIOC_PRIVATE_ISP_AF_REQ

Thanks and regards,

Ananth.

Viabhav,

This thread seems to have merged between 2 topics - but I'm interested in the answer to the original post about getting data from the CCDC to memory.  I've made some modifications in the 2.6.32 kernel provided with the DVSDK 4.02 release for OMAP/DM3730 to disable the previewer and resizer and hopefully get data directly from the CCDC to memory.  Now, I am getting the same errors as in the original post:

omap3isp omap3isp: isp_wait: wait is too much
omap3isp omap3isp: ccdc 1 won't become idle!

Has there been any solution to this problem?  I haven't found anything in other posts asking about this.

Thank you,

Jason

Jason,

You don't need to do any modifications, since DVSDK release (I believe based on kernel 2.6.32) doesn't enable resizer and previewer block. We have supported only TVP5146 => CCDC => Memory path.

If you could share your configuration and ISP CCDC register dump, I can review it for you.

Thanks,

Vaibhav

Vaibhav,

I apologize for not being more clear, but I am attempting to capture RAW bayer image data to memory using an OV14810 image sensor.  I have the Sensor->Prev->Resizer->Memory path working, but I need to bypass the Previewer and Resizer and get the Raw data in memory.  Please see this post for more information:

http://e2e.ti.com/support/dsp/omap_applications_processors/f/447/t/145740.aspx

I've attached my CCDC register dump and the key configuration files.  I'm asking for V4L2_PIX_FMT_SGRBG10 in my user application for output type.  I believe my configuration is close, I just can't understand why the CCDC is hanging.

Thanks,

Jason

Hi Vaibhav,

This seems to be old post, but I am facing similar issue.

I have OV5640 sensor outputting Raw8 bit data connected to Mistral DM3730 EVM platform. It seems OMAP3ISP supports only 10 bit Raw data and so I am configuring ISP with V4L2_PIX_FMT_SGRBG10 format. Sensor is connected to D0 - D7 pins of ISP. My use case is Sensor --> CCDC --> Memory. I am getting image but colours are off. Kernel version which we are using is 2.6.37.

Can you please let me know if there is latest ISP version where I can have 8 bit raw data support?

Thanks,

Srinivas

Srinivas -

Are you still having issues with this?  If so, I would suggest a separate post for your question, since it doesn't seem related to the original issue.  When you do, I would suggest including more details about what you mean when say that "colors are off".

If you are considering converting your image to YUV for display or encoding with the DSP, you might check out my post about a hardware issue I found that prevents use of 8-bit RAW data with the Previewer module:
http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/537/p/195213/715724.aspx

My suggestion would be that since the OV5640 is a 10-bit sensor, connecting it to D0-D9 on the Camera ISP will give you more flexibility.

Matt