TDA2PXEVM: ISS IPIPE changes to demosaic and convert RCCC RAW data into YUV frame

Norman S.

Part Number: TDA2PXEVM

Hello,

I would like to request assistance in supporting AR0132RCCC sensor to Vision SDK Capture link.
Following the required changes specified in http://www.ti.com/lit/an/spraca8/spraca8.pdf to support RCCC in TDA2xx ISP,
I modified pdk_01_10_02_07/packages/ti/drv/vps/src/vpslib/hal/src/vpshal_issipipe.c. with the following:

gIssHalIpipeDefCfg is modified as follows:

CFA Interpolation parameters
vpsissIpipeCfaConfig_t member value:

/* CFA Configuration */
{
VPS_ISS_IPIPE_CFA_MODE_2DIRAC,
8192, 167U, 100U, 1023U, 16U, 10U, 0U,
{
16U, 25U, 25U, 20U, 500U, 20U, 24U
}
},

Auto white balance : Disabled in ISS by setting ALGORITHMS_ISS_AEWB_MODE_NONE
Auto exposure : Disabled in ISS by setting ALGORITHMS_ISS_AEWB_MODE_NONE
Color Correction
vpsissIpipeRgb2RgbConfig_t members with unity matrix:

/* RGB2RGB 1 Configuration */
{
{
{256U, 0, 0},
{0, 256U, 0},
{0, 0, 256U}
},
{0U, 0U, 0U}
},
/* RGB2RGB 2 Configuration */
{
{
{256U, 0, 0},
{0, 256U, 0},
{0, 0, 256U}
},
{0U, 0U, 0U}
},

RGB2YUV conversion
vpsissIpipeRgb2YuvConfig_t member value:

/* RGB2YUV Configuration */
{
0U, /* Brightness */
16U, /* Contrast */
{
{0, 256U, 0},
{256U, 0, 0},
{0, 0, 0}
},
{0U, 128U, 128U}
},

However, the resulting frame is not expected.

Using RGGB interpolations configuration and coefficients, the result is as follows:

[BEFORE]

Image is monochrome (expected) but colors (red and green) appears to be washed out.
Separate issue on red patches appears on white pixels reported on a separate thread here: e2e.ti.com/.../846005

[AFTER]

The entire frame has blue overtone and the previous red patches are not blue.

Can anyone support me in configuring the ISS to convert RCCC RAW data into YUV properly?

over 5 years ago

0 Mayank Mangla over 5 years ago

TI__Mastermind 24525 points

TDA2P ISP software supports only Bayer format.

0 Norman S. over 5 years ago in reply to Mayank Mangla

Intellectual 690 points

Hello Mayank Mangla,

Thank you for the expert input on this matter.
As you have mentioned in the email and in the PDF report linked in the original post, TDA2P ISP does not support RCCC and re-design of the IPIPE, AE, AWB, and RGB2YUV are necessary.

Unfortunately, we need to make the changes to support RCCC in TDA2Px for an upcoming important demo.
Because of this, may I ask for your support on how to perform this re-design to support RCCC.

Is it possible to configure CFA interpolation module to ignore the R components and work with the monochrome clear pixel data instead?

0 Mayank Mangla over 5 years ago in reply to Norman S.

TI__Mastermind 24525 points

The changes you have done to CFA look OK. Please note that this will generate ISP output as a YUV420 Semi Planar image where:

Y component represents interpolated "C" channel at full resolution
U component represents "R" channel at 1/4 resolution
V component contains only zeros.

Display link will interpret the image as YUV420SP and try to convert it to RGB which causes the visual artifacts you are seeing on the screen. However, RCCC is primarily used for machine vision algorithms and visualization is not the primary goal. If visualization is important, please modify the display link to render only the luma channel.

0 Norman S. over 5 years ago in reply to Mayank Mangla

Intellectual 690 points

Thank you for this detailed information.

However, I would like to confirm the values of the CFA interpolation again, I have the following doubts on the values:

a) HPF Threshold for RCCC of 8192 is beyond the register maximum of 12bit value

b) What should be the value for HPF Slope for RCCC?

Thank you also for the information on how the Display link tries to render the YUV420SP into the screen.
I will check on how to change the Display link to not display the chroma channels.

0 Gang Hua over 5 years ago in reply to Norman S.

TI__Guru 59250 points

Hi Norman,

a) hpfThreshold is 13-bit.

Using 8192 is effectively the same as 0.

The actual value of hpfThreshold does not really matter for RCCC input.

b) you may try

hpfThreshold = 0;
hpfSlope = 0;
mixThreshold = 0;
mixSlope = 1023;
dirThreshold = 16;
dirSlope = 10;
nonDirWeight = 0;

Best,

Gang

0 Norman S. over 5 years ago in reply to Gang Hua

Intellectual 690 points

Hello Gang,

Thank you very much for the response.
On the value limit on hpfThreshold, it was my mistake. Thank you for correcting it.

I have used the CFA interpolation parameters you have provided which resulted to the following image:
RCCC2YUVframes45.dat

Image Format : RAW YUV420SP (NV21)
Dimension : 1280 x 1080

The unusual artifacts on hite pixels already exists in the Y channel, thus even if we ignore the UV channel on display, the white pixels appears as patches of black.

If such artifacts are due to CFA interpolation on TDA2px assumes bayer RGB format, for the purpose of demonstration, can we apply some filtering in ISS IPIPE to remove these black patches?

0 Gang Hua over 5 years ago in reply to Norman S.

TI__Guru 59250 points

Hi Norman,

I cannot open the attachment "RCCC2YUVframes45.dat" (3110400 bytes after downloading) properly as a 1280x1080 NV21 image.

Could you please check and upload both the RCCC raw image and the YUV output image?

BTW, you are right about hpfThreshold limit; it is just that hpfThreshold does not matter at all here given the other parameters.

Best,

Gang

RCCC2YUVframes45.dat

0 Norman S. over 5 years ago in reply to Gang Hua

Intellectual 690 points

Hello Gang,

My bad, the dimension should be 1920 x 1080.
I will dump the raw RCCC when I get back to office.

Regards,
Norman

0 Norman S. over 5 years ago in reply to Norman S.

Intellectual 690 points

Hello,

Attached is the RCCC raw data from IssCapture saved using ISSCAPTURE_LINK_CMD_GET_SAVE_FRAME_STATUS command.

The sensor parameter is as follows:

    info->width             = 1280;
    info->height            = 720;
    info->dataFormat        = SYSTEM_DF_BAYER_GRBG;
    info->bpp               = SYSTEM_BPP_BITS12;

isscaptureRAW.zip

0 Gang Hua over 5 years ago in reply to Norman S.

TI__Guru 59250 points

Hi Norman,

If you read the raw pixels out from the raw image (there may be 2 rows of embedded data at the top), the maximum pixel value in the raw images is clipped to 2047. Any pixel higher than 2047 are clipped to very small values.

It looks like bit-11 in your raw image is lost (only bit-0 through bit-10 are present).

This is not an issue in your ISP settings.

Please check if there is anything wrong receiving the raw data from sensor.

Best,
Gang

0 Norman S. over 5 years ago in reply to Gang Hua

Intellectual 690 points

Hello Gang,

Thank you for the details, but since I am not well versed in the raw pixels and how ISS capture dumps the raw pixels, I would like to confirm some points below.
# Apologies for asking for your time on this matter.

At first, to facilitate discussion, I configured the sensor registers with the following solid test pattern:

Red value = 0xFFFF
C1 value = 0xAAAA
C2 value = 0xBBBB
C3 value = 0xCCCC

The expectation is that the resulting view of the frame is all red (which was correct, the resulting view is indeed all red).
Attached is the resulting raw dump from ISS Capture.

Now to understand the missing 12-bit mentioned in your post above.

The pixel is transmitted as RAW12 from sensor to the ISP, thus the red value is expected as 0x0FFF (2047) while the CCC value should be 0x0AAA, 0x0BBB, and 0x0CCC.
Also note that due to read out configuration of the sensor the expected pattern of RCCC is actually CRCC.

As you mentioned, the first two rows might be embedded data so we will ignore the data from 0x0000 to 0x1400 (width x row x raw16 = 1280 x 2 x 2).

From my understanding, the expected data from 3rd row (0x1401 - 0x1E00) should be:

C=0x0AAA R=0x0FFF ---> AA 0A FF 0F ...

And the 4th row (0x1E01 -> 0x2800) should be:

C=0x0BBB C=0x0CCC ---> BB 0B CC 0C ...

However, the raw pixels in the dump shows:

C=0x02AA R=0x07FF ---> AA 02 FF 07 ...
C=0x03BB C=0x04CC ---> BB 03 CC 04 ...

From here, I can relate the missing 12-bit issue you mentioned.

Sorry for taking your time on this matter, but I have to confirm my understanding.
From my understanding I should be seeing 0x0FFF, 0x0AAA, 0x0BBB, and 0x0CCC in the raw dump right?
raw0000.raw.dat

0 Norman S. over 5 years ago in reply to Gang Hua

Intellectual 690 points

Hello Gang,

Wow! This resolved our issue. I cannot imagine how were you able to detect the missing 12 bit with the raw I provided.
Apparently, the bus width between the serializer and deserializer is not enough to transmit 12 bits with HS and VS. I modified the SerDes and now the problem is resolved.

I can now show the RCCC image as grayscale with red hints which is our target for the demo.

Thank you very much for the effective support!

Processors

Processors forum

TDA2PXEVM: ISS IPIPE changes to demosaic and convert RCCC RAW data into YUV frame