[TDA4] VPAC LDC function

Mesh LDC should be computed offline. Please refer to the post 3157.TDA3x_Mesh_LDC.pdf

Mesh LUT should be computed offline. Please refer to the post 3157.TDA3x_Mesh_LDC.pdf

1 At first,  Let me descibe what we want to do, I can use a simple graph to explain my target. 

input grayscale image -- > VPAC(ldc) --> output birdview image

Take input and output image for example:

input image

https://www.hindawi.com/journals/js/2016/4058093/fig22/

output image

https://www.hindawi.com/journals/js/2016/4058093/fig23/

Can I use VPAC ldc to achieve this goal, can you offer me an example

2 Second, I learn openvx api tivxVpacLdcNode from TIOVX User Guide.

However, I cannot understand every para clearly except  graph and configuration.

Can you tell me how to use every para in detail, or show me an example

3 As your reply, you say Mesh LDC and Mesh LUT should be computed offline.

 In my opinion, I guess your Mesh LDC means mesh_img, but I am not sure of that.

In my application, warp_matrix can be changed constantly, menwhild warp_img

should be changed constantly, can we make mesh_img compute online by VPAC,

not by A72 core, because we need high performance to need low time delay.

Yes, this operation can be done by VPAC LDC.

There is no example of this exact operation, but there are examples for correcting Fish Eye Lens Distortion. Please refer to LDC usage in the application

vision_apps/apps/basic_demos/app_single_cam

If you want to change warp matrix constantly, you can do it in 2 ways:

1. Precompute N number of Mesh LUTs and store them in memory. At runtime, keep switching MeshLUT pointer to the next warp matrix.

2. Instead of lookup table based correction, use the perspective transform. This takes only 8 parameters (A, B, C, D, E, F, G, H) which can be updated every frame. However, there is no SW support for this yet. You will need to modify OpenVX node for this.

Ok, upon your reply, I study the code for app_single_cam carefully, and I have 2 problems:

1. I find a ldc_lut array, which means your Mesh LUTs, but I don't know how to compute it, can you tell me how to do it?

2. for api tivxVpacLdcNode, the 3rd para is warp_matrix, I find description from TIOVX User Guide for it:  Must be 2x3 (affine) or 3x3 (perspective)

However, your reply refers perspective transform, which takes only 8 parameters (A, B, C, D, E, F, G, H).

In my opinion, perspective transform should use warp_matrix for 3x3 (perspective), just 9 items.

What is the relationship between warp_matrix and  (A, B, C, D, E, F, G, H), can you tell me in detail.

In order to find reason, at first we use cv2.warpPerspective interface to obtain birdview image by remap.

Then, we finish it succesfully with a proper 3*3 matrix. Let me describe the paras for cv2.warpPerspective.
1 [in] img: original cv::Mat input grayscale image

2 [in] matrix: input perspective matrix for 3*3

3 [in] (h w): output image size for width and height (1152*1008)

4 [in] flags: cv2.INTERNEST | cv2.WARP_INVERSE_MAP

Howerve, we use the same 3*3 matrix to use tivxVpacLdcNode interface to realize the same remap, but we cannot get right birdview image.

Let me descirbe the tivxVpacLdcNode paras that I use in detail, then you can help me to check it.
1 [in] graph: obj->graph = vxCreateGraph(obj->context);

2 [in] configuration: /* LDC Configuration */

tivx_vpac_ldc_params_init(&obj->ldc_params);
obj->ldc_params.luma_interpolation_type = 1;

3 [in] warp_matrix: the same 3*3 matrix used before

obj->matrix[0] = 89.8628;
obj->matrix[1] = -104.2030;
obj->matrix[2] = 59394.1733;
obj->matrix[3] = 0.0;
obj->matrix[4] = -21.0913;
obj->matrix[5] = 28274.7257;
obj->matrix[6] = 0.0;
obj->matrix[7] = -0.1662;
obj->matrix[8] = 177.2382;

4 [in] region_prms:

obj->region_params.out_block_width = 16;
obj->region_params.out_block_height = 16;
obj->region_params.pixel_pad = 0;

5. [in] mesh_prms:

memset(&obj->mesh_params, 0, sizeof(tivx_vpac_ldc_mesh_params_t));
obj->mesh_params.mesh_frame_width = 1152;
obj->mesh_params.mesh_frame_height = 1008;
obj->mesh_params.subsample_factor = 0;

6. [in] mesh_img: NULL

7. [in] dcc_cb: NULL

8. [in] in_img: 1280*720 NV12 vx_image( VX_DF_IMAGE_NV12)

9.[out] out0_img: ldc_img (1280*720 NV12 vx_image)

10. [out] out1_img: NULL

However, I cannot see anything for output image. Can you tell me what's wrong for me to use tivxVpacLdcNode interface?

I understand your question now. Your process looks correct. However, the format of the warp matrix coefficients looks wrong. Warp matrix is a 3x3 matrix with 9th value always = 1. Therefore there are only 8 parameters which are referred to as {A, B, C, D, E F, G, H}. For format is S16Q12 (signed number on 14 bits with 2 bits for integer and 12 bits of fraction)

You may find more details in the VPAC chapter of TRM. Please try specifying warp matrix in the correct format and let us know if it does not work.

I do experiment for your reply, however, the demo is failed again, and I have another 3 problems.

1. Referred {A, B, C, D, E F, G, H}

I cannot find explicit order to fill LDC perspective transformation in VPAC chapter of TRM.

If equation is as follows:

h = A * h1 + B * v1 + G;

v = C * h1 + D * v1 + H;

z = E * h1 + F * v1 + 1;

Now I fill the order is :

obj->matrix[0] = A;
obj->matrix[1] = B;
obj->matrix[2] = C;
obj->matrix[3] = D;
obj->matrix[4] = E;
obj->matrix[5] = F;
obj->matrix[6] = G;
obj->matrix[7] = H;
obj->matrix[8] =1;

I am not sure of this order, can you tell me the right order ?

2. What is the right format for G, H, 1?

In you reply, you just refered S16Q12 format, but I find S16Q12 and S16Q3 format on affine transformation for VPAC chapter of TRM.

Thus, for perspective transformation, I guess as follows:

G and H should be S16Q12 format, and 1 should be S16Q3, am I right ?

If 1 should be S16Q3, we use 8 to replace with 1 in actual compute.

Is it right for my understanding, can you tell me in detail?

3. S16Q12 format (signed number on 14bits with 2 bits for integer and 12 bits of fraction) 

I see upper information for VPAC chapter of TRM.

In that meaning, we just use 14bits to describe the value.

In my understanding, the first bit may be sign symbol flag, 0 for positive, and 1 for minus.

Then, what is the meaning for second bit?

Why not use 3 bits for integer? If integer part for my data is over 3, we cannot describe it for 2 bits,

how can I solve this problem, can you tell me in detail?

In TDA4,  we didn't find the file 3157.TDA3x_Mesh_LDC.pdf.

S16Q12 format (signed number on 14bits with 2 bits for integer and 12 bits of fraction) 

2 bits for integer: just means 00, 01, 10, 11 ?

12 bits of fraction: the precision is 1 / 2 ^ 12 ?

However, how signed flag is stored for S16Q12, can you tell me in detail ?

Does 14bits include signed flag ?

The parameters {a, b,d, e} are in S16Q12 format

The parameters {c, f, g, h} are in S16Q3 format

SnQm = n-bit signed number representing a signed fixed-point number with m fraction bits.

Range is [-2^(n-m-1) ~ 2^(n-m-1) – 2^(-m)]