TDA4VM: Bilateral NF tuning issue

Zs lei

Part Number: TDA4VM

Tool/software:

Hi TI expert,

I'm currently tuning NF block to improve denoise performance. My implementation is that add NF node to the pipeline of single camera app. Sigma values are set by tivx_vpac_nf_bilateral_sigmas_init function. I used single raw image inject to the pipelinr of single camera app.

The issue is either I set input_interleaved as 1 or 0, the output images are same. You can compare denoise result at flat area and sharpness and texture degradtion at bear and paper area.

For my understanding, input_interleaved is used to set denoising process on Y or UV plane. So, if 0, denoise only on Y plane and output Y + Original UV; if 1, denoise only on UV plane and output Original Y + UV. If I was right, my experiment results should be different. I have checked visually and compared them by using Beyond Compare tool but no difference.

My question is whether my understanding is correct or not? Please help correct me.
My purpose is to denoise with NF on both Y and UV plane, so what is the right procedure?

Thanks a lot.

Best regards,

Zhishen

6811.To_TI.zip

11 months ago

0 Gang Hua 11 months ago

TI__Guru 59580 points

Hi Zhishen,

Zs lei said:
The issue is either I set input_interleaved as 1 or 0, the output images are same.

The zip you shared has 2 test cases (with 2 different input images).

(1) "20240925_initial_params_Ydenoise_before_NF.yuv" and "20240925_initial_params_Ydenoise_after_NF.yuv"

(2) "20240927_initial_params_UVdenoise_beforeNF.yuv" and "20240927_initial_params_UVdenoise_afterNF.yuv"

Do you mean that you get the same output image for each test case if you flip "input_interleaved"?

Zs lei said:
For my understanding, input_interleaved is used to set denoising process on Y or UV plane.

https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/latest/exports/docs/tiovx/docs/user_guide/structtivx__vpac__nf__common__params__t.html#a6e38cb896530fe571c20685bbfc8c728

From tiovx document, it seems to be the format of your input image -- 0: NonInterleaved input mode; 1: Interleaved input mode (i.e. chroma plane of NV12)

You may take a look at the your application program to see if that parameter is actually used or overwritten by tiovx kernel (as it can see the image format from your vximage input).

Zs lei said:
My purpose is to denoise with NF on both Y and UV plane, so what is the right procedure?

In your shared files, I see Y and UV are both modified from input.
That means both Y and UV channels are already filtered.

0 Zs lei 11 months ago in reply to Gang Hua

Intellectual 660 points

Hi Hua,

Yes, I got the images with same denoising performance for each case if I filp "input_interleaved".

Let me clarify with you about the implication of this setting. In my test cases, the image input to NF node is NV12 format and NF is configured as Bilateral filter.

1. Either 0 or 1 of input_interleaved I will set, the input to NF node should keep same NV12. Is it right? It's unnecessary that separate Y and UV plane for each case. I refer to the case https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1230014/tda4vm-question-on-adding-bilateralnoisefilternode-to-app_single_cam/4645951?tisearch=e2e-sitesearch&keymatch=BLNF#4645951.

2. Please correct my understanding. Assume input is NV12.

If input_interleaved =0, only Y plane is filtered and combined with original UV to generate new NV12 as NF output.

Else if input_interleaved=1, UV is filtered and combined with original Y to generate new NV12 as NF output. It means that "the given hardware module can support either Luma (Non-Interleaved mode) OR Chroma mode (Interleaved mode) at a time" which is described at 6.10.7.4.7 Chroma Handling (Interleaved Mode) of TRM.

3. If I want to filer both Y and UV plane, I think I should run NF node twice with input_interleaved =0(for Y) and 1(for UV). Is it right? "The parameters for Chroma needs to programmed at every frame differently compared to Luma" which is at 6.10.7.4.7 Chroma Handling (Interleaved Mode) of TRM.

If my understanding was right, the outputs for input_interleaved = 0 and 1 should be different. I only run NF node 1 time but with different input_interleaved setting. You can separate output to Y, U, V, and then compare them between these 2 cases, you can see:

1. With visual checking the input of NF node, we can see most noise are at Y plane.

2. Outputs for different cases are almost 100% same. The output of input_interleaved = 1 should be much noiser if only UV is filtered. Shouldn't it be?

Best regards,

Zhishen

0 Gang Hua 11 months ago in reply to Zs lei

TI__Guru 59580 points

Hi Zhishen,

Thanks for the clarification!

Zs lei said:
In my test cases, the image input to NF node is NV12 format and NF is configured as Bilateral filter.

From the H/W perspective, BLNF can only process a Y plane or a UV plane in a single shot (not together).
The interleaved mode bit shall be set according to the format of the input image plane.

To support Y+UV image, PDK driver has to trigger BLNF H/W twice (one for Y and the other for UV).
The interleaved flag you set is probably ignored by the driver for NV12 input format.

+1 Gang Hua 11 months ago in reply to Zs lei

TI__Guru 59580 points

Zs lei said:
3. If I want to filer both Y and UV plane, I think I should run NF node twice with input_interleaved =0(for Y) and 1(for UV). Is it right?

I suppose PDK driver can do that for you in that case according the the e2e thread you referred to above (I am not familiar with driver details though).

In "vhwa_m2mNfApi.c", you can find that 2 iterations are needed for Y+UV.

static void Vhwa_nfSetCfgParams(Vhwa_M2mNfHandleObj *hObj,
                                const Vhwa_M2mNfConfig *nfCfg)
{
    uint32_t itrCnt;
    Vhwa_M2mNfConfig *nfConfig = NULL;


    /* Check for Null pointer */
    GT_assert(VhwaNfTrace, (NULL != hObj));
    GT_assert(VhwaNfTrace, (NULL != nfCfg));


    if ((FVID2_DF_YUV420SP_UV == nfCfg->inFmt.dataFormat) ||
        (FVID2_DF_YUV420SP_VU == nfCfg->inFmt.dataFormat))
    {
        /* Both Luma and Chroma Needs to be processed */
        hObj->numIter = 2U;
    }
    else
    {
        /* Only Chroma/Luma needs to be processed */
        hObj->numIter = 1U;
    }

In your shared BLNF output images, both Y and UV are modified from input.
I guess you have applied patch in the other e2e thread and the PDK driver has already triggered BLNF for both Y and UV.

0 Zs lei 11 months ago in reply to Gang Hua

Intellectual 660 points

Hi Gang,

Yes, I have applied patch in case 4645951 I linked in previous reply. The purpose is to support NV12 input.

I looked through the code you attached. It does iterate twice if input is NV12 to copy nfCfg content to different address of hObj ( nfConfig = &hObj->nfCfg[itrCnt]). But based on your code, the problem is nfCfg is not an array. Therefore, hObj->nfCfg[itrCnt] have same nf configs for 2 iterations. For my understanding, denoise parameters, e.g. sigma_r and sigma_s for Y and UV can be different. This code is showen that nf config is kept same even if input_interleaved is 1. I think it can explain why the dennoising results of my test cases are same comparing between input_interleaved is 0 and 1.

Please help correct me.

Thanks.

Best regards,

Zhishen

+1 Gang Hua 11 months ago in reply to Zs lei

TI__Guru 59580 points

Hi zs,

Zs lei said:
For my understanding, denoise parameters, e.g. sigma_r and sigma_s for Y and UV can be different.

If you need to change Y and UV parameters, it might be better for you to call BLNF for Y and UV separately with your desired parameters.

Zs lei said:
This code is showen that nf config is kept same even if input_interleaved is 1.

The "input_interleaved" field in H/W register is only for input format, it has no other effect on H/W.

Zs lei said:
I think it can explain why the dennoising results of my test cases are same comparing between input_interleaved is 0 and 1.

Since driver can see input image format, it does not need to check inpput_interleaved for anything.

Processors

Processors forum

TDA4VM: Bilateral NF tuning issue