J721S2XSOMXEVM: Weired output behavior of Draw Detection module on TIDL pipeline

Part Number: J721S2XSOMXEVM

Tool/software:

Hello TI experts,

I'm implementing a custom tidl application referring to app_tidl_od_cam & app_tidl_od demo applications.

The application receives nv12 images captured from another application which runs alongside.

The pipeline is as below

Received NV12 image -> Scaler Node --> (Output1) TIDL Node -> DrawDetections Node -> Mosaic Node -> Display Node
                                                              |                                          |
                                                              --> (Output2) ------------------

And the display is very weired, it looks like somehow the DrawDetection Node processes buffers with wrong index.

If I change the flow as the input of DrawDetection Node goes directly to Display Node, the display works find.

Received NV12 image -> Scaler Node --> (Output1) TIDL Node -> DrawDetections Node -> Mosaic Node
                                                              |                                        
                                                              --> (Output2) ----------------> Display Node

So I guess the output of scaler is okay, but somewhere in DrawDetection Node might have problem.

Here is the relevant code snippit below. (used kernel module APIs from vision_apps/modules and app_tidl_od_cam)

static vx_status app_init(AppObj *obj)
{
    vx_status status = VX_SUCCESS;

    /* Create OpenVx Context */
    obj->context = vxCreateContext();
    status = vxGetStatus((vx_reference) obj->context);

    if(status == VX_SUCCESS)
    {
        tivxHwaLoadKernels(obj->context);
        tivxImgProcLoadKernels(obj->context);
        tivxTIDLLoadKernels(obj->context);
        tivxVideoIOLoadKernels(obj->context);
    }

    /* Initialize modules */
    app_init_scaler(obj->context, &obj->scalerObj, "scaler_obj", 1, 2);
    app_init_tidl(obj->context, &obj->tidlObj, "tidl_obj", 1);
    app_update_pre_proc(obj->context, &obj->preProcObj, obj->tidlObj.config, 1);
    app_init_pre_proc(obj->context, &obj->preProcObj, "pre_proc_obj");
    app_update_draw_detections(&obj->drawDetectionsObj, obj->tidlObj.config);
    app_init_draw_detections(obj->context, &obj->drawDetectionsObj, "draw_detections_obj", 1);
    app_init_img_mosaic(obj->context, &obj->imgMosaicObj, "img_mosaic_obj", 4);
    app_init_display(obj->context, &obj->displayObj, "display_obj");

    return status;
}


static vx_status app_create_graph(AppObj *obj)
{
    vx_status status = VX_SUCCESS;
    vx_graph_parameter_queue_params_t graph_parameters_queue_params_list[2];
    vx_int32 graph_parameter_index;

    obj->graph = vxCreateGraph(obj->context);
    status = vxGetStatus((vx_reference)obj->graph);
    vxSetReferenceName((vx_reference)obj->graph, "app_tidl_od_cam_graph");

    app_create_graph_scaler_single(obj->context, obj->graph, &obj->scalerObj, obj->imgs[0]);
    app_create_graph_pre_proc(obj->graph, &obj->preProcObj, obj->scalerObj.output[0].arr);
    app_create_graph_tidl(obj->context, obj->graph, &obj->tidlObj, obj->preProcObj.output_tensor_arr);

    status = app_create_graph_draw_detections(obj->graph, &obj->drawDetectionsObj, obj->tidlObj.output_tensor_arr[0], obj->scalerObj.output[1].arr);
    obj->draw_detect_output_img = (vx_image)vxGetObjectArrayItem((vx_object_array)obj->drawDetectionsObj.output_image_arr, 0);

    vx_int32 idx = 0;
    obj->imgMosaicObj.input_arr[idx++] = obj->drawDetectionsObj.output_image_arr;
    obj->imgMosaicObj.num_inputs = idx;

    app_create_graph_img_mosaic(obj->graph, &obj->imgMosaicObj, NULL);
 
    //obj->display_image = (vx_image)vxGetObjectArrayItem(obj->scalerObj.output[1].arr, 0);
    obj->display_image = obj->draw_detect_output_img;
    //obj->display_image = obj->imgMosaicObj.output_image[0];
    status = app_create_graph_display(obj->graph, &obj->displayObj, obj->display_image);

    if(status == VX_SUCCESS)
    {
        graph_parameter_index = 0;
        add_graph_parameter_by_node_index(obj->graph, obj->scalerObj.node, 0);
        obj->scalerObj.graph_parameter_index = graph_parameter_index;
        graph_parameters_queue_params_list[graph_parameter_index].graph_parameter_index = graph_parameter_index;
        graph_parameters_queue_params_list[graph_parameter_index].refs_list_size = obj->imgs_num;
        graph_parameters_queue_params_list[graph_parameter_index].refs_list = (vx_reference*)&obj->imgs[0];
        graph_parameter_index++;

        vxSetGraphScheduleConfig(obj->graph,
                VX_GRAPH_SCHEDULE_MODE_QUEUE_AUTO,
                graph_parameter_index,
                graph_parameters_queue_params_list);

        tivxSetGraphPipelineDepth(obj->graph, APP_PIPELINE_DEPTH);

        tivxSetGraphPipelineDepth(obj->graph, 5);

        tivxSetNodeParameterNumBufByIndex(obj->scalerObj.node, 1, 4);
        tivxSetNodeParameterNumBufByIndex(obj->scalerObj.node, 2, 4);

        printf("Pipeline params setup done!\n");
    }
	
	return status;
}

 

Could you please let me know what causes the problem?

Regards,

Juhyun

  • Hi Junhyun,

    By any chance, are you using object array as an input or output to the draw node? 

    Regards,

    Brijesh

  • Hi Brijesh,

    Yes, both of the input and the output are all vx_object_arrays.

    But I used vx_image array for scaler input instead of vx_object_array which was intended for app_create_graph_scaler() API.

    The reason I changed it as vx_image was because that the source images came from the another application as vx_reference (vx_image),
    and at that time I didn't know how to create vx_object_array with already existing vx_images, and plus the camera channel was single.

    And then I found that app_create_graph_scaler() seemed to get and use only first element of vx_object_array passed as an input,
    so I decided to make a copy of the API that replaced the input type with vx_image. In the process I also removed vxReplicatedNode because it kept complaining some sort of missing vx_object_array type. I thought the channel was single so that would be okay.

    The related code is,

    app_create_graph_scaler_single(obj->context, obj->graph, &obj->scalerObj, obj->imgs[0]);

    And the copy of app_create_graph_scaler() is as below.

    The graph is almost similar to the one in app_tidl_od, so I tested on that application and I found that the change of scaler input type from vx_object_array to vx_image caused that weired flickering back and forth display issue.

    I still didn't know why it happened, but at least found out what caused it, so got into that vx_object_array problem.

    Fortunately, I could manage to solve it thank to .

    https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1463767/j721s2xsomxevm-vxobjectarray-with-existing-vx_reference/5624248?tisearch=e2e-sitesearch&keymatch=%2520user%253A611727#5624248

    After using vx_object_array for scaler input and original app_create_graph_scaler(), the display issue has been resolved.

    This must be due to my lack of knowlege about OpenVX/TIOVX framework, but It's still hard to understand why the vx_obejct_array thing causes the problem even though the channel is single.

    The problem is gone now, but could you please give me some explanation about this?

    Regards,

    Juhyun

  • Hi,

    The reason I changed it as vx_image was because that the source images came from the another application as vx_reference (vx_image),
    and at that time I didn't know how to create vx_object_array with already existing vx_images, and plus the camera channel was single.

    But is your node replicated in this case? If you are using it for single camera usecase, then please dont call replicateNode API for this node. 

    Regards,

    Brijesh

  • No it wasn't. I removed the vxReplicateNode in the app_create_graph_scaler_single()  which is a copy of app_create_graph_scaler().

    But in the rest of the module's creation APIs, like PreProc, TIDL, DrawDetection, vxReplicateNode is called in the original code regardless of the number of channel, and I didn't touch it, so those modules still called vxReplicatedNode.

    Actually I wonder about them, what is the purpose and the role of those vxReplicateNode in that context?

    Regards,

    Juhyun

  • Hi Junyun,

    Replicate node alone is not an issue, but replicating a node which takes an object array as argument and that object array is more than one element size, that could be an issue for single camera application. 

    Regards,

    Brijesh

  • Hi Brijesh,

    I used TIOVX kernels by module APIs that are provided in vision_apps/modules and vision_apps/apps/dl_demos/app_tidl_od, in a way that app_tidl_od uses.

    Most object arrays used for tiovx kernel api seem to be managed by module APIs itself, and I checked the sizes of those object arrays were all 1.
    And plus for the rest of the object arrays I didn't manually create with more than one element.

    What I just did is changing the vx_object_array input argument of app_create_graph_scaler() to vx_image and removing vxReplicateNode call in the routine.

    Regards,

    Juhyun

  • Hi Juhyun,

    Essentially, please make sure to use object array of size when replicate node is called, for all the parameters, which are replicated.. or for safer side, you can try disabling call to replicate node API. 

    Regards,

    Brijesh

  • Okay, I see. Thank for the advice.

    Regards,

    Juhyun