PROCESSOR-SDK-DRA8X-TDA4X: Multiple Instances of VISS with different tuning parameters for a custom Graph

Hi,

The graph above is not coming proper. So attaching in image format.

Regards,

Krishna

Krishna,

How much is the difference between configuration in viss0 and viss1?

Rgds,

Brijesh

Hi Brijesh,

VISS0 has to work on RGB frames, whereas VISS1 will be working on monochrome frames.

VISS0 will have following modules: BLC + NSF4 + WB + Demosaic + CCM + Gamma + YUV conversion + Defect pixel correction.

Whereas VISS1 will have BLC + NSF4 + Gamma + YUV conversion + Defect pixel correction

Regards,

Krishna

Hi Krishna,

Is NSF4 configuration different? Also is Gamma table different in both the cases? 

The reason i am asking these questions, we currently don't support different camera configurations, but if configuration is minimal, we can try calling some ioctl to get it working..  

NSF4 and gamma both the modules have big configuration registers, so there would be overhead in switching these configurations.. 

Regards,

Brijesh

Hi Brijesh,

The gamma table will be same for both the cases.

We prefer NSF4 configurations to be different for VISS1 and VISS2.

Hi Brijesh,

Any update on this?

In the meantime, I was trying to have a single instance of VISS which will switch between the configurations for very alternate frames. Currently, I am trying with very minimal configuration changes to avoid the overhead. It gets reflected on the display. But the problem with this is, each frame is going to the display. Whereas, I want just the RGB frame to go to the display.

So basically, VISS will work for each frame (60 FPS) and fill its output accordingly. The display also takes all the frames and display it.

1)Can you suggest how can I send the alternate frames to the display i.e effectively the display should operate at 30 FPS?

2) I am planning to break the graph in two parts.

First graph: Capture Node => VISS

Second graph: Scalar Node => Display.

At run time, I plan to dequeue the output of VISS (from first graph) and enqueue every alternate frame to the scalar node (second's graph input). But currently, the dequeue for Scalar node is not working and it hangs.

Can you please provide your inputs for this approach? Also, what is advisable, create two different openVX graphs or a single openVX graph would suffice?

Is there any better approach available which you would suggest?

Regards,

Krishna

Hi Krishna,

If you are ok keeping gamma table same. it should be easier. Yes, we can disable NSF in instance and enable it in another. 

Essentially, create two VISS instances, one with DCC file and next without DCC file. In the second instance, somehow, we need to set nsf4Bypass flag to true. 

Now with this change, second VISS instances will reuse all tuning parameters from first instance and since nsf4 is bypassed, driver will take care of switching it off.

Could you please try it out?

Regards,

Brijesh 

Hi Brijesh,

I gave it a quick try. The streaming is working fine but I could see continuous flickering on the display. I will check further on this and update you.

But even with the two instances of VISS, the problem is both the instances of VISS will be running continuously whereas we need each VISS to run alternatively. Basically, the camera and the custom module will be streaming at 60 FPS and we need both the VISS to run effectively at 30 FPS.

Regards,

Krishna

Hi Krishna,

That needs to be done outside of VISS. VISS can process the frames as it comes, but cannot schedule the processing. 

Regards,

Brijesh

Hi Krishna, 

Another probably Hacky way is to update VISS node to process alternate frames differently. But you still need to identify the alternate frame for further processing in the application.

Rgds,

Brijesh

Hi Brijesh,

Yes, I understand that this has to be done in the application as VISS would keep processing the frames as and when it receives them.

I wanted your input from application point of view on how to control this. I tried splitting the graph in two parts. Then dequeue the output of first sub-graph and as per the need, enqueue it to the input of second sub-graph. The downside of this approach is I need to do extra memcpy every time from the output of first sub-graph to the input of second sub-graph. Even if I skip the memcpy and just enqueue empty buffers to the seconds sub-graph, I can see performance drop.

Can you suggest on how can I optimize this or should I consider another approach all together.

Regards,

Krishna

Hi Krishna,

But why do you do memcpy? you could just dequeue from first graph and enqueue in the second graph same frame, no need to copy it.

Rgsd,

Brijesh 

Hi Brijesh,

I had tried that. But the openVX framework keeps a record of the graph parameter queue. So for any graph parameter, I can pass only the buffer which is associated with the queue with which that graph parameter was registered. If I try to pass any buffer which is not part of that queue, it gives me invalid ref error.

I am not able to give same queue as well to both the graph parameters.

Regards,

Krishna

Hi Krishna,

I currently don't see any other way, but let me check internally.

Rgds,

Brijesh

Hi Jesse,

Yes, I did try it. But even in this case, I was getting some error from the framework. But I was not sure of the correct way of doing it.

I have split my graph in 2 parts and the output of first sub graph goes to the input of second sub graph. My understanding is while creating the node, I have to pass the first instance of the set of buffers to the node and then use this set of buffers to register the graph parameter queue.

1) Same vx_graph object: First case is I use same graph object to contain both the graphs. Although, I dont see any error during graph verify, but at run time it hangs after processing few frames.

a) Since I dont want the graph to be interconnected, I have used 2 different buffers while creating the output node of first graph and the input node of seconds graph i.e one buffer is the output for the last node in first graph and second buffer is the input for the first node in second graph. That's why I had two set of buffers for the graph queue. Although I had tried giving same set of buffers to both the queues, I was not very hopeful as the first instance of buffer will be different for both the nodes during their creation.

2) Different vx_graph object: Here I use different graph objects for different graphs. Here, my first graph runs whereas second graph does not run at all. I am not sure why. In this case is it possible to give same buffer to the output node of first graph and input node of second graph. Generally, if it was part of same graph, and I give same buffer then the node gets connected. What will happen if I give same buffer to different nodes of first graph? I have not tried this, will try soon and update.

Regards,

Krishna

Hi Jesse,

I am adding a pseudo code to make my above statements more clear.

/* Graph 1: CaptureNode -> Custom Module
 * Graph 2: Custom Module ->           VISS (RGB) Node ------------------------>         Display
 *                                                                                                                              |
 *                                                                                                                             \ /
 * Graph 3: Custom Module -> VISS (Monochrome) Node ---------> NodeXYZ
 *
 * => Camera is running at 60 FPS.
 * => Custom Module will output rgb and monochrome raw frames alternatively
 * => Application will decide the output of Custom Module will go to which instance of VISS
 * => Consequently, both VISS will have to run at 30 FPS each.
 * => The output of VISS (RGB) Node and VISS (Monochrome) Node both goes to the last node (NodeXYZ)
 */

/* Case 1: Use single vx_graph to have all the sub-graphs shown above */

tivx_raw_image customOut[N], viss_input_rgb_raw[N], viss_input_mono_raw[N];
/* Create the raw images in a loop of N */

vx_graph graph = vxCreateGraph(context);
vx_node cap_node = tivxCaptureNode(graph, config, cap_frames[0]);
vx_node customMod = createCustomNode(graph, cap_frames_raw_in, customOut[0]); // cap_frames_raw_in is extracted from cap_frames which is object array
vx_node rgb_viss = tivxVpacVissNode(graph, .... viss_input_rgb_raw[0], .. viss_rgb_out ...); // viss_input_rgb_raw is input of viss and viss_rgb_out is the output
vx_node mono_viss = tivxVpacVissNode(graph, .... viss_input_mono_raw[0], .. viss_mono_out ...); // viss_input_mono_raw is input of viss and viss_mono_out is the output

/* .... Create other nodes */

vx_graph_parameter_queue_params_t parameter[4];
parameter[0].refs_list = (vx_reference*)&cap_frames[0];
parameter[1].refs_list = (vx_reference*)&customOut[0];
parameter[2].refs_list = (vx_reference*)&viss_input_rgb_raw[0];
parameter[3].refs_list = (vx_reference*)&viss_input_mono_raw[0];

/* Graph execution */
while(1)
{
    raw_image image, rgbImage, monoImage;
    image = dequeue(CustomMod);
    rgbImage = checkDequeue and dequeue(rgb_viss);
    monoImage = checkDequeue and dequeue(mono_viss);

    if(image is monochrome)
    {
        copy(monoImage, image);
        enqueue(mono_viss, &monoImage);
    }
    else
    {
        copy(rgbImage, image);
        enqueue(rgb_viss, &rgbImage);
    }
    enqueue(customMod, &image);
}

This runs for few frames and hangs.

My understanding is while creating the node, I have to pass the first instance of the buffer. For example, while creating "rgb_viss" node, I have given viss_input_rgb_raw[0] as argument which is the first instance of the buffer list which I have used in "parameter[2].refs_list = (vx_reference*)&viss_input_rgb_raw[0];". That's why I get error if I give some buffer at node creation but a different set of buffers as the queue. That's why I am not able to register same queue for the graph parameters.

/* Case 2: Use multiple vx_graph for all the different graphs shown above */

// Here I just use different vx_graph graph objects for each pipeline. In this case just the first graph runs and other two does not run at all

Regards,

Krishna

Krishna,

Krishna Mohan said:

/* Case 1: Use single vx_graph to have all the sub-graphs shown above */

* => Consequently, both VISS will have to run at 30 FPS each.

[/quote]

Krishna Mohan said:

=> Camera is running at 60 FPS.

Case 1 (all in same graph) will not work if you have capture node running at 60fps, and you want to conditionally execute one VISS node or the other at half that rate.  For each invocation of the first node, all other nodes need to run at same rate.  For this reason, you have to split the graph into pieces where application can control the flow rate of each graph.

Therefore, I am interested in the logic you used for case 2, but you didn't show it in your previous post.  Can you share the Case 2 logic?

I assume for case 2, graph 1 with capture node will have 2 graph parameters:

parameter_g1[0].refs_list = (vx_reference*)&cap_frames[0];
parameter_g1[1].refs_list = (vx_reference*)&customOut[0];

Then for graph 2 (rgb) and graph 3 (monochrome): you can reuse the same ref list from the output of graph 1, though it will be a different graph parameter, associated with each of those 2 graphs:

parameter_g2[0].refs_list = (vx_reference*)&customOut[0];

parameter_g3[0].refs_list = (vx_reference*)&customOut[0];

Then, in runtime, the application should dequeue frames from g1 parameter[1], and enqueue that buffer into either g2 or g3, as application determines.

Because you will be needing to manage enqueueing and dequeuing buffers for 3 different graphs at different rates, it may be easier to have an event handler which is triggered on vxEventWait() and selects which graph parameter to enqueue/dequeue based on the event ID you assign it.  Please reference the Event handling section of the pipelining extension spec for example code and API description: