TDA4VM: Segmentation Model TIDL Output Dumps

Hi Vyom,

Could you tell me a bit about your application? Are you using any standard SDK segmentation demos?

How many classes are there in your model?

Is it an 8bit, 16bit or float model that you are executing?

Regards,

Nikhil

Dear Nikhil,

I am using t Standard SDK 8.02

There are 14 classes

It is a 8bit model

Thanks and Regards,

Vyom Mishra

May I know which demo are you running? 

Is it app_tidl_seg demo ?

Dear Sir,

No, it is app_tidl_seg_cam, but modified for the required palette and with our ISX016 cameras Integrated

Thanks and Regards,

Vyom Mishra

Hi Vyom,

In the application, the color_map is hardcoded to [5]. I believe you have changed this right?

May I know what are the other modifications done to the application?

Have you tested the ISX016 cameras with the single or multi-cam application and ensure that they are streaming correctly?

Regards,

Nikhil

Dear Sir,

Yes, we have changed the hardcoded value.

Yes, we have tested the streaming, we are also running the application with the mosaic display.

Visualization on display is correctly displayed, only the issue lies in dumps.

existing application app_seg_cam , we have

- added ISX016 camera changes

- X Node is added after the TIDL node is filled by post-processing of segmentation

Camera > Scalar > Pre-processing> TIDL > X Node > Post-processing -> Mosaic-> Display

Thanks and Regards,

Vyom Mishra

Hi,

To understand your issue better.

You are able to stream the modified segmentation demo and get the expected output on the display whereas when you take an image dump from the node, you are seeing the image as attached above?

If my understanding is correct, Is the output dump from mosaic node matching with the display?

Could you help me understand exactly which node output are you taking in your graph?

Regards,

Nikhil

Dear Sir,

Thanks for the response!

Yeah, your understanding is correct.

I am taking dumps from TIDL Node, which is coming as attached above. Mosaic dumps are correct and even on the display it looks as expected.

Thanks and Regards,

Vyom Mishra

Hi Vyom,

The output from the TIDL node, (i.e. tensor.bin file) would only contain the information of about the classes (i.e. it is just a mask). The post processing node tivxPixelVizNode() would give color to each of the mask and overlay it with the original image and it would be same as the one you are seeing on the display.

Hence, output of TIDL node cannot be viewed directly.

Regards,

Nikhil

Dear Sir,

Thanks for the response!

I understand the same, so we have dumped TIDL Output and replaced the class values with a palette in Python for visualization.

So the output attached above is the visualized from the Python script.

Thanks and Regards,

Vyom Mishra

Hello Sir,

Please find the observation:

Case I:

Capture->Scalar->Pre-processing->TIDL

Dumps of TIDL come as expected.

Case II:

Capture->Scalar->Pre-processing->TIDL->X Node

X Node is replicated Node which takes input from TIDL Ouput.

When we dump, the TIDL dumps come wrong.

Can you please help us to know the reason for the same?

Thanks and Regards,

Vyom Mishra

Hello Sir,

The issue is resolved,

Thanks and Regards,

Vyom Mishra

Thank you for the confirmation. 

If you don't mind, could you please let me know what resolved this issue?

Regards,

Nikhil

Dear Sir,

We ran the application for a longer time (>10 minutes) and then checked the dumps.

Earlier we were running it for 2-3mins.

I know this observation is strange for us too.

Thanks and Regards,

Vyom Mishra

Understood. Thank you for sharing this.

Closing this thread.

Regards,

Nikhil

Hello Sir,

There is one observation that needs your suggestions and support

Our Use case has to be this

Capture->Scalar->Pre-processing->TIDL->X Node

This use-case is not stable for dumping the TIDL and X Node dumps

We observed that

if we enable (Post-processing -> Mosiac -> Display) with our expected pipeline

Capture->Scalar->Pre-processing->TIDL-> X Node

                                                                 -> Post-processing -> Mosiac -> Display

(here, TIDL output is shared to X -Node and also Post-processing)

 Post-processing and mosaic with display it comes correctly.

If we disable post-proc and Mosaic with display the dumps don't come as expected

this observation is a little confusing and unexpected.

Could you please let me know your say on this?

Thanks and Regards,

Vyom Mishra

Hi Vyom,

I thought the X-node was part of the pipeline. 

Is X node an end node with no further outputs?

Vyom Mishra1 said:

 Post-processing and mosaic with display it comes correctly.

If we disable post-proc and Mosaic with display the dumps don't come as expected

In both the cases are you taking the dump from output of TIDL node itself?

Could you please confirm if your X node is working as expected when  post-proc and Mosaic with display is enabled?

Regards,

Nikhil

Hello Sir,

We have taken the reference of app_seg_cam application.

Yes, the X-Node is the last node but its output is important to us.

We have added X-Node to the pipeline after TIDL Node and X-Node is the end of the pipeline.

For experiment purposes, we enabled the post-processing, mosaic and display and found that dumps(from the TIDL Node and from X Node) comes properly.

In both cases, we are taking dumps from TIDL Node and X-Node.

Yes, Our X Node is working as expected and giving proper dumps.

Just for Summary:

Our current design is 

Capture->Scalar->Pre-processing->TIDL->X Node

We need to dump the TIDL and X Node output dumps

In this case we do not get proper dumps as shared above.

If we enable the( Post-processing -> Mosiac -> Display) of app_seg_cam

Capture->Scalar->Pre-processing->TIDL-> X Node

                                                                 -> Post-processing -> Mosiac -> Display

Here, X Node output is end, TIDL output is shared to X-Node and Post-processing for experiment purpose, and it gives correct TIDL and X-Node dumps

Thanks and Regards,

Vyom Mishra

Hi Vyom,

Could you please remind me again what are you doing inside X node? I know that it takes in input LUT and output tensor of the TIDL node. May I know what are you doing with this two information internal to X node?

Regards,

Nikhil

Hello Sir,

We are doing a view conversion of the scene.

Thanks and Regards,

Vyom Mishra

Hi Vyom,

I mean, how are these inputs being used?

For eg. Are you modifying the pointers to the tidl output tensor or something else?

I'm trying to understand how the X node is affecting the output dump of TIDL node.

Regards,

Nikhil

Hello Sir,

We are resizing in the X Node.

I ran the sample app_seg file-based application and observed a similar issue at runtime in Mosaic.

I am sharing the images for your reference

Here you can see this image matches the case I mentioned above without even any modification in the segmentation sample application file. It is a default application.

Thanks and Regards,

Vyom Mishra

Hi,

Vyom Mishra1 said:

Here you can see this image matches the case I mentioned above without even any modification in the segmentation sample application file. It is a default application.

No modificaition? You mean without X node being added to the file based seg demo?

Could you tell me what is left and right images in this case?

i.e. When running this demo, when do you get images shown in left and when do you get images shown in right?

Regards,

Nikhil

Dear Sir,

Yeah, these dumps are taken from the default file-based segmentation sample application.

Both the images were taken simultaneously while the application was running.

I have attached the right side image to showcase the frame it comes exactly the way we have experienced in our dumps after adding X Node

Thanks and Regards,

vyom Mishra