TDA4VH-Q1: About ISP delay and how to optimize it

Hi wang zhili,

Are you seeing any issue in ISP performance? ISP is running at very high speed, so should not have performance issue. 

wang zhili said:

1. Do you have the data of ISP delay of TDA4? Or is there a way to test the ISP delay on TDA4?

You can get these details by running single camera or multi camera application on EVM. It prints average, min and max delay through ISP as part of performance statistics.

wang zhili said:

2. Do you know any methods of optimizing ISP delay on TDA4?

Lets first see what issue you are facing. 

Regards,

Brijesh 

Hi Brijesh,

The delay (the time from the incoming of a raw image to the outgoing of a YUV image) I test on TDA4VH board is about 102ms, the test method I used is that I opened a second chronograph on PC screen and pointed the camera at the screen, capture the DP output image of the TDA4VH board and the PC screen simultaneously, then calculated the second chronograph values from both screens in the same frame.

But in the application of CMS on car, the system delay must be less than 50ms, so I want to know if there are any ways to optimize the delay on TDA4 or AM62A?

The picture below is the statistics come from single cam app when press p:

The delay of graph is small, does this represent the delay of the ISP?

Hi wang zhili,

I guess you mean latency from capture to the display. You seem to be running single camera application and it requires lot of optimizations in order to reduce the overall latency. As you can see, VISS just takes 4ms to complete a frame. Its not VISS specific issue. Its an application issue. 

Regards,

Brijesh

Hi Brijesh,

Brijesh Jadav said:

it requires lot of optimizations in order to reduce the overall latency

Can you specify these optimizations?

Can the delay of graph (sum of 5 nodes) be considered as all the delays between TDA4 receiving image data and output image data?

Thanks.

wang zhili said:

Can the delay of graph (sum of 5 nodes) be considered as all the delays between TDA4 receiving image data and output image data?

No, this does not include let's say capture time + display time + any of the pipeline delay in the node due to multiple buffers etc. What's the components that you want to use it the example? Is it just capture -> VISS -> DSS? and are those capture and DSS running at 60fps? 

Regards,

Brijesh

Does capture time mean the delay of capturing images in camera? 

Brijesh Jadav said:

What's the components that you want to use it the example? Is it just capture -> VISS -> DSS?

If the "capture" means the camera image capture and "VISS" means the ISP in TDA4 chip, the example is capture -> VISS -> DSS. 

Brijesh Jadav said:

are those capture and DSS running at 60fps? 

Yes.

Do you have any idea how much of the pipeline delay in the node due to multiple buffers etc. is?

wang zhili said:

Does capture time mean the delay of capturing images in camera? 

yes, which is 16.67ms for 60fps capture.

wang zhili said:

If the "capture" means the camera image capture and "VISS" means the ISP in TDA4 chip, the example is capture -> VISS -> DSS. 

ok, so you just want to capture, do isp and display ??

wang zhili said:

Do you have any idea how much of the pipeline delay in the node due to multiple buffers etc. is?

In this case, 16ms + 16ms + 5ms(depending on ISP time) + 16ms(if capture/display not in sync) ~= 55ms.

Regards,

Brijesh

Brijesh Jadav said:

ok, so you just want to capture, do isp and display ?

Yes, that is what I care most and also is the case I use most. 

Brijesh Jadav said:

In this case, 16ms + 16ms + 5ms(depending on ISP time) + 16ms(if capture/display not in sync) ~= 55ms.

Do these delays include the graph delays (5 nodes) mentioned above? Are these 4 delays between the 5 nodes in the graph?

The display monitor is connected with TDA4 board through a DP line, are capture and display synchronized in this case?

Thanks.

wang zhili said:

Do these delays include the graph delays (5 nodes) mentioned above? Are these 4 delays between the 5 nodes in the graph?

but there are no 5 nodes in the above graph. There are only three nodes, csirx -> viss -> dss. There is not any graph delay, also since all three nodes are running on the same core, there is not even IPC delay.

wang zhili said:

The display monitor is connected with TDA4 board through a DP line, are capture and display synchronized in this case?

No, above timings does not consider capture and display synchronized. 

Brijesh Jadav said:

but there are no 5 nodes in the above graph. There are only three nodes, csirx -> viss -> dss

In the picture I upload above (also place here), there are 5 nodes: CAPTURE2, VPAC_VISS1, MCU2-0, VPAC_MSC1, DISPLAY1, and they are included in graph_85.

So I am confused that you say there are only three nodes: csirx -> viss -> dss, could you please explain this for me?

And what is the relationship between the delay showed in picture above and "16ms + 16ms + 5ms(depending on ISP time) + 16ms(if capture/display not in sync) ~= 55ms"? Can I consider the sum of the delay showed in picture above and 55ms as the delay between TDA4 ISP receiving image data and output image data?

Thanks.

Yes, but you wanted to different path, isn't it? CSIRX -> viss -> DSS? 

This application uses csirx -> viss -> msc -> dss

                                               aewb

Regards,

Brijesh

Ok, for 16ms + 16ms + 5ms(depending on ISP time) + 16ms(if capture/display not in sync) ~= 55ms, does these mean that 16ms (capture) + 16ms (csirx) + 5ms (viss) + 16ms (dss)?

Additional 16ms is due to capture and display not running in sync. Ideally it should be 16 csirx + 5ms viss + 16ms, but i used additional 16ms for sync issue.

Does this is all of ISP delay (from obtain raw data to output yuv) in TDA4? 

Yes, i mean its approximate time for single channel processing. 

in the current multi-camera demo, it takes around 4.5ms to process a around 2MP frame.. 

Rgds,

Brijesh 

Is this delay optimal? Can it be promoted?

Yes, this is optimal ISP processing time. 

Thank you.

If I want to test the specific value of the TDA4 ISP delay (from obtain raw data to output yuv), is there a way to do that? Such as adding some timestamp in SDK?

Yes, you can use/add the timestamp, but that's how SDK captures the ISP performance. For which exactly step, do you want to add timestamp?

I want to know the specific delay of the whole ISP, so is there a way to use only one timestamp that start at node CAPTURE2 and end at node DISPLAY1?

I guess you mean latency from capture to display, you would have to add code to measure latency. It is not available in the sdk. 

Fine. So there is no way to test the specific latency from ISP receiving one frame to outputting this frame?

but that's what the openVX performance stats provides, isn't it? When you see min/max/avg time for ISP in the OpenVX framework, it provides time from when frame arrives in ISP to when frame leaves ISP. 

Ok, I got it. Thanks.

As for what you said that "any of the pipeline delay in the node due to multiple buffers etc.", how to understand this delay?

that typically does not happen for the ISP. Its more for the display and capture. Lets say if you have enqueued more than 1 buffer to the display, then the next buffer will definitely take time for the display. You need to make sure there is exactly one buffer enqueued to the display. 

Brijesh Jadav said:

You need to make sure there is exactly one buffer enqueued to the display. 

Do you know where I can find how many buffers have been used?

That you would be enqueueing from the application. 

Is 16ms used for sync issue because the camera is running at 60fps?

Yes, the additional 16ms is due to sync issue between capture and display.. 

Is csirx equivalent to CAPTURE2 node? As we discussed before, the delay of csirx is 16ms, but in the picture I uploaded above, the delay of CAPTURE2 node is only about 4ms, do you know what this is about?

Yes, they are same. It cannot be 4.2ms, if you are running at 60fps, it should be around 16.6ms. Can you please let it run for longer time and then check the performance? 

I keep the camera running for a longer time, but the performance showed the same delay of CAPTURE2 node.

When I changed the camera running mode to 30fps, the delay of CAPTURE2 node is about 33.2ms, but the delay of DISPLAY1 node is only about 8.4ms, while it is about 16.6ms in the case of 60fps. 

Do you know how to make this right?

That's strange. it should not happen.

Are both capture and display running at the same fps? Are there any difference in fps? Lets say display is running bit faster, than when we enqueue a buffer to the display, it might be possible that it return the completed buffer faster, because display vsync could be early. 

Regards,

Brijesh 

Brijesh Jadav said:

Are both capture and display running at the same fps?

Perhaps the display is always refreshing at 60fps, regardless of whether the incoming image is 30fps or 60fps.

In the normal case, the delay of DISPLAY1 node should correspond to the frame rate (such as 16.7ms for 60fps, 33.3ms for 30fps)?

Well in a way yes, but it also depends on when display buffer is enqueued. The process API in the display node first enqueues a buffer to the display driver and then tries to dequeue a buffer from the node. Lets say if the buffer is enqueued very near to vsync, then dequeue will also be called very near to vsync, and so display timings can be shorter in this case.

Regards,

Brijesh