V4L2 Capture buffers with old data DM368

I'm interested in this as well as I'm seeing something similar.  I wasn't going to post a question until I checked my logic, but so far I can't see a reason.  Here's a description of my problem....

When I initiate a capture and encode everything is fine and the video looks good.  I've designed my "encoder" as a C++ class that gets instantiated when encoding is started and deleted when encoding is stopped.  So basically everything is created "new" on and encode and everything is closed and deleted when encode is stopped.  I've looked through my code to make sure the DMAI interfaced is closed/deleted/destroyed when encoding is stopped.

The problem is that the first encode is fine, then stopped.  The second encode is fine then stopped.  On the third encode I am seeing exactly what you described.  There is a stuck frame that has a picture that appears to be from a previous encode.  It keeps cycling through the encoder and appears in the video as a flashing image.  Subsequent encodes seem to be the same or perhaps add an addition stuck frame or a corrupted frame.

Hopefully, someone has some ideas on this.

John A

Hi John,

This behavior possibiliy can come if the RSZ output port could not keep up with the pixel clock and do not get enough DDR bandwidth to write the data out. Other possibility is that a VSYNC or HSYNC got missed out.

I would suggest a few things to try that can take us a little closer to the problem:

1. Can you profile the VDINT0/VDINT1 interrupt routines and see the frequency of interrupts?

2. Can you also try decoupling RSZ and Capture, i mean make capture in single shot mode and just make sure that you are getting a new frame everytime. You can pre-fill some location of the capture buffers with say 0xABCDEF12 and can compare the values after getting capture buffer. This will ensure if there is a problem at capture end or at resizer end.

Regards,

Anshuman

Hi, Anshuman,

 

Did you have any ideas on the issue I'm seeing? The issue seems to be that the system flows through as if a frame was captured and the buffer is ready (i.e. the ISRs, select(), and DQBUF), except that once in a while we get old data in the capture buffer. We've further instrumented the system such that when we are done with a capture buffer, we write unique data relative to the index of the buffers, to a certain location in the "used frame". On each capture, we check for the data and if if it exists, we assume the frame is bad because it hasn't been overwritten on the next DQBUF.

Here is an example of the bad frames when we find them:

frame: 3400, period: 33332 usec ( 30.00 fps)                                   
frame: 3500, period: 33332 usec ( 30.00 fps)                                   
frame: 3600, period: 33332 usec ( 30.00 fps)                                   
frame: 3700, period: 33332 usec ( 30.00 fps)                                   
frame: 3800, period: 33332 usec ( 30.00 fps)                                   
JITTER detect .... requeuing frame 3860                                        
JITTER detect .... requeuing frame 3862                                        
JITTER detect .... requeuing frame 3864                                        
JITTER detect .... requeuing frame 3866                                        
JITTER detect .... requeuing frame 3868                                        
JITTER detect .... requeuing frame 3870                                        
JITTER detect .... requeuing frame 3872                                        
JITTER detect .... requeuing frame 3874                                        
JITTER detect .... requeuing frame 3876                                        
JITTER detect .... requeuing frame 3878                                        
JITTER detect .... requeuing frame 3880                                        
JITTER detect .... requeuing frame 3882                                        
JITTER detect .... requeuing frame 3884                                        
JITTER detect .... requeuing frame 3886                                        
JITTER detect .... requeuing frame 3888                                        
JITTER detect .... requeuing frame 3890                                        
JITTER detect .... requeuing frame 3892                                        
JITTER detect .... requeuing frame 3894                                        
JITTER detect .... requeuing frame 3896                                        
JITTER detect .... requeuing frame 3898                                        
JITTER detect .... requeuing frame 3900                                        
JITTER detect .... requeuing frame 3860                                        
JITTER detect .... requeuing frame 3862                                        
JITTER detect .... requeuing frame 3864                                        
JITTER detect .... requeuing frame 3866                                        
JITTER detect .... requeuing frame 3868                                        
JITTER detect .... requeuing frame 3870                                        
JITTER detect .... requeuing frame 3872                                        
JITTER detect .... requeuing frame 3874                                        
JITTER detect .... requeuing frame 3876                                        
JITTER detect .... requeuing frame 3878                                        
JITTER detect .... requeuing frame 3880                                        
JITTER detect .... requeuing frame 3882                                        
JITTER detect .... requeuing frame 3884                                        
JITTER detect .... requeuing frame 3886                                        
JITTER detect .... requeuing frame 3888                                        
JITTER detect .... requeuing frame 3890                                        
JITTER detect .... requeuing frame 3892                                        
JITTER detect .... requeuing frame 3894                                        
JITTER detect .... requeuing frame 3896                                        
JITTER detect .... requeuing frame 3898                                        
frame: 3900, period: 39982 usec ( 25.01 fps)

Anshuman Saxena said:

Hi John,

This behavior possibiliy can come if the RSZ output port could not keep up with the pixel clock and do not get enough DDR bandwidth to write the data out. Other possibility is that a VSYNC or HSYNC got missed out.

I would suggest a few things to try that can take us a little closer to the problem:

1. Can you profile the VDINT0/VDINT1 interrupt routines and see the frequency of interrupts?

2. Can you also try decoupling RSZ and Capture, i mean make capture in single shot mode and just make sure that you are getting a new frame everytime. You can pre-fill some location of the capture buffers with say 0xABCDEF12 and can compare the values after getting capture buffer. This will ensure if there is a problem at capture end or at resizer end.

Regards,

Anshuman

Anshuman, I kind of put this on the back burner while converting my code to perform interlaced encoding.  Now that I'm encoding interlaced the capture and RSZ is decoupled.  I just tried restarting the encoder a bunch of times and now I am not experiencing the problem.

Can you elaborate on how to profile the interrupt routines? Do you mean to edit the code, put in some counters, and compare them against the time?

It seems odd to me that the issue would be the RSZ not keeping up.  It works the first time every time.  It's only after a restart, which entails closing everything and reopening.  Seems like a buffer is circulating but not getting filled by the capture.

John A

 

It seems we have been able to "fix" the problem by providing a buffer of pixels (8) around each edge of the frame. With this in place, we no longer get capture buffers with old data.

This still doesn't explain why the RSZ_INT_DMA IRQ fires when there isn't any data being written to RAM (even at the same 30Hz interval).

Are you saying that you made the buffers 16 pixels wider and taller, then set and X & Y coordinates to 8 before capturing?

John A

Hi John,

Yes, we made our image sensor provide an extra 16 pixels in x and y, move them through the pipeline, and essentially have the resizer "crop" the pixels by starting at the 8th pixel and line of the image. So the SRC_[V|H]PS  is 0, and RZ[A|B]_I_[V|H]PS are set to 7.

 

It seems that this problem has reared its ugly head again, even with our original 8-pixel border "fix". Below is a description of what we see:

We are experiencing intermittent issues with the D36x resizer output (our application is heavily based on DVSDK 3.1), which we think is related to race conditions between the capture interrupts and/or timing available for the resizer to complete its write before the next frame starts. The effect is a capture buffer that hasn't been filled with up-to-date information, yet V4L2 allows the buffer to be de-queued. This seems to happen more often when the resizer is traversing the entire fame for a full-frame downscale (1792x1344 to a simultaneous 640x480 AND 320x240 for example).

The DVSDK capture software uses two ISRs that ping-pong. One is the VDINT0 (triggered on VD but delayed for the number of lines in the valid frame), and the other is the RSZ_INT_DMA. We think that both ISRs are happening, due to the way the ISR code expects buffer pointer states to be in before action is taken.The job for RSZ_INT_DMA is to change the output address for the next resized frame, and the job of VDINT0 is to ready the resizer by triggering RSZ_ENA/B. However, it seems the dma complete ISR triggers before it is actually finished.

The way we tell frames are bad is as follows. Before we requeue a capture buffer, we write an impossible YUV value into the beginning of the frame, and a different one at the end of the frame. Every time we dequeue a buffer, we check these values to see if the capture buffer has been updated. Most of the time, the capture buffer is updated, but in the worst case, we see that 1 in about 100 frames doesn't get written to the capture buffer. Moreover, we see that the data at the start of the frame has been overwritten, but the data at the end of the frame has not. Thus we concluded that the resizer is starting, but not finishing, but sill firing the dma complete interrupt to singal that frame complete.

Several questions:

1. The VPFE documentation describes an Output Bandwidth Regulator, but I cannot find any registers where I can check the values for the resizer A and B bandwidths (256/N bits per cycle, where N is a settable variable via register). Can this be changed via register access?
2. Our sensor is outputting 1792x1344 @ 30fps, with a frame blanking of ~1.8ms (80MHz clock). Does the blanking need to be higher in order to allow the resizer to finish writing to SDRAM before the next frame starts?
3. Are there any other priority adjustments we can set to make sure the resizer finishes in time? In the worst-case, we seem to get bad frames once in about 100 frames, but once a frame is bad, we can get many in a row.
4. What is the expected behavior if the resizer cannot complete the SDRAM writes before the VDINT0 interrupt fires. The code suggests that if the dma interrupt hasn't fired, then the VDINT0 ISR will just skip through.

 

Derek,

Were you ever able to resolve this?  I seem to be running into something similar on a DM365 and a 2.6.37 kernel, but it's only presented itself after I made the required changes to enable the 2nd output.

-M

We've not been able to fix the problem, but have successfully gotten around it by writing impossible YUV values to the first and last couple of pixels. We check each captured buffer for these values, and if they exist, we throw the buffer away.

Interesting that you've discovered this with the 2nd output; we are in fact also using the 2nd output, and didn't notice (but weren't looking for it either) when we only had one output working.

We traced it down to the the frame complete interrupt firing without data actually being complete iirc.

Derek,

I'll keep that in mind as I go through this.  Naturally, my goal is to never lose a frame.

I captured 4xD1 channels at 15fps for many hours without any resizer problems.  Only after I added the second output (which is not supported in the 2.6.37 kernel for some reason) did I start having issues.  I was kind of hoping that it was a problem with how I added the second output, but others have seen this issue as well.

Mark