Delay between capture and display of the frame

Dmitriy,

Be aware that if you reduce the number of buffers then you will see tearing and/stuttering in the displayed video due to the input and output frame rates being asynchronous to each other.

If you are OK with this then you should be able to configure a single buffer. I am not a software expert though so will need to defer on exactly how to configure for a single buffer.

BR,

Steve

Hello.

The problem in that I request of system only 1 buffer, and it persistently sets up 3 buffers.

Hi,

Driver, by default, always allocates 3 buffers. Because of it, you always get 3 buffers even if you are requesting for single buffer. I think there will be some boot time parameters by which we can specify the number of buffers to be allocated at boot time. See if it is available and try to set it to 0, so that no buffers will be allocated at boot time. Now you will be able to allocate single buffer from application. There is still a problem with the single buffer. As far as i remember, driver always keeps one buffer with itself, so it will not work with the single buffer. This needs lot of changes in the driver to make it work with the single buffer. Is it ok for your to work with two buffers?

Thanks,

Brijesh Jadav

Hi.

 OK. I willl try to set in bootargs, that the driver would use 0 buffers (I did not try this variant). Since I tried to set at occupancy of system (bootargs) to use only 1 buffer system all allocated 3 buffers. I can probably solve (or to improve a situation) the problem with a delay if I willl use 2 buffers, it is necessary to experiment, but I did not manage to get from system of 2 buffers. If I set to system to use 1 or 2 buffers and as if I request of drivers through VIDIOC_REQBUFS 1 or 2 buffers, I all receive 2 buffers.

With best regards, Dmitriy

I am trying to have a consistent amount of buffers delay between running the application and want to find out if it is possible in the driver?  Did you manage to get a constant buffer delay with setting up only one buffer?

Thanks, Jinh T.

Hello.

At present I did not manage to get from system 1 buffer (it was possible to receive minimally 3 buffers).

Hi,

It is difficult to work with the one buffer. For example, if you fill the buffer, which is getting displayed also, you will see artifacts.  My suggestion is to use two buffers i.e. one given to the driver and one is for application to process. I think this also needs some driver changes. Current driver always expects 3 minimum buffers. You will have to change some macros inside the driver to make use 2 buffers.

Thanks,

Brijesh Jadav

Hi Jinh,

I think you will need to have same pool of buffers for both capture and display and rotate these buffers between capture and display. I think this should make the consistant delay between capture and display.

Thanks,

Brijesh Jadav

If the video input rate and the video output rate are different then you can never make the delay constant, it will always fluctuate. between 2 discrete delays depending on a) which is the faster rate and b) how many buffers you have allocated.

If you only have 2 buffers then you will see more skip/repeat than with 3 buffers.

Basically...

1 buffer will give a delay between 0 and 1 frame periods but will show tearing when the read and write pointers overlap.

2 buffers will give a delay between 1 and 2 frame periods, will not show tearing but the skip/repeat frequency will be high causing un-smooth display.

3 buffers will give a delay between 1 and 2 frame periods if the reading is faster than writing or between 2 and 3 frame periods if reading is slower than writing, will not show tearing but the skip/repeat frequency will be a minimum hence video display will be as smooth as possible (without doing frame locking or interpolation anyhow)

BR,

Steve

Steve,

I agree with you, It is impossible to make constant delay if rate of input and output are difference, even if one rate is integer divisor of other, synchronizing vsync is really very difficult. 

When i meant i could be done, i considered both are running at the same rate.

Thanks,

Brijesh Jadav

Understood :)

If both source and sink are frame locked (running at exactly the same rate) then it is possible to use a single frame buffer without causing tearing or skip/repeat.

To be 100% certain you do need to be able to control the phase relationship, hence delay, between write and read though. This could possibly be achieved by controlling the reset of either the source or the sink.

This would also control the absolute delay between source and sink.

BR,

Steve

Hi Brijesh,

Yes, the same pool of buffers between capture and display is used - being dequeued and queue by the application.  In my case the lenght of the delay is not a problem but it should be the same every time I run the application and this is where I need to find a way to sync.  I am using PCLK mode where PCLK from the VPFE drives the VPBE - I assume this is what Steve describes as "frame locked" - so the output and input is running at the same rate.  My camera is synchronised to a 50Hz signal that is also phase locked to a 1Hz signal which is also driving a strobe light and another part of the system is sending me UDP data for each field (this is encoded as a barcode on the left edge - and for my testing set up to start on zero after every 1Hz and counting to 49 within a PAL frame) so my mission is to see the field exposing the strobe every time I reset the application with the same phase offset relative to the UDP data coming in.  When I was not using a ringbuffer on the UDP which can jitter i was able to expose the flash in synch with the comms but due to the jitter and no buffering I was getting data duplication/loss every now and then which made it unusable.  What I thus need to find a way to do is to sync the processes when STREAMON is called and at this point probably need to know/force which video buffer is used and the same with the comms relative to the 1Hz mother-sync that determines all the other phases in the system.  Any ideas around this or how to sync the video buffers when I initialise my run would be appreciated.

Thanks, Jinh T.

Jinh,

By frame locked I mean that the camera FRAME rate and the display frame rates are exactly the same (derived from the same clock).

I am getting a little confused here now though. I thought your initial issue was the relationship between the camera data coming in and the image being displayed, but it appears your issue is more related to the image coming in from the camera and additional data coming from another source? Is this correct?

What I think you are then trying to do is only display the image which was captured when the strobe fired? Is this correct?

BR,

Steve

Hi Steve,

With my VPBE running from PCLK from the VPFE I have the output and input video sync'd but am unsure if this is what you mean with frame locked.  Before enabling the pclk link the output video what not phase locked to the input and drifting relative to each other.  Is there another way to frame lock it so that "hardware" controls the offset between in and out phase?  I would assume not, as the processing in between the front end and back end is going under software contrl?  This was just one step towards my mission of getting everything that is in sync but running "asynchronously" back in sync as I will explain in the next paragraph to give more detail on my application.

The whole system is sync'd to IRIG time signal with Synchronised Time code Generators locking onto this signal and putting out my 1Hz and 50Hz (and a bunch of other frequencies) reference signals thats all in sync on the falling adge of the 1Hz signal.  This 1Hz (really a 1pulse per second with 80/20 duty cycle) is driving my strobe light and also resetting a counter in my data to start counting again from 0 to 49 for 50 fields in a period of one second.  My camera is sync'd to this 50pps signal as well as my data transmission signal that transmits a UDP packet containing data about the previous field 1.9ms after the falling edge of the 50Hz signal.  What I need to do now is to take the buffered video and the buffered data and insert a left-edge encoded barcode on a video frame (at 25Hz containing 2 fields) but I need this all to have a fixed offset/phase every time I run the application but due to drivers and buffering and probably user-space things happening I get a "between runs jitter" of 2 fields or 2 frames.  The latency of inserting the data can be fixed in post processing, I need to have it "jitter" free with the data always running with a fixed offset to the video.  During a run I don't get any frames dropped or data dropped but although the front end of the system is synchronised it looses it after the video buffering.  An idea is to use the 1Hz signal that is going into my GPIO0 interrupt which I use to put a special marker in the next frame I process after the 1pps event to put unique reference marks on the two fields, as a synchronising event to resynch all buffers but somewhere in the video I still get a shift of 3 fields.

Can I maybe email you my code offline to have a look at what a problem could be.  It is based on the fbdev_loopback demo from evmdm365 running still on dvsdk_2_10_01_18.  I am trying to resolve this issue before upgrading to the latest DVSDK but don't know if any newer drivers would make a difference anyway?

Thanks for the help, Jinh T.

Jinh,

I am not really too familiar with the software side of things to be honest so your code would not be too helpful to me, sorry.

So, what I think is fundamentally the issue here is that you are not currently able to synchronize the phase of the camera and the display.

What you are doing with the clocks is good though, and will ensure there is no drift. Since all the timings are the same for the camera and display (I believe this is true given what you say above), and the clocks are the same then you are correct that the two are therefore frame locked. The phase is not fixed, but this is what we now need to fix :)

One possibility is to trigger a camera reset at some defined point in time. This could be a one off event and triggered by a display vertical sync interrupt. The result would be an (almost) accurate phase relationship between the camera and the display. I would expect the relationship to be well behaved after this since the variation would be small and unless the reset occurs close to a critical processing point it would not matter.

In the above you mention that you seem to have a good lock between the camera and display and it is only the initial relationship which is causing issues now. Does this mean that for a given execution run that the relationship is constant and does not loose sync, or do the camera and display drift apart over time?

BR,

Steve

Hallo Steve,

Yes, you are correct, I am having problems to sync the camera with the display - where display for me really means the point where I take the data that I have received over ethernet and convert it to edge barcode and insert it into the frame being processed.  Your last statement is correct, for a given execution the sync relationship is stable and also mostly the same offset between camera and display for different executions but every now and then the run is on a different offset (giving me a jitter of 3 fields effectively between video and data with regardss to the strobe event).  So, no camera and display drift and it does not lose sync.  Before having a proper ringbuffer on my data reception side, I was putting the data received into the display on the fly and then it was always stable between executions except for two things: the camera is field sync'd to the 1PPS so I got my strobe on data count 0 or 49 depending on the FID being sync'd f1 or f2 with relation to my 1PPS's falling edge (the mother sync event) (for this only the software is restarted and not the camera for when the camera is restarted the FID has a 50% chance being f1 first of f2 first relative to 1PPS) and the second thing being that the data was not buffered so ever so often the asynchronousity (if I can call it like that) caught up and the data captured to the display is jittering inside the run - showing duplicate counts on consecutive fields.  Buffering the udp data solved this so now it is like you said to fix the phase.  I was thinking last night to use my mother-sync-event (a 1PPS interrupt) once-off like you said to use it as the sync-point as up to now I still didn't have a externally referenced sync point and only tried to do it sort-of on-the-fly by getting to a point in the software where the streaming gets started (async in effect) and then clearing my video buffers to use the next buffer coming in as well as the next data coming in but this also gave me jitter between executions.  We are also looking at sending data at 25Hz and not 50Hz as we are essentially processing frames and not fields like in the old analog days (which we are upgrading from a system that used dualport RAM and got the previous' fields data on the fly and clocked it into the video signal) and we could thus have a hand-shake signal on ethernet to also sync the data being sent from the measurement computer to the video-insertion-board (evmdm365).

Doing the camera trigger (which does have an asynchronous reset input not being used now) is also a possibility which will definately solve my FID jitter I mentioned above but will complicate the system a bit and we will look into it as well.

Hope you understand my explanation but please ask if you need more info and thanks for your help on this.

Thanks, Jinh T.

OK, so this is now looking like a synchronization issue between the camera and the ethernet, and not really with the display. The display is just a way for you to check your results?

What I think you need to do therefore is issue a reset to the camera based on an initializer from the ethernet (possibly always do this periodically just to ensure you are always in sync)

I don't think you are going to be able to get around issuing a reset to the camera.

BR,

Steve

You are correct, need need to have a stable sync between the camera and data coming in.  I have added a sync point in my application that resyncs the buffers to a 1PPS event and am sitting at the same point than before using the buffering on the comms and the data seems to be handled correctly now.  What I can see in my data vs video relationship is that it is basically 1 field difference as I did tests with the camera running on a f1-first after 1pps and a f2-first after 1pps.  One thing I spoke about to AntBer is to use the FID which is also reflected in the MODESET.MDFS register in VPFE in order to read the camera phase and do the adjustment in software relative to what I calibrate it as.  We are trying to keep the systme interaction as little as possible because another feature we are looking at is to use our 1PPS signal and the VD from the tvp5146 in order to put a time-reference to a non-synchronised camera but that is another story to be tackled when this one is sorted out.

I view the "display" as my point of operation inside the application and am still confused with regards to the capture driver buffering and it's effect on my synchronisation.  It does look now that it is stable as I am waiting for a whole second at a fixed point in the code and after this point I drain all the buffers from the queue and start from this point then which seems to give a fixed delay relative to capture and data bar the one field offset I get from the camera alignment relative to 1PPS.

I am going to work on the FID status bit now and also try to do the 25Hz data rate from ethernet.   What happens with the data from the display is that it is recorded during a run on a DVD recorder and during the later process of data reduction, going into a frame grabber and the whole recording is processed field-by-field and dring this process the relative offset between the data and video is offset again to align the real event with its data.

Thanks for your input but let me know if you think of anything that can help, Jinh T.

I think fundamentally that the issue will be that you have three timing critical processes occurring and it may not be possible to align them.

The display sub-system does not have any means to be a slave to any other timing, so it must be the master of the system if you want it to be part of the synchronized flow. It is simple to trigger an interrupt at either the v-syncs or at specific lines in the output cycle which software can then use to re-sync both the ethernet and camera, but they must both be capable of effectively being a slave.

This becomes a bigger issue when you move to an asynchronous camera source such as the TVP5146.

In this case your only choice will be to sync the ethernet to the camera input, then triple buffer between the output of your image manipulation and the display output.

BR,

Steve

Hi Steve, thanks for your valuable input.

I am using the tvp5146 on the EVMDM365 board so here goes the rest of my "bigger" issue which looks promising though.

I did some measurements again yesterday afternoon (and after struggling to find bit 15 in MODESET for a while due to too much running around with other things) where I read the FID from MDFS in the MODESET register and thus aligning my data with the camera field offset and with the relative offset I chose I got most of my measurements aligned to the same data packet (all relative to 1PPS) irrespective of the field alignment of the camera.  There were 2 mearuements with a different offset though (the one was 2 fields and the other 4 fields too early on the udp count) so I will revisit the whole flow of things to see that everything is in a suitable place.

As you said, my display subsystem (start_loop function calling the display_frame functions in my modified version of fbdev_loopback) is sort of he master in the system as this is where I aling to 1PPS and clear buffers and set my udp-read-buffer-pointer.  I will ponder on your statement to use v-sync to re-sync the ethernet and camera and let you know.  I am using triple buffering on the capture side and a ringbuffer on the udp which is very long just to be safe now.

Effectively the ethernet is sync'd to the camera as we are transmitting data always at a delay of 1700us after the 50PPS falling edge which is also the signal driving the camera and these are still aligned with 1PPS.  Does this not make the system to not have 3 async processes but only two - being then camera/udp and display processes?  I am a bit confused as to the buffering of the display manupilation and will ponder on it too.

Will let you know what happens , thanks, Jinh T.

Jinh,

When I mentioned three async processes I was really meaning worst case. In your current position you are able to sync 2 of them (camera and UDP) so it reduced the number of async processes to two, yes.

Sorry for the confusion.

BR.

Steve

Hi Steve,

done some more calibration test with the code as it is now using my UDP simulator and it looks good. I calibrated against the state of the software to align my data going onto the display.  I will go out to the field early next week to do more measurements on the real optical system and hope for good results there also.

I am still pondering on the v-sync issue just to use whatever data there is to make the application "bullet-proof" (for our environment it sounds a bit better than "water-tight".

Thanks for your input and enjoy the weekend, Jinh T.

Yeah... field tests are always "fun" !!! :) Hopefully all goes well for you though.

Have a great weekend yourself.

BR,

Steve

Just an update on my application:  got the processes synchronising nicely now at my 1PPS event point after having some hassles with missing the first frame about 7% of the time but found the "bug" and fixed it by using a tight 40ms window after sync to make sure it gets that first frame before carrying on.  Now I need to add some on-the-fly checking as well, as dropping any frames of video or data is not desirable as that will cause the whole system to get out of sync again.

Thanks again for all assitance, Jinh T.

Excellent. Glad you have this working how you need it.

BR,

Steve