HDVPSS Freezing when DDR is Overused

Hello,

What is the use case that you are using here (pipeline)?

What is the software release as well?

BR

Margarita

The pipeline is essentially (note i do it programatically hence the non-exact syntax):

V4L2src /dev/video0 always-copy=false --> Video Caps (x-raw-yuv,NV12,1920x1080,60,8 buffers) --> omxbufferalloc (8 buffers) --> omxh264enc --> mpegtsmux-->udpsink

V4L2src /dev/video2 always-copy=false --> Video Caps (x-raw-yuv,NV12,1920x1080,60,8 buffers) --> omxbufferalloc (8 buffers) --> omxh264enc --> mpegtsmux-->udpsink

 The VPSS freezing happens regardless of bitrate and # of buffers.  I've tried many different combinations so with no avail.  Video always stops after a certain amount of time.  Hence why i don't think its directly related to the software and more likely related to VPSS crashing due to something like memory.  But again I do not have any way to properly debug this.

Using:

EZSDK 5_05_02_00

OpenMAX IL 0.3

Gstreamer 0.10

yes i am doing dual capture.  the pipelines listed are actually a single pipeline.

Hello,

So, You observed this hang only in case of dual capture, correct?

The default EZSDK does not support dual capture.

BR

Margarita

Apologies Margarita.

For the dual capture part of gstreamer we use the RidgeRun SDK. 

Again I believe the issue to be related to DDR not being optimized and causing the VPSS to fault and I'd really like some assistance on how to more properly debug the VPSS layer (resetting individual components, etc) and potentially better ways to debug DDR not running optimally (mtest passes but maybe something more intensive?)

Margarita,

I haven't had issues getting dual 30fps streams working.  As mentioned in my original post, I have dual 1080p 30FPS working without issue and can run as many items as i want and it will not freeze.

Only when performing dual 60fps do issues arise, and only when running many items in the background. 
So both 60fps streams come through, but then simply stop when performing heavy background operations while streaming. 

The only way to fix this is to reboot or rmmod all the video coprocessor drivers (vpss, ti81xxvin, syslink, cmemk) and re insmod them. 

Without reinitializing all these drivers, this freezing/no video input detected stays even after restarting my gstreamer based application.

Hence why I wanted to know if there is any way to reset just individual parts of the VPSS so i can at least figure out what part is locking up.

RidgeRun says they haven't had issues on the eval board, but again they aren't running my exact software in the background. 

Margarita,

I just determined that this isn't strictly related to 60fps and dual, it is solely related to what bitrate the encoded stream(s) is outputted as. 

I originally that it was solely a 60fps dual issue, but that was only because when doing dual 60fps with a variable bitrate the encoder simply can't constrain the bitrate.  Hence if the stream is 1080p60 the minimum bitrate sent out is always at least 6Mbps regardless of what the actual target bitrate is set at. 

And it seems if the total bitrate is 15mbps or above, the VPSS is inclined to freeze.

So when running a single 1080p60 stream at 15-25mbps, the VPSS will always eventually freeze.

Hence the VPSS lockup is unrelated to the video input, it is solely related to the amount of data being used and hence how overloaded my DDR is.

Any Ideas?  Again i do not know if this is related to my board solely or not.  still determining.

Hey Margarita,

Where are the recommended values for various resolutions?

Also if the recommended value for 1080p60 is 10Mbps, does that mean it gets split in half for dual capture/encode (5Mbps each)?

My bigger issue here is that when the system gets in this "lock up state" it freezes and can't recover.  And with these higher resolutions, it is hard to contain these bitrates when there is high motion on the screen (even with a constant bitrate setup).

So if high motion arises (often in our environment) this results in a sudden bitrate spike and hence a freeze.

-Chris

Margarita,

I implemented a constrained constant bitrate system to ensure the bitrate doesn't bleed higher. 

I have determined the following:

           1x 720p60 at 10Mbps doesn’t lock up(unless heavy background operations are being performed)

            1x 720p60 at 15Mbps usually locks up after 10 min

            1x 720p60 at 20Mbps usually locks up after 5 min.

  2x 720p60 at combined 10mbps doesn’t lock up(unless heavy background operations are being performed)

            2x 720p60 at combined 12Mbps usually locks up after 2 min

            2x 720p60 at combined 20Mbps will lock up right away

 again the lock up situation is when the VPSS layer essentially stops processing video with no errors.

Hello,

Could you confirm that the HDVICP (h264 encoder) is hanging. You could check this if you have the overlay package. You could check it by VIDENC2_process () function in omx_venc.c is returning or not by using traces.

Best Regards,

Margarita

Hey,

I will try and check that out. 

I was able to narrow down that it seems related to the mpegtsmux.  I previously had tested w/o the mux but I ended up doing a faulty test (other issues at that time) so it wasn't accurate.

But it does seem removing the mux and replacing it with a qtmux doesn't show the issues.

Obviously i need this mpegtsmux though.  I think the mux simply is not keeping up with the amount of frames the encoder is feeding and eventually the encoder freezes up.  Again I believe it isn't the mpegtsmux locking up because restarting the application doesn't alleviate the issue (only reseting the full vpss drivers and firmware). 

So basically it seems my only options are somehow get this mpegtsmux to be more optimized or find a way for the encoder to not get in that state.

I put in a very large leaky queue in between the 2 elements and this prevented the lock up (since the encoder could keep passing data) but this queue leaks in packets instead of frames.  Hence the displayed data looks awful. 

I know on previous SOCs (DM368) we didn't have the issue since the encoder would just drop the frame if the mux couldn't accept it which visually looks like a stutter but that is fine. 

It seems here it is doing the same thing (frame drops) but eventually it just stops and that is what Im trying to alleviate.

Hello,

Could you confirm that when your board hang is because the encoder hangs?

You could check the H264_Encoder_HDVICP2_UserGuide.pdf for the allowFrameSkip settings.

Mpegtsmux is non a TI element you could search in the net for similar issue with the mpegtsmux. The mpegtsmux is part of the bad plugin. In the default EZSDK we have gst-plugins-bad-0.10.21 version.

BR
Margarita

I do not how to confirm 100% the encoder is the one hanging, but based on these items i have made that assumption.

• When the freeze happens, encoding no longer works even after application restart. 
  
  • After freeze, running v4l2src -->perf-->fakesink shows data is still being sent all the way through
  • After freeze, running v4l2src-->omxbufferalloc-->perf-->fakesink shows data is still being sent all the way through
  • But after freeze, running v4l2src-->omxbufferalloc-->omx_h264enc-->perf-->fakesink shows data stopping.
  • Hence the omx_h264enc element is causing no data to traverse
• Only way to recover is to unload the vpss, ti81xxvin, and firmware and reload them all.
  
  • Then h.264enc element works again since this essentially forces a reset
• putting a queue leaky queue between the mux and encoder prevents the crash (since encoder always has space to write to)

Basically mpegtsmux not keeping up is an issue, but the main issue is the encoder freezing when this happens.  I don't think we can ever get the mux to a point where it will keep up 1:1 with the encoder, it just doesn't seem possible.  So the only other solution is to prevent the encoder from freezing or have it drop frames in a manor which doesnt cause horrid visual loss (hence frames would have to drop pre-encode)

I enabled allowFrameSkip and the freeze still happened.