vdec2_process() function call hangs indefinitely

Hi alok,

The error message is basically saying that the VIDDEC2_process function call failed with extended error code 0. You would have to refer to the decoder's documentation to find out what it means.

Is this h262 decoder a TI codec? If so I'd suggest you post this question on the Multimedia Software Codecs Forum. Folks over there may have more insight into the internals of your particular decoder.

Best regards,

Vincent

Hi Alok,

If not already, please share the stresm which hangs the codec indefintely. I assume it is reporducibel on the stand alone decoder. 

Best Regards

Feroz

Hi David,

 As discussed, 

Is your stream (elementary) consistently failing the “standalone” codec? 

Can you cut it into a few MB before the hang and reproduce/share please? Or does it need 1.75 GB to be passed for the crash? 

Best Regards

Feroz

Our Application which shows the reported error behaviours is based on the dvsdk sample applications. Of course, there are some modifications that we have made for displaying to display device etc but the core streaming framework is the same as in the dvsdk sample applications.

We will provide a stripped minimal data dump of the stream on which we have seen these behaviours. However the reported problems happen intermittently and so we cannot say for sure that it will be reproduced with the data dump and after how long. At our end we could see the problem occurring after a run of 3 hrs but there are times when it has failed much sooner. The total length of the video playback is about 20 minutes which is streamed in the "repeat" mode. So from this particular failure case, we can infer that the same video was successfully played back about 8-9 times before hanging on the vdec2_process() call.

So to summarize, we need to find out what causes vdec2_process() to hang. We will need to go beyond the vdec2_process() call invocation to the internals to understand in what situations this happens. We think that the problems occur when the state of the decoder or vdec2 becomes invalid/corrupted over a long run. It would be hard for us to specifically pinpoint the codec as the problem area. It could also be the DVSDK framework since our application calls only DMAI functions such as vdec2_process(). So it needs to be checked if the same errors can be reproduced at your end.

As for speedy resolution of the reported problems it will be of great help if we can get in direct contact with the development teams of vdec2 (dvsdk) and codec over a teleconference so that they can provide some insight on the internal handling in dvsdk and codec’s in which the reported scenarios might occur (as per their respective  design).

Hi Feroz,

We tried to strip down the data dump(1.75 GB) to about 500MB but after stripping it We are not able to play the stripped version in VLC. We were able to play the original data dump of 1.75GB in VLC. Seems VLC cannot detect some of the headers which are needed to start the playback. So, We are not sure if this stripped data dump be useful. Please let us know if the stripped data can be used and will suffice in the meanwhile, we will try uploading the original dump to the TI FTP.

Hi Alok,

Unfortunately, it will be extremely difficult for the codec team (or anybody in TI) to do something unless there is a crashing test case.  

Is it under erroneous conditions that you see the hang? Or for any normal stream? I would request you please try to see what scenarios codec crashes and capture the ingoing stream. if required infuse errors artificially.

Are you able to reproduce the issue when you strip down the system to only H.264 decode? It may not be codec issue at all. 

We will need to isolate and root cause.  I see no other way.

Best Regards

Feroz

Hi David, Alok,

Any updates on this? Could you isolate the issue for crash yet? 

BTW, did the latest HD H264 codec work with McASP simultaneously with SD video? That issue is resolved I hope.

This is the same latest H264  codec isn't it? For Mpeg2/SD playback there is no such hang issue, correct? Please confirm.

From what I understand the H264 process call is not hanging but somewhere in the vdec2_process the code is hanging. is my understanding correct?

Best

Feroz

Hi Alok,

The trace just shows non fatal error. Is it with this special stream that your application crashes too ?

Could you put in more traces within vdec2_process around the H264 process call?  

We need to ensure the same issue is seen both systems before we can attack DVSDK.. Else the effort will be futile. 

Also suggest you use the latest codec.

Best Regards,

Feroz

Hi Feroz,

Thanks for the discussion on the teleconference.

Can you provide us the exact location in code where you want us to put debug prints inside the vdec2_process so as to convincingly conclude whether it is the codec or the vdec2_process that hangs indefinitely. Since this is part of the TI code We just want to be 100% sure that we put the debug prints in the precise location.

Best Regards,

Hi Alok,

I guess you will have to do a trial and error as to how much trace you want to generate and where. Firstly perhaps you may want to see the buffers’ status before the call and after the call to vdec2_process to get an overall picture and frame number.

As suggested, please keep the inputs stream constant every run, so the errors are predictable and not random.

You may want to put some traces around the key functions that you talked about (getFreebuffs getdisplayBufs etc) and the buffers returned or marked by them.

This can be the start and you can analyze further depending on the results. There is no great need for a precision. Please use your gut feel on where the buffers are getting locked up.

It is really important that you can get a thorough hold of the vdec2 layer in order to root cause/debug the issues of your system. So it is worth spending time to understand the exact control and data flows of the system. For designing Avsync , this is going to be crucial.

Best Regards,

Feroz

We have traced the Vdec2_process call as follows:

Vdec2_process :
dvsdk_demos_3_10_00_16/dm6467/decode/video.c ----Application level call to Vdec2_process

VIDDEC2_process :
 dmai_2_10_00_12/packages/ti/sdo/dmai/ce/Vdec2.c

VISA_enter :
codec_engine_2_25_05_16/packages/ti/sdo/ce/visa.c

fxns->process :
codec_engine_2_25_05_16/packages/ti/sdo/ce/video2/viddec2.c
( process call will enter codec_engine_2_25_05_16/packages/ti/sdo/ce/video2/split/viddec2back_stubs.c or codec_engine_2_25_05_16/packages/ti/sdo/ce/video2/split/viddec2front_stubs.c or
codec_engine_2_25_05_16/packages/ti/sdo/ce/video2/viddec2_stubs.c )

VISA_exit : codec_engine_2_25_05_16/packages/ti/sdo/ce/visa.c

***************************************************************************************************************
Tried using TraceUtil (as given in Codec Engine Application Developers Guide):
<http://processors.wiki.ti.com/index.php/Codec_Engine_Application_Developers_Guide>

Added following line in dvsdk_demos_3_10_00_14/dm6467/decode/decode.cfg

/* Enable SoCrates tracing in the demo */
var TraceUtil = xdc.useModule('ti.sdo.ce.utils.trace.TraceUtil');
TraceUtil.attrs = TraceUtil.SOCRATES_TRACING;

Enabled tracing in the application code in the "videoThrFxn()" as :
TraceUtil_start(envp->engineName);

DSP/BIOS logging enabled in DSP Server image by adding the following to cs2dm6467_1_00_00_08/packages/ti/sdo/server/cs/server.cfg
/* activate BIOS logging module */
var LogServer = xdc.useModule('ti.sdo.ce.bioslog.LogServer');

Expecting:
GPP-side trace information is stored in the /tmp/cearmlog.txt file, DSP-side trace information is placed in /tmp/cedsp0log.txt
but we get
root@dm6467t-evm:/home/ALOK# ls -al /tmp/
drwxrwxrwt    2 root     root         4096 Jan  9  2012 .
drwxrwxr-x    7 root     root         4096 Jun 21  2010 ..
-rw-r--r--    1 root     root            0 Oct 22  2010 .automount-hda1
-rw-r--r--    1 root     root            0 Oct 22  2010 .automount-hda2
-rw-r--r--    1 root     root            0 Jan  9  2012 bioslog.dat
-rw-r--r--    1 root     root           89 Jan  9  2012 cearmlog.txt
prw-rw-rw-    1 root     root            0 Jan  9  2012 cecmdpipe
-rw-r--r--    1 root     root            0 Jan  9  2012 cedsp0log.txt

Just to summarize, we have not been able to get any trace from within the vdec2_process in DVSDK (even after enabling TraceUtil). Could you please tell us what could be going wrong?
***************************************************************************************************************

Can you please provide any update on the problems mention in our previous post. We are stuck at this point and any clues for solving the same will be very helpful.

Alok,

Could you implement the "age logic" which I had told you about? That should have root caused the issue or even resolved this.

Also, if trace is not working, could you try some other means like may be printfs or your own print dumps? Not sure why trace is a deadlaock

Best Regards

Feroz

I'd forget about TraceUtil for now - has some good use cases, but for what you want, getting trace should be as simple as setting CE_DEBUG and running your app.

http://processors.wiki.ti.com/index.php/CE_DEBUG

What version of CE are you using?

Chris

Hi Chris,

We are using the following dvsdk and codec server:   

dvsdk_3_10_00_16/ codec_engine_2_25_01_06 

Setting CE_DEBUG = 3 does not show us any additional logs as stated in the documentation link. Do we need to re-compile the dvsdk with any particular setting?

Hi,

Can you please provide any update/information on the previous query.

Shouldn't have to recompile - those trace msgs are built into the CE libraries.  During runtime, within the CERuntime_init() fxn, CE looks at the CE_DEBUG environment variable and, if set, enables the appropriate trace.

Depending on your shell, "how you set the environment variable" may be different(?).  Or if your app is complex, it may be that you need to set CE_DEBUG in a different process (e.g. if CERuntime_init() is called from a daemon process rather than your app).

Chris