MCFW freezes periodically

Dmitry Bodnya

Expert 1140 points

Other Parts Discussed in Thread: DM388

Hi!

I am working with DM388 IPNC board from APPRO.

RDK version is 3.8.0. Sensor type is SONY IMX136.

The firmware flashed in nand is original binaries from RDK(I flashed it special - to see problem in original soft).

When I am running RDK everyrhing is working - 3 channels are compressed - FullHD/D1/JPEG.

To make a problem I am inserting SD card, waiting while sys_server will mount it and then ejecting SD card.

From the moment I eject card frame skipping begins in MCFW. When it happens I see such statistics:

[m3video] CH | In Recv In Skip In User Out Latency
[m3video] Num | FPS FPS Skip FPS FPS Min / Max
[m3video] --------------------------------------------
[m3video] 0 | 30 2 0 27.49 127 / 172
[m3video] 1 | 30 2 0 27.37 10 / 44
[m3video] 2 | 30 2 22 4.60 139 / 177

"2' in skip column doesn't meen only 2 frames skipped. It meens that during 30 seconds average skip ratio is 2!!!

After that such skipping happens with different periods. If I inserting SD card back everything comes ok.

After some debug I observed that video freezes at some link during ~1-2 seconds. No frames compressed at this time.

The fact is video on display(HDMI) doesn't stop, but video on RTSP stream freezes for such period.

Also I observed(through the "top" app, through telnet) that in such moments of freezing ipnc_rdk_mcfw.out app is taking 100% of cpu time.

I added some debug prints in bios6 side of links and I observed that frames skipping is happend in encLink on bios6 side. In file mcfw/src_bios6/links_m3video/iva_enc/encLink_common.c:1580 there is comment:

/* Free the input frame if processQue is full */

So, here skipping is happened; Some link dosn't take ready frames/bitbufs in time and it raises skipping.

For me it looks like some link, working on A8 is getting paused for too long. It can be a part of Linux system that polls SD card appearing. It can be a strange ipnc_rdk_mcfw.out app behaviour when it take all cpu time. But I can't figure out what exactly happens.

Could somebody help me?

Does anybody see such freezing?

Thank you.

over 11 years ago

Badri Narayanan over 11 years ago

TI__Guru 59700 points

I am not familiar with IPNC RDK but am familiar with MCFW links and chains. Your analysis is correct.

The typical link connection is

encLink -> ipcBitsOut (VideoM3) -> ipcBitsIn(A8)

If encLink finds no free output buffers it will free the input frames to avoid propagation of frame drop to previous links.

If your cortex A8 load is 100% then ipcBitsIn link will not get a chance to run and your application will not be invoking Venc_getBitsstreamBuffers/Venc_putBitstreamBuffer API. Since bitstream buffers are not being consumed encLink will not have free output buffers and it will result in frame drop at input of encLink. You will see discontinuity in encoded stream.

As you mentioned it looks like busy loop inside kernel driver code that is causing 100% CPU load when card is removed. You could try using strace/oprofile when card disconnect occurs to see if you can isolate where busy waiting happens.

Also refer this thread where Pavel discusses SD card disconnect detection and handling

RE: DM8148 Using CD_DAT3 as SD Card Detection pin

Dmitry Bodnya over 11 years ago in reply to Badri Narayanan

Expert 1140 points

Hi Badri.

Thanks for reply.

I am not familiar with strace/oprofile but I will try. May be you know is it somewhere ready binaries for arm?

And I can't understand.. Video freezes for 1-2 seconds. If the root of the problem is in system code working around SD card - I can't believe it's normal behaviour..

Freezing process doesn't depend on FPS. It can happen with 60, 30, 15 fps..

Anand Kulkarni71405 over 11 years ago in reply to Dmitry Bodnya

TI__Expert 8920 points

Hi Dmitry,

I am able to reproduce the problem on DM388 IPNC.

Also i could figure out the root cause.

I am working on the proper fix for the issue.

I will get back to you with the fix soon.

regards,

Anand

Dmitry Bodnya over 11 years ago in reply to Anand Kulkarni71405

Expert 1140 points

Hi Anand!

Glad to hear good news!

Will wait for your fix.

Thank you!

Anand Kulkarni71405 over 11 years ago in reply to Dmitry Bodnya

TI__Expert 8920 points

Hi Dmitry,

Can you test with the attached uImage?

regards,

Anandhttp://e2e.ti.com/cfs-file.ashx/__key/communityserver-discussions-components-files/716/2514.uImage.7z

Dmitry Bodnya over 11 years ago in reply to Anand Kulkarni71405

Expert 1140 points

Anand,

I will be able to check it only on Monday, unfortunatly..

I will write you back as soon as I can.

Thank you.

Dmitry Bodnya over 11 years ago in reply to Anand Kulkarni71405

Expert 1140 points

Hi, Anand!

I did a test and with your kernel I don't see frame skipping in any case - with or without SD card!

That's great!

What fix did you made? Do you have patch or something to move it to my own kernel build?

Anand Kulkarni71405 over 11 years ago in reply to Dmitry Bodnya

TI__Expert 8920 points

Hi Dmitry,

Pl. find the modified file attached (..\ipnc_psp_arago\kernel\drivers\mmc\host\omap_hsmmc.c).

The change is in the function 'omap_hsmmc_reset_controller_fsm()'.

2402.omap_hsmmc.c

regards,

Anand

Dmitry Bodnya over 11 years ago in reply to Anand Kulkarni71405

Expert 1140 points

Yes, it works!

Thank you, Anand!

P.S. How did you find it? Did you use strace/oprofile?

Processors

Processors forum

MCFW freezes periodically