H.264 on dm36x - Adaptive Longterm frame insertion

Alex,

We will look into this and get back(by middle of next week only). Can you pls help us know whcih codec version you are using. You can find it in the release notes as build id (pls search for h264_encoder_dm365_releasenotes.pdf in the codec package)?

Is it possible for you to share some streams where you have used LT frame ?

regards

Yashwant

Yashwant,

Thanks for offering to look into this. I believe we're on the absolute latest encoder; Build ID: 02.20.00.01. The rdecoder is also latest: Build ID: 02.00.00.10. I will try to capture some streams for you. In the mean time perhaps you could clarify some basic questions about the parameters?

1) What does LongTermRefreshInterval=30 actually do? Does it work the same was as intraFrameInterval? i.e. every 30 frames encoded, 1 frame will automatically be marked as a long term frame. Does it have to be any particular type of frame? I-frame? IDR-frame?

2) If LongTermRefreshInterval does work like intraFrameInterval is it necessary to use SetLongTermFrame at all? Perhaps this is only if you wanted to force a particular frame as a long term frame say on a scene change?

3) Does the h.264 decoder definitely support this mode? It's only mentioned in the encoder documentation. Is there anything in the decoder settings required to make it work?

Many thanks,

Alex

Replies -

1. Working is similar to intraFrameInterval. Frame after that refresh cycle will get marked as LT frame. LT frame can be I, IDR or P frame. But all I or IDR frame will automatically become LT frame.

2. use of SetLongTermFrame is more for system which does not want periodic LT frame insertion or for channels which has QoS feedback. Ezample if a periodic LT frame gets corrupted and encoder wants to send one more.

3. Decoder is universal and does not need any special parameter for this mode.

regards

Yashwant

Thanks Yashwant, sounds like I'm going about this the right way. Just one more question. After an error is detected, my decoder signals back to the encoder that it should encode the next frame relative to the long term frame, Does it need to keep doing this for every frame until the next periodic IDR frame, or is a single frame encoded against the longterm frame enough to stop error propagation?

I think if all P frames are only encoded relative to the previous frame, one long term frame should be enough to stop error propagation. However if some P frames reference frames older than the one immediately prior we need to carry on encoding against the long term frame until an IDR frame comes along to reset everything.

Thanks,

Alex

Alex,

TI DM36x encoder only refers to previous P frame only. So, you need to only set useLongTermFrame for 1 frame only. After that normal encoding can proceed(P frame referring to prev frame). Need not repateadedly send this command for all frames till next IDR.

regards

Yashwant

Hi Alex,

What is the delay expected between  decoder and encoder? i.e. if decoder is deocding frame "n" and encoder in encoding frame "m", what is the difference between "m" and "n". Maybe you can reduce the value of LongTermRefreshInterval to a lower value (say 15). LongTermRefreshInterval should be greater than delay between encoder and decoder.

What is the value intraframeinterval/IDRframeinterval? All intra/IDR frames are marked as longterm.

>>The error propagation appears to have been halted by instructing the encoder to "UseLongTermFrame=1". However the picture looks awful.

This could be due to the reference frame being temporally far the current frame being encoded. One solution could be to reduce LongTermRefreshInterval

Can you share a stream?It will be helpful

Hi Adithya,

>What is the delay expected between  decoder and encoder? i.e. if decoder is deocding frame "n" and encoder in encoding frame "m", what is the difference between "m" and "n".

I'm testing using an almost direct connection. The delay is typically less than 3 frames.

>Maybe you can reduce the value of LongTermRefreshInterval to a lower value (say 15).

I have tried the values of 30 and 10. The result looks the same in both cases.

>What is the value intraframeinterval/IDRframeinterval? All intra/IDR frames are marked as longterm.

When I first tried to use this feature intraframeinterval was 75. I also tried it with intraframeinterval=0 in case there was some kind of conflict between generating long term frames and I frames.

>This could be due to the reference frame being temporally far the current frame being encoded. One solution could be to reduce LongTermRefreshInterval

Hi Alex,

It should work fine if CABAC, transform8X8  and high profile are enabled. There are no such restrictions. Can you share the encoder parameters like bitrate, resolution, rate control mode.

Once we look at the stream it will easier to comment on what could be the issue

Hi Adithya & Yashwant,

I recorded a stream for you but when I played it back using Movie Player on my Ubuntu desktop the picture looked fine. I think this proves it's a decoder problem because the picture looked quite corrupted when it was decoded by the dm368.

In the attached 20 second steam you should see an initial IDR frame followed by all P frames. On at least two occasions the encoder was instructed to use the long term frame for reference as a result of packet loss detected at the far end. The long term frame settings were:

EnableLongTermFrame = 1;

LongTermRefreshInterval = 10;

Note - to upload the file I had to change the extension to .txt otherwise the forum wouldn't let me post it. The original extension was .264 and the file is a raw h264 stream with no headers.

4331.videoEncFile.txt

Thanks,

Alex

Hi Alex,

The streams looks fine in all the analyzers? This is not an encoder issue.

I am assuming you are using TI decoder? What is the value of "IH264VDEC_Params->frame_closedloop_flag". Please set it to '0' The output will be corrupted if it is set to 1.

Hi Adithya,

Thanks for checking the steam fro me. We did have frame_closedloop_flag=1, but I've just tried setting it to 0 and it makes no difference. Long term reference frames still produce corruption in the decoder output.

Are you able to decode the stream I sent using a DM368 ?

Thanks,

Alex

Hi Alex,

That is strange. frame_closedloop_flag=0 corresponds to universal decoder. I will check it and let you know

Hi Alex,

We decoded the stream setting frame_closedloop_flag to 0 and decoded video is fine. There is no corruption.

Are you sure the flag is set to 0. Which version of decoder are you using. You can get the latest decoder from

http://software-dl.ti.com/dsps/dsps_public_sw/codecs/DM36x/index_FDS.html

Hi Adithya,

Sounds like something strange is going on. I was on decoder version 2.00.00.10 so I upgraded to  2.00.00.13 but I'm afraid it didn't make any difference. I can't see anything in the release notes to suggest it should have done either.

I believe I'm setting the closedloop flag correctly, but here is how I create the decoder so you can check: 

memcpy(&pObj->h264Params, &IH264VDEC_PARAMS, sizeof(pObj->h264Params));

  pObj->params.size               = sizeof(VIDDEC2_Params);

  pObj->params.maxHeight          = create->maxHeight;

  pObj->params.maxWidth           = create->maxWidth;

  pObj->params.maxFrameRate       = 0;

  pObj->params.maxBitRate         = ALG_VID_DEC_MAX_BITRATE;

  pObj->params.dataEndianness     = XDM_BYTE;

  pObj->params.forceChromaFormat  = 9;

  pObj->h264Params.displayDelay = 0;

  pObj->h264Params.hdvicpHandle = NULL;

  pObj->h264Params.disableHDVICPeveryFrame = 0;

  pObj->h264Params.levelLimit= 0;

  pObj->h264Params.frame_closedloop_flag= 0; //0 - Universal; 1 - closed loop

  pObj->h264Params.inputDataMode= 1;  

  pObj->h264Params.sliceFormat= IH264VDEC_TI_BYTESTREAM;

  pObj->h264Params.viddecParams = pObj->params;

  pObj->h264Params.viddecParams.size = sizeof(pObj->h264Params);

  /* Create video decoder instance */

  pObj->hDecode = VIDDEC2_create(gALG_hEngine, pObj->algName, (VIDDEC2_Params*)&pObj->h264Params);

  if (pObj->hDecode == NULL) {

    OSA_ERROR("Failed to open video decode algorithm (%s)\n", pObj->algName);

    OSA_memFree(pObj);

    return NULL; 

  }

Later I set the dynamic params like this:

    pObj->dynamicParams.size          = sizeof(VIDDEC2_DynamicParams);

    pObj->dynamicParams.decodeHeader  = XDM_DECODE_AU;

    pObj->dynamicParams.displayWidth  = displayWidth;

    pObj->dynamicParams.frameSkipMode = IVIDEO_NO_SKIP;

    pObj->dynamicParams.frameOrder    = IVIDDEC2_DECODE_ORDER;

    h264DynParams.viddecDynamicParams = pObj->dynamicParams;

    h264DynParams.viddecDynamicParams.size = sizeof(h264DynParams);

    h264DynParams.resetHDVICPeveryFrame = 1; // Set to 1 because we are running both the encoder and decoder in parallel

    h264DynParams.dataSyncHandle = NULL;

    h264DynParams.getDataFxn = NULL;    

    status = VIDDEC2_control(pObj->hDecode, XDM_SETPARAMS, (VIDDEC2_DynamicParams*)&h264DynParams, &pObj->decStatus);

    if (status != VIDDEC2_EOK) {

      OSA_ERROR("XDM_SETPARAMS failed, status=%ld\n", status);

      return OSA_EFAIL;

    }

Hi Adithya,

A couple of other thoughts occurred to me.

1) Is there any debug output from the decoder I could turn on that would confirm it's running with closedloop = 0 and/or shed more light on what's going wrong?

2) From the way the decoded video looks, it's as though the decoder's copy of the long term frame is corrupt, so when it tries to decode a new P-frame relative to that buffer the image become distorted. I'm wondering if that could be caused by a memory problem. Is there anything I can do to make sure the decoder is allocating all the memory it needs and that memory isn't being overwritten?

I don't have an easy way to screen capture a distorted image, but the best way I can describe it is that small patches of the screen that would have contained movement are garbled. I think the patches are macroblocks encoded in the new P-frame which must have been applied to a poor reference point. The rest of the image is fine. So for example if the video shows a person talking, the background behind the person is unaffected, but areas of movement on their face suddenly show the wrong colours and shapes. Since all the following frames are still P-frames, the error persist across all the frames that follow. 

Hope that's useful,

Alex