This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

H.264 on dm36x - Adaptive Longterm frame insertion

The H.264 platinum codecs feature support for "Adaptive Longterm Frame Insertion", see here:

http://processors.wiki.ti.com/index.php/H.264_DM36x_Ver_2.0_Codec#Adaptive_Longterm_Frame_Insertion

 

 I'm attempting to use this feature in a DM368-based system but I'm not sure I'm setting it up correctly. With the feature turned off and a simulated packet loss of 0.1% the decoder behaves as expected. I get a decode error on the frame with the dropped packet and then every frame after that reports extended error code 0x4f "The current picture refers to an already decoded error frame" ...until the next I-frame comes along and stops the error being propagated. All as expected.

 

With the feature enabled the reporting from the decoder looks correct. I still get the initial error from the frame with the dropped packet, then a couple of 0x4f warnings and then they stop. The error propagation appears to have been halted by instructing the encoder to "UseLongTermFrame=1". However the picture looks awful. It looks to me like setting UseLongTermFrame=1 on the encoder immediately corrupts the image. The result is much worse than the original error left to propagate. I'm assuming I'm not setting LongTermFrames correctly, or there's a mismatch between the references used by encode and decode.

 

My settings are;

EnableLongTermFrame = 1;

LongTermRefreshInterval = 30;

UseLongTermFrame is set to 1 whenever the decoder signals an error

SetLongTermFrame = 0; // Do I have to set this explicitly if I have set the refresh interval?

 

I've also tried setting LongTermRefreshInterval = 0 because the user guide says that I frames are automatically counted as long term frames. However this makes no difference.

 

Hope someone can shed some light on the right way to use this feature.

 

Alex 

  • Alex,

    We will look into this and get back(by middle of next week only). Can you pls help us know whcih codec version you are using. You can find it in the release notes as build id (pls search for h264_encoder_dm365_releasenotes.pdf in the codec package)?

    Is it possible for you to share some streams where you have used LT frame ?

    regards

    Yashwant

  • Yashwant,

     

    Thanks for offering to look into this. I believe we're on the absolute latest encoder; Build ID: 02.20.00.01. The rdecoder is also latest: Build ID: 02.00.00.10. I will try to capture some streams for you. In the mean time perhaps you could clarify some basic questions about the parameters?

    1) What does LongTermRefreshInterval=30 actually do? Does it work the same was as intraFrameInterval? i.e. every 30 frames encoded, 1 frame will automatically be marked as a long term frame. Does it have to be any particular type of frame? I-frame? IDR-frame?

    2) If LongTermRefreshInterval does work like intraFrameInterval is it necessary to use SetLongTermFrame at all? Perhaps this is only if you wanted to force a particular frame as a long term frame say on a scene change?

    3) Does the h.264 decoder definitely support this mode? It's only mentioned in the encoder documentation. Is there anything in the decoder settings required to make it work?

     

    Many thanks,


    Alex

     

  • Replies -

    1. Working is similar to intraFrameInterval. Frame after that refresh cycle will get marked as LT frame. LT frame can be I, IDR or P frame. But all I or IDR frame will automatically become LT frame.

    2. use of SetLongTermFrame is more for system which does not want periodic LT frame insertion or for channels which has QoS feedback. Ezample if a periodic LT frame gets corrupted and encoder wants to send one more.

    3. Decoder is universal and does not need any special parameter for this mode.

     

    regards

    Yashwant

  • Thanks Yashwant, sounds like I'm going about this the right way. Just one more question. After an error is detected, my decoder signals back to the encoder that it should encode the next frame relative to the long term frame, Does it need to keep doing this for every frame until the next periodic IDR frame, or is a single frame encoded against the longterm frame enough to stop error propagation?

    I think if all P frames are only encoded relative to the previous frame, one long term frame should be enough to stop error propagation. However if some P frames reference frames older than the one immediately prior we need to carry on encoding against the long term frame until an IDR frame comes along to reset everything.

    Thanks,

     

    Alex

  • Alex,

    TI DM36x encoder only refers to previous P frame only. So, you need to only set useLongTermFrame for 1 frame only. After that normal encoding can proceed(P frame referring to prev frame). Need not repateadedly send this command for all frames till next IDR.

    regards

    Yashwant

  • Hi Alex,

    What is the delay expected between  decoder and encoder? i.e. if decoder is deocding frame "n" and encoder in encoding frame "m", what is the difference between "m" and "n". Maybe you can reduce the value of LongTermRefreshInterval to a lower value (say 15). LongTermRefreshInterval should be greater than delay between encoder and decoder.

    What is the value intraframeinterval/IDRframeinterval? All intra/IDR frames are marked as longterm.

    >>The error propagation appears to have been halted by instructing the encoder to "UseLongTermFrame=1". However the picture looks awful.

    This could be due to the reference frame being temporally far the current frame being encoded. One solution could be to reduce LongTermRefreshInterval

    Can you share a stream?It will be helpful

  • Hi Adithya,

    >What is the delay expected between  decoder and encoder? i.e. if decoder is deocding frame "n" and encoder in encoding frame "m", what is the difference between "m" and "n".

    I'm testing using an almost direct connection. The delay is typically less than 3 frames.

    >Maybe you can reduce the value of LongTermRefreshInterval to a lower value (say 15).

    I have tried the values of 30 and 10. The result looks the same in both cases.

    >What is the value intraframeinterval/IDRframeinterval? All intra/IDR frames are marked as longterm.

    When I first tried to use this feature intraframeinterval was 75. I also tried it with intraframeinterval=0 in case there was some kind of conflict between generating long term frames and I frames.

    >This could be due to the reference frame being temporally far the current frame being encoded. One solution could be to reduce LongTermRefreshInterval

    I wondered about this which is why I tried the LongTermRefreshInterval of 10. Surely that's good enough?
    Are there any encoder settings which prevent the correct use of long term frames? For example CABAC encoding is on and I'm encoding high profile.
    Many thanks for trying to help me. I will try to record a stream for you.
    Alex

     

  • Hi Alex,

    It should work fine if CABAC, transform8X8  and high profile are enabled. There are no such restrictions. Can you share the encoder parameters like bitrate, resolution, rate control mode.

    Once we look at the stream it will easier to comment on what could be the issue

  • Hi Adithya & Yashwant,

     

    I recorded a stream for you but when I played it back using Movie Player on my Ubuntu desktop the picture looked fine. I think this proves it's a decoder problem because the picture looked quite corrupted when it was decoded by the dm368.

     

    In the attached 20 second steam you should see an initial IDR frame followed by all P frames. On at least two occasions the encoder was instructed to use the long term frame for reference as a result of packet loss detected at the far end. The long term frame settings were:

    EnableLongTermFrame = 1;

     

    LongTermRefreshInterval = 10;

     

    Note - to upload the file I had to change the extension to .txt otherwise the forum wouldn't let me post it. The original extension was .264 and the file is a raw h264 stream with no headers.

     

    4331.videoEncFile.txt

    Thanks,

     

    Alex

     

     

  • Hi Alex,

    The streams looks fine in all the analyzers? This is not an encoder issue.

    I am assuming you are using TI decoder? What is the value of "IH264VDEC_Params->frame_closedloop_flag". Please set it to '0' The output will be corrupted if it is set to 1.

  • Hi Adithya,

    Thanks for checking the steam fro me. We did have frame_closedloop_flag=1, but I've just tried setting it to 0 and it makes no difference. Long term reference frames still produce corruption in the decoder output.

    Are you able to decode the stream I sent using a DM368 ?

    Thanks,

    Alex

  • Hi Alex,

    That is strange. frame_closedloop_flag=0 corresponds to universal decoder. I will check it and let you know

  • Hi Alex,

    We decoded the stream setting frame_closedloop_flag to 0 and decoded video is fine. There is no corruption.

    Are you sure the flag is set to 0. Which version of decoder are you using. You can get the latest decoder from

    http://software-dl.ti.com/dsps/dsps_public_sw/codecs/DM36x/index_FDS.html

  • Hi Adithya,

     

    Sounds like something strange is going on. I was on decoder version 2.00.00.10 so I upgraded to  2.00.00.13 but I'm afraid it didn't make any difference. I can't see anything in the release notes to suggest it should have done either.

     

    I believe I'm setting the closedloop flag correctly, but here is how I create the decoder so you can check: 

     

    memcpy(&pObj->h264Params, &IH264VDEC_PARAMS, sizeof(pObj->h264Params));

      pObj->params.size               = sizeof(VIDDEC2_Params);

      pObj->params.maxHeight          = create->maxHeight;

      pObj->params.maxWidth           = create->maxWidth;

      pObj->params.maxFrameRate       = 0;

      pObj->params.maxBitRate         = ALG_VID_DEC_MAX_BITRATE;

      pObj->params.dataEndianness     = XDM_BYTE;

      pObj->params.forceChromaFormat  = 9;


      pObj->h264Params.displayDelay = 0;

      pObj->h264Params.hdvicpHandle = NULL;

      pObj->h264Params.disableHDVICPeveryFrame = 0;

      pObj->h264Params.levelLimit= 0;

      pObj->h264Params.frame_closedloop_flag= 0; //0 - Universal; 1 - closed loop

      pObj->h264Params.inputDataMode= 1;  

      pObj->h264Params.sliceFormat= IH264VDEC_TI_BYTESTREAM;

      pObj->h264Params.viddecParams = pObj->params;

      pObj->h264Params.viddecParams.size = sizeof(pObj->h264Params);


      /* Create video decoder instance */

      pObj->hDecode = VIDDEC2_create(gALG_hEngine, pObj->algName, (VIDDEC2_Params*)&pObj->h264Params);

      if (pObj->hDecode == NULL) {

        OSA_ERROR("Failed to open video decode algorithm (%s)\n", pObj->algName);

        OSA_memFree(pObj);

        return NULL; 

      }

     

    Later I set the dynamic params like this:

     

        pObj->dynamicParams.size          = sizeof(VIDDEC2_DynamicParams);

        pObj->dynamicParams.decodeHeader  = XDM_DECODE_AU;

        pObj->dynamicParams.displayWidth  = displayWidth;

        pObj->dynamicParams.frameSkipMode = IVIDEO_NO_SKIP;

        pObj->dynamicParams.frameOrder    = IVIDDEC2_DECODE_ORDER;

        h264DynParams.viddecDynamicParams = pObj->dynamicParams;

        h264DynParams.viddecDynamicParams.size = sizeof(h264DynParams);

        h264DynParams.resetHDVICPeveryFrame = 1; // Set to 1 because we are running both the encoder and decoder in parallel

        h264DynParams.dataSyncHandle = NULL;

        h264DynParams.getDataFxn = NULL;    

            

        status = VIDDEC2_control(pObj->hDecode, XDM_SETPARAMS, (VIDDEC2_DynamicParams*)&h264DynParams, &pObj->decStatus);

        if (status != VIDDEC2_EOK) {

          OSA_ERROR("XDM_SETPARAMS failed, status=%ld\n", status);

          return OSA_EFAIL;

        }

     

     

  • Hi Adithya,

     

    A couple of other thoughts occurred to me.

    1) Is there any debug output from the decoder I could turn on that would confirm it's running with closedloop = 0 and/or shed more light on what's going wrong?

    2) From the way the decoded video looks, it's as though the decoder's copy of the long term frame is corrupt, so when it tries to decode a new P-frame relative to that buffer the image become distorted. I'm wondering if that could be caused by a memory problem. Is there anything I can do to make sure the decoder is allocating all the memory it needs and that memory isn't being overwritten?

     

    I don't have an easy way to screen capture a distorted image, but the best way I can describe it is that small patches of the screen that would have contained movement are garbled. I think the patches are macroblocks encoded in the new P-frame which must have been applied to a poor reference point. The rest of the image is fine. So for example if the video shows a person talking, the background behind the person is unaffected, but areas of movement on their face suddenly show the wrong colours and shapes. Since all the following frames are still P-frames, the error persist across all the frames that follow. 

     

    Hope that's useful,

     

    Alex

     

  • Hi Alex,

    It looks like decoder is not getting the correct reference. Are you allocating sufficient number of buffers?

    You can also check with decoder application in the package app/client/test/src/TestappDecoder_ih264_arm926.c. We have tested your stream with this application.

    You can also refer to the link for more about buffer mechanism

    http://www.ti.com/lit/an/sprab88/sprab88.pdf

  • Hi Adithya,

    Thanks for your help here. I've marked it as verified because it was our mistake with buffer management that was causing the corruption. The link you provided made it clear what we needed to do to fix.

    However running in universal mode is bad news for us as it has a significant impact on performance. I thought the idea of closed loop was that it could decode anything the TI encoder produced That is clearly not the case if the decoder can't decode long term reference frames with closedloop=1. Please could we raise a bug report/feature request against this?

    Many thanks,

    Alex

  • Hi Alex,

    We will look into this but please dont expect any near term fix. Closed loop decoder was meant to decode streams of ver 2.0 encoder. We added more features in later version of encoder but the closed loop decoder did not get upgraded.

    regards

    Yashwant

  • Hi Yashwant,

    Thanks for the honest assessment. I see that the last release for the decoder was almost a year ago. Shouldn't a new release be due by now?

    Also, is there anything I can do to speed up the universal decoder since I'm not using lots of other features it has? Perhaps by specifying a lower levelIdc in the encoder?

    All the best,

    Alex