This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

H.264 decoder buffers leak

Hi,

We use H.264 HP decoder 01.01.04.00. In some cases it does not release all the frames.

You can find demo that reproduces the issue by the following URL:

drive.google.com/.../edit

Facts about the demo:

- It runs on 8 cores

- Decoder works on first 4 cores

- Input is read in cycle from 0x90000000 address. You should load stream.264 file (from archive) that contains demo H.264 elementary stream by this address. Do it before running application using Load Memory tool.

- You can see from logs that decoder does not release all acquired input buffers. That is why with time application runs out of memory:

Frames hold by decoder 42
Reading input decoder buffer from beginning
inputID 148
[C66xx_3] Decoder process error -1. Extended error 0x1095 - non-fatal error
[C66xx_0] outputID 129
[C66xx_2] Decoder process error -1. Extended error 0x1095 - non-fatal error
[C66xx_0] freeBufID 129
freeBufID 148
Frames hold by decoder 41
Reading input decoder buffer from beginning
inputID 149
Frames hold by decoder 42
Reading input decoder buffer from beginning
inputID 150
freeBufID 150
Frames hold by decoder 42
Reading input decoder buffer from beginning
inputID 151
Frames hold by decoder 43
Reading input decoder buffer from beginning
inputID 152
[C66xx_3] Decoder process error -1. Extended error 0x1095 - non-fatal error
[C66xx_0] outputID 133
[C66xx_2] Decoder process error -1. Extended error 0x1095 - non-fatal error
[C66xx_0] freeBufID 133
freeBufID 152
Frames hold by decoder 42
Reading input decoder buffer from beginning
inputID 153
Frames hold by decoder 43
Reading input decoder buffer from beginning
inputID 154
freeBufID 154
Frames hold by decoder 43
Reading input decoder buffer from beginning
inputID 155
Frames hold by decoder 44
Reading input decoder buffer from beginning
[t=0x00000003:1157c42f] ti.sdo.ipc.heaps.HeapMemMP: ERROR: line 744: E_memory: Out of memory: handle=0x8460c8, size=2289280
ti.sdo.ipc.heaps.HeapMemMP: line 744: E_memory: Out of memory: handle=0x8460c8, size=2289280
Failed to create frame
Failed to create frame
[C66xx_1] Error in decoder on another core

Full log is also in archive.

I use the following components:
- XDCtools 3.30.3.47
- EDMALLD 2.11.11
- FC 3.30.0.06
- IPC 3.22.2.11
- SYS/BIOS 6.40.2.27
- H.264 decoder 01.01.04.00

Please help to resolve the issue.

Regards,

Andrey Lisnevich

  • Hi Andrey,

    We downloaded the demo and when we try to import the project it is throwing following error.

    "Error: Import failed for project 'decode_demo' because its meta-data cannot be interpreted. Please contact support "

    Did we miss anything here ?

    Thanks,

    Praveen

  • Hi Praveen,

    I use CCS v6 for Linux. Just created empty workspace, successfully imported and built the project. Nothing missing.

    Regards,

    Andrey Lisnevich

  • Hi Andrey, we have a newer release which is currently under validation. Some issues were fixed there so I was wondering if you can give a quick try to see if your issue was one of those.

    4011.C66x_h264hpvdec_01_01_04_02_ELF_32bit.bin

    8640.C66x_h264hpvdec_01_01_04_02_ELF_32bit.exe

    thank you,

    Paula

  • Hi Paula,

    Newer release just hangs with this input. Also my original implementation of lock reports assertion error:

    int32_t TranscodeComponent_lockAcquire(int32_t user_id, void* lockHandle)
    {
        assert(lockHandle != NULL);

        int32_t* lockUserId = (int32_t*) lockHandle;
        GateMP_Handle gate = GlobalGates_getTranscodeComponentGate();

        while (1) {
            IArg gateKey = GateMP_enter(gate);

            Cache_inv(lockUserId, sizeof(*lockUserId), Cache_Type_ALL, TRUE);

            if (*lockUserId == -1) { // Lock is released
                *lockUserId = user_id;
                Cache_wb(lockUserId, sizeof(*lockUserId), Cache_Type_ALL, TRUE);
                GateMP_leave(gate, gateKey);
                return IVIDMC3_LOCK_ERROR_NONE;
            } else {
                assert(*lockUserId != user_id);
            }

            GateMP_leave(gate, gateKey);

            // Wait until lock is released
            do {
                Cache_inv(lockUserId, sizeof(*lockUserId), Cache_Type_ALL, TRUE);
            } while (*lockUserId != -1);
        }
    }

    assert(*lockUserId != user_id) - means that decoder tries to acquire lock twice.

    Regards,

    Andrey Lisnevich

  • HI Andrey,

    We fixed issue with lock acquiring twice and attached the updated library. 3022.H264_4xLib_20140820.zip

    Compared to version 01.01.02.07, in this library we added support for BP concealment features( DDR3 increased by 2 MB to add those features) and fixed some bugs. 

    Also, this library we tested on all the combination of cores, so it should run on any 4-cores, please check.

    Thanks,

    Praveen

  • Hi,

    I experience the issue again. When there is problems in input stream and decoder process call returns non-fatal error sometimes buffers are not released.

    I am using C66x h264hpvdec 01.01.04.03

    Regards,
    Andrey Lisnevich
  • Hi Andrey,


    Is it possible to share a stream to replicate the issue ?

    And, can you please check with the latest attached library ?

    Thanks,

    Praveen.8228.H264decoder_4core_library.zip

  • Hi Praveen,

    Latest attached library also leaks buffers.

    I am not able to reproduce the issue in demo. But let me share with you logs and stream:

    https://drive.google.com/folderview?id=0Byw88ezNrM71YXEwX0VuenNfb28&usp=sharing

    From logs you can see that two buffers leaked (after about 5 minutes of working): 7829 and 7817

    buffers in use: 7829 14777 14773 7817 14765 14771 14775

    Buffers leak very rarely (not always) and only when there are problems in input stream (i.e. non-fatal errors in decoder)

    Regards,

    Andrey Lisnevich

  • Hi Praveen,

    Do you have any news on the issue?

    Regards,
    Andrey Lisnevich

  • Hi Andrey,

    I am currently working on this , not able to reproduce the issue at our end with the stream you shared. But by using logs, I am doing code inspection to find the bug. I will keep you posted on the status.

    Meanwhile if it is possible for you to share a stream which reproduces the issue will be very helpful for me to debug quickly.

    And, is this issue observed only when using 4-cores ? Is 1-core or 2-core decoder working fine?

    Thanks in advance,

    Praveen

  • Hi Andrey,

    We have fixed one issue in free buffer logic and attached the updated library. Please use the attached library and let us know the results.

    7484.H264d_4core_lib_20150122.zip

    Thanks,

    Praveen.

  • Hi Praveen,

    Unfortunately the library doesn't fixes this buffer leak.

    But good news is that we've found out how to reproduce - input the same h.264 frame twice.

    For example:

    Iteration1: input frames a,b

    Iteration2: input frames b,c

    In this case output buffer (most likely relatet to frame b) may leak.

    Regards,

    Andrey Lisnevich

  • Hi Praveen,

    Are you able to reproduce the issue using the description above?

    Regards,
    Andrey Lisnevich
  • HI Andrey,

    I am not able to reproduce the issue. The below are the steps I followed to reproduce the issue.

    a. I am running decoder on 4-cores

    b. I am feeding same h264 frame twice as follows. As there is a dummy process call for every alternate actual process call, we feed two frames of input data (frames 1, 2) for both calls. So now i am feeding the same input buffer (frames 1,2) to the next two process calls also. Because the application don't know the starting point of input frame 2. 

    C. For the next two process calls, i am giving next two frames of input data (frames 3, 4) and same input buffer (frames 3,4) to the next two process calls also. Like this i am feeding same input buffer twice. With this setup i am not able to reproduce the issue. 

    Is that is correct, please let me know if i miss anything

    Thanks for you help,

    Praveen

         

  • Hi Praveen,

    > Because the application don't know the starting point of input frame 2.

    As we see it is essential to feed frames' data like 1,2 and then 2,3. To do it we know size of each frame and feed application frame by frame.

    For example you can use "ffprobe -show_frames stream.h264" to see frame sizes.

    Regards,
    Andrey Lisnevich
  • Hi Andrey,

    Got it, I will try this and let you once I reproduce the issue.

    Thanks,
    Praveen.
  • Hi Andrey,

    I have extracted the frame offsets and feed to the decoder as you suggested (frames 1,2 and then 2,3), but did not observe any buffer leakage.

    I have decoded around 1000 frames, there is no issue. After decoding how many frames you observe the leakage issue ?

    If possible can you share the output logs ?

    Thanks,
    Praveen
  • Hi Praveen,

    This is the demo: drive.google.com/.../view

    Just run it on 8 cores. Stream that causes leak already built in into demo - DecoderDemoVector.h. For each frame it contains small header with frame size and then raw frame data. Run the demo less than a minute and you will see logs of buffers in use like this:

    [ 89 271 453 635 817 894 902 904 910 912 914 ]

    In this case buffers 89, 271, 453, 635 and 817 leaked.

    Regards,
    Andrey Lisnevich

  • Hi Andrey,

    Thanks for sharing the demo.

    We could able to replicate the issue now by using the demo. We are debugging the issue and we will revert back with fix.

    Thanks for your help,

    Praveen

  • Hi Andrey,


    We fixed the issue and validated the library in the demo. I have attached the updated library. Please validate the library at your end and let me know  for any questions

    1145.H264d_4core_lib_20150206.zip

    Thanks a lot,

    Praveen

  • Thanks Praveen,

    This build fixes the buffer leak.

    Regards,
    Andrey Lisnevich