memory leak in EZSDK 05.05 (gst-omx)

I found the memory leak is still there when media file is removed and gs-launch is run (error situation).

Any pointer from Ti experts where the loss might be happening?

Salim

Hello,

I will check it and I will let you know the results.

What is the EZSDK version that you are using here 5.05.01 or 5.05.02?

Best Regards,

Margarita

Thanks Margarita,

The version is 5.05.02.

Kindest Regards,

Salim

Any findings Margarita?

Thanks and

Kindest Regards,

Salim

I even tried to free the cached memory via cat 3 > /proc/sys/vm/drop_caches, the result is the same.

Regards,

Salim

Hello Salim,

I will check it for memory leak.

An easy way to debug the memory leak of a GStreamer program that is running is by looking at the reference counters. In most of the cases the memory leaks on gstreamer happened because reference counters on one or more objects never go back to zero.

One of the debug variables is GST_REFCOUNTING. With that, you can get a log of all the refcount changes that happened inside a gstreamer.

Setting GST_REFCOUNTING

Add the following to your pipeline's command line parameters:

--gst-debug=GST_REFCOUNTING:5

Another option is to set the GStreamer debugging shell variable, then run your pipeline:

export GST_DEBUG=GST_REFCOUNTING:5

Filtering debug output

Here are  example on how to filter the debug output.

You can see which objects disappear, meaning their reference count reaches zero:

gst-launch --gst-debug=GST_REFCOUNTING:5 videotestsrc num-buffers=3 ! fakesink 2>&1 | grep "\->0" | cut -d' ' -f 19

You could try to debug it.

Best Regards,
Margarita

Thanks Margarita,

I will wait for any results from your side and at the same time perform some more tests on my side.

BTW, will GST_REFCOUNTING be halpful to detect an leakage caused by gst-omx as well as omx components? I doubt that.

What is good way to measure the memory usage / leakage by gst-omx and omx?

Also, in above reported issue, the leak is observed after application/process exit, the leakage observed might not be from gstreamer / omx but from other shared services like syslink that might not be encapsulated by the process but used by process. (However this does not mean gst-omx and omx do nt have linux side memory leakage. We will need different method to check for gst-mx / omx side memory. Please share if you know of reliable and effective way).

Rgds,

Salim

Hello,

After few tests I can say that, the gstreamer plugin and OMX are not the cause for the problem.

I will let you know when I have something.

Best Regards,

Margarita

Thanks Margarita,

I too had same inference. This is because, the memory depletion is present even after the process invoking gstreamer and omx is killed.

High suspicion is on Syslink.

Surprisingly, VPSS omx components too use Syslink but do not demonstrate this issue.

So, I think that VICP2 side syslink specifically might be causing the issue.

Kindest Regards,

Salim

Hi Margarita,

We are facing same issue in our DM8168 with EZsdk setup.

After multiple restart of gstreamer pipeline, we are getting error

 CRITICAL **: g_omx_port_allocate_buffers: assertion `port->buffers[i]' failed.

Do you have any update on this memory leak issue?

Thanks,

Abeesh.

HI Abeesh,

I do not know about your test environment, but few hints may help you:

The error CRITICAL **: g_omx_port_allocate_buffers: assertion `port->buffers[i]' failed.

will happen only if previous playback has not be closed in clean way (decoder resources mainly shared video buffer) have been released.

There could be many reason for previous playback not getting close normally:

- Is you are using gst-launch command line, in the middle of the playback you press ^C and sometimes the pipeline stops without releasing the video buffers

- There may be timestamped problem in the media file, where EOS might not be received as expected.

Hence, check in your pipeline when you stop is are all resources begin released before being re-allocated in the next playback.

Rgds,

Salim

Hello,

It would be better to post this issue in the DM8168 forum.

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/717.aspx

Best Regards,

Margarita

Hi Salim,

We are using C application instead of using gst-launch.Before starting pipeline again, am doing this much state changes in pipeline and unref the pipeline.

       gst_element_set_state (hsc1_fact_live.Bin, GST_STATE_PAUSED);
        gst_element_set_state (hsc1_fact_live.Bin, GST_STATE_READY);
        gst_element_set_state (hsc1_fact_live.Bin, GST_STATE_NULL);
        gst_object_unref (hsc1_fact_live.Bin);

Is there anything extra needs to be done ?

Thanks,
Abeesh

HI Abeesh,

After each of gst_element_set_... try to use gst_element_get with NULL as parameter instead of timeout.

Kindest Regards,
Salim

Hi salim,

Am using that, The code snippet I have sent u was old one,New one is like this,

int gst_state_check(GstElement *element,int trans_state)
{
    int timeout=0,state;

    while(1)
    {
        state = gst_element_get_state(element,NULL,NULL,-1);
        if(trans_state !=GST_STATE_PAUSED)
        {
            if(state ==  GST_STATE_CHANGE_SUCCESS){
                return 1;
            }
        }
        else {
            if((state ==  GST_STATE_CHANGE_SUCCESS ) || state == GST_STATE_CHANGE_NO_PREROLL){
                return 1;
            }
        }
        usleep(500);
        timeout++;
        if(timeout > 10)
        {
            g_print("ERROR!!!!!!!!!!! state change failed: state %d for state_transition=%d\n",state, trans_state);
            return -1;
        }
    }
}

        gst_element_send_event(hsc1_fact_rtp_stream.Bin, gst_event_new_eos ());
        gst_element_set_state(hsc1_fact_rtp_stream.Bin, GST_STATE_PAUSED);
        gst_state_check(hsc1_fact_rtp_stream.Bin ,GST_STATE_PAUSED);
        gst_element_set_state(hsc1_fact_rtp_stream.Bin, GST_STATE_READY);
        gst_state_check(hsc1_fact_rtp_stream.Bin, GST_STATE_READY);
        gst_element_set_state(hsc1_fact_rtp_stream.Bin, GST_STATE_NULL);
        gst_state_check(hsc1_fact_rtp_stream.Bin, GST_STATE_NULL);
        gst_object_unref (hsc1_fact_rtp_stream.Bin);

Hi, 

Have you tried with the Ridgerun plugins? they are based on the latest gstreamer 0.10.36, the version on the EZSDK is older so you get the latest fixes available in the gstreamer libraries. Now, we have an improved version of the plugins included in the EZSDK (option 1) in the below post, we have fixed several bug fixes, It is included in the Ridgerun Eval SDK 

http://e2e.ti.com/support/dsp/davinci_digital_media_processors/f/717/t/327198.aspx

Ridgerun Eval SDKs:

http://www.ridgerun.com/www/index.php/download-center.html

However, those already have problems to move between states and if you keep destroying instances of the plugins that are implemented on the VPSS you will see a problem, in the above thread.

Now, we are creating a new set of plugins (option 3) in the thread you can see how you could use them from the eval as well.

@ Salim Mohamed: Did you solve your problem? We can try to reproduce it at Ridgerun.

@ ABEESH M P: As Salim mentioned the allocate buffers assertion happens when something was not closed properly, you can verify running sys_top that the amount of memory used while running the pipeline is not freed, we solved several of those errors in our version of the plugins too. Could you share the pipeline that you are using? I will try to reproduce it with our plugins.

-David