SK-AM62A-LP: Issue with edgeai application

Hi Sajan,

Can you please share the complete config that you are trying to run? And I assume this issue is from apps_python in our SDK? If not, please provide which application it is you are trying to run, and if the issue is present with apps_python or apps_cpp as well

Regards,
Jay

Hi Jay,

Jay Goyal said:

please share the complete config that you are trying to run?

title: "Preview"

inputs:
    input0:
        source: /opt/video/sample.mp4
        format: h264
        width: 1280
        height: 720
        framerate: 20
        loop: True

models:
    model0:
        model_path: models/ONR-OD-8220-yolox-s-lite-mmdet-coco-640x640
        viz_threshold: 0.5

outputs:
    output0:
        sink: kmssink
        width: 1280
        height: 720
        framerate: 20

flows:
    flow0: [input0, model0, output0]

Jay Goyal said:

And I assume this issue is from apps_python in our SDK?

No, I did forked the application apps_python and created OD only from that. 

Video looping is now working.

diff --git a/apps_python/gst_wrapper.py b/apps_python/gst_wrapper.py
index bed3734..44f222c 100644
--- a/apps_python/gst_wrapper.py
+++ b/apps_python/gst_wrapper.py
@@ -78,58 +78,72 @@ class GstPipe:
         sink.set_caps(caps)
         return sink
 
-    def pull_frame(self, src, loop):
+    def _restart_src_pipe(self, pipe_index):
         """
-        Pull a frame from gst pipeline
-        Args:
-            src: gst src element from which the frame is pulled
-            loop: If src need to be looped after eos
+        Destroy and recreate src pipeline for looping
         """
+        pipe = self.src_pipe[pipe_index]
+        pipe.set_state(Gst.State.NULL)
+        # Wait for NULL state to complete fully
+        pipe.get_state(Gst.CLOCK_TIME_NONE)
+        pipe.set_state(Gst.State.PLAYING)
+        ret = pipe.get_state(5 * Gst.SECOND)
+        if ret[0] == Gst.StateChangeReturn.FAILURE:
+            print("[ERROR] Failed to restart pipeline for loop")
+            return False
+        return True
+
+    def pull_frame(self, src, loop):
         sample = src.try_pull_sample(5000000000)
         if type(sample) != Gst.Sample:
             if src.is_eos():
                 if loop:
-                    # Seek can be called from various sources hence putting lock
                     with self.mutex:
-                        src.seek_simple(Gst.Format.TIME, Gst.SeekFlags.FLUSH, 0)
+                        restarted = False
+                        for i, pipe in enumerate(self.src_pipe):
+                            if pipe.get_by_name(src.get_name()):
+                                restarted = self._restart_src_pipe(i)
+                                break
+                        if not restarted:
+                            return None
                         sample = src.try_pull_sample(5000000000)
+                        if type(sample) != Gst.Sample:
+                            print("[ERROR] Error pulling frame after restart")
+                            return None
                 else:
                     return None
             else:
-                print("[ERROR] Error pulling frame from GST Pipeline")
+                print(
+                    f"[ERROR] pull timeout: eos={src.is_eos()} "
+                )
                 return None
         caps = sample.get_caps()
-
         struct = caps.get_structure(0)
         width = struct.get_value("width")
         height = struct.get_value("height")
-
         buffer = sample.get_buffer()
         _, map_info = buffer.map(Gst.MapFlags.READ)
         frame = copy.deepcopy(np.ndarray((height, width, 3), np.uint8, map_info.data))
         buffer.unmap(map_info)
-
         return frame
 
     def pull_tensor(self, src, loop, width, height, layout, data_type):
-        """
-        Pull a frame from gst pipeline
-        Args:
-            src: gst src element from which the frame is pulled
-            loop: If src need to be looped after eos
-            width: width of the tensor
-            height: height of the tensor
-            layout: data layout (NHWC or NCHW)
-            data_type: data type of the tensor
-        """
         sample = src.try_pull_sample(5000000000)
         if type(sample) != Gst.Sample:
             if src.is_eos():
                 if loop:
-                    # Seek can be called from various sources hence putting lock
                     with self.mutex:
-                        src.seek_simple(Gst.Format.TIME, Gst.SeekFlags.FLUSH, 0)
+                        restarted = False
+                        for i, pipe in enumerate(self.src_pipe):
+                            if pipe.get_by_name(src.get_name()):
+                                restarted = self._restart_src_pipe(i)
+                                break
+                        if not restarted:
+                            return None
                         sample = src.try_pull_sample(5000000000)
+                        if type(sample) != Gst.Sample:
+                            print("[ERROR] Error pulling tensor after restart")
+                            return None
                 else:
                     return None
             else:
@@ -141,8 +155,8 @@ class GstPipe:
             frame = np.ndarray((1, height, width, 3), data_type, map_info.data)
         elif layout == "NCHW":
             frame = np.ndarray((1, 3, height, width), data_type, map_info.data)
+        frame = copy.deepcopy(frame)
         buffer.unmap(map_info)
-
         return frame
 
     def push_frame(self, frame, sink):
@@ -1675,3 +1689,4 @@ def get_gst_pipe(flows, outputs):
                     sink_player = add_and_link(o.gst_disp_elements, player=sink_player)
                     link_elements(s.gst_post_proc_elements[-1], o.gst_disp_elements[0])
     return src_players, sink_player
+

Problem:
Now the actual problem is at random time after running the video for more than 5 minutes, memory increase exponentially from ~580MB and the system becomes stuck because of no memory.

Could you please provide instruction on debugging the memory leak.

Warm Regards,
Sajan

Hi Sajan,

Is this with the Gstreamer upgraded to version 1.24 (which resolves the memory leak)?

Best Regards

Suren

Hello Suren,

Suren Porwar said:

Is this with the Gstreamer upgraded to version 1.24 (which resolves the memory leak)?

No.

I can't upgrade to SDK 11 right now because a lot of effort has already gone into the current setup, including model training and integration. Also, SDK 11 retraining/recompilation is not available through Model Composer, and I tried the EdgeAI TensorLab CLI approach as well, but the training is not working.

I tested with two out-of-the-box applications, and stream switching is not working. The pipeline you provided only reopens the existing camera stream, which works as expected, but switching to a different camera stream has not been working for a long time.

Best Regards,
Sajan

Hello Sajan,

So, If I understand correctly you have two heterogeneous camera connected to AM62A board and you are trying to switch between these cameras and encoding it? Is this correct. This would help us reproduce the issue on our end.

Also, you are trying to decode the sample.mp4 in a loop and see memory leak?

Are these two separate issues?

Best Regards

Suren

Hello Suren,

Connected two imx219 camera into am62a via arducam v3 link board. Tried to switch the streams that is fed into the rtsp using od and cl applications.

Suren Porwar said:

Are these two separate issues?

Yes.

Also my od application for a long run seems:
[ERROR] Error pulling tensor from GST Pipeline
I used the exact pull_frame() and pull_tensor() from https://github.com/TexasInstruments/edgeai-gst-apps/blob/main/apps_python/gst_wrapper.py#L81

But seems pull frame or pull tensor error when running the video in loop for 5 hours or more.

Warm Regards,
Sajan

This is the exact error I seems after running the app for 2 hours and 28 minutes.

[ERROR] Error pulling tensor from GST Pipeline

From my understanding, I thought this is not video eos or jump to start related error. Because the error happens at any time of the video

That not a video stream related issue. I tried different videos.
Below is the system log when the error occurred.

    PID     ELAPSED %CPU %MEM   RSS    VSZ
   3770    02:28:30  111  9.1 311344 1789244

Hi Sajan,

I want to respond based on the stream switch issue. Can you please share exactly what you are trying to do?

Are trying to switch the input sensor for them on the fly? Or are you trying to do something else?

Regards,
Jay

Hi Suren,

Yes, The cam0 provides the CL application output frame and cam1 provides OD. I tried switching the stream by changing the port number.

Warm Regards,
Sajan

Hi Sajan,

To understand your issue correctly. You are using IMX219 cam0 with classification and IMX219 cam1 with object detection workflows. This works. When you start the applications with the cameras switched, it fails. Is this correct?

If yes, please share the output of media-ctl -p, and the input and output pipelines for both the cases.

Regards,
Jay

Hello Jay,

Jay Goyal said:

You are using IMX219 cam0 with classification and IMX219 cam1 with object detection workflows.

This statement is correct.

outputs:
    output0:
        sink: remote
        host: 127.0.0.1
        port: 8081
        encoding: h264
        width: 1280
        height: 720
        bitrate: 4000000
        gop-size: 15

    output0:
        sink: remote
        host: 127.0.0.1
        port: 8082
        encoding: h264
        width: 1280
        height: 720
        bitrate: 4000000
        gop-size: 15

Like this udpsinks, the frame would send to 8081(CL) and 8082(OD).

Then started up a script for switch the stream from this port numbers.

pipeline used at receiver end is:

            f"udpsrc port={cfg['port']} address=127.0.0.1 "
            "caps=\"application/x-rtp,media=video,encoding-name=H264,payload=96\" ! "
            "rtph264depay ! h264parse config-interval=-1 ! "
            "v4l2h264dec capture-io-mode=dmabuf ! "
            "kmssink sync=false force-modesetting=true"

Hi Sajan,

Jay Goyal said:

When you start the applications with the cameras switched, it fails. Is this correct?

Can you please confirm this part? What is the inherent issue?

I am still unclear about what switching the stream means. Are you changing which camera feeds which model on the fly, or are you doing something else. It might be helpful if you can drop the part of your codebase that handles this. Please remember that this is a public forum.

Regards,
Jay

Hi Jay,

Jay Goyal said:

Are you changing which camera feeds which model on the fly, or are you doing something else.

No. There is nothing about the model OD and CL apps. I am saying about the OD and CL apps only do put the frames into the ports assigned. It is about the issue in showing the streams from accessing the port.

Code I used to switch the port is https://docs.google.com/document/d/1fGB0UB5NguTpstUQpELoFXumcoMxu2BUg2zMCF5azEc/edit?usp=sharing

This will crash after sometime of switching.

Warm Regards,
Sajan

Hi Sajan,

This should theoretically work, but it requires a decent amount of boilerplate to make it work with the existing edgeai-gst-apps infra. If that is done correctly and you are seeing error pulling tensor from pipeline, here is what I think is happening:

1. When you switch the pipeline, the task for pulling the frame from the pipeline fires, because that is running on a different thread, and that crashes because it can't pull the frame.
2. I am not sure about the entire architecture, but the underlying GStreamer pipeline and the associated nodes would also change. So, you need to synchronize these variables with the task.

If you are seeing a different issue, please let me know.

Regards,
Jay