PROCESSOR-SDK-AM62A: Where can I find logic of saving the output (apps_python)

Hello Qutaiba,

I tried modifying filesink by splitmuxsink but seems the apps is not running.
Modified gst_wrapper code snippet below

    elif sink == "video":
        property = {}
        if (gst_element_map["h264enc"]["element"] == "v4l2h264enc"):
            property = {"extra-controls": Gst.Structure.from_string(enc_extra_ctrl)[0]}

        sink_elements += make_element(gst_element_map["h264enc"], property=property)

        for i in video_enc[sink_ext]:
            sink_elements += make_element(i[0], property=i[1])

        property={ "location": output.sink.replace(".mp4", "_%03d.mp4"), "max-size-time": 5 * 1000000000, "name": sink_name}
        sink_elements += make_element("splitmuxsink", property=property)

Best Regards,
Sajan

Hi Sajan,

I am not sure that I can walk with you through the steps of changing the gst_wrapper code. My suggestion is to first add the "splitmuxsink" plugin directly to the gstreamer pipeline. When you have a fully working pipeline and have better understanding of how this new plugin interact with the rest of the plugins, then go back to change the gst_wrapper. 

You can use the optiflow to generate the basic pipeline which you can use as a starting point to add the "splitmuxsink". The optiflow is part of the edgeai-gst-apps (https://github.com/TexasInstruments/edgeai-gst-apps/tree/main/optiflow) it uses the same yamil file as configurable inputs. Some examples of the pipelines which you can also use as a starting point are here: https://software-dl.ti.com/processor-sdk-linux/esd/AM62AX/latest/exports/edgeai-docs/common/edgeai_dataflows.html#optiflow. 

In this approach you can first focus on adding the "splitmuxsink" and then change the gst_wrapper as needed. 

Best regards,

Qutaiba

Hi Qutaiba,

Qutaiba Saleh said:

optiflow to generate the basic pipeline which you can use as a starting point to add the "splitmuxsink"

        sink_cmd += (                                                   
            ' ! h264parse ! '                                          
            'splitmuxsink muxer-factory=matroskamux '                   
            'max-size-time=3000000000 max-files=10 '               
            'location=' + output.sink.replace('.mp4', '_%03d.mp4')      
        )

By adding the above line in gst_wrapper.py sink == 'video' condition I got the video output as 3 second files. and 6 th file is a 52 second duration video.

My question is why I can't write the same code in apps_python. If I add it in apps_python in throws some error or app will struck.

Warm Regards,
Sajan

Hi Sajan,

I am glad that the optiflow worked for your use case. 

Sajan Saji said:

My question is why I can't write the same code in apps_python. If I add it in apps_python in throws some error or app will struck.

The apps_python uses more complicated logic to generate the final gst pipeline. Unlike the optiflow, the apps_python does not directly generate the final string for the pipeline. The code here is intended to demo purposes and you are modifying it a bit beyond. For such specific use cases, I suggest that you build your own gst wrapper similar to this example: https://github.com/TexasInstruments-Sandbox/edgeai-gst-apps-retail-checkout/blob/main/retail-shopping/gst_configs.py. In this case, you don't have to worry about the extra logic considering all other input and output formats.

If you are still interested in modifying the apps_python, my help will be at a consultation capacity. I see the changes you made to the gst_wrapper.py in the apps_python. They look fine from the first look. As I don't see exactly where it stopped, I can not provide any suggestion. You can provide the logs of where it stopped. I will be happy to look at it and see if I can provide any beneficial suggestion. 

Best regards,

Qutaiba

Hi Qutaiba,

Qutaiba Saleh said:

provide the logs of where it stopped

I will provide two logs of different codes.

1st is 

    elif sink == "video":          
        property = {}                                                                   
        if (gst_element_map["h264enc"]["element"] == "v4l2h264enc"):
            property = {"extra-controls": Gst.Structure.from_string(enc_extra_ctrl)[0]}
                           
        sink_elements += make_element(gst_element_map["h264enc"], property=property)
                                     
        for i in video_enc[sink_ext]:
            sink_elements += make_element(i[0], property=i[1])
           
        mux = Gst.ElementFactory.make("mp4mux", f"{sink_name}-muxer")
       
        split = Gst.ElementFactory.make("splitmuxsink", sink_name)
                                                             
        split.set_property(             
            "location",
            output.sink.rstrip(".mp4") + "_%05d.mp4"
        )                                   
        split.set_property("max-size-time", 60 * Gst.SECOND)
        split.set_property("muxer", mux)             
                            
        sink_elements += [mux, split]                

libtidl_onnxrt_EP loaded 0x30387660 
Final number of subgraphs created are : 1, - Offloaded Nodes - 124, Total Nodes - 124 
APP: Init ... !!!
   247.479601 s: MEM: Init ... !!!
   247.479671 s: MEM: Initialized DMA HEAP (fd=5) !!!
   247.479924 s: MEM: Init ... Done !!!
   247.479961 s: IPC: Init ... !!!
   247.497610 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   247.504067 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   247.508018 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
   247.508070 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
   247.508081 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
   247.510685 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-0 
   247.510887 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-1 
   247.511000 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-2 
   247.511098 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-3 
   247.511114 s:  VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
   247.511136 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO

** (python3:1626): WARNING **: 00:04:08.938: Could not add muxer element - splitmuxsink will not work

** (python3:1626): WARNING **: 00:04:08.945: Could not add muxer element - splitmuxsink will not work

2nd 

    elif sink == "video":
        property = {}
        if (gst_element_map["h264enc"]["element"] == "v4l2h264enc"):
            property = {"extra-controls": Gst.Structure.from_string(enc_extra_ctrl)[0]}

        sink_elements += make_element(gst_element_map["h264enc"], property=property)

        for i in video_enc[sink_ext]:
            sink_elements += make_element(i[0], property=i[1])

        muxer = Gst.ElementFactory.make("mp4mux", None)
        if not muxer:
            raise RuntimeError("Failed to create mp4mux")

        splitmuxsink = Gst.ElementFactory.make("splitmuxsink", sink_name)
        if not splitmuxsink:
            raise RuntimeError("Failed to create splitmuxsink")

        splitmuxsink.set_property("location", output.sink.replace(".mp4", "_%05d.mp4"))
        splitmuxsink.set_property("max-size-time", 3_000_000_000)  # 3 seconds
        splitmuxsink.set_property("max-files", 10)
        splitmuxsink.set_property("muxer", muxer)

        sink_elements.append(splitmuxsink)

libtidl_onnxrt_EP loaded 0x3016a6a0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 124, Total Nodes - 124 
APP: Init ... !!!
   685.756972 s: MEM: Init ... !!!
   685.757115 s: MEM: Initialized DMA HEAP (fd=5) !!!
   685.757293 s: MEM: Init ... Done !!!
   685.757325 s: IPC: Init ... !!!
   685.775078 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   685.779531 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   685.779689 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
   685.779710 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
   685.779722 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
   685.781205 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-0 
   685.781513 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-1 
   685.781774 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-2 
   685.782046 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-3 
   685.782081 s:  VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
   685.782121 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

v4l2src device=/dev/video-imx219-cam1 io-mode=5 pixel-aspect-ratio=None ! queue leaky=2 ! capsfilter caps="video/x-bayer, width=(int)1920, height=(int)1080, format=(string)rggb;" ! tiovxisp dcc-isp-
file=/opt/imaging/imx219/linear/dcc_viss.bin sensor-name=SENSOR_SONY_IMX219_RPI ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler name=split_01
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1920, height=(int)1080;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers
=2 drop=True name=sen_0
split_01. ! queue ! capsfilter caps="video/x-raw, width=(int)1188, height=(int)668;" ! tiovxmultiscaler target=1 ! capsfilter caps="video/x-raw, width=(int)454, height=(int)256;" ! tiovxdlcolorconve
rt out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! videobox qos=True left=115 right=115 top=16 bottom=16 ! tiovxdlpreproc out-pool-size=4 data-type=3 ! capsfilter caps="applic
ation/x-tensor-tiovx;" ! appsink max-buffers=2 drop=True name=pre_0


==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=True format=3 block=True is-live=True name=post_0 ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! v4l2h264enc ext
ra-controls="controls, frame_level_rate_control_enable=(int)1, video_bitrate=(int)10000000, video_gop_size=(int)30;" ! h264parse ! mp4mux faststart-file=/tmp/qtmux1039052340 ! 

 +--------------------------------------------------------------------------+
 | Test Preview                                                             |
 +--------------------------------------------------------------------------+
 +--------------------------------------------------------------------------+
 | Input Src: /dev/video-imx219-cam1                                        |
 | Model Name: TEST-CL-SAMPLE                                               |
 | Model Type: classification                                               |
 +--------------------------------------------------------------------------+
 | dl-inference                     :        17.66 ms   from   5    samples |
 | total time                       :        50.04 ms   from   3    samples |
 | framerate                        :        19.98 fps  from   3    samples |
 | Detected Class                   :                                       |
 +--------------------------------------------------------------------------+

Also is it possible for me to make an app that can be used for similar purpose? I mean how about its difficulty level.

Thanks and Regards,
Sajan 

Hi Qutaiba,

I look forward to hearing from you.

Thanks and Regards,
Sajan 

Hi Sajan, 

Whey are you using Gst.ElementFactory.make? Did you try just updating these two lines https://github.com/TexasInstruments/edgeai-gst-apps/blob/main/apps_python/gst_wrapper.py#L983C8-L984C8 with something like: