TDA4VM: [TDA4VM-SK] Build custom camera GUI on Linux SDK

Hi Takuma Fujiwara 
Thanks for the response, I am able to modify the source code.

Need guidance on flow configurations for multiple inference view.

We wanted to show this view when clicking a user button on edgeai-gui-app,

We tried with following pipeline definition and passed this to backend and triggered the pipeline, but the view is not getting triggered at all and is stuck

static string j721e_multi_inf_pipeline = "multifilesrc location=/opt/oob-demo-assets/oob-gui-video2.h264 loop=true stop-index=-1 ! h264parse ! v4l2h264dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! video/x-raw, format=NV12 ! \
                                            tiovxmultiscaler name=split_01 src_0::roi-startx=0 src_0::roi-starty=0 src_0::roi-width=1280 src_0::roi-height=764 target=0 \
                                            \
                                            split_01. ! queue ! video/x-raw, width=320, height=320 ! tiovxdlpreproc model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320  out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 ! post_0.tensor \
                                            split_01. ! queue ! video/x-raw, width=480, height=272 ! post_0.sink \
                                            tidlpostproc name=post_0 model=/opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320 alpha=0.2 viz-threshold=0.500000 top-N=5 display-model=true ! queue ! mosaic_0. \
                                            \
                                            multifilesrc location=/opt/oob-demo-assets/oob-gui-video2.h264 loop=true stop-index=-1 ! h264parse ! v4l2h264dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! video/x-raw, format=NV12 ! \
                                            tiovxmultiscaler name=split_11 src_0::roi-startx=0 src_0::roi-starty=0 src_0::roi-width=1280 src_0::roi-height=768 target=1 \
                                            \
                                            split_11. ! queue ! video/x-raw, width=416, height=416 ! tiovxdlpreproc model=/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416  out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416 ! post_1.tensor \
                                            split_11. ! queue ! video/x-raw, width=480, height=272 ! post_1.sink \
                                            tidlpostproc name=post_1 model=/opt/model_zoo/ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416 alpha=0.2 viz-threshold=0.500000 top-N=5 display-model=true ! queue ! mosaic_0. \
                                            \
                                            multifilesrc location=/opt/oob-demo-assets/oob-gui-video2.h264 loop=true stop-index=-1 ! h264parse ! v4l2h264dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! video/x-raw, format=NV12 ! \
                                            tiovxmultiscaler name=split_21 src_0::roi-startx=0 src_0::roi-starty=0 src_0::roi-width=1280 src_0::roi-height=768 target=0 \
                                            \
                                            split_21. ! queue ! video/x-raw, width=224, height=224 ! tiovxdlpreproc model=/opt/model_zoo/ONR-CL-6360-regNetx-200mf  out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/ONR-CL-6360-regNetx-200mf ! post_2.tensor \
                                            split_21. ! queue ! video/x-raw, width=480, height=272 ! post_2.sink \
                                            tidlpostproc name=post_2 model=/opt/model_zoo/ONR-CL-6360-regNetx-200mf alpha=0.2 viz-threshold=0.500000 top-N=5 display-model=true ! queue ! mosaic_0. \
                                            \
                                            multifilesrc location=/opt/oob-demo-assets/oob-gui-video2.h264 loop=true stop-index=-1 ! h264parse ! v4l2h264dec capture-io-mode=5 ! tiovxmemalloc pool-size=12 ! video/x-raw, format=NV12 ! \
                                            tiovxmultiscaler name=split_31 src_0::roi-startx=0 src_0::roi-starty=0 src_0::roi-width=1280 src_0::roi-height=768 target=1 \
                                            \
                                            split_31. ! queue ! video/x-raw, width=512, height=512 ! tiovxdlpreproc model=/opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512  out-pool-size=4 ! application/x-tensor-tiovx ! tidlinferer target=1 model=/opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512 ! post_3.tensor \
                                            split_31. ! queue ! video/x-raw, width=480, height=272 ! post_3.sink \
                                            tidlpostproc name=post_3 model=/opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512 alpha=0.2 viz-threshold=0.500000 top-N=5 display-model=true ! queue ! mosaic_0. \
                                            \
                                            \
                                            tiovxmosaic name=mosaic_0 target=1 src::pool-size=4 \
                                            sink_0::startx=\"<0>\" sink_0::starty=\"<0>\" sink_0::widths=\"<480>\" sink_0::heights=\"<272>\" \
                                            sink_1::startx=\"<480>\" sink_1::starty=\"<0>\" sink_1::widths=\"<960>\" sink_1::heights=\"<272>\" \
                                            sink_2::startx=\"<0>\" sink_2::starty=\"<272>\" sink_2::widths=\"<960>\" sink_2::heights=\"<272>\" \
                                            sink_3::startx=\"<480>\" sink_3::starty=\"<272>\" sink_3::widths=\"<960>\" sink_3::heights=\"<272>\" \
                                            ! video/x-raw,format=NV12, width=1920, height=1080 ! queue ! tiperfoverlay main-title=null title=\"\" ! ";

Please guide us on the same.

Regards,

Soorya

Hi Soorya,

A couple questions:

1. Does a single input and single inference pipeline run when passing to backend?
2. Does the pipeline run with gst-launch-1.0 commandline tool?

And as a general pointer for troubleshooting GStreamer, I recommend starting from a smaller pipeline and then gradually adding more to the pipeline to determine where the pipeline fails. The simplest pipeline to start from being "videotestsrc ! fakesink" where the source is a file output and the sink is a "fake" sink that consumes most input format and does not render output.

Regards,

Takuma

 I tried to run same model ONR-OD-8200-yolox-nano-lite-mmdet-coco-416x416 on all 4 inputs, it was working properly. When I tried to change models for the other 3 inputs, it stopped working, I also corrected pre processing stage by updating the image resolution as per the model, still the same result