• State
  • Locked Locked
  • Replies 13 replies
  • Subscribers 99 subscribers
  • Views 1010 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-TDA4VM: Single input multi infer example with industrial camera

Ugo Bertacchini
Intellectual 300 points
Part Number: SK-TDA4VM


Hi,

I have followed the instruction contained in the Edge AI Academy (Part7: Connect your own industrial camera) and I was able to connect one generic industrial camera to the SK-TDA4VM and run the app_edgeai demos as shown in the following image.

Unfortunately, when I configure the flow in the yaml file in order to have two models running on the same image I got the following results.

The same yaml config file is working if I pick another type of image source (see next image).

The GStreamer pipeline that the application is trying to run is the following:

[GST SRC STR]
[FLOW 0]
customsrc exposureauto=1 ! video/x-raw, format=RGB ! videoscale ! videoconvert ! queue ! tee name=tee_split0
tee_split0. ! queue ! tiovxmultiscaler name=split_01
tee_split0. ! queue ! tiovxmultiscaler name=split_02
split_01. ! queue ! video/x-raw, width=640, height=640 ! tiovxdlpreproc data-type=10 target=0 channel-order=0 mean-0=0.000000 mean-1=0.000000 mean-2=0.000000 scale-0=1.000000 scale-1=1.000000 scale-2=1.000000 tensor-format=bgr out-pool-size=4 ! application/x-tensor-tiovx ! appsink name=pre_0 max-buffers=2 drop=true
split_01. ! queue ! video/x-raw, width=640, height=360 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! appsink name=sen_0 max-buffers=2 drop=true
split_02. ! queue ! video/x-raw, width=360, height=320 ! tiovxmultiscaler ! video/x-raw, width=320, height=320 ! tiovxdlpreproc data-type=3 target=1 channel-order=1 tensor-format=rgb out-pool-size=4 ! application/x-tensor-tiovx ! appsink name=pre_1 max-buffers=2 drop=true
split_02. ! queue ! video/x-raw, width=640, height=360 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! appsink name=sen_1 max-buffers=2 drop=true

[GST SINK STR]
appsrc format=GST_FORMAT_TIME is-live=true block=true do-timestamp=true name=post_0 ! tiovxdlcolorconvert ! video/x-raw,format=NV12, width=640, height=360 ! queue ! mosaic_0.sink_0
appsrc format=GST_FORMAT_TIME is-live=true block=true do-timestamp=true name=post_1 ! tiovxdlcolorconvert ! video/x-raw,format=NV12, width=640, height=360 ! queue ! mosaic_0.sink_1
tiovxmosaic name=mosaic_0 background=/tmp/background_0
sink_0::startx=320 sink_0::starty=360 sink_0::width=640 sink_0::height=360
sink_1::startx=960 sink_1::starty=360 sink_1::width=640 sink_1::height=360
! video/x-raw,format=NV12, width=1920, height=1080 ! kmssink sync=false driver-name=tidss

Any suggestion about the cause that impact the input image if I change the flow with multiple models?

Thanks for your help,

Ugo

  • 0
    TI__Genius 10355 points

    Ugo,

    Can you check the camera source resolution? It looks from the artifact that what camera is streaming and what [width, height] resolution mentioned in the yaml file seems to be different.

    Regards,
    Shyam

  • 0 in reply to Shyam Jagannathan
    TI__Expert 4259 points

    Hi Ugo,

    Can you please attach the yaml config file you are using
    I want to know what is the resolution coming out of the camera

    regards
    Rahul T R

  • 0 in reply to Shyam Jagannathan
    Intellectual 300 points

    Hello Shyam, thanks for your quick feedback.

    I agree with you it seems like the camera resolutions do not match.

    I have tried to double check it and I confirm that the resolution is set properly: the camera is streaming at 1440x928 in both cases (single model and multpiple model running).

    Here you can find the yaml file I am using.

    title: "TIDL Demo"
log_level: 2
inputs:
    input0:
        source: /dev/video2
        format: jpeg
        width: 1280
        height: 720
        framerate: 30
    input1:
        source: /opt/edge_ai_apps/data/videos/video_0001_h264.mp4
        format: h264_sw
        width: 1280
        height: 720
        framerate: 30
        loop: True
    input2:
        source: /opt/edge_ai_apps/data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
    input3:
        source: custom_rgb
        width: 1440
        height: 928
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/ONR-KD-7060-human-pose-yolox-s-640x640
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/ONR-OD-8100-yolov5-s6-ti-lite-coco-640x640
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6        
    model3:
        model_path: /opt/model_zoo/ONR-SS-8610-deeplabv3lite-mobv2-ade20k32-512x512
        alpha: 0.4
    model4:
        model_path: /opt/model_zoo/TFL-CL-0000-mobileNetV1-mlperf
        topN: 5
    model5:
        model_path: /opt/model_zoo/TVM-CL-3410-gluoncv-mxnet-mobv2
        topN: 5
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
    output1:
        sink: /opt/edge_ai_apps/data/output/videos/output_video.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edge_ai_apps/data/output/images/output_image_%04d.jpg
        width: 1920
        height: 1080

flows:
    flow0:
        input: input3
        models: [model0,model2]
        outputs: [output0,output0]
        mosaic:
            mosaic0:
                width:  640
                height: 360
                pos_x:  320
                pos_y:  360
            mosaic1:
                width:  640
                height: 360
                pos_x:  960
                pos_y:  360

    Basically it works when I put "input1" (instead of "input3") as a flow0 input or when I keep input3 and I put just one model and one output in the flow0 models and outputs array.

    It seems like the image got corrupted after the "tee" Gstreamer operation.

    Do you know why we can have such behaviour?

  • 0 in reply to Ugo Bertacchini
    TI__Expert 4259 points

    Hi Ugo,

    Can you please try below pipeline and check if get
    correct output

    gst-launch-1.0 customsrc exposureauto=1 ! video/x-raw, format=RGB ! videoscale ! videoconvert ! queue ! tee name=tee_split0 \
    tee_split0. ! queue ! fakesink \
    tee_split0. ! queue ! kmssink sync=false

    This will help in isolating the issue

    Regards
    Rahul T R

  • 0 in reply to Rahul Ravikumar
    Intellectual 300 points

    Hello Rahul, sure.

    I have just tried and it works.

    Regards,

    Ugo

  • 0 in reply to Ugo Bertacchini
    TI__Expert 4259 points

    So Input is coming properly

    gst-launch-1.0 \
    customsrc exposureauto=1 ! video/x-raw, format=RGB ! videoscale ! videoconvert ! queue ! tee name=tee_split0 \
    tee_split0. ! queue ! tiovxmultiscaler name=split_01 \
    tee_split0. ! queue ! tiovxmultiscaler name=split_02 \
    split_01. ! queue ! video/x-raw, width=640, height=640 ! tiovxdlpreproc data-type=10 target=0 channel-order=0 mean-0=0.000000 mean-1=0.000000 mean-2=0.000000 scale-0=1.000000 scale-1=1.000000 scale-2=1.000000 tensor-format=bgr out-pool-size=4 ! application/x-tensor-tiovx ! fakesink \
    split_01. ! queue ! video/x-raw, width=640, height=360 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! fakesink \
    split_02. ! queue ! video/x-raw, width=360, height=320 ! tiovxmultiscaler ! video/x-raw, width=320, height=320 ! tiovxdlpreproc data-type=3 target=1 channel-order=1 tensor-format=rgb out-pool-size=4 ! application/x-tensor-tiovx ! fakesink \
    split_02. ! queue ! video/x-raw, width=640, height=360 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! kmssink sync=false

    Can you check this, to see if some issue after scaling

    Regards
    Rahul T R

  • 0 in reply to Rahul Ravikumar
    Intellectual 300 points

    This is replicating the issue I see in the demo.

    You can see the result in this picture.

  • 0 in reply to Ugo Bertacchini
    TI__Expert 4259 points

    gst-launch-1.0 customsrc exposureauto=1 ! video/x-raw, format=RGB ! videoscale ! videoconvert ! tiovxmultiscaler ! video/x-raw, width=640, height=360 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! kmssink sync=false

    Can you check if above pipeline reproduces the issue

    Regards
    Rahul T R

  • 0 in reply to Rahul Ravikumar
    Intellectual 300 points

    This last pipeline have not reproduced the issue, the video is good.

  • 0 in reply to Ugo Bertacchini
    TI__Expert 4259 points

    gst-launch-1.0 customsrc exposureauto=1 ! video/x-raw, format=RGB ! videoscale ! videoconvert ! tee ! tiovxmultiscaler ! video/x-raw, width=640, height=360 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! kmssink sync=false

    Can you please try this

    Regards
    Rahul T R

  • 0 in reply to Rahul Ravikumar
    Intellectual 300 points

    Now it is corrupted again.

  • 0 in reply to Ugo Bertacchini
    TI__Expert 4259 points

    Hi Ugo,

    adding tee element before tiovxmultiscaler, is causing the issue
    I will get back to you on this issue

    Thanks
    Rahul T R

  • 0 in reply to Rahul Ravikumar
    Intellectual 300 points

    Yes, it seems you are right.

    Lookng forward to hear some feedback about that.

    Thanks for the support.

    Ugo