TDA4VM: edge ai object detection

Hankyu Kim

Prodigy 10 points

Part Number: TDA4VM

Hello, I am TDA4VM user with CSI camera for running edge ai - object detection

I finished and checked camera setting with the gstreamer. (There is no problem with camera)

I also checked with /opt/edge_ai_apps/init_script.sh

+ installed all the libraries that are required to run edge ai with the setup_script.sh

and when I run the object detection

Fullscreen

1
sudo /opt/edge_ai_apps/apps_python/app_edgeai.py configs/object_detection.yaml
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

sudo /opt/edge_ai_apps/apps_python/app_edgeai.py configs/object_detection.yaml

and here is the result, but looks like have some error somewhere.. how to fix?

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
debian@myname:/opt/edge_ai_apps$ sudo /opt/edge_ai_apps/apps_python/app_edgeai.py configs/object_detection.yaml 
libtidl_onnxrt_EP loaded 0x12923ee0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 391, Total Nodes - 391 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  9408.066545 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  9408.082145 s:  VX_ZONE_INIT:Enabled
  9408.082202 s:  VX_ZONE_ERROR:Enabled
  9408.082209 s:  VX_ZONE_WARNING:Enabled
  9408.090617 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  9408.090889 s:  VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!
[GST SRC STR]
[FLOW 0]
v4l2src device=/dev/video2 ! queue leaky=2 ! video/x-bayer, width=640, height=480, format=rggb ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin format-msb=7 ! video/x-raw, format=NV12 !tiovxmultiscaler name=split_01 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

debian@myname:/opt/edge_ai_apps$ sudo /opt/edge_ai_apps/apps_python/app_edgeai.py configs/object_detection.yaml 
libtidl_onnxrt_EP loaded 0x12923ee0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 391, Total Nodes - 391 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  9408.066545 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  9408.082145 s:  VX_ZONE_INIT:Enabled
  9408.082202 s:  VX_ZONE_ERROR:Enabled
  9408.082209 s:  VX_ZONE_WARNING:Enabled
  9408.090617 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
  9408.090889 s:  VX_ZONE_INIT:[tivxHostInitLocal:86] Initialization Done for HOST !!!
[GST SRC STR]
[FLOW 0]
v4l2src device=/dev/video2 ! queue leaky=2 ! video/x-bayer, width=640, height=480, format=rggb ! tiovxisp sink_0::device=/dev/v4l-subdev2 sensor-name=SENSOR_SONY_IMX219_RPI dcc-isp-file=/opt/imaging/imx219/dcc_viss.bin sink_0::dcc-2a-file=/opt/imaging/imx219/dcc_2a.bin format-msb=7 ! video/x-raw, format=NV12 !tiovxmultiscaler name=split_01 
split_01. ! queue ! video/x-raw, width=640, height=480 ! videoscale ! video/x-raw, width=640, height=640 ! tiovxdlpreproc data-type=3 target=0 channel-order=0 tensor-format=rgb out-pool-size=4 ! application/x-tensor-tiovx ! appsink name=pre_0 max-buffers=2 drop=true 
split_01. ! queue ! video/x-raw, width=320, height=320 ! tiovxdlcolorconvert target=1 out-pool-size=4 ! video/x-raw, format=RGB ! appsink name=sen_0 max-buffers=2 drop=true 

[GST SINK STR]
appsrc format=GST_FORMAT_TIME is-live=true block=true do-timestamp=true name=post_0 ! tiovxdlcolorconvert ! video/x-raw,format=NV12, width=320, height=320 ! queue ! mosaic_0.sink_0 
tiovxmosaic name=mosaic_0 background=/tmp/background_0
sink_0::startx=0  sink_0::starty=0  sink_0::width=320   sink_0::height=320  
! video/x-raw,format=NV12, width=320, height=320 ! kmssink sync=false driver-name=tidss 

  9415.252221 s:  VX_ZONE_WARNING:[vxReleaseContext:1050] Found a reference 0xffff7f6f64e0 of type 00000816 at external count 1, internal count 0, releasing it
  9415.252464 s:  VX_ZONE_WARNING:[vxReleaseContext:1052] Releasing reference (name=user_data_object_155) now as a part of garbage collection
  9415.253495 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:100] De-Initialization Done for HOST !!!
  9415.297187 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
APP: Deinit ... !!!
REMOTE_SERVICE: Deinit ... !!!
REMOTE_SERVICE: Deinit ... Done !!!
IPC: Deinit ... !!!
IPC: DeInit ... Done !!!
MEM: Deinit ... !!!
MEM: Alloc's: 45 alloc's of 54151372 bytes 
MEM: Free's : 45 free's  of 54151372 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
APP: Deinit ... Done !!!

over 1 year ago

0 Pratik Kedar over 1 year ago

TI__Mastermind 24041 points

Which SDK version you are using ?

Can you check if any std object detection demo with cam input is running ? (webcam based etc)

Can you run the demo as mentioned here :

https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-edgeai/TDA4VM/09_00_01/exports/docs/common/sample_apps.html#python-c-apps

0 Hankyu Kim over 1 year ago in reply to Pratik Kedar

Prodigy 10 points

Here is my sdk version : imaging-REL.PSDK.JACINTO.09.00.00.02

furthermore, it worked with demo video.

but when I run it with usb cam, it dies or gets stuck there all the time..

with csi cam, just get error there (doesn't work)

0 Pratik Kedar over 1 year ago in reply to Hankyu Kim

TI__Mastermind 24041 points

Hi,

I believe you have downloaded and flashed the sdk from https://www.ti.com/tool/PROCESSOR-SDK-J721E for your current exp.

Can you share the log of USB cam demo run ?

Please make sure you run the demo as mentioned here :

Pratik Kedar said:
Can you run the demo as mentioned here :

https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-edgeai/TDA4VM/09_00_01/exports/docs/common/sample_apps.html#python-c-apps

Make sure you run command is visible in logs.

Also can you share the config file that you are using for object detection ? Have you made correct change to take accept cam based input ?

Also can you share which USB cam you are using ? Here is the info about supported cam : https://software-dl.ti.com/jacinto7/esd/processor-sdk-linux-edgeai/TDA4VM/09_00_01/exports/docs/devices/TDA4VM/linux/getting_started.html

0 Hankyu Kim over 1 year ago in reply to Pratik Kedar

Prodigy 10 points

[deleted]

0 Hankyu Kim over 1 year ago in reply to Hankyu Kim

Prodigy 10 points

Thank you for your help.

I followed your steps and I got the below result.

But I would like to know why there are 3 input videos and what does it mean?

Fullscreen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
title: "Object Detection Demo"
log_level: 2
inputs:
    input0:
        source: /dev/video2
        format: rggb
        width: 1920
        height: 1080
        sen_id: imx219
        framerate: 30
    input1:
        source: /opt/edge_ai_apps/data/videos/video_0000_h264.mp4
        format: h264_sw
        width: 1280
        height: 720
        framerate: 30
        loop: True
    input2:
        source: /opt/edge_ai_apps/data/images/%04d.jpg
        width: 1280
        height: 720
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

title: "Object Detection Demo"
log_level: 2
inputs:
    input0:
        source: /dev/video2
        format: rggb
        width: 1920
        height: 1080
        sen_id: imx219
        framerate: 30
    input1:
        source: /opt/edge_ai_apps/data/videos/video_0000_h264.mp4
        format: h264_sw
        width: 1280
        height: 720
        framerate: 30
        loop: True
    input2:
        source: /opt/edge_ai_apps/data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/TVM-OD-5020-yolov3-mobv1-gluon-mxnet-416x416
        viz_threshold: 0.6
    model1:
        model_path: /opt/model_zoo/TFL-OD-2020-ssdLite-mobDet-DSP-coco-320x320
        viz_threshold: 0.6
    model2:
        model_path: /opt/model_zoo/ONR-OD-8050-ssd-lite-regNetX-800mf-fpn-bgr-mmdet-coco-512x512
        viz_threshold: 0.6
    model3:
        model_path: /opt/model_zoo/ONR-OD-8120-yolov5-m6-ti-lite-coco-640x640
        viz_threshold: 0.6
        
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080
    output1:
        sink: /opt/edge_ai_apps/data/output/videos/output_video.mkv
        width: 1920
        height: 1080
    output2:
        sink: /opt/edge_ai_apps/data/output/images/output_image_%04d.jpg
        width: 1920
        height: 1080

flows:
    flow0:
        input: input0
        models: [model3]
        outputs: [output0]
        mosaic:
            mosaic0:
                width:  1920
                height: 1080
                pos_x: 0
                pos_y: 0

+ one more problem, it only work with 1920x1080 resolution.

how to make 640x480 resolution work?

I could find dcc_2a_640x480.bin & dcc_viss_640x480.bin on imaging-REL.PSDK.JACINTO.09.00.00.02. and put it on /opt/imaging/imx219, but didn't work.