This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: Failed to run app_edgeai with self-compiled edgeai-yolov5 pretrained model.

Part Number: TDA4VM

Hi team,

I've run following command on TDA4VM:

Fullscreen
1
root@tda4vm-sk:/opt/edgeai-gst-apps/apps_cpp# ./bin/Release/app_edgeai ../configs/single_input_multi_infer.yaml
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

It ended up with Segmentation fault. Following are logs after running the command:

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@tda4vm-sk:/opt/edgeai-gst-apps/apps_cpp# ./bin/Release/app_edgeai ../configs/single_input_multi_infer.yaml
Number of subgraphs:1 , 107 nodes delegated out of 107 nodes
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
6029.148206 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
6029.148295 s: VX_ZONE_INIT:Enabled
6029.148302 s: VX_ZONE_ERROR:Enabled
6029.148308 s: VX_ZONE_WARNING:Enabled
6029.149071 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
6029.149243 s: VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
6029.183827 s: VX_ZONE_ERROR:[ownContextSendCmd:799] Command ack message returned failure cmd_status: -1
6029.183880 s: VX_ZONE_ERROR:[ownContextSendCmd:835] tivxEventWait() failed.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

single_input_multi_infer.yaml is as follows:

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
title: "Single Input, Multi Inference"
log_level: 2
inputs:
input0:
source: /dev/video2
format: jpeg
width: 1280
height: 720
framerate: 30
input1:
source: /opt/edgeai-test-data/videos/video_0000_h264.h264
format: h264
width: 1280
height: 720
framerate: 30
loop: True
input2:
source: /opt/edgeai-test-data/images/%04d.jpg
width: 1280
height: 720
index: 0
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The error came from 4th model("yolov5s6_640_ti_lite_37p4_56p0") only, which was downloaded from https://github.com/TexasInstruments/edgeai-yolov5 and compiled with:

Fullscreen
1
python3 onnxrt_ep.py -c
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

The compilation finished with following warning message:

Fullscreen
1
2
3
4
5
6
------------------ Network Compiler Traces -----------------------------
successful Memory allocation
INFORMATION: [TIDL_ResizeLayer] Resize_107 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_123 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] Resize_139 Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
****************************************************
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Also, the auto-generated param.yaml from compilation is:

Fullscreen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
postprocess:
data_layout: NCHW
detection_thr: 0.3
preprocess:
crop:
- 640
- 640
data_layout: NCHW
mean:
- 0
- 0
- 0
resize:
- 640
- 640
scale:
- 0.003921568627
- 0.003921568627
- 0.003921568627
session:
artifacts_folder: ''
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Thank you in advance for your support!

Best regards,

Ann

  • Same result when i tried this model

  • From the above logs, the occurred error is "6029.183927 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed"

    Please check our FAQ : https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1227571/faq-sk-tda4vm-error-while-running-custom-compiled-model?tisearch=e2e-sitesearch&keymatch=faq%3Atrue

    Regards,

    Pratik

  • How to re compile with same sdk version as that of target tda4vm. I go through the edgeaibechmark github but the instruction is not very clear to step wise compilation of pretained yolov5 model. Is there any detailed documentation and supportive steps to help me to compile model.

  • I'm not sure that the "Segmentation fault" is caused by "Graph verify failed error".

    When I tested other already-compiled-model provided by TI, the "Graph verify failed" also occurred. However, there were no "Segmentation fault" in log as follows:

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    root@tda4vm-sk:/opt/edgeai-gst-apps/apps_cpp# ./bin/Release/app_edgeai ../configs/single_input_multi_infer.yaml
    Number of subgraphs:1 , 107 nodes delegated out of 107 nodes
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=5) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
    5800.299044 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
    5800.299128 s: VX_ZONE_INIT:Enabled
    5800.299135 s: VX_ZONE_ERROR:Enabled
    5800.299140 s: VX_ZONE_WARNING:Enabled
    5800.299889 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
    5800.300072 s: VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
    5800.323022 s: VX_ZONE_ERROR:[ownContextSendCmd:799] Command ack message returned failure cmd_status: -1
    5800.323048 s: VX_ZONE_ERROR:[ownContextSendCmd:835] tivxEventWait() failed.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Is there any other possible reason that cause Segmentation fault?

    Many thanks,

    Ann

  • Hi,

    From the above logs, the occurred error is "6029.183927 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed"

    Please check our FAQ : https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1227571/faq-sk-tda4vm-error-while-running-custom-compiled-model?tisearch=e2e-sitesearch&keymatch=faq%3Atrue

    Regards,

    Pratik

    Could you please share run log of vx_app_arm_remote_log.out

    Please follow instructions mentioned in FAQ

    Regards,

    Pratik

  • Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    root@tda4vm-sk:/opt/vision_apps# ./vx_app_arm_remote_log.out &
    [1] 3179
    root@tda4vm-sk:/opt/vision_apps# [MCU2_0] 3.813979 s: CIO: Init ... Done !!!
    [MCU2_0] 3.814052 s: ### CPU Frequency = 1000000000 Hz
    [MCU2_0] 3.814100 s: CPU is running FreeRTOS
    [MCU2_0] 3.814127 s: APP: Init ... !!!
    [MCU2_0] 3.814152 s: SCICLIENT: Init ... !!!
    [MCU2_0] 3.814410 s: SCICLIENT: DMSC FW version [8.6.3--v08.06.03 (Chill Capybar]
    [MCU2_0] 3.814459 s: SCICLIENT: DMSC FW revision 0x8
    [MCU2_0] 3.814513 s: SCICLIENT: DMSC FW ABI revision 3.1
    [MCU2_0] 3.814559 s: SCICLIENT: Init ... Done !!!
    [MCU2_0] 3.814590 s: UDMA: Init ... !!!
    [MCU2_0] 3.815900 s: UDMA: Init ... Done !!!
    [MCU2_0] 3.815963 s: MEM: Init ... !!!
    [MCU2_0] 3.816009 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ d9000000 of size 16777216 bytes !!!
    [MCU2_0] 3.816089 s: MEM: Created heap (L3_MEM, id=1, flags=0x00000000) @ 3600000 of size 262144 bytes !!!
    [MCU2_0] 3.816152 s: MEM: Init ... Done !!!
    [MCU2_0] 3.816178 s: IPC: Init ... !!!
    [MCU2_0] 3.816239 s: IPC: 6 CPUs participating in IPC !!!
    [MCU2_0] 3.816287 s: IPC: Waiting for HLOS to be ready ... !!!
    [MCU2_0] 14.151892 s: IPC: HLOS is ready !!!
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    This are my output when running ./vx_app_arm_remote_log.out &

  • Hi,

    From above link, I did encounter an version mismatch error as follows:

    Fullscreen
    1
    [C7x_1 ] 149.776057 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:705] Network version - 0x20221206, Expected version - 0x20230208
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    After downgrade SDK to https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-SK-TDA4VM/08.05.00.03 (0x20221206), the version mismatch error is solved, but there are still error as follows:

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    [07:58:02.000.000000]:ERROR:[getConfig:0147] Mean value specification missing.
    [07:58:02.000.000000]:ERROR:[initialize:1437] getConfig() failed.
    [07:58:02.000.000536]:ERROR:[getConfig:0147] Mean value specification missing.
    (app_edgeai:1225): GLib-GObject-WARNING **: 07:58:02.264: value "((GstTIOVXDLPreProcDataType) 0)" of type 'GstTIOVXDLPreProcDataType' is invalid or out of range for property 'data-type' of type 'GstTIOVXDLPreProcDataType'
    [07:58:02.019.019185]:ERROR:[makePostprocessImageObj:0088] Invalid post-processing task type.
    [07:58:02.019.019237]:ERROR:[createPostprocCntxt:1571] PostprocessImage::makePostprocessImageObj() failed.
    [07:58:02.019.019250]:ERROR:[initialize:1857] createPostprocCntxt() failed.
    Segmentation fault (core dumped)
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Also, running ./vx_app_arm_remote_log.out & gave same error.

    Logs:

    without ./vx_app_arm_remote_log.out:

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    cd /opt/edge_ai_apps/apps_cpp
    root@tda4vm-sk:/opt/edge_ai_apps/apps_cpp# ./bin/Release/app_edgeai ../configs/single_input_multi_infer.yaml
    libtidl_onnxrt_EP loaded 0x115d0ae0
    Final number of subgraphs created are : 1, - Offloaded Nodes - 320, Total Nodes - 320
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=4) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
    101.577654 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
    101.582418 s: VX_ZONE_INIT:Enabled
    101.582448 s: VX_ZONE_ERROR:Enabled
    101.582454 s: VX_ZONE_WARNING:Enabled
    101.583556 s: VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
    101.584717 s: VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
    [07:58:07.000.000000]:ERROR:[getConfig:0147] Mean value specification missing.
    [07:58:07.000.000000]:ERROR:[initialize:1437] getConfig() failed.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    with ./vx_app_arm_remote_log.out:

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    root@tda4vm-sk:/opt/edge_ai_apps/apps_cpp# ./bin/Release/app_edgeai ../configs/single_input_multi_infer.yaml
    libtidl_onnxrt_EP loaded 0x239baae0
    Final number of subgraphs created are : 1, - Offloaded Nodes - 320, Total Nodes - 320
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=4) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
    96.969500 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
    96.974270 s: VX_ZONE_INIT:Enabled
    96.974298 s: VX_ZONE_ERROR:Enabled
    96.974303 s: VX_ZONE_WARNING:Enabled
    96.975335 s: VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
    96.976640 s: VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
    [07:58:02.000.000000]:ERROR:[getConfig:0147] Mean value specification missing.
    [07:58:02.000.000000]:ERROR:[initialize:1437] getConfig() failed.
    [07:58:02.000.000536]:ERROR:[getConfig:0147] Mean value specification missing.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • log of ./vx_app_arm_remote_log.out:

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    root@tda4vm-sk:/opt/vision_apps# ./vx_app_arm_remote_log.out &
    [1] 1223
    root@tda4vm-sk:/opt/vision_apps# [MCU2_0] 4.206037 s: CIO: Init ... Done !!!
    [MCU2_0] 4.206104 s: ### CPU Frequency = 1000000000 Hz
    [MCU2_0] 4.206145 s: CPU is running FreeRTOS
    [MCU2_0] 4.206171 s: APP: Init ... !!!
    [MCU2_0] 4.206195 s: SCICLIENT: Init ... !!!
    [MCU2_0] 4.206461 s: SCICLIENT: DMSC FW version [8.5.2--v08.05.02 (Chill Capybar]
    [MCU2_0] 4.206510 s: SCICLIENT: DMSC FW revision 0x8
    [MCU2_0] 4.206558 s: SCICLIENT: DMSC FW ABI revision 3.1
    [MCU2_0] 4.206598 s: SCICLIENT: Init ... Done !!!
    [MCU2_0] 4.206626 s: UDMA: Init ... !!!
    [MCU2_0] 4.208139 s: UDMA: Init ... Done !!!
    [MCU2_0] 4.208197 s: MEM: Init ... !!!
    [MCU2_0] 4.208239 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ d9000000 of size 16777216 bytes !!!
    [MCU2_0] 4.208314 s: MEM: Created heap (L3_MEM, id=1, flags=0x00000000) @ 3600000 of size 262144 bytes !!!
    [MCU2_0] 4.208374 s: MEM: Init ... Done !!!
    [MCU2_0] 4.208400 s: IPC: Init ... !!!
    [MCU2_0] 4.208460 s: IPC: 6 CPUs participating in IPC !!!
    [MCU2_0] 4.208508 s: IPC: Waiting for HLOS to be ready ... !!!
    [MCU2_0] 15.839688 s: IPC: HLOS is ready !!!
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • Hi Ann,

    Could you please share your latest param.yaml file for our reference ?

    Regards,

    Pratik

  • Also Ann can u share the model and artifacts what u compiled so i can verify that issue what iam facing is version dependencies of SDK                                         

    Regards,

    N

  • Hi Pratik,

    param.yaml is as follows:

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    postprocess:
    data_layout: NCHW
    detection_thr: 0.3
    preprocess:
    crop:
    - 640
    - 640
    data_layout: NCHW
    mean:
    - 0
    - 0
    - 0
    resize:
    - 640
    - 640
    scale:
    - 0.003921568627
    - 0.003921568627
    - 0.003921568627
    session:
    artifacts_folder: artifacts
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

    Thanks for you help!

    Best regards,

    Ann

  • Hi

    Ann can u share the model and artifacts what u compiled so i can verify that issue what iam facing is version dependencies of SDK                                         

    Regards,

  • Hi Ann,

    Appears to me that param.yaml is not generated correctly.

    From the above posted logs,

    [07:58:02.000.000000]:ERROR:[getConfig:0147] Mean value specification missing.
    [07:58:02.000.000000]:ERROR:[initialize:1437] getConfig() failed.
    [07:58:02.000.000536]:ERROR:[getConfig:0147] Mean value specification missing.

     

    The mean values are set to 0 in your generated param.yaml file.

    Could try setting it random and check if this issue appears after that ?

    Regards,

    Pratik