This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-TDA4VM: Error when running custom compiled model

Part Number: SK-TDA4VM
Other Parts Discussed in Thread: TDA4VM

Hello experts,

I encountered an error while attempting to run this compiled depth estimation model on the TDA4VM board. I have attached the error log file, which provides more detailed information about the issue.

Additionally, I have also included the config yaml file that I used for running the model. It would be great if someone could review them and provide any insights or suggestions on what might be causing the error.

I would appreciate any help or guidance in troubleshooting this issue. If you have any experience with FastDepth model compilation, your input would be highly valuable.

Thank you in advance for your support!

Best regards,
Hareendran

Error when running the script.

root@tda4vm-sk:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/fast_depth.yaml
libtidl_onnxrt_EP loaded 0x206ee990
Final number of subgraphs created are : 1, - Offloaded Nodes - 87, Total Nodes - 87
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=4) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   193.811299 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   193.811472 s:  VX_ZONE_INIT:Enabled
   193.811519 s:  VX_ZONE_ERROR:Enabled
   193.811563 s:  VX_ZONE_WARNING:Enabled
   193.814806 s:  VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
   193.816037 s:  VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
   193.824784 s:  VX_ZONE_ERROR:[ownContextSendCmd:802] Command ack message returned failure cmd_status: -1
   193.824812 s:  VX_ZONE_ERROR:[ownContextSendCmd:838] tivxEventWait() failed.
   193.824945 s:  VX_ZONE_ERROR:[ownNodeKernelInit:525] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
   193.824956 s:  VX_ZONE_ERROR:[ownNodeKernelInit:526] Please be sure the target callbacks have been registered for this core
   193.824992 s:  VX_ZONE_ERROR:[ownNodeKernelInit:527] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
   193.825086 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl:1:1 ... failed !!!
   193.825140 s:  VX_ZONE_ERROR:[vxVerifyGraph:2055] Node kernel init failed
   193.825150 s:  VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
^C^Z
[2]+  Stopped(SIGTSTP)        ./app_edgeai.py ../configs/fast_depth.yaml
root@tda4vm-sk:/opt/edge_ai_apps/apps_python#

fast_depth.yaml

title: "Depth Map"
log_level: 2
inputs:
    input0:
        source: /opt/edge_ai_apps/data/images/%04d.jpg
        width: 1280
        height: 720
        index: 0
        framerate: 1
        loop: True
models:
    model0:
        model_path: /opt/model_zoo/de-7300_onnxrt_nyudepthv2_fast-depth_fast-depth_onnx
        # topN: 5
outputs:
    output0:
        sink: kmssink
        width: 1920
        height: 1080

flows:
    flow0: [input0,model0,output0,[320,180,1280,720]]

  • Hi,

    Could you please share which SDK version you are using ?

    Regards,

    Pratik

  • Hi Pratik,

    I'm using 8.5.0 SDK in TDA4VM and i compiled using the master branch of edgeai-benchmark repo with the run_custom_pc.sh script.

    Regards,
    Hareendran

  • Hi,

    Thanks for details.

    Could you please share the console log after running vx_app_arm_remote_log.out script located at /opt/vision_apps directory.

    Make sure you run your application first ,

    root@tda4vm-sk:/opt/edge_ai_apps/apps_python# ./app_edgeai.py ../configs/fast_depth.yaml

    Then run the vx_app_arm_remote_log.out as mentioned below,

    root@tda4vm-sk:/opt/edge_ai_apps/apps_python# ./vx_app_arm_remote_log.out &

    Regards,

    Pratik

  • HI Pratik,

    Please find the log below.

    root@tda4vm-sk:/opt/vision_apps# ./vx_app_arm_remote_log.out &
    [1] 1625
    root@tda4vm-sk:/opt/vision_apps# [MCU2_0]      3.829730 s: CIO: Init ... Done !!!
    [MCU2_0]      3.829801 s: ### CPU Frequency = 1000000000 Hz
    [MCU2_0]      3.829847 s: CPU is running FreeRTOS
    [MCU2_0]      3.829873 s: APP: Init ... !!!
    [MCU2_0]      3.829895 s: SCICLIENT: Init ... !!!
    [MCU2_0]      3.830158 s: SCICLIENT: DMSC FW version [8.5.2--v08.05.02 (Chill Capybar]
    [MCU2_0]      3.830207 s: SCICLIENT: DMSC FW revision 0x8  
    [MCU2_0]      3.830242 s: SCICLIENT: DMSC FW ABI revision 3.1
    [MCU2_0]      3.830297 s: SCICLIENT: Init ... Done !!!
    [MCU2_0]      3.830326 s: UDMA: Init ... !!!
    [MCU2_0]      3.831816 s: UDMA: Init ... Done !!!
    [MCU2_0]      3.831873 s: MEM: Init ... !!!
    [MCU2_0]      3.831912 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ d9000000 of size 16777216 bytes !!!
    [MCU2_0]      3.831987 s: MEM: Created heap (L3_MEM, id=1, flags=0x00000000) @ 3600000 of size 262144 bytes !!!
    [MCU2_0]      3.832050 s: MEM: Init ... Done !!!
    [MCU2_0]      3.832075 s: IPC: Init ... !!!
    [MCU2_0]      3.832136 s: IPC: 6 CPUs participating in IPC !!!
    [MCU2_0]      3.832183 s: IPC: Waiting for HLOS to be ready ... !!!
    [MCU2_0]     15.919575 s: IPC: HLOS is ready !!!
    [MCU2_0]     15.934929 s: IPC: Init ... Done !!!
    [MCU2_0]     15.934992 s: APP: Syncing with 5 CPUs ... !!!
    [MCU2_0]     16.062424 s: APP: Syncing with 5 CPUs ... Done !!!
    [MCU2_0]     16.062673 s: REMOTE_SERVICE: Init ... !!!
    [MCU2_0]     16.064233 s: REMOTE_SERVICE: Init ... Done !!!
    [MCU2_0]     16.064313 s: FVID2: Init ... !!!
    [MCU2_0]     16.064385 s: FVID2: Init ... Done !!!
    [MCU2_0]     16.064419 s: VHWA: VPAC Init ... !!!
    [MCU2_0]     16.064447 s: SCICLIENT: Sciclient_pmSetModuleState module=290 state=2
    [MCU2_0]     16.064853 s: SCICLIENT: Sciclient_pmSetModuleState success
    [MCU2_0]     16.064892 s: VHWA: LDC Init ... !!!
    [MCU2_0]     16.072062 s: VHWA: LDC Init ... Done !!!
    [MCU2_0]     16.072124 s: VHWA: MSC Init ... !!!
    [MCU2_0]     16.086993 s: VHWA: MSC Init ... Done !!!
    [MCU2_0]     16.087054 s: VHWA: NF Init ... !!!
    [MCU2_0]     16.088965 s: VHWA: NF Init ... Done !!!
    [MCU2_0]     16.089024 s: VHWA: VISS Init ... !!!
    [MCU2_0]     16.100010 s: VHWA: VISS Init ... Done !!!
    [MCU2_0]     16.100076 s: VHWA: VPAC Init ... Done !!!
    [MCU2_0]     16.100123 s:  VX_ZONE_INIT:Enabled
    [MCU2_0]     16.100154 s:  VX_ZONE_ERROR:Enabled
    [MCU2_0]     16.100183 s:  VX_ZONE_WARNING:Enabled
    [MCU2_0]     16.101366 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target MCU2-0 
    [MCU2_0]     16.101609 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_NF 
    [MCU2_0]     16.101830 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_LDC1 
    [MCU2_0]     16.102046 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_MSC1 
    [MCU2_0]     16.102271 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_MSC2 
    [MCU2_0]     16.102577 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target VPAC_VISS1 
    [MCU2_0]     16.102840 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE1 
    [MCU2_0]     16.103094 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE2 
    [MCU2_0]     16.103365 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DISPLAY1 
    [MCU2_0]     16.103623 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DISPLAY2 
    [MCU2_0]     16.103850 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CSITX 
    [MCU2_0]     16.104112 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE3 
    [MCU2_0]     16.104375 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE4 
    [MCU2_0]     16.104634 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE5 
    [MCU2_0]     16.104888 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE6 
    [MCU2_0]     16.105141 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE7 
    [MCU2_0]     16.105406 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target CAPTURE8 
    [MCU2_0]     16.105640 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DSS_M2M1 
    [MCU2_0]     16.105864 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DSS_M2M2 
    [MCU2_0]     16.106086 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DSS_M2M3 
    [MCU2_0]     16.106328 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DSS_M2M4 
    [MCU2_0]     16.106388 s:  VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
    [MCU2_0]     16.106426 s: APP: OpenVX Target kernel init ... !!!
    [MCU2_0]     16.125550 s: APP: OpenVX Target kernel init ... Done !!!
    [MCU2_0]     16.125602 s: VISS REMOTE SERVICE: Init ... !!!
    [MCU2_0]     16.125678 s: VISS REMOTE SERVICE: Init ... Done !!!
    [MCU2_0]     16.125713 s: UDMA Copy: Init ... !!!
    [MCU2_0]     16.127548 s: UDMA Copy: Init ... Done !!!
    [MCU2_0]     16.127644 s: APP: Init ... Done !!!
    [MCU2_0]     16.127683 s: APP: Run ... !!!
    [MCU2_0]     16.127709 s: IPC: Starting echo test ...
    [MCU2_0]     16.130283 s: APP: Run ... Done !!!
    [MCU2_0]     16.131779 s: IPC: Echo status: mpu1_0[x] mcu2_0[s] mcu2_1[.] C66X_1[P] C66X_2[.] C7X_1[.] 
    [MCU2_0]     16.131900 s: IPC: Echo status: mpu1_0[x] mcu2_0[s] mcu2_1[P] C66X_1[P] C66X_2[.] C7X_1[.] 
    [MCU2_0]     16.132001 s: IPC: Echo status: mpu1_0[x] mcu2_0[s] mcu2_1[P] C66X_1[P] C66X_2[P] C7X_1[.] 
    [MCU2_0]     16.132096 s: IPC: Echo status: mpu1_0[x] mcu2_0[s] mcu2_1[P] C66X_1[P] C66X_2[P] C7X_1[P] 
    [MCU2_1]      3.846572 s: CIO: Init ... Done !!!
    [MCU2_1]      3.846642 s: ### CPU Frequency = 1000000000 Hz
    [MCU2_1]      3.846683 s: CPU is running FreeRTOS
    [MCU2_1]      3.846710 s: APP: Init ... !!!
    [MCU2_1]      3.846733 s: SCICLIENT: Init ... !!!
    [MCU2_1]      3.846993 s: SCICLIENT: DMSC FW version [8.5.2--v08.05.02 (Chill Capybar]
    [MCU2_1]      3.847045 s: SCICLIENT: DMSC FW revision 0x8  
    [MCU2_1]      3.847081 s: SCICLIENT: DMSC FW ABI revision 3.1
    [MCU2_1]      3.847131 s: SCICLIENT: Init ... Done !!!
    [MCU2_1]      3.847161 s: UDMA: Init ... !!!
    [MCU2_1]      3.848743 s: UDMA: Init ... Done !!!
    [MCU2_1]      3.848802 s: MEM: Init ... !!!
    [MCU2_1]      3.848843 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ da000000 of size 16777216 bytes !!!
    [MCU2_1]      3.848914 s: MEM: Created heap (L3_MEM, id=1, flags=0x00000001) @ 3640000 of size 262144 bytes !!!
    [MCU2_1]      3.848975 s: MEM: Init ... Done !!!
    [MCU2_1]      3.849000 s: IPC: Init ... !!!
    [MCU2_1]      3.849058 s: IPC: 6 CPUs participating in IPC !!!
    [MCU2_1]      3.849105 s: IPC: Waiting for HLOS to be ready ... !!!
    [MCU2_1]     16.046898 s: IPC: HLOS is ready !!!
    [MCU2_1]     16.062297 s: IPC: Init ... Done !!!
    [MCU2_1]     16.062370 s: APP: Syncing with 5 CPUs ... !!!
    [MCU2_1]     16.062424 s: APP: Syncing with 5 CPUs ... Done !!!
    [MCU2_1]     16.062463 s: REMOTE_SERVICE: Init ... !!!
    [MCU2_1]     16.064252 s: REMOTE_SERVICE: Init ... Done !!!
    [MCU2_1]     16.064320 s: FVID2: Init ... !!!
    [MCU2_1]     16.064396 s: FVID2: Init ... Done !!!
    [MCU2_1]     16.064434 s: VHWA: DMPAC: Init ... !!!
    [MCU2_1]     16.064460 s: SCICLIENT: Sciclient_pmSetModuleState module=48 state=2
    [MCU2_1]     16.064978 s: SCICLIENT: Sciclient_pmSetModuleState success
    [MCU2_1]     16.065020 s: SCICLIENT: Sciclient_pmSetModuleState module=305 state=2
    [MCU2_1]     16.065448 s: SCICLIENT: Sciclient_pmSetModuleState success
    [MCU2_1]     16.065487 s: VHWA: DOF Init ... !!!
    [MCU2_1]     16.076551 s: VHWA: DOF Init ... Done !!!
    [MCU2_1]     16.076615 s: VHWA: SDE Init ... !!!
    [MCU2_1]     16.080670 s: VHWA: SDE Init ... Done !!!
    [MCU2_1]     16.080734 s: VHWA: DMPAC: Init ... Done !!!
    [MCU2_1]     16.080784 s:  VX_ZONE_INIT:Enabled
    [MCU2_1]     16.080814 s:  VX_ZONE_ERROR:Enabled
    [MCU2_1]     16.080840 s:  VX_ZONE_WARNING:Enabled
    [MCU2_1]     16.082047 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target MCU2-1 
    [MCU2_1]     16.082301 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DMPAC_SDE 
    [MCU2_1]     16.082523 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:66] Added target DMPAC_DOF 
    [MCU2_1]     16.082580 s:  VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
    [MCU2_1]     16.082618 s: APP: OpenVX Target kernel init ... !!!
    [MCU2_1]     16.082884 s: APP: OpenVX Target kernel init ... Done !!!
    [MCU2_1]     16.082928 s: UDMA Copy: Init ... !!!
    [MCU2_1]     16.085546 s: UDMA Copy: Init ... Done !!!
    [MCU2_1]     16.085614 s: APP: Init ... Done !!!
    [MCU2_1]     16.085649 s: APP: Run ... !!!
    [MCU2_1]     16.085675 s: IPC: Starting echo test ...
    [MCU2_1]     16.088234 s: APP: Run ... Done !!!
    [MCU2_1]     16.089519 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[s] C66X_1[P] C66X_2[.] C7X_1[.] 
    [MCU2_1]     16.089636 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[s] C66X_1[P] C66X_2[P] C7X_1[.] 
    [MCU2_1]     16.089722 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[s] C66X_1[P] C66X_2[P] C7X_1[P] 
    [MCU2_1]     16.131064 s: IPC: Echo status: mpu1_0[x] mcu2_0[P] mcu2_1[s] C66X_1[P] C66X_2[P] C7X_1[P] 
    [C6x_1 ]      3.906760 s: CIO: Init ... Done !!!
    [C6x_1 ]      3.906785 s: ### CPU Frequency = 1350000000 Hz
    [C6x_1 ]      3.906796 s: CPU is running FreeRTOS
    [C6x_1 ]      3.906803 s: APP: Init ... !!!
    [C6x_1 ]      3.906811 s: SCICLIENT: Init ... !!!
    [C6x_1 ]      3.907022 s: SCICLIENT: DMSC FW version [8.5.2--v08.05.02 (Chill Capybar]
    [C6x_1 ]      3.907034 s: SCICLIENT: DMSC FW revision 0x8  
    [C6x_1 ]      3.907044 s: SCICLIENT: DMSC FW ABI revision 3.1
    [C6x_1 ]      3.907053 s: SCICLIENT: Init ... Done !!!
    [C6x_1 ]      3.907062 s: UDMA: Init ... !!!
    [C6x_1 ]      3.908674 s: UDMA: Init ... Done !!!
    [C6x_1 ]      3.908694 s: MEM: Init ... !!!
    [C6x_1 ]      3.908708 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ dc000000 of size 16777216 bytes !!!
    [C6x_1 ]      3.908725 s: MEM: Created heap (L2_MEM, id=2, flags=0x00000001) @ 800000 of size 229376 bytes !!!
    [C6x_1 ]      3.908740 s: MEM: Created heap (DDR_SCRATCH_MEM, id=4, flags=0x00000001) @ dd000000 of size 50331648 bytes !!!
    [C6x_1 ]      3.908757 s: MEM: Init ... Done !!!
    [C6x_1 ]      3.908765 s: IPC: Init ... !!!
    [C6x_1 ]      3.908785 s: IPC: 6 CPUs participating in IPC !!!
    [C6x_1 ]      3.908799 s: IPC: Waiting for HLOS to be ready ... !!!
    [C6x_1 ]     14.822376 s: IPC: HLOS is ready !!!
    [C6x_1 ]     14.826401 s: IPC: Init ... Done !!!
    [C6x_1 ]     14.826428 s: APP: Syncing with 5 CPUs ... !!!
    [C6x_1 ]     16.062423 s: APP: Syncing with 5 CPUs ... Done !!!
    [C6x_1 ]     16.062440 s: REMOTE_SERVICE: Init ... !!!
    [C6x_1 ]     16.063111 s: REMOTE_SERVICE: Init ... Done !!!
    [C6x_1 ]     16.063147 s:  VX_ZONE_INIT:Enabled
    [C6x_1 ]     16.063157 s:  VX_ZONE_ERROR:Enabled
    [C6x_1 ]     16.063167 s:  VX_ZONE_WARNING:Enabled
    [C6x_1 ]     16.063960 s:  VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
    [C6x_1 ]     16.063974 s: APP: OpenVX Target kernel init ... !!!
    [C6x_1 ]     16.064237 s: APP: OpenVX Target kernel init ... Done !!!
    [C6x_1 ]     16.064253 s: UDMA Copy: Init ... !!!
    [C6x_1 ]     16.068880 s: UDMA Copy: Init ... Done !!!
    [C6x_1 ]     16.068902 s: APP: Init ... Done !!!
    [C6x_1 ]     16.068911 s: APP: Run ... !!!
    [C6x_1 ]     16.068920 s: IPC: Starting echo test ...
    [C6x_1 ]     16.070121 s: APP: Run ... Done !!!
    [C6x_1 ]     16.070473 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[s] C66X_2[x] C7X_1[P] 
    [C6x_1 ]     16.070721 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[s] C66X_2[P] C7X_1[P] 
    [C6x_1 ]     16.089231 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[P] C66X_1[s] C66X_2[P] C7X_1[P] 
    [C6x_1 ]     16.130936 s: IPC: Echo status: mpu1_0[x] mcu2_0[P] mcu2_1[P] C66X_1[s] C66X_2[P] C7X_1[P] 
    [C6x_2 ]      3.989956 s: CIO: Init ... Done !!!
    [C6x_2 ]      3.989983 s: ### CPU Frequency = 1350000000 Hz
    [C6x_2 ]      3.989994 s: CPU is running FreeRTOS
    [C6x_2 ]      3.990002 s: APP: Init ... !!!
    [C6x_2 ]      3.990010 s: SCICLIENT: Init ... !!!
    [C6x_2 ]      3.990244 s: SCICLIENT: DMSC FW version [8.5.2--v08.05.02 (Chill Capybar]
    [C6x_2 ]      3.990257 s: SCICLIENT: DMSC FW revision 0x8  
    [C6x_2 ]      3.990266 s: SCICLIENT: DMSC FW ABI revision 3.1
    [C6x_2 ]      3.990277 s: SCICLIENT: Init ... Done !!!
    [C6x_2 ]      3.990286 s: UDMA: Init ... !!!
    [C6x_2 ]      3.991914 s: UDMA: Init ... Done !!!
    [C6x_2 ]      3.991936 s: MEM: Init ... !!!
    [C6x_2 ]      3.991950 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ e0000000 of size 16777216 bytes !!!
    [C6x_2 ]      3.991967 s: MEM: Created heap (L2_MEM, id=2, flags=0x00000001) @ 800000 of size 229376 bytes !!!
    [C6x_2 ]      3.991983 s: MEM: Created heap (DDR_SCRATCH_MEM, id=4, flags=0x00000001) @ e1000000 of size 50331648 bytes !!!
    [C6x_2 ]      3.991999 s: MEM: Init ... Done !!!
    [C6x_2 ]      3.992008 s: IPC: Init ... !!!
    [C6x_2 ]      3.992029 s: IPC: 6 CPUs participating in IPC !!!
    [C6x_2 ]      3.992043 s: IPC: Waiting for HLOS to be ready ... !!!
    [C6x_2 ]     14.975027 s: IPC: HLOS is ready !!!
    [C6x_2 ]     14.981217 s: IPC: Init ... Done !!!
    [C6x_2 ]     14.981244 s: APP: Syncing with 5 CPUs ... !!!
    [C6x_2 ]     16.062423 s: APP: Syncing with 5 CPUs ... Done !!!
    [C6x_2 ]     16.062441 s: REMOTE_SERVICE: Init ... !!!
    [C6x_2 ]     16.063127 s: REMOTE_SERVICE: Init ... Done !!!
    [C6x_2 ]     16.063158 s:  VX_ZONE_INIT:Enabled
    [C6x_2 ]     16.063169 s:  VX_ZONE_ERROR:Enabled
    [C6x_2 ]     16.063180 s:  VX_ZONE_WARNING:Enabled
    [C6x_2 ]     16.063969 s:  VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
    [C6x_2 ]     16.063984 s: APP: OpenVX Target kernel init ... !!!
    [C6x_2 ]     16.064253 s: APP: OpenVX Target kernel init ... Done !!!
    [C6x_2 ]     16.064269 s: UDMA Copy: Init ... !!!
    [C6x_2 ]     16.069030 s: UDMA Copy: Init ... Done !!!
    [C6x_2 ]     16.069054 s: APP: Init ... Done !!!
    [C6x_2 ]     16.069064 s: APP: Run ... !!!
    [C6x_2 ]     16.069072 s: IPC: Starting echo test ...
    [C6x_2 ]     16.070358 s: APP: Run ... Done !!!
    [C6x_2 ]     16.070718 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[P] C66X_2[s] C7X_1[.] 
    [C6x_2 ]     16.070759 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[P] C66X_2[s] C7X_1[P] 
    [C6x_2 ]     16.089257 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[P] C66X_1[P] C66X_2[s] C7X_1[P] 
    [C6x_2 ]     16.130957 s: IPC: Echo status: mpu1_0[x] mcu2_0[P] mcu2_1[P] C66X_1[P] C66X_2[s] C7X_1[P] 
    [C7x_1 ]      4.543074 s: CIO: Init ... Done !!!
    [C7x_1 ]      4.543087 s: ### CPU Frequency = 1000000000 Hz
    [C7x_1 ]      4.543098 s: CPU is running FreeRTOS
    [C7x_1 ]      4.543106 s: APP: Init ... !!!
    [C7x_1 ]      4.543114 s: SCICLIENT: Init ... !!!
    [C7x_1 ]      4.543338 s: SCICLIENT: DMSC FW version [8.5.2--v08.05.02 (Chill Capybar]
    [C7x_1 ]      4.543352 s: SCICLIENT: DMSC FW revision 0x8  
    [C7x_1 ]      4.543362 s: SCICLIENT: DMSC FW ABI revision 3.1
    [C7x_1 ]      4.543372 s: SCICLIENT: Init ... Done !!!
    [C7x_1 ]      4.543381 s: UDMA: Init ... !!!
    [C7x_1 ]      4.544641 s: UDMA: Init ... Done !!!
    [C7x_1 ]      4.544653 s: MEM: Init ... !!!
    [C7x_1 ]      4.544664 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ 117000000 of size 268435456 bytes !!!
    [C7x_1 ]      4.544684 s: MEM: Created heap (L3_MEM, id=1, flags=0x00000001) @ 70020000 of size 8159232 bytes !!!
    [C7x_1 ]      4.544702 s: MEM: Created heap (L2_MEM, id=2, flags=0x00000001) @ 64800000 of size 458752 bytes !!!
    [C7x_1 ]      4.544720 s: MEM: Created heap (L1_MEM, id=3, flags=0x00000001) @ 64e00000 of size 16384 bytes !!!
    [C7x_1 ]      4.544736 s: MEM: Created heap (DDR_SCRATCH_MEM, id=4, flags=0x00000001) @ 100000000 of size 385875968 bytes !!!
    [C7x_1 ]      4.544755 s: MEM: Init ... Done !!!
    [C7x_1 ]      4.544764 s: IPC: Init ... !!!
    [C7x_1 ]      4.544777 s: IPC: 6 CPUs participating in IPC !!!
    [C7x_1 ]      4.544791 s: IPC: Waiting for HLOS to be ready ... !!!
    [C7x_1 ]     15.009698 s: IPC: HLOS is ready !!!
    [C7x_1 ]     15.011855 s: IPC: Init ... Done !!!
    [C7x_1 ]     15.011870 s: APP: Syncing with 5 CPUs ... !!!
    [C7x_1 ]     16.062425 s: APP: Syncing with 5 CPUs ... Done !!!
    [C7x_1 ]     16.062443 s: REMOTE_SERVICE: Init ... !!!
    [C7x_1 ]     16.062603 s: REMOTE_SERVICE: Init ... Done !!!
    [C7x_1 ]     16.062626 s:  VX_ZONE_INIT:Enabled
    [C7x_1 ]     16.062637 s:  VX_ZONE_ERROR:Enabled
    [C7x_1 ]     16.062646 s:  VX_ZONE_WARNING:Enabled
    [C7x_1 ]     16.062794 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1 
    [C7x_1 ]     16.062863 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_2 
    [C7x_1 ]     16.062999 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_3 
    [C7x_1 ]     16.063069 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_4 
    [C7x_1 ]     16.063136 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_5 
    [C7x_1 ]     16.063209 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_6 
    [C7x_1 ]     16.063306 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_7 
    [C7x_1 ]     16.063374 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:59] Added target DSP_C7-1_PRI_8 
    [C7x_1 ]     16.063397 s:  VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
    [C7x_1 ]     16.063409 s: APP: OpenVX Target kernel init ... !!!
    [C7x_1 ]     16.063549 s: APP: OpenVX Target kernel init ... Done !!!
    [C7x_1 ]     16.063564 s: APP: Init ... Done !!!
    [C7x_1 ]     16.063573 s: APP: Run ... !!!
    [C7x_1 ]     16.063581 s: IPC: Starting echo test ...
    [C7x_1 ]     16.063747 s: APP: Run ... Done !!!
    [C7x_1 ]     16.070546 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[P] C66X_2[.] C7X_1[s] 
    [C7x_1 ]     16.070712 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[x] C66X_1[P] C66X_2[P] C7X_1[s] 
    [C7x_1 ]     16.089279 s: IPC: Echo status: mpu1_0[x] mcu2_0[x] mcu2_1[P] C66X_1[P] C66X_2[P] C7X_1[s] 
    [C7x_1 ]     16.130993 s: IPC: Echo status: mpu1_0[x] mcu2_0[P] mcu2_1[P] C66X_1[P] C66X_2[P] C7X_1[s] 
    [C7x_1 ]    280.494495 s:  VX_ZONE_ERROR:[tivxKernelTIDLCreate:705] Network version - 0x20230208, Expected version - 0x20221206
    
    

    The below error keep gettting whenever i ran the model.
    [C7x_1 ] 678.441216 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:705] Network version - 0x20230208, Expected version - 0x20221206

    Regards,

    Hareendran

  • Hi,

    Thanks for sharing the file.

    [C7x_1 ] 678.441216 s: VX_ZONE_ERROR:[tivxKernelTIDLCreate:705] Network version - 0x20230208, Expected version - 0x20221206

    As I can see this issue is occurring because of sdk version miss match.

    The  model artifacts are built on sdk tools 8.6 from edgeai-benchmark.

    The default tidl tools version is set to the latest TIDL_TOOLS_RELEASE_NAME=${1:-r8.6}

    Refer here : https://github.com/TexasInstruments/edgeai-benchmark/blob/master/setup_pc.sh

    However you are using sdk 8.5 on target for inferencing purpose.

    You can resolve this either by recompiling model on 8.5 tools or using 8.6 sdk on target board.

    Regards,

    Pratik

  • Hi Pratik,

    Following your suggestion, I updated the TIDL_TOOLS_RELEASE_NAME variable to ${1:-r8.5} in the setup_pc.sh file. Then, I reinstalled the Python package by executing ./setup_pc.sh.

    After the successful installation, I compiled it using ./run_tutorials_pc.sh TDA4VM, which runs the tutorials/tutorial_detection.ipynb notebook. Please refer to the file below.

    # %% [markdown]
    # ## Model Compilation Jupyter Notebook Example
    # 
    # This notebook shows the example of model compilation using edgeai-benchmark.
    # 
    # This script uses TIDL to compile a model and output in a format that edgeai-sdk can understand.
    # 
    # edgeai_benchmark is a python package provided in edgeai-benchmark that provides several functions to assist model compilation.
    
    # %%
    import os
    import tempfile
    import argparse
    import cv2
    from edgeai_benchmark import *
    
    # %%
    # the cwd must be the root of the respository
    if os.path.split(os.getcwd())[-1] in ('scripts', 'tutorials'):
        os.chdir('../')
    #
    print(os.environ['TIDL_TOOLS_PATH'])
    print(os.getcwd())
    
    # %% [markdown]
    # #### Create a temporary directory. 
    # 
    # This is were the compiled artifacts will be stored.
    
    # %%
    # modelartifacts_tempdir = tempfile.TemporaryDirectory()
    current_directory = os.getcwd()
    final_directory = os.path.join(current_directory, r'compile_op_8_5')
    if not os.path.exists(final_directory):
       os.makedirs(final_directory)
    print(final_directory)
    modelartifacts_custom = os.path.join(final_directory, 'modelartifacts')
    
    # %% [markdown]
    # #### Read settings from settings_import_on_pc.yaml
    # 
    # Modify the settings as necessary in the constructor of settings.ConfigSettings()
    
    # %%
    settings = config_settings.ConfigSettings('./settings_import_on_pc.yaml', 
                    modelartifacts_path=modelartifacts_custom,
                    calibration_frames=10, calibration_iterations=10, num_frames=100)
    
    work_dir = os.path.join(settings.modelartifacts_path, f'{settings.tensor_bits}bits')
    print(f'work_dir = {work_dir}')
    
    # %% [markdown]
    # #### Create Dataset Reader classes
    # 
    # Change the dataset paths according to your dataset location
    
    # %%
    dataset_calib_cfg = dict(
        path=f'{settings.datasets_path}/coco',
        split='val2017',
        shuffle=True,
        num_frames=min(settings.calibration_frames,5000),
        name='coco'
    )
    
    # dataset parameters for actual inference
    dataset_val_cfg = dict(
        path=f'{settings.datasets_path}/coco',
        split='val2017',
        shuffle=False, # can be set to True as well, if needed
        num_frames=min(settings.num_frames,5000),
        name='coco'
    )
    
    # calib_dataset = datasets.COCODetection(**dataset_calib_cfg, download=True)
    # val_dataset = datasets.COCODetection(**dataset_val_cfg, download=True)
    
    # %% [markdown]
    # #### Session runtime_options
    # 
    # The default runtime_options can be overriden by passing a runtime_options dict to this function
    
    # %%
    # choose one session_name depending on the model type
    # tflitert for tflite models, onnxrt for onnx models, tvmdlr for mxnet models.
    # session_name = constants.SESSION_NAME_TFLITERT
    session_name = constants.SESSION_NAME_ONNXRT
    #session_name = constants.SESSION_NAME_TVMDLR
    
    session_type = settings.get_session_type(session_name)
    runtime_options = settings.get_runtime_options(session_name, is_qat=False)
    
    print(session_type)
    print(runtime_options)
    
    # %%
    preproc_transforms = preprocess.PreProcessTransforms(settings)
    postproc_transforms = postprocess.PostProcessTransforms(settings)
    
    # these session cfgs also has some default input mean and scale. 
    # if your model needs a difference mean and scale, update the session cfg dict being used with those values
    # onnx_session_cfg = sessions.get_onnx_session_cfg(settings, work_dir=work_dir)
    onnx_session_cfg = sessions.get_onnx_session_cfg(settings, work_dir=work_dir, input_mean=(0, 0, 0), input_scale=(1/255, 1/255, 1/255))
    # tflite_session_cfg = sessions.get_tflite_session_cfg(settings, work_dir=work_dir)
    
    # %% [markdown]
    # #### Create pipeline_configs
    # 
    # pipeline_configs is nothing but a dict with the various model configs that we want to compile
    
    # %%
    if settings.dataset_cache is None:
        settings.dataset_cache = datasets.get_datasets(settings)
        
    pipeline_configs = {
        'de-7300': dict(
            task_type='depth_estimation',
            calibration_dataset=settings.dataset_cache['nyudepthv2']['calibration_dataset'],
            input_dataset=settings.dataset_cache['nyudepthv2']['input_dataset'],
            dataset_category=datasets.DATASET_CATEGORY_NYUDEPTHV2,
            preprocess=preproc_transforms.get_transform_jai((246,246), (224,224), backend='cv2', interpolation=cv2.INTER_NEAREST),
            session=session_type(**onnx_session_cfg,
                runtime_options=utils.dict_update(settings.runtime_options_onnx_p2(),
                    {'advanced_options:output_feature_16bit_names_list':'233, 424'}),
                model_path=f'{settings.models_path}/vision/depth_estimation/nyudepthv2/fast-depth/fast-depth.onnx'),
            postprocess=postproc_transforms.get_transform_depth_estimation_onnx(),
            metric=dict(disparity=False, scale_shift=False),
            model_info=dict(metric_reference={'accuracy_delta_1%':77.1}, model_shortlist=None)
        )
    }
    print(pipeline_configs)
    
    # %% [markdown]
    # #### Model Compilation
    # 
    # This will take a few minutes. Please be patient...
    
    # %%
    # run the model compliation/import and inference
    tools.run_accuracy(settings, work_dir, pipeline_configs)
    
    # %% [markdown]
    # #### Package artifacts
    # 
    # Package the artifacts into a .tar.gz file, keeping only the necessary files for inference.
    
    # %%
    out_dir = f'{work_dir}_package'
    tools.package_artifacts(settings, work_dir, out_dir)
    



    Once the compilation was completed, I copied the output to TDA4VM and executed the model using ./app_edgeai.py ../configs/fast_depth.yaml. However, it only displayed the following output.

    Unfortunately, pressing ctrl + c does not allow us to exit; instead, we need to use ctrl + z.

    Moreover, when running the ./vx_app_arm_remote_log.out & command, no errors were reported.

    Additionally, after running the fast_depth demo, we encountered difficulties running any other working demo. The following logs were displayed.


    Furthermore, there were no errors in the output of the ./vx_app_arm_remote_log.out & command.

    Regards,

    Hareendran

     

  • Hi,

    We will get back to you on this issue by Monday.

    Regards,

    Pratik

  • Hi Pratik,

    Any update?

    Regards,
    Hareendran

  • Hi,

    As per your shared config file I can see that, output is configured to kmssink, can you reverify that the your board is connected to the display ?

    Additionally, after running the fast_depth demo, we encountered difficulties running any other working demo.

    Since you have halted the initial process abruptly, the deinits didn't happened.

    You can reboot the board and re run the other demos.

    Regards,

    Pratik 

  • Hi Pratik,

    The display is connected to the board and other demos are working fine.

    Regards,

    Hareendran

  • Hi,

    When i ran the cpp application with the custom compiled fast depth model, below errors are getting.

    root@tda4vm-sk:/opt/edge_ai_apps/apps_cpp/bin/Release# ./app_edgeai ../../../configs/fast_depth.yaml
    libtidl_onnxrt_EP loaded 0x2d4bc370
    Final number of subgraphs created are : 1, - Offloaded Nodes - 87, Total Nodes - 87
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=4) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
      1111.699111 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
      1111.699254 s:  VX_ZONE_INIT:Enabled
      1111.699266 s:  VX_ZONE_ERROR:Enabled
      1111.699308 s:  VX_ZONE_WARNING:Enabled
      1111.700010 s:  VX_ZONE_INIT:[tivxInitLocal:145] Initialization Done !!!
      1111.701119 s:  VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
    [10:29:45.000.000000]:ERROR:[makePostprocessImageObj:0088] Invalid post-processing task type.
    [10:29:45.000.000080]:ERROR:[createPostprocCntxt:1571] PostprocessImage::makePostprocessImageObj() failed.
    [10:29:45.000.000123]:ERROR:[initialize:1857] createPostprocCntxt() failed.
    Segmentation fault (core dumped)
    root@tda4vm-sk:/opt/edge_ai_apps/apps_cpp/bin/Release#


    From the error, it shows that post-processing task type not supports for the custom model.

    It only supports classification, detection, segmentation and human_pose_estimation. Please refer the post_process_image.cpp file


    Regards,
    Hareendran

  • Hi,

    We support fixed set of post processing tasks, they are object detection, segmentation, classification in our sdk.

    Have you added your custom post processing for this model ?

    Regards,

    Pratik

  • Hi Pratik,
    I didn't added any custom post processing for my model.

    Regards,
    Hareendran

  • Hi,

    Could you please share param.yaml file with us.

    Regards,

    Pratik

  • Hi Pratik,
    Please see the param.yaml file below.

    calibration_dataset:
      name: nyudepthv2
      num_classes: 151
      num_frames: 10
      path: ./dependencies/datasets/nyudepthv2
      shuffle: true
      split: val
    dataset_category: nyudepthv2
    input_dataset:
      name: nyudepthv2
      num_classes: 151
      num_frames: 100
      path: ./dependencies/datasets/nyudepthv2
      shuffle: false
      split: val
    metric:
      disparity: false
      run_dir: /home/vboxuser/edgeai-benchmark/compile_op_8_5/modelartifacts/8bits/de-7300_onnxrt_nyudepthv2_fast-depth_fast-depth_onnx
      scale_shift: false
    model_info:
      metric_reference:
        accuracy_delta_1%: 77.1
      model_shortlist: null
    postprocess:
      data_layout: NCHW
      save_output: false
    preprocess:
      add_flip_image: false
      backend: cv2
      crop:
      - 224
      - 224
      data_layout: NCHW
      interpolation: 0
      pad_color: 0
      resize:
      - 246
      - 246
      resize_with_pad: false
      reverse_channels: false
    session:
      artifacts_folder: artifacts
      extra_inputs: null
      input_data_layout: NCHW
      input_mean: null
      input_optimization: true
      input_scale: null
      input_shape:
        input.1Net_IN:
        - 1
        - 3
        - 224
        - 224
      model_file: /home/vboxuser/edgeai-benchmark/compile_op_8_5/modelartifacts/8bits/de-7300_onnxrt_nyudepthv2_fast-depth_fast-depth_onnx/model/fast-depth.onnx
      model_folder: model
      model_id: de-7300
      model_path: model/fast-depth.onnx
      model_type: null
      num_inputs: 1
      num_tidl_subgraphs: 16
      output_shape: null
      run_dir: de-7300_onnxrt_nyudepthv2_fast-depth_fast-depth_onnx
      run_dir_tree_depth: 3
      run_suffix: null
      runtime_options:
        accuracy_level: 1
        advanced_options:activation_clipping: 1
        advanced_options:add_data_convert_ops: 3
        advanced_options:bias_calibration: 1
        advanced_options:calibration_frames: 10
        advanced_options:calibration_iterations: 10
        advanced_options:channel_wise_quantization: 0
        advanced_options:high_resolution_optimization: 0
        advanced_options:output_feature_16bit_names_list: 233, 424
        advanced_options:params_16bit_names_list: ''
        advanced_options:pre_batchnorm_fold: 1
        advanced_options:quantization_scale_type: 0
        advanced_options:weight_clipping: 1
        artifacts_folder: /home/vboxuser/edgeai-benchmark/compile_op_8_5/modelartifacts/8bits/de-7300_onnxrt_nyudepthv2_fast-depth_fast-depth_onnx/artifacts
        debug_level: 0
        import: 'no'
        platform: J7
        priority: 0
        tensor_bits: 8
        tidl_tools_path: /home/vboxuser/edgeai-benchmark/tools/TDA4VM/tidl_tools
        version: '8.2'
      session_name: onnxrt
      supported_machines: null
      target_device: null
      target_machine: pc
      tensor_bits: 8
      tidl_offload: true
      tidl_tools_path: /home/vboxuser/edgeai-benchmark/tools/TDA4VM/tidl_tools
      work_dir: /home/vboxuser/edgeai-benchmark/compile_op_8_5/modelartifacts/8bits
    task_type: depth_estimation

    I also changed the

    postprocess:
      data_layout: NCHW
      save_output: false
    section to
    postprocess:{}
    and tried. But no change. I'am also attaching the generated artifacts folder for your reference.
    de-7300_onnxrt_nyudepthv2_fast-depth_fast-depth_onnx.tar.gz

    Regards,

    Hareendran

  • Hi,

    From the shared param.yaml file i can see that task_type is set to depth_estimation.

    As mentioned earlier we support limited post processing layers, which are object classification, object segmentation, object detection as part of sdk offering.

    The depth_estimation is custom post processing task which is not supported by inside sdk.

    We would recommend, based on your project requirement you can either experiment using standard supported post processing layers or you can implement your custom.

    Please refer below mentioned github link for custom post processing implementation :

    https://github.com/TexasInstruments/edgeai-gst-apps-6d-pose

    Regards,

    Pratik

  • Hi Pratik,

    Thanks for the github link for object 6D pose estimation. I'll try the custom post processing implementation.

    Best regards,
    Hareendran

  • Sure, thanks for letting us know.