J722SXH01EVM: Out of the box demos in custom build.

Part Number: J722SXH01EVM
Other Parts Discussed in Thread: AM67A

Tool/software:

Hello, 

I'm starting on a project targeting the AM67A. Using the prebuilt wic file, the Edge AI gallery functions as expected. But when I build the edgeai Yocto image from scratch (host: Ubuntu 22.04, building sdk 11.0), the Edge AI gallery examples crash. echo $EDGEAI_VERSION returns 11.0.

Here's a call to one of the out of the box examples and the resulting error:

root@j722s-evm:/opt/edgeai-gst-apps/apps_python# python3 app_edgeai.py ../configs/object_detection.yaml     
libtidl_onnxrt_EP loaded 0x381c9c70 
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
APP: Init ... !!!
   338.949354 s: MEM: Init ... !!!
   338.949438 s: MEM: Initialized DMA HEAP (fd=5) !!!
   338.949640 s: MEM: Init ... Done !!!
   338.949669 s: IPC: Init ... !!!
   339.005974 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   339.012924 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   339.013124 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
   339.013146 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
   339.013157 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
   339.014156 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-0 
   339.014520 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-1 
   339.014799 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-2 
   339.015051 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-3 
   339.015080 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
   339.015119 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
   339.029888 s:  VX_ZONE_ERROR: [ownContextSendCmd:1001] Command ack message returned failure cmd_status: -1
   339.029936 s:  VX_ZONE_ERROR: [ownNodeKernelInit:704] Target kernel, TIVX_CMD_NODE_CREATE failed for node node_84
   339.029949 s:  VX_ZONE_ERROR: [ownNodeKernelInit:705] Please be sure the target callbacks have been registered for this core
   339.029961 s:  VX_ZONE_ERROR: [ownNodeKernelInit:706] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
   339.029976 s:  VX_ZONE_ERROR: [ownGraphNodeKernelInit:793] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
   339.030029 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Node kernel init failed
   339.030043 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Graph verify failed
Traceback (most recent call last):
  File "/opt/edgeai-gst-apps/apps_python/app_edgeai.py", line 67, in <module>
    main(sys.argv)
  File "/opt/edgeai-gst-apps/apps_python/app_edgeai.py", line 46, in main
    demo = EdgeAIDemo(config)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/edgeai-gst-apps/apps_python/edge_ai_class.py", line 108, in __init__
    model_obj.create_runtime()
  File "/usr/lib/python3.12/site-packages/edgeai_dl_inferer.py", line 315, in create_runtime
    self.run_time = RunTime(self.artifacts,
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/edgeai_dl_inferer.py", line 170, in __init__
    self.interpreter = _onnxruntime.InferenceSession(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 387, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 439, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Create state function failed. Return value:-1

I trained a YOLO network using the edgeai-modelmaker repo on branch r11.0 to make sure it wasn't a version compatibility issue - I got about the same error passing an array of 0's into the network. Can you please advise on what may be going on? I see there's now also an SDK 11.01.07 configuration, but wasn't sure if that would work with the r11.0 branch of edgeai-tensorlab. 

  • Hi Derek,

    How did you build the edgai image? Did you set the ARAGO_BRAND to edgeai?

    Best,
    Jared

  • Yes, I added ARAGO_BRAND = "edgeai" at the bottom of local.conf, set the machine to j722s-evm, limit the threads to 8 so my machine doesn't crash, then bitbake tisdk-edgeai-image (without the -k flag, so I can debug any recipes that fail). 

    I've tried both configs/processor-sdk-analytics/processor-sdk-analytics-11.00.00-config.txt and configs/processor-sdk-analytics/processor-sdk-analytics-11.01.07.05-config.txt

    .

  • Hi Derek,

    It looks like the kernel hasn't been loaded to the C7x correctly.

    Can you try running:

    cd /opt/vision_apps
    source ./vision_apps_init.sh

    Best,
    Jared

  • Nice, that gets me to a new error. 

    root@j722s-evm:/opt/edgeai-gst-apps/apps_python# python3 app_edgeai.py ../configs/object_detection.yaml 
    libtidl_onnxrt_EP loaded 0xa74f3c0 
    Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
    APP: Init ... !!!
       203.672271 s: MEM: Init ... !!!
       203.672349 s: MEM: Initialized DMA HEAP (fd=5) !!!
       203.672551 s: MEM: Init ... Done !!!
       203.672583 s: IPC: Init ... !!!
       203.729138 s: IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
       203.735875 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
       203.736090 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
       203.736119 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
       203.736132 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
       203.737231 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-0 
       203.737585 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-1 
       203.738003 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-2 
       203.738412 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-3 
       203.738465 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
       203.738504 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
       203.752897 s:  VX_ZONE_ERROR: [ownContextSendCmd:1001] Command ack message returned failure cmd_status: -1
       203.752945 s:  VX_ZONE_ERROR: [ownNodeKernelInit:704] Target kernel, TIVX_CMD_NODE_CREATE failed for node node_84
       203.752958 s:  VX_ZONE_ERROR: [ownNodeKernelInit:705] Please be sure the target callbacks have been registered for this core
       203.752970 s:  VX_ZONE_ERROR: [ownNodeKernelInit:706] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
       203.752985 s:  VX_ZONE_ERROR: [ownGraphNodeKernelInit:793] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
       203.753038 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Node kernel init failed
       203.753050 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Graph verify failed
    Traceback (most recent call last):
      File "/opt/edgeai-gst-apps/apps_python/app_edgeai.py", line 67, in <module>
        main(sys.argv)
      File "/opt/edgeai-gst-apps/apps_python/app_edgeai.py", line 46, in main
        demo = EdgeAIDemo(config)
               ^^^^^^^^^^^^^^^^^^
      File "/opt/edgeai-gst-apps/apps_python/edge_ai_class.py", line 108, in __init__
        model_obj.create_runtime()
      File "/usr/lib/python3.12/site-packages/edgeai_dl_inferer.py", line 315, in create_runtime
        self.run_time = RunTime(self.artifacts,
                        ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/site-packages/edgeai_dl_inferer.py", line 170, in __init__
        self.interpreter = _onnxruntime.InferenceSession(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 387, in __init__
    [C7x_1 ]    203.752620 s:  VX_ZONE_ERROR: [tivxKernelTIDLCreate:964] Network version - 0x20250630, Expected version - 0x20250429
        self._create_inference_session(providers, provider_options, disabled_optimizers)
      File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 439, in _create_inference_session
        sess.initialize_session(providers, provider_options, disabled_optimizers)
    onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Create state function failed. Return value:-1
    

    Doesn't this mean the wrong branch of AI tools was used to compile the model? echo $EDGEAI_VERSION returns 11.0, and both the model that came with the build and the YOLO I compiled with edgeai-modelmaker throw the same error...and both are apparently version 0x20250630. What SDK version is 0x20250429? 

  • Hi Derek,

    Doesn't this mean the wrong branch of AI tools was used to compile the model? echo $EDGEAI_VERSION returns 11.0, and both the model that came with the built and the YOLO I compiled with edgeai-modelmaker throw the same error...and both are apparently version 0x20250630. What SDK version is 0x20250429? 

    I don't know the TIDL version specifics for each SDK release. The software manifest doesn't seem to have the specific hex version dates.

    My main concern is in "203.752985 s: VX_ZONE_ERROR: [ownGraphNodeKernelInit:793] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!"

    This means that the C7x isn't loaded with the TIDL kernel, which will break any model you want to run.

    I will test building and running a Yocto generated image.

    Can you also send the output log of the:

    source ./vision_apps_init.sh

    Best,
    Jared

  • Sure, here it is. 

    root@j722s-evm:/opt/vision_apps# source ./vision_apps_init.sh                                                                                                                                                      
    root@j722s-evm:/opt/vision_apps# [MCU2_0]      5.385048 s: CIO: Init ... Done !!!
    [MCU2_0]      5.385086 s: CPU is running FreeRTOS
    [MCU2_0]      5.385098 s: APP: Init ... !!!
    [MCU2_0]      5.385109 s: SCICLIENT: Init ... !!!
    [MCU2_0]      5.385169 s: SCICLIENT: DMSC FW version [11.0.9--v11.00.09+ (Fancy Rat)]
    [MCU2_0]      5.385185 s: SCICLIENT: DMSC FW revision 0xb  
    [MCU2_0]      5.385199 s: SCICLIENT: DMSC FW ABI revision 4.0
    [MCU2_0]      5.385213 s: SCICLIENT: Init ... Done !!!
    [MCU2_0]      5.385225 s: UDMA: Init ... !!!
    [MCU2_0]      5.385402 s: UDMA: Init ... Done !!!
    [MCU2_0]      5.385418 s: MEM: Init ... !!!
    [MCU2_0]      5.385432 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ b5800000 of size 33554432 bytes !!!
    [MCU2_0]      5.385460 s: MEM: Init ... Done !!!
    [MCU2_0]      5.385472 s: IPC: Init ... !!!
    [MCU2_0]      5.385486 s: IPC: 4 CPUs participating in IPC !!!
    [MCU2_0]      5.385684 s: IPC: Waiting for HLOS to be ready ... !!!
    [MCU2_0]     16.365761 s: IPC: HLOS is ready !!!
    [MCU2_0]     16.365826 s: IPC: Init ... Done !!!
    [MCU2_0]     16.365841 s: APP: Syncing with 3 CPUs ... !!!
    [MCU2_0]     16.365859 s: APP: Syncing with 3 CPUs ... Done !!!
    [MCU2_0]     16.365874 s: REMOTE_SERVICE: Init ... !!!
    [MCU2_0]     16.366790 s: REMOTE_SERVICE: Init ... Done !!!
    [MCU2_0]     16.366812 s: FVID2: Init ... !!!
    [MCU2_0]     16.366840 s: FVID2: Init ... Done !!!
    [MCU2_0]     16.366863 s: VHWA: VPAC Init ... !!!
    [MCU2_0]     16.366907 s: SCICLIENT: Sciclient_pmSetModuleState module=219 state=2
    [MCU2_0]     16.366988 s: SCICLIENT: Sciclient_pmSetModuleState success
    [MCU2_0]     16.367021 s: VHWA: LDC Init ... !!!
    [MCU2_0]     16.367124 s: VHWA: LDC Init ... Done !!!
    [MCU2_0]     16.367143 s: VHWA: MSC Init ... !!!
    [MCU2_0]     16.367842 s: VHWA: MSC Init ... Done !!!
    [MCU2_0]     16.367862 s: VHWA: VISS Init ... !!!
    [MCU2_0]     16.368968 s: VHWA: VISS Init ... Done !!!
    [MCU2_0]     16.368994 s: VHWA: FC Init ... !!!
    [MCU2_0]     16.369082 s: VHWA: FC Init ... Done !!!
    [MCU2_0]     16.369095 s: VHWA: VPAC Init ... Done !!!
    [MCU2_0]     16.369108 s: VHWA: DMPAC: Init ... !!!
    [MCU2_0]     16.369161 s: SCICLIENT: Sciclient_pmSetModuleState module=277 state=2
    [MCU2_0]     16.369284 s: SCICLIENT: Sciclient_pmSetModuleState success
    [MCU2_0]     16.369298 s: VHWA: DOF Init ... !!!
    [MCU2_0]     16.369495 s: VHWA: DOF Init ... Done !!!
    [MCU2_0]     16.369516 s: VHWA: SDE Init ... !!!
    [MCU2_0]     16.369603 s: VHWA: SDE Init ... Done !!!
    [MCU2_0]     16.369621 s: VHWA: DMPAC: Init ... Done !!!
    [MCU2_0]     16.369644 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
    [MCU2_0]     16.369661 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
    [MCU2_0]     16.369678 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
    [MCU2_0]     16.369899 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.cmd_timeout_test on target MCU2-0
    [MCU2_0]     16.369971 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.tiovx_overhead on target MCU2-0
    [MCU2_0]     16.370018 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_sink on target MCU2-0
    [MCU2_0]     16.370115 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_source on target MCU2-0
    [MCU2_0]     16.370178 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_sink2 on target MCU2-0
    [MCU2_0]     16.370222 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_source2 on target MCU2-0
    [MCU2_0]     16.370268 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_intermediate on target MCU2-0
    [MCU2_0]     16.370311 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_intermediate_2 on target MCU2-0
    [MCU2_0]     16.370352 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_source_error on target MCU2-0
    [MCU2_0]     16.370395 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_source_obj_array on target MCU2-0
    [MCU2_0]     16.370438 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_sink_obj_array on target MCU2-0
    [MCU2_0]     16.370482 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_intermediate on target MCU2-0
    [MCU2_0]     16.370523 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_source on target MCU2-0
    [MCU2_0]     16.370566 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_sink on target MCU2-0
    [MCU2_0]     16.370605 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.test_target on target MCU2-0
    [MCU2_0]     16.370645 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.image_intermediate on target MCU2-0
    [MCU2_0]     16.370690 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.ext.obj_array_split on target MCU2-0
    [MCU2_0]     16.370785 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MCU2-0 
    [MCU2_0]     16.370850 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target VPAC_LDC1 
    [MCU2_0]     16.370912 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target VPAC_MSC1 
    [MCU2_0]     16.370970 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target VPAC_MSC2 
    [MCU2_0]     16.371073 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target VPAC_VISS1 
    [MCU2_0]     16.371152 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target CAPTURE1 
    [MCU2_0]     16.371223 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target CAPTURE2 
    [MCU2_0]     16.371304 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target CAPTURE3 
    [MCU2_0]     16.371378 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target CAPTURE4 
    [MCU2_0]     16.371451 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DISPLAY1 
    [MCU2_0]     16.371535 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DISPLAY2 
    [MCU2_0]     16.371600 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target CSITX 
    [MCU2_0]     16.371662 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target CSITX2 
    [MCU2_0]     16.371723 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DMPAC_SDE 
    [MCU2_0]     16.371783 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DMPAC_DOF 
    [MCU2_0]     16.371881 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target VPAC_FC 
    [MCU2_0]     16.371905 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
    [MCU2_0]     16.371924 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
    [MCU2_0]     16.371938 s: APP: OpenVX Target kernel init ... !!!
    [MCU2_0]     16.377397 s: APP: OpenVX Target kernel init ... Done !!!
    [MCU2_0]     16.377421 s: VISS REMOTE SERVICE: Init ... !!!
    [MCU2_0]     16.377462 s: VISS REMOTE SERVICE: Init ... Done !!!
    [MCU2_0]     16.377477 s: UDMA Copy: Init ... !!!
    [MCU2_0]     16.377759 s: UDMA Copy: Init ... Done !!!
    [MCU2_0]     16.377786 s: APP: Init ... Done !!!
    [MCU2_0]     16.377799 s: APP: Run ... !!!
    [MCU2_0]     16.377810 s: IPC: Starting echo test ...
    [MCU2_0]     16.377874 s: APP: Run ... Done !!!
    [MCU2_0]     16.378652 s: IPC: Echo status: a530-0[.] main-r5f0-0[s] c75ss0[.] c75ss1[P] 
    [MCU2_0]     16.378772 s: IPC: Echo status: a530-0[.] main-r5f0-0[s] c75ss0[P] c75ss1[P] 
    [C7x_1 ]      5.556337 s: CIO: Init ... Done !!!
    [C7x_1 ]      5.556356 s: CPU is running FreeRTOS
    [C7x_1 ]      5.556368 s: APP: Init ... !!!
    [C7x_1 ]      5.556379 s: SCICLIENT: Init ... !!!
    [C7x_1 ]      5.556439 s: SCICLIENT: DMSC FW version [11.0.9--v11.00.09+ (Fancy Rat)]
    [C7x_1 ]      5.556457 s: SCICLIENT: DMSC FW revision 0xb  
    [C7x_1 ]      5.556471 s: SCICLIENT: DMSC FW ABI revision 4.0
    [C7x_1 ]      5.556484 s: SCICLIENT: Init ... Done !!!
    [C7x_1 ]      5.556496 s: UDMA: Init ... !!!
    [C7x_1 ]      5.556522 s: UDMA: Init ... Done !!!
    [C7x_1 ]      5.556534 s: MEM: Init ... !!!
    [C7x_1 ]      5.556548 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ 108000000 of size 67108864 bytes !!!
    [C7x_1 ]      5.556573 s: MEM: Init ... Done !!!
    [C7x_1 ]      5.556584 s: IPC: Init ... !!!
    [C7x_1 ]      5.556595 s: IPC: 4 CPUs participating in IPC !!!
    [C7x_1 ]      5.556910 s: IPC: Waiting for HLOS to be ready ... !!!
    [C7x_1 ]     15.444699 s: IPC: HLOS is ready !!!
    [C7x_1 ]     15.444777 s: IPC: Init ... Done !!!
    [C7x_1 ]     15.444793 s: APP: Syncing with 3 CPUs ... !!!
    [C7x_1 ]     16.365862 s: APP: Syncing with 3 CPUs ... Done !!!
    [C7x_1 ]     16.365879 s: REMOTE_SERVICE: Init ... !!!
    [C7x_1 ]     16.366750 s: REMOTE_SERVICE: Init ... Done !!!
    [C7x_1 ]     16.366779 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
    [C7x_1 ]     16.366799 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
    [C7x_1 ]     16.366817 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
    [C7x_1 ]     16.367331 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel vx_tutorial_graph.phase_rgb on target DSP_C7-1
    [C7x_1 ]     16.367391 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_sink on target DSP_C7-1
    [C7x_1 ]     16.367442 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_source on target DSP_C7-1
    [C7x_1 ]     16.367495 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_sink2 on target DSP_C7-1
    [C7x_1 ]     16.367543 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_source2 on target DSP_C7-1
    [C7x_1 ]     16.367633 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_intermediate on target DSP_C7-1
    [C7x_1 ]     16.367726 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_intermediate_2 on target DSP_C7-1
    [C7x_1 ]     16.367798 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_source_error on target DSP_C7-1
    [C7x_1 ]     16.367854 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_source_obj_array on target DSP_C7-1
    [C7x_1 ]     16.367940 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_sink_obj_array on target DSP_C7-1
    [C7x_1 ]     16.368031 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_intermediate on target DSP_C7-1
    [C7x_1 ]     16.368077 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_source on target DSP_C7-1
    [C7x_1 ]     16.368161 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_sink on target DSP_C7-1
    [C7x_1 ]     16.368248 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.test_target on target DSP_C7-1
    [C7x_1 ]     16.368291 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.image_intermediate on target DSP_C7-1
    [C7x_1 ]     16.368373 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.multi_in_out on target DSP_C7-1
    [C7x_1 ]     16.368416 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.test_target on target DSP_C7-1
    [C7x_1 ]     16.368459 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.tiovx_overhead on target DSP_C7-1
    [C7x_1 ]     16.368583 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1 
    [C7x_1 ]     16.368685 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1_PRI_2 
    [C7x_1 ]     16.368784 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1_PRI_3 
    [C7x_1 ]     16.368881 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1_PRI_4 
    [C7x_1 ]     16.368977 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1_PRI_5 
    [C7x_1 ]     16.369075 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1_PRI_6 
    [C7x_1 ]     16.369171 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1_PRI_7 
    [C7x_1 ]     16.369266 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-1_PRI_8 
    [C7x_1 ]     16.369293 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
    [C7x_1 ]     16.369334 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
    [C7x_1 ]     16.369376 s: APP: OpenVX Target kernel init ... !!!
    [C7x_1 ]     16.369978 s: APP: OpenVX Target kernel init ... Done !!!
    [C7x_1 ]     16.369996 s: APP: Init ... Done !!!
    [C7x_1 ]     16.370007 s: APP: Run ... !!!
    [C7x_1 ]     16.370016 s: IPC: Starting echo test ...
    [C7x_1 ]     16.370143 s: APP: Run ... Done !!!
    [C7x_1 ]     16.379023 s: IPC: Echo status: a530-0[.] main-r5f0-0[P] c75ss0[s] c75ss1[.] 
    [C7x_1 ]     16.379062 s: IPC: Echo status: a530-0[.] main-r5f0-0[P] c75ss0[s] c75ss1[P] 
    [C7x_2 ]      5.729098 s: CIO: Init ... Done !!!
    [C7x_2 ]      5.729116 s: CPU is running FreeRTOS
    [C7x_2 ]      5.729128 s: APP: Init ... !!!
    [C7x_2 ]      5.729138 s: SCICLIENT: Init ... !!!
    [C7x_2 ]      5.729199 s: SCICLIENT: DMSC FW version [11.0.9--v11.00.09+ (Fancy Rat)]
    [C7x_2 ]      5.729217 s: SCICLIENT: DMSC FW revision 0xb  
    [C7x_2 ]      5.729231 s: SCICLIENT: DMSC FW ABI revision 4.0
    [C7x_2 ]      5.729244 s: SCICLIENT: Init ... Done !!!
    [C7x_2 ]      5.729256 s: UDMA: Init ... !!!
    [C7x_2 ]      5.729282 s: UDMA: Init ... Done !!!
    [C7x_2 ]      5.729296 s: MEM: Init ... !!!
    [C7x_2 ]      5.729309 s: MEM: Created heap (DDR_LOCAL_MEM, id=0, flags=0x00000004) @ 118000000 of size 67108864 bytes !!!
    [C7x_2 ]      5.729334 s: MEM: Init ... Done !!!
    [C7x_2 ]      5.729345 s: IPC: Init ... !!!
    [C7x_2 ]      5.729356 s: IPC: 4 CPUs participating in IPC !!!
    [C7x_2 ]      5.729669 s: IPC: Waiting for HLOS to be ready ... !!!
    [C7x_2 ]     15.620452 s: IPC: HLOS is ready !!!
    [C7x_2 ]     15.620526 s: IPC: Init ... Done !!!
    [C7x_2 ]     15.620541 s: APP: Syncing with 3 CPUs ... !!!
    [C7x_2 ]     16.365862 s: APP: Syncing with 3 CPUs ... Done !!!
    [C7x_2 ]     16.365879 s: REMOTE_SERVICE: Init ... !!!
    [C7x_2 ]     16.366008 s: REMOTE_SERVICE: Init ... Done !!!
    [C7x_2 ]     16.366032 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
    [C7x_2 ]     16.366052 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
    [C7x_2 ]     16.366069 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
    [C7x_2 ]     16.366625 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel vx_tutorial_graph.phase_rgb on target DSP_C7-2
    [C7x_2 ]     16.366695 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_sink on target DSP_C7-2
    [C7x_2 ]     16.366760 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_source on target DSP_C7-2
    [C7x_2 ]     16.366802 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_sink2 on target DSP_C7-2
    [C7x_2 ]     16.366847 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_source2 on target DSP_C7-2
    [C7x_2 ]     16.366927 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.scalar_intermediate on target DSP_C7-2
    [C7x_2 ]     16.366970 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_intermediate_2 on target DSP_C7-2
    [C7x_2 ]     16.367013 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_source_error on target DSP_C7-2
    [C7x_2 ]     16.367058 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_source_obj_array on target DSP_C7-2
    [C7x_2 ]     16.367102 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.scalar_sink_obj_array on target DSP_C7-2
    [C7x_2 ]     16.367148 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_intermediate on target DSP_C7-2
    [C7x_2 ]     16.367196 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_source on target DSP_C7-2
    [C7x_2 ]     16.367251 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.pyramid_sink on target DSP_C7-2
    [C7x_2 ]     16.367301 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.test_target on target DSP_C7-2
    [C7x_2 ]     16.367352 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.capture.image_intermediate on target DSP_C7-2
    [C7x_2 ]     16.367401 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.test_target on target DSP_C7-2
    [C7x_2 ]     16.367471 s:  VX_ZONE_INFO: [ownAddTargetKernelInternal:189] registered kernel com.ti.test_kernels.tiovx_overhead on target DSP_C7-2
    [C7x_2 ]     16.367631 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2 
    [C7x_2 ]     16.367730 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2_PRI_2 
    [C7x_2 ]     16.367826 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2_PRI_3 
    [C7x_2 ]     16.367925 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2_PRI_4 
    [C7x_2 ]     16.368042 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2_PRI_5 
    [C7x_2 ]     16.368158 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2_PRI_6 
    [C7x_2 ]     16.368260 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2_PRI_7 
    [C7x_2 ]     16.368356 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target DSP_C7-2_PRI_8 
    [C7x_2 ]     16.368381 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
    [C7x_2 ]     16.368399 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
    [C7x_2 ]     16.368421 s: APP: OpenVX Target kernel init ... !!!
    [C7x_2 ]     16.369204 s: APP: OpenVX Target kernel init ... Done !!!
    [C7x_2 ]     16.369264 s: APP: Init ... Done !!!
    [C7x_2 ]     16.369276 s: APP: Run ... !!!
    [C7x_2 ]     16.369286 s: IPC: Starting echo test ...
    [C7x_2 ]     16.369426 s: APP: Run ... Done !!!
    [C7x_2 ]     16.378664 s: IPC: Echo status: a530-0[.] main-r5f0-0[P] c75ss0[.] c75ss1[s] 
    [C7x_2 ]     16.378937 s: IPC: Echo status: a530-0[.] main-r5f0-0[P] c75ss0[P] c75ss1[s] 
    

  • Hi Derek,

    Thank you for the log. I will run my own Yocto build and see if I can replicate your issue.

    Best,
    Jared

  • Thank you! 

  • HI Derek,

    I've successfully recreated your issue.

    Best,
    Jared

  • Hi Jared, 

    Great, I'll revert SDK versions for now and await a solution to go to 11.

    Derek

  • Hi Derek,

    I'm currently testing between the prepackaged .config in the SDK vs the 11.00 config from the repository.

    I will let you know if the prepackaged .config works.

    Best,
    Jared

  • Hi Derek,

    I am running into issues with the prepackaged processor-sdk-analytics-11_00-config.txt

    # This file takes repo entries in the format
    # repo name,repo uri,repo branch,repo commit[,layers=layer1:layer2...:layern]
    
    bitbake,https://git.openembedded.org/bitbake,2.8,696c2c1ef095f8b11c7d2eff36fae50f58c62e5e
    meta-arago,https://git.yoctoproject.org/meta-arago,scarthgap,c04e85dd207dd24387edac10526b9ec971df9496,layers=meta-arago-distro:meta-arago-extras:meta-arago-test
    meta-qt6,https://code.qt.io/yocto/meta-qt6.git,6.9,fce7cf8c3aa54b3650a30a79dce705838c32f3c3,layers=
    meta-virtualization,https://git.yoctoproject.org/meta-virtualization,scarthgap,9e040ee8dd6025558ea60ac9db60c41bfeddf221,layers=
    meta-openembedded,https://git.openembedded.org/meta-openembedded,scarthgap,e92d0173a80ea7592c866618ef5293203c50544c,layers=meta-networking:meta-python:meta-oe:meta-gnome:meta-filesystems:meta-multimedia
    meta-ti,https://git.yoctoproject.org/meta-ti,scarthgap,a98a2e1029870b25e117d4c9410e75bfd955f35a,layers=meta-ti-extras:meta-ti-bsp
    meta-arm,https://git.yoctoproject.org/meta-arm,scarthgap,8e0f8af90fefb03f08cd2228cde7a89902a6b37c,layers=meta-arm:meta-arm-toolchain
    meta-clang,https://github.com/kraj/meta-clang,scarthgap,eaa08939eaec9f620b14742ff3ac568553683034,layers=
    oe-core,https://git.openembedded.org/openembedded-core,scarthgap,b214cc84a922f7a3fb7ebbc501189ce25e8bd2bd,layers=meta
    meta-tisdk,https://git.ti.com/git/ti-sdk-linux/meta-tisdk.git,scarthgap,1f96dccc01f10c73295e0bac6a15a97d3d4562fd,layers=meta-ti-foundational
    meta-edgeai,https://git.ti.com/git/edgeai/meta-edgeai.git,scarthgap,81d1b80b2a314814c6d900e82ea2da93eee3f57d,layers=
    OECORELAYERCONF=./sample-files/bblayers.conf.sample
    OECORELOCALCONF=./sample-files/local-arago64-v2.conf.sample
    BITBAKE_INCLUSIVE_VARS=yes

    Specifically cors, express, and nodejs. I may just need to decrease the number of threads running at a time, however.

    Can you try building and testing the config on your end as well?

    Best,
    Jared

  • Hi Jared,

    I just started the build, though cors and express have given me issues before - typically fetching, not compiling. I'll let you know how it goes. I'm only allowing 8 threads so I can continue working otherwise on this system. 

    Derek

  • Hi Derek,

    Let me know how it goes.

    Best,
    Jared

  • Hi Jared,

    I've had trouble with cors and express as well, do_fetch seems to fail. 

    Derek

  • Hi Derek,

    I've fixed the fetcher issues. The issues and solution are the same as observed in the following thread: PROCESSOR-SDK-J721E: Fetch Error on express and cors modules during Yocto SDK 9.1 bitbake build  

    Unfortunately, the same behavior is still observed:

    root@j722s-evm:/opt/edgeai-gst-apps# ./apps_python/app_edgeai.py configs/object_detection.yaml 
    libtidl_onnxrt_EP loaded 0x2aae3b30 
    Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
    APP: Init ... !!!
       836.542174 s: MEM: Init ... !!!
       836.542444 s: MEM: Initialized DMA HEAP (fd=5) !!!
       836.542690 s: MEM: Init ... Done !!!
       836.542722 s: IPC: Init ... !!!
       836.598260 s: IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
       836.607368 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
       836.611694 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
       836.611757 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
       836.611768 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
       836.613349 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-0 
       836.613563 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-1 
       836.613684 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-2 
       836.613841 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:169] Added target MPU-3 
       836.613859 s:  VX_ZONE_INFO: [tivxInitLocal:202] Initialization Done !!!
       836.613884 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
       836.632287 s:  VX_ZONE_ERROR: [ownContextSendCmd:1001] Command ack message returned failure cmd_status: -1
       836.632336 s:  VX_ZONE_ERROR: [ownNodeKernelInit:704] Target kernel, TIVX_CMD_NODE_CREATE failed for node node_84
       836.632349 s:  VX_ZONE_ERROR: [ownNodeKernelInit:705] Please be sure the target callbacks have been registered for this core
       836.632361 s:  VX_ZONE_ERROR: [ownNodeKernelInit:706] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
       836.632376 s:  VX_ZONE_ERROR: [ownGraphNodeKernelInit:793] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
       836.632429 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Node kernel init failed
       836.632442 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Graph verify failed
    Traceback (most recent call last):
      File "/opt/edgeai-gst-apps/./apps_python/app_edgeai.py", line 67, in <module>
        main(sys.argv)
      File "/opt/edgeai-gst-apps/./apps_python/app_edgeai.py", line 46, in main
        demo = EdgeAIDemo(config)
               ^^^^^^^^^^^^^^^^^^
      File "/opt/edgeai-gst-apps/apps_python/edge_ai_class.py", line 108, in __init__
        model_obj.create_runtime()
      File "/usr/lib/python3.12/site-packages/edgeai_dl_inferer.py", line 315, in create_runtime
        self.run_time = RunTime(self.artifacts,
                        ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/site-packages/edgeai_dl_inferer.py", line 170, in __init__
        self.interpreter = _onnxruntime.InferenceSession(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 387, in __init__
        self._create_inference_session(providers, provider_options, disabled_optimizers)
      File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 439, in _create_inference_session
        sess.initialize_session(providers, provider_options, disabled_optimizers)
    onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Create state function failed. Return value:-1


    I've created a ticket (to fix the issue) and have asked for the specific config used to build the wic image on ti.com. 

    Best,
    Jared