This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

SK-TDA4VM: Executing yolox model on TI SOC failed.

Part Number: SK-TDA4VM

Tool/software:

Hello,

I am trying to perform inference on a TI SoC using the OSRT approach with a YOLOX model. I have used a model from yolox-ti-edgeai, and I have also tried models from the model zoo. I could compile the ONNX models using edgeai-tidl-tools on my host PC. However, when I execute the models on the target, I got the following error:

2024-06-11 16:15:49.917614036 [E:onnxruntime:, sequential_executor.cc:494 ExecuteKernel] Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: /onnx/onnxruntime/onnxruntime/core/framework/execution_frame.cc:170 onnxruntime::common::Status onnxruntime::IExecutionFrame::GetOrCreateNodeOutputMLValue(int, int, const onnxruntime::TensorShape*, OrtValue*&, const onnxruntime::Node&) shape && tensor.Shape() == *shape was false. OrtValue shape verification failed. Current shape:{1,3549,6} Requested shape:{1,1,1,1,3549,6}


TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
************ TIDL_subgraphRtCreate done ************
 [ERROR] Unable to run session: Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: /onnx/onnxruntime/onnxruntime/core/framework/execution_frame.cc:170 onnxruntime::common::Status onnxruntime::IExecutionFrame::GetOrCreateNodeOutputMLValue(int, int, const onnxruntime::TensorShape*, OrtValue*&, const onnxruntime::Node&) shape && tensor.Shape() == *shape was false. OrtValue shape verification failed. Current shape:{1,3549,6} Requested shape:{1,1,1,1,3549,6}

Could someone help by mentioning the potential issues or steps to troubleshoot this problem?

Thanks

  • Hi,

    Thank you for the question. Could you provide the following information so we could debug further?

    Could you clarify what version of edgeai-tidl-tools you are seeing this behavior?

    I have used a model from yolox-ti-edgeai, and I have also tried models from the model zoo

    From my understanding you are running models that we offer out of the box, from edgeai-modelzoo and edgeai-yolox which are contained in edgeai-tensorlab (https://github.com/TexasInstruments/edgeai-tensorlab/tree/main). Can you specify what models you are seeing this issue with?

    Best,

    Asha

  • Hi,

    Thanks for the response. The version of edge-tidl-tools is 09_02_06_00. The model is own yolox model from https://github.com/TexasInstruments/edgeai-yolox. 

  • Hi Abdulghani,

    I do apologize for the delay in getting back to you. Is this still a blocking issue for you currently? 

    When you mean own yolex model, could you specify more? Is this one of the pretrained models (if so can you specify the name)?

    From my understanding, the inference on target is failing. Have you checked the inference on host emulation (running on x86 PC)? Does this result in any errors?

    Can you provide the compilation logs and the inference logs with debug_level = 2 as a txt file for further debug?

    Best,

    Asha

  • Hello,

    • Actually I got the same error when trying to compile the model edgeai-yolox-nano-ti-lite-onnx from model zoo and integrate it to the edge_ai_apps on beagleboard-ai64.
    • I did not try doing inference on host pc.

    Note:

    - I have used edgeai-tidl-tools (with another version as mentioned above) and edgeai-benchmark to compile the model this time.
    - The model compile successfully but it does not execute on target with the example. (The example could run but no bounding boxes appear on screen)

    thanks.

    Fullscreen
    1
    2
    3
    4
    5
    6
    7
    8
    33668.604213 s: VX_ZONE_ERROR:[ownContextSendCmd:815] Command ack message returned failure cmd_status: -1
    33668.604464 s: VX_ZONE_ERROR:[ownContextSendCmd:851] tivxEventWait() failed.
    33668.604536 s: VX_ZONE_ERROR:[ownNodeKernelInit:538] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
    33668.604597 s: VX_ZONE_ERROR:[ownNodeKernelInit:539] Please be sure the target callbacks have been registered for this core
    33668.604656 s: VX_ZONE_ERROR:[ownNodeKernelInit:540] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
    33668.604719 s: VX_ZONE_ERROR:[ownGraphNodeKernelInit:583] kernel init for node 0, kernel com.ti.tidl ... failed !!!
    33668.604797 s: VX_ZONE_ERROR:[vxVerifyGraph:2055] Node kernel init failed
    33668.604868 s: VX_ZONE_ERROR:[vxVerifyGraph:2109] Graph verify failed
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • hello,

    Also when compiling the model I got the warning:

    Fullscreen
    1
    WARNING: [TIDL_E_DATAFLOW_INFO_NULL] ti_cnnperfsim.out fails to allocate memory in MSMC. Please look into perfsim log. This model can only be used on PC emulation, it will get fault on target.
    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  • Hi,

    Copying your question from the other thread:

    And the model cannot run on target with the edge_ai_apps. According to this post TDA4VM: TIDL ONNX import error with 32-bit numParamBits the problem is solved by setting the parameter numParamBits = 32.

    Where can I set this parameter in the edgeai-tidl-tools (version 08_02_00_05) 

    Running on EVM does not support 32-bit. You can use the tensor_bits parameter in edgeai-tidl-tools to modify this (either 8 or 16 bit to run on target). Also, it is recommended to run on host emulation first to determine functionality.

    Best,

    Asha

  • Hello
    thanks for response. The tensor_bit parameter is set already to 8 by default. Despite the warning I could run the model on target.
    The problem was solved by setting the num_class of coco dataset from 80 to 90.

  • Hi Abdulghani,

    Good to know that you were able to resolve the issue. Can I close this ticket?

    Best,

    Asha

  • Hello,

    When we could consider the warning as obsolete, then yes.

    Thanks again for help Slight smile