• State
  • Locked Locked
  • Replies 3 replies
  • Subscribers 101 subscribers
  • Views 181 views
  • Users 0 members are here
Support feedback
Options
Options
Similar topics

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM62A3: AM62A: [edgeai-tidl-tools] Execution of Jupyter examples gives a segmentation fault.

Stefan Werner
Prodigy 30 points
Part Number: AM62A3

Tool/software:

I am trying to run the Jupyter notebook examples in the edgeai-tidl-tools repository but have encountered segmentation faults when running them.

The segmentation faults are thrown once the InferenceSession is created for the models (in the onnx examples). I am able to run the examples provided under examples/osrt_python, so the edgeai-tidl-tools should be installed. Ive used the docker based installation (SOC=am62a) and have followed the outlined Jupyter notebook setup instructions. I have tried with the latest SDK version as well as one that is compatible with my devboard (version 9.0.0). 

Any suggestions? 

  • 0
    TI__Genius 11106 points

    Hi Stefan,

    Could you mention which specific jupyter notebook is giving you this issue and which model? I can try to reproduce on my side

    • There is a set of delegate_options (a python dictionary) passed to the InferenceSession during initialization. If you include debug_level=2 (with correct Py syntax) in this dict, it will increase the verbosity.

    Is this during compilation / import or during inference?

    • If it's on the dev board, then I assume inference. Import/compile is only for x86
    • You can tell which one by looking at the ExecutionProvider (for ONNX) or Delegate (for TFlite). If  'Compilation' is in the name, then it's trying to invoke the x86-compatible import tool


      Stefan Werner said:
      I have tried with the latest SDK version as well as one that is compatible with my devboard (version 9.0.0). 

      Are you using the AM62A-SK-LP board (the official EVM from TI site) or a different development board? As noted in our previous thread, the compiled models ("artifacts" in our jargon) are tied to an SDK version. The most recent 10.0 release has made improvements on logging and verbosity. If it's possible to reflash the most recent SDK, I would recommend so. If you are using a 3rd party SOM or dev board, then please stick with their provided BSP/SDK

      BR,
      Reese

  • 0 in reply to Reese Grimsley
    Prodigy 30 points

    Hi ,

    Ive just gotten back to looking at this issue. I have issues with multiple notebooks, but I am looking at "custom-model-onnx" now. As soon as I create the onnx inference session (sess = rt.InferenceSession(onnx_model_path ,providers=EP_list, provider_options=[compile_options, {}], sess_options=so) ) the kernel just dies. Increasing the debugging level doesnt help, since the kernel dies before any outputs are logged. This is during the compilation workflow.

    The osrt_python/ort example doesnt work for me as well (although the kernel doesnt die right away). The tflite scripts worked without issues.

    I am using a AM62A-SK-LP board. Right now, I need to stick to the 9.0.0 version. I would greatly appreciate some help here.

    BR,

    Stefan

  • 0 in reply to Stefan Werner
    TI__Genius 11106 points

    Hi Stefan,

    Stefan Werner said:
    Increasing the debugging level doesnt help, since the kernel dies before any outputs are logged. This is during the compilation workflow.

    This is a weakness of Jupyter, I believe -- it doesn't handle segfaults or similar lower level failures well. Debugging a model in this workflow is rather challenging, which is why I recommend the explicit python scripts for this. 

    Stefan Werner said:
    The osrt_python/ort example doesnt work for me as well (although the kernel doesnt die right away

    I can help more with this -- can you post a log of this failure? I assume this model is a custom architecture. If you're willing to, please post the model file (random weights are fine; the weights don't matter much for import, so best to avoid any IP leak)

    Stefan Werner said:

    I am using a AM62A-SK-LP board. Right now, I need to stick to the 9.0.0 version. I would greatly appreciate some help here.

    9.0.0, got it. Some of the debugging logs are not as helpful as the more recent SW release, although SDK version is tightly linked to the TIDL tools version. I understand if it's not acceptable to upgrade.