• State
  • Locked Locked
  • Replies 3 replies
  • Subscribers 97 subscribers
  • Views 602 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Issue running with ONNX quantized model with TIDLExecutionProvider

Lucas FIscher
Prodigy 10 points

Hi,

I have an issue using the https://dev.ti.com/edgeaisession/ and the benchmarker.

I have developped a quantization method for neural networks using onnx and I wanted to test it with the TIDLExecutionProvider as I need to deploy it for one of my customer

I have used the custom-model-onnx and I uploaded the following model based on Mobilenet.

As soon as I try to load it with the inference engine of TI.

The jupyter kernel dies.

I have then tried with a non quantized model but with no more success, the jupyter dies as soon as I try to use the TIDLExecutionProvider

Don't really know how to move forward.

I can add the model for anyone to test but I am not sure how to do it 

Thanks

  • 0
    Prodigy 10 points

    For anyone who wants to try to DL the model, please take a look at the github issue : github.com/.../16

  • 0 in reply to Lucas FIscher
    TI__Genius 9065 points

    Hi Lucas,

    TIDL supports tflite quantized models but currently does not support onnx quantized models. As for the issue even with floating point model, thanks for sharing the model, will check for the issue and get back.

    In the meantime, can you run with "debug_level":1 set as part of compilation options and share the log?

    Regards,

    Anand

  • 0 in reply to Anand Pathak
    Prodigy 10 points

    Hi Anand,

    Thanks for the reply, I have multiple question for you:

    1 - I have done what you have asked by creating a logger

    from scripts.utils import loggerWritter

    log_dir = Path("logs").mkdir(parents=True, exist_ok=True)
    output_dir = 'logs/output_dir'
    with loggerWritter("logs/custon-model-onnx"):

    # model compilation options
    compile_options = {
    'tidl_tools_path' : os.environ['TIDL_TOOLS_PATH'],
    'artifacts_folder' : output_dir,
    'debug_level' : 1,
    }


    and infered it with the following code: 

    so = rt.SessionOptions()
    EP_list = ['TIDLCompilationProvider','CPUExecutionProvider']
    sess = rt.InferenceSession(onnx_model_path ,providers=EP_list, provider_options=[compile_options, {}], sess_options=so)

    input_details = sess.get_inputs()

    I went to the artifact folder and here is the ouput in the file

    custon-model-onnx_er.log I get the following text 

    ''' terminate called after throwing an instance of 'std::exception'
    what(): std::exception '''

    and in 

    custon-model-onnx_out.log, I get the following text:

    """I

    Internal Error: do_recv() expected MSG_ID 5005, got 0!

    Stack trace:
    [bt] (0) /usr/lib/libti_inference_client.so(StackTrace[abi:cxx11](unsigned long, unsigned long)+0x1ed) [0x7fb507fa005d]
    [bt] (1) /usr/lib/libti_inference_client.so(send_once(unsigned int, void*, int, progressbar*, progressbar*)+0x838) [0x7fb507f9fac8]
    [bt] (2) /usr/local/lib/python3.6/dist-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x8074c) [0x7fb50292774c]
    [bt] (3) /usr/local/lib/python3.6/dist-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x80aa9) [0x7fb502927aa9]
    [bt] (4) /usr/local/lib/python3.6/dist-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x880ef) [0x7fb50292f0ef]
    [bt] (5) /usr/local/lib/python3.6/dist-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0xa5a67) [0x7fb50294ca67]
    [bt] (6) /usr/local/lib/python3.6/dist-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x74433) [0x7fb50291b433]
    [bt] (7) /usr/bin/python3(_PyCFunction_FastCallDict+0x35c) [0x5675fc]
    [bt] (8) /usr/bin/python3() [0x5953c1]
    [bt] (9) /usr/bin/python3() [0x54a545]
    [bt] (10) /usr/bin/python3() [0x551f81]
    [bt] (11) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(+0x3d8e3) [0x7fb5168dc8e3]
    [bt] (12) /usr/bin/python3(_PyObject_FastCallKeywords+0x19c) [0x5aa2ac]
    [bt] (13) /usr/bin/python3() [0x50a7f3]
    [bt] (14) /usr/bin/python3(_PyEval_EvalFrameDefault+0x444) [0x50c274]
    [bt] (15) /usr/bin/python3() [0x509989]

    """ 

    Don't really know what is happening and of course the inference is killing the notebook session

    2-.

    As show in this picture, it seems that you can suport quantized model right in ONNX ? I can generate quantized model with QDQ or QlinearOPS with ease but I am not sure I understand your answer compared to the image in the notebook

    Thanks a lot for your help