This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: Runtime error with TFLite model

Part Number: TDA4VM

Hello, 

I am trying to load a model with OSRT(TFLite). However, this model throws a segmentation fault during compilation (see issue).

To test, I simplified the model and used TF MNIST training example and converted it to TFLite model. 

I tried the following models:

- Normal training and conversion to TFLite from saved model format(model.tflite)

- Normal training and conversion to TFLite float16 model with quantization without calibration dataset using TFLite options(model_16.tflite)

- Quantization Aware training (Model name: model_8.tflite)

I have uploaded all the models here: https://drive.google.com/drive/u/0/folders/1DRigofV7T-YCg9NkhLxVfNFT786DlxBH

All the models result in error when trying to compile(error log)

Traceback (most recent call last):
  File "/home/test.py", line 54, in <module>
    'tidl_model_import_tflite.so'), compile_options)])
  File "/usr/local/lib/python3.6/dist-packages/tflite_runtime/interpreter.py", line 497, in __init__
    delegate._get_native_delegate_pointer())  # pylint: disable=protected-access
RuntimeError

  • Hi Ashay, thanks for sharing models, I will take a look. One option, which could give you more information about the error is to convert your jupyter notebook to a python script and run it from your cloud's workspace terminal.

    An example below

    in terminal you can type bash 

    user@1c26caa7764f:/home/root/notebooks$ jupyter nbconvert --to script custom-model-onnx-yolox.ipynb
    [NbConvertApp] Converting notebook custom-model-onnx-yolox.ipynb to script
    [NbConvertApp] Writing 7765 bytes to custom-model-onnx-yolox.py
    user@1c26caa7764f:/home/root/notebooks$ python3 custom-model-yolox.py

    thank you,

    Paula

  • Hi Paula, thanks for the tip, I had tried it in form of Python script as well. However it did not provide me any extra information.

    Looking forward to hear your feedback on the models.

    Thanks

    Ashay

  • Hi Ashay, apologize for my late reply. Thanks a lot for creating these small models. However, they don't seem to be correct models. 

    I tried to run model,tflite and model_tflite using edgeai-tidl-tools python's scripts for easier debugging, but not even by disabling offload they work. Attached log just for your reference. Could you please help me to fix them?, so they can run in CPU mode (w/o offloading)?. After that, as they are small test models we can try using "allow_list" and see which layer(s) is (are) causing issues.

    ~/Desktop/edgeai-tidl-tools/examples/osrt_python/tfl$ python3 tflrt_delegate.py -c -d
    Running 1 Models - ['model_8']
    
    
    Running_Model :  model_8
    Traceback (most recent call last):
      File "tflrt_delegate.py", line 271, in <module>
        run_model(model, mIdx)
      File "tflrt_delegate.py", line 193, in run_model
        imgs, output, proc_time, sub_graph_time, ddr_write, ddr_read, new_height, new_width  = infer_image(interpreter, input_images, config)
      File "tflrt_delegate.py", line 90, in infer_image
        channel = input_details[0]['shape'][3]
    IndexError: index 3 is out of bounds for axis 0 with size 3
    ~/Desktop/edgeai-tidl-tools/examples/osrt_python/tfl$ python3 tflrt_delegate.py -c -d
    Running 1 Models - ['model']
    
    
    Running_Model :  model
    Traceback (most recent call last):
      File "tflrt_delegate.py", line 271, in <module>
        run_model(model, mIdx)
      File "tflrt_delegate.py", line 193, in run_model
        imgs, output, proc_time, sub_graph_time, ddr_write, ddr_read, new_height, new_width  = infer_image(interpreter, input_images, config)
      File "tflrt_delegate.py", line 90, in infer_image
        channel = input_details[0]['shape'][3]
    IndexError: index 3 is out of bounds for axis 0 with size 3
    ~/Desktop/edgeai-tidl-tools/examples/osrt_python/tfl$ 
    

    thank you,

    Paula

  • Hi Paula, sure I will have a look at them today and fix them. Thanks for your response.

    Best

    Ashay

  • Hi Paula, I think I know why it did not run for you. The model's input is (1,28,28). There is no channel axis for this model and hence you encounter the error.

    Without offloading or using acceleration delegates I used the following code:

    import tflite_runtime.interpreter as tflite
    import os
    
    compile_options = {"tidl_tools_path":os.environ["TIDL_TOOLS_PATH"],
                       "artifacts_folder":"/home/tda4/output",
                       "tensor_bits":8,
                       "accuracy_level": 0,
                       "debug_level":1
    }
    
    shared_ob_path = os.path.join(os.environ["TIDL_TOOLS_PATH"], 
                                  "tidl_model_import_tflite.so")
    
    tidl_delegate = [tflite.load_delegate(shared_ob_path, compile_options)]
    
    print("Interpreter Created")
    
    interpreter = tflite.Interpreter(model_path="/home/tda4/model_8.tflite")
    interpreter.allocate_tensors()
    
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()
    print(input_details)
    print(output_details)

    and got the following output:

    root@46b1a2294618:/home/tda4# python3 /home/tda4/test_tflite_model.py
    tidl_tools_path                                 = /home/root/tidl_tools 
    artifacts_folder                                = /home/tda4/output 
    tidl_tensor_bits                                = 8 
    debug_level                                     = 1 
    num_tidl_subgraphs                              = 16 
    tidl_denylist                                   = 
    tidl_denylist_layer_name                        = 
    tidl_denylist_layer_type                         = 
    tidl_allowlist_layer_name                        = 
    model_type                                      =  
    tidl_calibration_accuracy_level                 = 64 
    tidl_calibration_options:num_frames_calibration = 20 
    tidl_calibration_options:bias_calibration_iterations = 50 
    mixed_precision_factor = -1.000000 
    model_group_id = 0 
    power_of_2_quantization                         = 2 
    enable_high_resolution_optimization             = 0 
    pre_batchnorm_fold                              = 1 
    add_data_convert_ops                          = 0 
    output_feature_16bit_names_list                 =  
    m_params_16bit_names_list                       =  
    reserved_compile_constraints_flag               = 1601 
    ti_internal_reserved_1                          = 
    Interpreter Created
    INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
    [{'name': 'serving_default_input_1:0', 'index': 0, 'shape': array([ 1, 28, 28], dtype=int32), 'shape_signature': array([-1, 28, 28], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
    [{'name': 'StatefulPartitionedCall:0', 'index': 19, 'shape': array([ 1, 10], dtype=int32), 'shape_signature': array([-1, 10], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]

    however, adding the delegate I run into RuntimeError.

    If needed I can create toy example with 3 channels.

    Best

    Ashay

  • Hi Ashay, I tried previous shared model with some script modifications but running into some issues. So, instead of debugging unrelated issues please help me to create a 3 channels toy example.

    thank you,

    Paula

  • Sure, I will create them and let you know, once they have been uploaded.

    Best

    Ashay

  • HI Ashay, thanks for sharing models. From the models shared all except one (saved_model_fp16.tflite) compiled ok. Let me share console logs. I compile them and run host emulation inference using edgeai-tidl-tools

    I will check saved_model_fp16.tflite segmentation fault later and come back to you. In any case, other versions of the models seem to be OK.

    8081.logs.zip

    thank you,

    Paula

  • Thanks for providing these. Is there a possibility that the models when trained with quantization aware training(Tensorflow) and then converted to TFLite, we avoid the calibration within edge ai tidl framework. What compile options should be used to do this?

    Also, were you able to find the reason why fp16 model did not work?

    Thank You!

    Best

    Ashay

  • Hi Ashay, in order to bypass TIDL's PTQ you can set "advanced_options:quantization_scale_type = 3"

    https://github.com/TexasInstruments/edgeai-tidl-tools/tree/master/examples/osrt_python

    Thak you,

    Paula