• State Resolved
  • Locked Locked
  • Replies 4 replies
  • Subscribers 97 subscribers
  • Views 566 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: TIDL inference fails in 8.5 but works in 8.4 for some TFLITE models

Christophe ALLART
Prodigy 135 points
Part Number: TDA4VM

Hello,

I have the TDA4VM board and use the  ti-processor-sdk-rtos-j721e-evm  version 08_05   under Ubuntu 18.04.

I am using a Python script to import and run inference using OSRT ( TFLITE ) API .

On several CNN models (TFLITE ) , i got an error during inference as follows :

 ------- config file  ---------
{'model_path': '../SHARE/ydet/ydet_reduced_int8.tflite', 'artifact_folder': './ARTIF/artif_YDET', 'in_range_min': 0.0, 'in_range_max': 1.0, 'calib_nbTensors': 1, 'calib_accuy': 0, 'calib_nbIterations': 1, 'calib_tensorBits': 8, 'calib_debug_level': 0, 'calib_deny_list': ' 114 , 6 ', 'calib_high_resolution_optimization': 0, 'random_mode': 0, 'inence_debug_level': 0, 'images_folder': '', 'image_inf': '', 'image_inf_constval': 0.5, 'model_type': 1}
 ----------------
 running inference on TIDL , and on OSRT
INFERENCE   tidl_tools_path=/home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools , artifact_folder=./ARTIF/artif_YDET , model=../SHARE/ydet/ydet_reduced_int8.tflite
run inference  ... use_tidl backend =  True

 Number of subgraphs:1 , 73 nodes delegated out of 77 nodes

The soft limit is 2048
The hard limit is 2048
MEM: Init ... !!!
MEM: Init ... Done !!!
 0.0s:  VX_ZONE_INIT:Enabled
 0.11s:  VX_ZONE_ERROR:Enabled
 0.15s:  VX_ZONE_WARNING:Enabled
 0.2134s:  VX_ZONE_INIT:[tivxInit:184] Initialization Done !!!
 0.7465s:  VX_ZONE_ERROR:[tivxAlgiVisionCreate:332] Calling ialg.algInit failed with status = -1120
Segmentation fault (core dumped)

This error happens on SDK version 08_05  but not on version 08_04.

I checked in TIDL source code and found  a problem related to memory allocation that fails during init ( in  tiovx/kernels/ivision/common/tivx_alg_vision.c  )
The crash itself is probably simply a bug after error detection.

Here is the backtrace in GDB :

Thread 7 "python3" received signal SIGSEGV, Segmentation fault.
	[Switching to Thread 0x7fffea8f9700 (LWP 13696)]
	0x00007fffc0fd3ea4 in TIDL_removePriorityObject(void*, IALG_MemRec*) () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	(gdb) bt
	#0  0x00007fffc0fd3ea4 in TIDL_removePriorityObject(void*, IALG_MemRec*) () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#1  0x00007fffc0eed99a in tivxAlgiVisionDeleteAlg () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#2  0x00007fffc0eeddcf in tivxAlgiVisionCreate () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#3  0x00007fffc0eece6d in tivxKernelTIDLCreate () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#4  0x00007fffc0ee9a80 in ownTargetKernelCreate () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#5  0x00007fffc0ee1715 in ownTargetNodeDescNodeCreate () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#6  0x00007fffc0ee3154 in ownTargetTaskMain () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#7  0x00007fffc0eea558 in tivxTaskMain () from /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools/libvx_tidl_rt.so
	#8  0x00007ffff77ca6db in start_thread (arg=0x7fffea8f9700) at pthread_create.c:463
	#9  0x00007ffff7b0361f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

 

As said, i have this error on some models ( pre-quantized TFLITE models ).

Thanks for your help.  

  • 0
    TI__Genius 9065 points

    Hi,

    Can you please confirm what value you are setting for "quantization_scale_type" compilation option? Also, can you confirm compilation has happened successfully without any error, please share the compilation log as well.

    Regards,

    Anand

  • 0 in reply to Anand Pathak
    Prodigy 135 points

    Hi,

    advanced_options:quantization_scale_type = 0    ( non power of 2 )
    Here is the log of import 8 bits (import with a single input tensor with random values ), which shows no error.
    
tidl_tools_path                                 = /home/ti-proc-rtos-8-5/tidl_j721e_08_05_00_16/tidl_tools
artifacts_folder                                = ./ARTIF/artif_YDET
tidl_tensor_bits                                = 8
debug_level                                     = 0
num_tidl_subgraphs                              = 16
tidl_denylist                                   = 114   6
tidl_denylist_layer_name                        =
tidl_denylist_layer_type                         =
model_type                                      =
tidl_calibration_accuracy_level                 = 64
tidl_calibration_options:num_frames_calibration = 1
tidl_calibration_options:bias_calibration_iterations = 1
mixed_precision_factor = -1.000000
model_group_id = 0
power_of_2_quantization                         = 2
enable_high_resolution_optimization             = 0
pre_batchnorm_fold                              = 1
add_data_convert_ops                          = 0
output_feature_16bit_names_list                 =
m_params_16bit_names_list                       =
reserved_compile_constraints_flag               = 1601
ti_internal_reserved_1                          =

 Number of subgraphs:1 , 73 nodes delegated out of 77 nodes

WARNING : Pad layer won't be merged in the succeeding layer, it will be treated as a stand alone layer
WARNING : Pad layer won't be merged in the succeeding layer, it will be treated as a stand alone layer
WARNING : Pad layer won't be merged in the succeeding layer, it will be treated as a stand alone layer
WARNING : Pad layer won't be merged in the succeeding layer, it will be treated as a stand alone layer
WARNING : Pad layer won't be merged in the succeeding layer, it will be treated as a stand alone layer

 ************** Frame index 1 : Running float import *************
IMPORT nb operations : Total Giga Macs : 5.2293   , network file ./ARTIF/artif_YDET/tempDir/136_tidl_net.bin_netLog.txt
INFORMATION: [TIDL_ResizeLayer] model/lambda/resize/ResizeNearestNeighbor Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] model/lambda_1/resize/ResizeNearestNeighbor Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
WARNING: [TIDL_E_DATAFLOW_INFO_NULL] Network compiler returned with error or didn't executed, this model can only be used on PC/Host emulation mode, it is not expected to work on target/EVM.
****************************************************
**          3 WARNINGS          0 ERRORS          **
****************************************************
The soft limit is 2048
The hard limit is 2048
MEM: Init ... !!!
MEM: Init ... Done !!!
 0.0s:  VX_ZONE_INIT:Enabled
 0.11s:  VX_ZONE_ERROR:Enabled
 0.15s:  VX_ZONE_WARNING:Enabled
 0.1689s:  VX_ZONE_INIT:[tivxInit:184] Initialization Done !!!
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
------debug ---------
TENSOR details : input  name=inputs:0   index=137  shape=[  1 512 608   3]  type=<class 'numpy.float32'>  nb_of_elem=933888
TENSOR details : output  name=Identity_2:0   index=138  shape=[ 1 64 76  5]  type=<class 'numpy.float32'>  nb_of_elem=24320
TENSOR details : output  name=Identity_1:0   index=139  shape=[ 1 32 38  5]  type=<class 'numpy.float32'>  nb_of_elem=6080
TENSOR details : output  name=Identity:0   index=140  shape=[ 1 16 19  5]  type=<class 'numpy.float32'>  nb_of_elem=1520
TENSOR=input calibration tensor 0   size=933888 shape=(1, 512, 608, 3) min=0.000 max=1.000  :   0.1171 0.9406 0.0156 0.6484 0.5904  ... 0.1409 0.2720  ... 0.7644 0.8286 0.7213 0.8195

 ************ Frame index 1 : Running fixed point mode for calibration ****************
IMPORT nb operations : Total Giga Macs : 5.2293   , network file ./ARTIF/artif_YDET/tempDir/136_tidl_net.bin_netLog.txt
IMPORT nb operations : Total Giga Macs : 5.2293   , network file ./ARTIF/artif_YDET/tempDir/136_tidl_net.bin_netLog.txt

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/calll81z/jacinto/ARTIF/artif_YDET/tempDir/136_tidl_io_.qunat_stats_config.txt
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   15607.59  .... ..... ... .... .....IMPORT nb operations : Total Giga Macs : 5.2293   , network file ./ARTIF/artif_YDET/tempDir/136_tidl_net.bin_netLog.txt

~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~

Processing config file #0 : /home/calll81z/jacinto/ARTIF/artif_YDET/tempDir/136_tidl_io_.qunat_stats_config.txt
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . .. T   16733.15  .... ..... ... .... .....


 *****************   Calibration iteration number 0 completed ************************



IMPORT nb operations : Total Giga Macs : 5.2293   , network file ./ARTIF/artif_YDET/tempDir/136_tidl_net.bin_netLog.txt

------------------ Network Compiler Traces -----------------------------
successful Memory allocation
Rerunning network compiler

------------------ Network Compiler Traces -----------------------------
successful Memory allocation
INFORMATION: [TIDL_ResizeLayer] model/lambda/resize/ResizeNearestNeighbor Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
INFORMATION: [TIDL_ResizeLayer] model/lambda_1/resize/ResizeNearestNeighbor Any resize ratio which is power of 2 and greater than 4 will be placed by combination of 4x4 resize layer and 2x2 resize layer. For example a 8x8 resize will be replaced by 4x4 resize followed by 2x2 resize.
****************************************************
**          2 WARNINGS          0 ERRORS          **
****************************************************
 import done ... copying model  ../SHARE/ydet/ydet_reduced_int8.tflite  to artifact dir ./ARTIF/artif_YDET

 end import
IMPORT num_of_calib_tensors=1  accuracy_level=0  calibration_iterations=1


MEM: Deinit ... !!!
MEM: Alloc's: 29 alloc's of 339652644 bytes
MEM: Free's : 29 free's  of 339652644 bytes
MEM: Open's : 0 allocs  of 0 bytes
MEM: Deinit ... Done !!!
... end tool ...
    Regards,
    Christophe
  • 0 in reply to Christophe ALLART
    Prodigy 135 points

    Hello,

    I checked again on these models with the new  SDK  08_06  version.

    Now the import goes well .

    So we can close this ticket.

    Thanks.