TDA4VM: DataConvert Layer in TIDL subgraph cost too much time in inference

Yongchao Ding

Part Number: TDA4VM

I have a model with user-define-op such as pointpillar network. and my input is (150000, 5), and the type of model input and output is float, Then the model will split into 3part, The first one is user-define-op, the Second one is Transpose, the last one is conv+conv which transposed to tidl subgraph as follows.

Then my problem is in tidl subgraph, I using debugTraceLevel=3 to print the Layer cycle of the subraph. And I found that the first layer and last layer DataConverter Cost a lot of time, nearly 22ms, and the 2 conv Layer cost nearly 1.4ms.

So I am Confused if it is normal? and I want to know the detail operations in DataConverter, if there just has quantize and dequantize operation?

[C7x_1 ] 171901.116484 s: Network Cycles 24430318
[C7x_1 ] 171901.116584 s:  Layer,   Layer Cycles,kernelOnlyCycles, coreLoopCycles,LayerSetupCycles,dmaPipeupCycles, dmaPipeDownCycles, PrefetchCycles,copyKerCoeffCycles,LayerDeinitCycles,LastBlockCycles, paddingTrigger,    paddingWait,LayerWithoutPad,LayerHandleCopy,   BackupCycles,  RestoreCycles,
[C7x_1 ] 171901.116708 s:      1,        8470622,        8411329,        8424861,           2036,          13546,                19,              0,                 0,            366,         350254,            572,            231,        8447244,              0,              0,              0,
[C7x_1 ] 171901.116830 s:      2,         724793,         502190,         511665,         168414,            904,             16985,              0,                 0,            515,          17164,            362,             30,         704798,           2367,              0,              0,
[C7x_1 ] 171901.116952 s:      3,         686062,         166335,         483109,         165724,          13698,                12,              0,                 0,            409,           2864,            258,             30,         666797,           2165,              0,              0,
[C7x_1 ] 171901.117074 s:      4,       14310284,       14190972,       14259900,           2617,           1633,             19678,              0,                 0,            356,         147836,            555,            196,       14289742,              0,              0,              0,
[C7x_1 ] 171901.117121 s:  Sum of Layer Cycles 24191761

model_svg.zip

over 3 years ago

0 Anshu Jain over 3 years ago

TI__Guru 56820 points

Hi,

Can you confirm whether you got any warning while compiling the model. A warning like below :

"Warning : Optimized implementation for <Layer number> layer is not supported"

Regards,

Anshu

0 Yongchao Ding over 3 years ago in reply to Anshu Jain

Prodigy 40 points

Our model has user-defined op, such as PrepareBevCoordsAndFeatures, But The part of model in blue box is support by TIDL, So those will be converted into one subgraph without Warning. The Log File is as follows

Fullscreen convert.log Download

RelayImportDebug: In TIDL_relayAllowNode: RelayImportDebug: name: nn.conv2d RelayImportDebug: In TIDL_relayAllowNode: RelayImportDebug: name: nn.batch_norm RelayImportDebug: In TIDL_relayAllowNode: RelayImportDebug: name: nn.conv2d RelayImportDebug: In TIDL_relayAllowNode: RelayImportDebug: name: nn.batch_norm RelayImportDebug: In TIDL_relayImportInit subgraph_id=0 RelayImportDebug: in[0]: scale=1.000000 zp=0 t=6 n=1 c=8 h=150000 w=1 nchw=1 RelayImportDebug: out[0]: scale=1.000000 zp=0 t=6 nchw=1 RelayImportDebug: Layer 0, subgraph input 0, name=tidl_0_i0 RelayImportDebug: In TIDL_relayImportNode: RelayImportDebug: op name: nn.conv2d, num_args: 2 RelayImportDebug: args[0] dims: [1, 8, 150000, 1] RelayImportDebug: args[1] dims: [48, 8, 1, 1] RelayImportDebug: zp_len=1, scale_len=1 RelayImportDebug: zp[0]=0, scale[0]=1.000000 RelayImportDebug: In TIDL_relayImportLinkNode: RelayImportDebug: Layer 1, name: tidl_0_2, num_ins: 1, num_outs: 1 RelayImportDebug: in: tidl_0_i0 RelayImportDebug: out: tidl_0_7 RelayImportDebug: Layer 1, input 0 name tidl_0_i0 RelayImportDebug: In TIDL_relayImportNode: RelayImportDebug: op name: nn.batch_norm, num_args: 5 RelayImportDebug: args[0] dims: [1, 48, 150000, 1] RelayImportDebug: args[1] dims: [48] RelayImportDebug: args[2] dims: [48] RelayImportDebug: args[3] dims: [48] RelayImportDebug: args[4] dims: [48] RelayImportDebug: zp_len=1, scale_len=1 RelayImportDebug: zp[0]=0, scale[0]=1.000000 RelayImportDebug: In TIDL_relayImportLinkNode: RelayImportDebug: Layer 2, name: tidl_0_7, num_ins: 1, num_outs: 1 RelayImportDebug: in: tidl_0_2 RelayImportDebug: out: tidl_0_9 RelayImportDebug: Layer 2, input 0 name tidl_0_2 RelayImportDebug: In TIDL_relayImportNode: RelayImportDebug: op name: nn.relu, num_args: 1 RelayImportDebug: args[0] dims: [1, 48, 150000, 1] RelayImportDebug: zp_len=1, scale_len=1 RelayImportDebug: zp[0]=0, scale[0]=1.000000 RelayImportDebug: In TIDL_relayImportLinkNode: RelayImportDebug: Layer 3, name: tidl_0_9, num_ins: 1, num_outs: 1 RelayImportDebug: in: tidl_0_7 RelayImportDebug: out: tidl_0_11 RelayImportDebug: Layer 3, input 0 name tidl_0_7 RelayImportDebug: In TIDL_relayImportNode: RelayImportDebug: op name: nn.conv2d, num_args: 2 RelayImportDebug: args[0] dims: [1, 48, 150000, 1] RelayImportDebug: args[1] dims: [48, 48, 1, 1] RelayImportDebug: zp_len=1, scale_len=1 RelayImportDebug: zp[0]=0, scale[0]=1.000000 RelayImportDebug: In TIDL_relayImportLinkNode: RelayImportDebug: Layer 4, name: tidl_0_11, num_ins: 1, num_outs: 1 RelayImportDebug: in: tidl_0_9 RelayImportDebug: out: tidl_0_16 RelayImportDebug: Layer 4, input 0 name tidl_0_9 RelayImportDebug: In TIDL_relayImportNode: RelayImportDebug: op name: nn.batch_norm, num_args: 5 RelayImportDebug: args[0] dims: [1, 48, 150000, 1] RelayImportDebug: args[1] dims: [48] RelayImportDebug: args[2] dims: [48] RelayImportDebug: args[3] dims: [48] RelayImportDebug: args[4] dims: [48] RelayImportDebug: zp_len=1, scale_len=1 RelayImportDebug: zp[0]=0, scale[0]=1.000000 RelayImportDebug: In TIDL_relayImportLinkNode: RelayImportDebug: Layer 5, name: tidl_0_16, num_ins: 1, num_outs: 1 RelayImportDebug: in: tidl_0_11 RelayImportDebug: out: tidl_0_18 RelayImportDebug: Layer 5, input 0 name tidl_0_11 RelayImportDebug: In TIDL_relayImportNode: RelayImportDebug: op name: nn.relu, num_args: 1 RelayImportDebug: args[0] dims: [1, 48, 150000, 1] RelayImportDebug: zp_len=1, scale_len=1 RelayImportDebug: zp[0]=0, scale[0]=1.000000 RelayImportDebug: In TIDL_relayImportLinkNode: RelayImportDebug: Layer 6, name: tidl_0_o0, num_ins: 1, num_outs: 0 RelayImportDebug: in: tidl_0_16 RelayImportDebug: Layer 6, input 0 name tidl_0_16 RelayImportDebug: Layer 6, single output node of subgraph RelayImportDebug: In TIDL_relayOptimizeNet: RelayImportDebug: AddSingleOutDataLayer for layer 6 Running tidl_optimizeNet printing Current net 0|TIDL_DataLayer | |tidl_0_i0 | 0| 0| 1|TIDL_ConvolutionLayer |tidl_0_i0 |tidl_0_2 | 0| 1| 2|TIDL_BatchNormLayer |tidl_0_2 |tidl_0_7 | 1| 2| 3|TIDL_ReLULayer |tidl_0_7 |tidl_0_9 | 2| 3| 4|TIDL_ConvolutionLayer |tidl_0_9 |tidl_0_11 | 3| 4| 5|TIDL_BatchNormLayer |tidl_0_11 |tidl_0_16 | 4| 5| 6|TIDL_ReLULayer |tidl_0_16 |tidl_0_o0 | 5| 6| 7|TIDL_DataLayer |tidl_0_o0 |tidl_0_o0 | 6| 0| Completed tidl_optimizeNet RelayImportDebug: In TIDL_relayPostProcessNet: Raw data format used is same as inElementType 6 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Raw data format used is same as inElementType 0 Num of Layer Detected : 6 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Num|TIDL Layer Name |Out Data Name |Group |#Ins |#Outs |Inbuf Ids |Outbuf Id |In NCHW |Out NCHW |MACS | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0|TIDL_DataLayer |tidl_0_i0 | 0| -1| 1| x x x x x x x x | 0 | 0 0 0 0 | 1 8 150000 1 | 0 | 1|TIDL_DataConvertLayer |tidl_0_i0_netFormat | 0| 1| 1| 0 x x x x x x x | 1 | 1 8 150000 1 | 1 8 150000 1 | 1200000 | 2|TIDL_ConvolutionLayer |tidl_0_9 | 0| 1| 1| 1 x x x x x x x | 2 | 1 8 150000 1 | 1 48 150000 1 | 57600000 | 3|TI Processing config file #0 : /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.qunat_stats_config.txt Input : dataId=0, name=tidl_0_i0, scale=1.000000 Ouput : dataId=4, name=tidl_0_o0, scale=1.000000 38464, 0.037 0x56173f7493f0 worstCaseDelay for Pre-emption is 0.0000000 Network File Read done Calling algAlloc TIDL_initDebugTraceParams Done Alg Alloc for Layer # - 0 Alg Alloc for Layer # - 1 Alg Alloc for Layer # - 2 Alg Alloc for Layer # - 3 Alg Alloc for Layer # - 4 Alg Alloc for Layer # - 5 TIDL Memory requiement MemRecNum , Space , Attribute , SizeinBytes 0 , DDR , Persistent, 15208 1 , DDR , Persistent, 136 2 , DDR , Scratch , 16384 3 , DDR , Scratch , 4096 4 , DDR , Scratch , 57344 5 , DDR , Persistent, 2176 6 , DDR , Scratch , 62924928 7 , DDR , Scratch , 256 8 , DDR , Scratch , 43200128 9 , DDR , Scratch , 230403072 10 , DDR , Persistent, 5431680 11 , DDR , Persistent, 172416 12 , DDR , Scratch , 256 13 , DDR , Persistent, 128 NOTE: Memory requirement in host emulation can be different from the same on EVM To get the actual TIDL memory requirement make sure to run on EVM with writeTraceLevel = 0 Num, Space, SizeinBytes, SineInMB 0, 17, 15208, 0.015 0x56173f752a40 1, 17, 136, 0.000 0x56173f744490 2, 17, 16384, 0.016 0x56173f7565b0 3, 17, 4096, 0.004 0x56173f75a5c0 4, 17, 57344, 0.055 0x56173f75b5d0 5, 17, 2176, 0.002 0x56173f7695e0 6, 17, 62924928, 60.010 0x7fc037a1b010 7, 17, 256, 0.000 0x56173f740840 8, 17, 43200128, 41.199 0x7fc0350e8010 9, 17, 230403072, 219.729 0x7fc02752d010 10, 17, 5431680, 5.180 0x7fc026ffe010 11, 17, 172416, 0.164 0x7fc0d0b67010 12, 17, 256, 0.000 0x56173f748cd0 13, 17, 128, 0.000 0x56173f7404c0 Total External Memory (DDR) Size = 342228208, 326.374 Alg Init for Layer # - 0 out of 5 Alg Init for Layer # - 1 out of 5 Alg Init for Layer # - 2 out of 5 Alg Init for Layer # - 3 out of 5 Alg Init for Layer # - 4 out of 5 Alg Init for Layer # - 5 out of 5 Instance created for /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.qunat_stats_config.txt Processing Cnt : 0, InstCnt : 0 /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/subgraph0_net.bin! 9600000, 9.155 0x7fc0246ca010 117600000, 112.152 0x7fc01d6a3010 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------ # 0 . ..Skipping static gen-set function TIDL_activate is called with handle : 3f752a40 Starting Layer # - 1 Processing Layer # - 1 End of Layer # - 1 with outPtrs[0] = 0x7fc037a1b010 Starting Layer # - 2 Processing Layer # - 2 End of Layer # - 2 with outPtrs[0] = 0x7fc037eaee10 Starting Layer # - 3 Processing Layer # - 3 End of Layer # - 3 with outPtrs[0] = 0x7fc039a26210 Starting Layer # - 4 Processing Layer # - 4 End of Layer # - 4 with outPtrs[0] = 0x7fc01d6a3010 TIDL_process is completed with handle : 3f752a40 T 148.75 Skipping static gen-set function .... ..... ... .... .....TIDL_deactivate is called with handle : 3f752a40 DL_ConvolutionLayer |tidl_0_o0_netFormat | 0| 1| 1| 2 x x x x x x x | 3 | 1 48 150000 1 | 1 48 150000 1 | 345600000 | 4|TIDL_DataConvertLayer |tidl_0_o0 | 0| 1| 1| 3 x x x x x x x | 4 | 1 48 150000 1 | 1 48 150000 1 | 7200000 | 5|TIDL_DataLayer |tidl_0_o0 | 0| 1| -1| 4 x x x x x x x | 0 | 1 48 150000 1 | 0 0 0 0 | 0 | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total Giga Macs : 0.4116 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Num of Layer Detected : 6 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Num|TIDL Layer Name |Out Data Name |Group |#Ins |#Outs |Inbuf Ids |Outbuf Id |In NCHW |Out NCHW |MACS | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0|TIDL_DataLayer |tidl_0_i0 | 0| -1| 1| x x x x x x x x | 0 | 0 0 0 0 | 1 8 150000 1 | 0 | 1|TIDL_DataConvertLayer |tidl_0_i0_netFormat | 0| 1| 1| 0 x x x x x x x | 1 | 1 8 150000 1 | 1 8 150000 1 | 1200000 | 2|TIDL_ConvolutionLayer |tidl_0_9 | 0| 1| 1| 1 x x x x x x x | 2 | 1 8 150000 1 | 1 48 150000 1 | 57600000 | 3|TIDL_ConvolutionLayer |tidl_0_o0_netFormat | 0| 1| 1| 2 x x x x x x x | 3 | 1 48 150000 1 | 1 48 150000 1 | 345600000 | 4|TIDL_DataConvertLayer |tidl_0_o0 | 0| 1| 1| 3 x x x x x x x | 4 | 1 48 150000 1 | 1 48 150000 1 | 7200000 | 5|TIDL_DataLayer |tidl_0_o0 | 0| 1| -1| 4 x x x x x x x | 0 | 1 48 150000 1 | 0 0 0 0 | 0 | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total Giga Macs : 0.4116 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ cd /workspace/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/tidl_j721e_08_02_00_11/ti_dl/test && ./PC_dsp_test_dl_algo.out s:/workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.qunat_stats_config.txt Num of Layer Detected : 6 ------------------------------------------------------------------------------------------------------------------------------------------ Processing config file #0 : /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.qunat_stats_config.txt Input : dataId=0, name=tidl_0_i0, scale=1.000000 Ouput : dataId=4, name=tidl_0_o0, scale=1.000000 30240, 0.029 0x5583661ed3f0 worstCaseDelay for Pre-emption is 0.0000000 Network File Read done Calling algAlloc TIDL_initDebugTraceParams Done Alg Alloc for Layer # - 0 Alg Alloc for Layer # - 1 Alg Alloc for Layer # - 2 Alg Alloc for Layer # - 3 Alg Alloc for Layer # - 4 Alg Alloc for Layer # - 5 TIDL Memory requiement MemRecNum , Space , Attribute , SizeinBytes 0 , DDR , Persistent, 15208 1 , DDR , Persistent, 136 2 , DDR , Scratch , 16384 3 , DDR , Scratch , 4096 4 , DDR , Scratch , 57344 5 , DDR , Persistent, 2176 6 , DDR , Scratch , 16124928 7 , DDR , Scratch , 256 8 , DDR , Scratch , 43200128 9 , DDR , Scratch , 230403072 10 , DDR , Persistent, 5431680 11 , DDR , Persistent, 172416 12 , DDR , Scratch , 256 13 , DDR , Persistent, 128 NOTE: Memory requirement in host emulation can be different from the same on EVM To get the actual TIDL memory requirement make sure to run on EVM with writeTraceLevel = 0 Num, Space, SizeinBytes, SineInMB 0, 17, 15208, 0.015 0x5583661f4a20 1, 17, 136, 0.000 0x5583661e8490 2, 17, 16384, 0.016 0x5583661f8590 3, 17, 4096, 0.004 0x5583661fc5a0 4, 17, 57344, 0.055 0x5583661fd5b0 5, 17, 2176, 0.002 0x55836620b5c0 6, 17, 16124928, 15.378 0x7f3d0579a010 7, 17, 256, 0.000 0x5583661e4840 8, 17, 43200128, 41.199 0x7f3d02e67010 9, 17, 230403072, 219.729 0x7f3cf52ac010 10, 17, 5431680, 5.180 0x7f3cf4d7d010 11, 17, 172416, 0.164 0x7f3d9bc44010 12, 17, 256, 0.000 0x5583661eccd0 13, 17, 128, 0.000 0x5583661e44c0 Total External Memory (DDR) Size = 295428208, 281.742 Alg Init for Layer # - 0 out of 5 Alg Init for Layer # - 1 out of 5 Alg Init for Layer # - 2 out of 5 Alg Init for Layer # - 3 out of 5 Alg Init for Layer # - 4 out of 5 Alg Init for Layer # - 5 out of 5 Instance created for /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.qunat_stats_config.txt Processing Cnt : 0, InstCnt : 0 /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/subgraph0_net.bin! 9600000, 9.155 0x7f3cf2449010 117600000, 112.152 0x7f3ceb422010 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------ # 0 . ..Skipping static gen-set function TIDL_activate is called with handle : 661f4a20 Starting Layer # - 1 Processing Layer # - 1 End of Layer # - 1 with outPtrs[0] = 0x7f3d0579a010 Starting Layer # - 2 Processing Layer # - 2 End of Layer # - 2 with outPtrs[0] = 0x7f3d058bef90 Starting Layer # - 3 Processing Layer # - 3 End of Layer # - 3 with outPtrs[0] = 0x7f3d05f9cc90 Starting Layer # - 4 Processing Layer # - 4 End of Layer # - 4 with outPtrs[0] = 0x7f3ceb422010 TIDL_process is completed with handle : 661f4a20 T 179.34 Skipping static gen-set function .... ..... ... .... .....TIDL_deactivate is called with handle : 661f4a20 -------------------------------------------------- Num|TIDL Layer Name |Out Data Name |Group |#Ins |#Outs |Inbuf Ids |Outbuf Id |In NCHW |Out NCHW |MACS | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0|TIDL_DataLayer |tidl_0_i0 | 0| -1| 1| x x x x x x x x | 0 | 0 0 0 0 | 1 8 150000 1 | 0 | 1|TIDL_DataConvertLayer |tidl_0_i0_netFormat | 0| 1| 1| 0 x x x x x x x | 1 | 1 8 150000 1 | 1 8 150000 1 | 1200000 | 2|TIDL_ConvolutionLayer |tidl_0_9 | 0| 1| 1| 1 x x x x x x x | 2 | 1 8 150000 1 | 1 48 150000 1 | 57600000 | 3|TIDL_ConvolutionLayer |tidl_0_o0_netFormat | 0| 1| 1| 2 x x x x x x x | 3 | 1 48 150000 1 | 1 48 150000 1 | 345600000 | 4|TIDL_DataConvertLayer |tidl_0_o0 | 0| 1| 1| 3 x x x x x x x | 4 | 1 48 150000 1 | 1 48 150000 1 | 7200000 | 5|TIDL_DataLayer |tidl_0_o0 | 0| 1| -1| 4 x x x x x x x | 0 | 1 48 150000 1 | 0 0 0 0 | 0 | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total Giga Macs : 0.4116 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~ cd /workspace/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/tidl_j721e_08_02_00_11/ti_dl/test && ./PC_dsp_test_dl_algo.out s:/workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.qunat_stats_config.txt ***************** Calibration iteration number 0 completed ************************ Num of Layer Detected : 6 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Num|TIDL Layer Name |Out Data Name |Group |#Ins |#Outs |Inbuf Ids |Outbuf Id |In NCHW |Out NCHW |MACS | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0|TIDL_DataLayer |tidl_0_i0 | 0| -1| 1| x x x x x x x x | 0 | 0 0 0 0 | 1 8 150000 1 | 0 | 1|TIDL_DataConvertLayer |tidl_0_i0_netFormat | 0| 1| 1| 0 x x x x x x x | 1 | 1 8 150000 1 | 1 8 150000 1 | 1200000 | 2|TIDL_ConvolutionLayer |tidl_0_9 | 0| 1| 1| 1 x x x x x x x | 2 | 1 8 150000 1 | 1 48 150000 ------------------ Network Compiler Traces ----------------------------- successful Memory allocation Files already downloaded and verified type tensor_int8_t { tensor_nil_int8, tensor0_int8(int8), tensor1_int8(Tensor[(?), int8]), tensor2_int8(Tensor[(?, ?), int8]), tensor3_int8(Tensor[(?, ?, ?), int8]), tensor4_int8(Tensor[(?, ?, ?, ?), int8]), tensor5_int8(Tensor[(?, ?, ?, ?, ?), int8]), tensor6_int8(Tensor[(?, ?, ?, ?, ?, ?), int8]), } type tensor_float32_t { tensor_nil_float32, tensor0_float32(float32), tensor1_float32(Tensor[(?), float32]), tensor2_float32(Tensor[(?, ?), float32]), tensor3_float32(Tensor[(?, ?, ?), float32]), tensor4_float32(Tensor[(?, ?, ?, ?), float32]), tensor5_float32(Tensor[(?, ?, ?, ?, ?), float32]), tensor6_float32(Tensor[(?, ?, ?, ?, ?, ?), float32]), } type Tree[A] { Rose(A, List[Tree[A]]), } type tensor_float16_t { tensor_nil_float16, tensor0_float16(float16), tensor1_float16(Tensor[(?), float16]), tensor2_float16(Tensor[(?, ?), float16]), tensor3_float16(Tensor[(?, ?, ?), float16]), tensor4_float16(Tensor[(?, ?, ?, ?), float16]), tensor5_float16(Tensor[(?, ?, ?, ?, ?), float16]), tensor6_float16(Tensor[(?, ?, ?, ?, ?, ?), float16]), } type tensor_uint8_t { tensor_nil_uint8, tensor0_uint8(uint8), tensor1_uint8(Tensor[(?), uint8]), tensor2_uint8(Tensor[(?, ?), uint8]), tensor3_uint8(Tensor[(?, ?, ?), uint8]), tensor4_uint8(Tensor[(?, ?, ?, ?), uint8]), tensor5_uint8(Tensor[(?, ?, ?, ?, ?), uint8]), tensor6_uint8(Tensor[(?, ?, ?, ?, ?, ?), uint8]), } type Option[A] { Some(A), None, } type tensor_int32_t { tensor_nil_int32, tensor0_int32(int32), tensor1_int32(Tensor[(?), int32]), tensor2_int32(Tensor[(?, ?), int32]), tensor3_int32(Tensor[(?, ?, ?), int32]), tensor4_int32(Tensor[(?, ?, ?, ?), int32]), tensor5_int32(Tensor[(?, ?, ?, ?, ?), int32]), tensor6_int32(Tensor[(?, ?, ?, ?, ?, ?), int32]), } type tensor_int64_t { tensor_nil_int64, tensor0_int64(int64), tensor1_int64(Tensor[(?), int64]), tensor2_int64(Tensor[(?, ?), int64]), tensor3_int64(Tensor[(?, ?, ?), int64]), tensor4_int64(Tensor[(?, ?, ?, ?), int64]), tensor5_int64(Tensor[(?, ?, ?, ?, ?), int64]), tensor6_int64(Tensor[(?, ?, ?, ?, ?, ?), int64]), } type List[A] { Cons(A, List[A]), Nil, } type tensor_float64_t { tensor_nil_float64, tensor0_float64(float64), tensor1_float64(Tensor[(?), float64]), tensor2_float64(Tensor[(?, ?), float64]), tensor3_float64(Tensor[(?, ?, ?), float64]), tensor4_float64(Tensor[(?, ?, ?, ?), float64]), tensor5_float64(Tensor[(?, ?, ?, ?, ?), float64]), tensor6_float64(Tensor[(?, ?, ?, ?, ?, ?), float64]), } type tensor_uint16_t { tensor_nil_uint16, tensor0_uint16(uint16), tensor1_uint16(Tensor[(?), uint16]), tensor2_uint16(Tensor[(?, ?), uint16]), tensor3_uint16(Tensor[(?, ?, ?), uint16]), tensor4_uint16(Tensor[(?, ?, ?, ?), uint16]), tensor5_uint16(Tensor[(?, ?, ?, ?, ?), uint16]), tensor6_uint16(Tensor[(?, ?, ?, ?, ?, ?), uint16]), } type tensor_int16_t { tensor_nil_int16, tensor0_int16(int16), tensor1_int16(Tensor[(?), int16]), tensor2_int16(Tensor[(?, ?), int16]), tensor3_int16(Tensor[(?, ?, ?), int16]), tensor4_int16(Tensor[(?, ?, ?, ?), int16]), tensor5_int16(Tensor[(?, ?, ?, ?, ?), int16]), tensor6_int16(Tensor[(?, ?, ?, ?, ?, ?), int16]), } def @main(%nqlib_compute_graph_input_0_out: Tensor[(150000, 5), float32], %nqlib_compute_graph_input_1_out: Tensor[(150000, 1), float32], %voxel_encoder_pfn_layers_0_linear.weight: Tensor[(48, 8, 1, 1), float32], %voxel_encoder_pfn_layers_0_norm.weight: Tensor[(48), float32], %voxel_encoder_pfn_layers_0_norm.bias: Tensor[(48), float32], %voxel_encoder_pfn_layers_0_norm.running_mean: Tensor[(48), float32], %voxel_encoder_pfn_layers_0_norm.running_var: Tensor[(48), float32], %voxel_encoder_pfn_layers_1_linear.weight: Tensor[(48, 48, 1, 1), float32], %voxel_encoder_pfn_layers_1_norm.weight: Tensor[(48), float32], %voxel_encoder_pfn_layers_1_norm.bias: Tensor[(48), float32], %voxel_encoder_pfn_layers_1_norm.running_mean: Tensor[(48), float32], %voxel_encoder_pfn_layers_1_norm.running_var: Tensor[(48), float32]) { %0 = ailice.PrepareBevCoordsAndFeatures(%nqlib_compute_graph_input_0_out, %nqlib_compute_graph_input_1_out, out_dtype="", x_min=-70f, y_min=-60f, voxel_size_x=0.15625f, voxel_size_y=0.15625f, bev_size_x=896, bev_size_y=768, z_offset=1.55f); %1 = %0.0; %2 = %0.1; %3 = (%1, %2); %4 = %3.1; %5 = expand_dims(%4, axis=0); %6 = expand_dims(%5, axis=-2); %7 = transpose(%6, axes=[0, 3, 1, 2]); %8 = nn.conv2d(%7, %voxel_encoder_pfn_layers_0_linear.weight, padding=[0, 0, 0, 0], channels=48, kernel_size=[1, 1]); %9 = nn.batch_norm(%8, %voxel_encoder_pfn_layers_0_norm.weight, %voxel_encoder_pfn_layers_0_norm.bias, %voxel_encoder_pfn_layers_0_norm.running_mean, %voxel_encoder_pfn_layers_0_norm.running_var, epsilon=0.001f); %10 = %9.0; %11 = nn.relu(%10); %12 = nn.conv2d(%11, %voxel_encoder_pfn_layers_1_linear.weight, padding=[0, 0, 0, 0], channels=48, kernel_size=[1, 1]); %13 = nn.batch_norm(%12, %voxel_encoder_pfn_layers_1_norm.weight, %voxel_encoder_pfn_layers_1_norm.bias, %voxel_encoder_pfn_layers_1_norm.running_mean, %voxel_encoder_pfn_layers_1_norm.running_var, epsilon=0.001f); %14 = %13.0; nn.relu(%14) } Generating subgraph boundary tensors for calibration... Building graph on host for tensor data collection... Running graph on host for tensor data collection... Importing subgraph into TIDL... TIDL import of 1 Relay IR subgraphs succeeded. TIDL artifacts are stored at ./panolidar_export/ Building C7x tvm deployable module: generating c files... Building C7x tvm deployable module: building... (log in c7x_deploy_mod.log) make TVM_ROOT=/workspace/TI/TVM_TI TVM_C7X_ROOT=/workspace/TI/TVM_TI/src/runtime/contrib/tidl/c7x QUIET= -C /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir -f /workspace/TI/TVM_TI/src/runtime/contrib/tidl/c7x/Makefile.c7x_mod -j$(nproc) Creating Arm wrapper tvm module... type tensor_int8_t { tensor_nil_int8, tensor0_int8(int8), tensor1_int8(Tensor[(?), int8]), tensor2_int8(Tensor[(?, ?), int8]), tensor3_int8(Tensor[(?, ?, ?), int8]), tensor4_int8(Tensor[(?, ?, ?, ?), int8]), tensor5_int8(Tensor[(?, ?, ?, ?, ?), int8]), tensor6_int8(Tensor[(?, ?, ?, ?, ?, ?), int8]), } type tensor_float32_t { tensor_nil_float32, tensor0_float32(float32), tensor1_float32(Tensor[(?), float32]), tensor2_float32(Tensor[(?, ?), float32]), tensor3_float32(Tensor[(?, ?, ?), float32]), tensor4_float32(Tensor[(?, ?, ?, ?), float32]), tensor5_float32(Tensor[(?, ?, ?, ?, ?), float32]), tensor6_float32(Tensor[(?, ?, ?, ?, ?, ?), float32]), } type Tree[A] { Rose(A, List[Tree[A]]), } type tensor_float16_t { tensor_nil_float16, tensor0_float16(float16), tensor1_float16(Tensor[(?), float16]), tensor2_float16(Tensor[(?, ?), float16]), tensor3_float16(Tensor[(?, ?, ?), float16]), tensor4_float16(Tensor[(?, ?, ?, ?), float16]), tensor5_float16(Tensor[(?, ?, ?, ?, ?), float16]), tensor6_float16(Tensor[(?, ?, ?, ?, ?, ?), float16]), } type tensor_uint8_t { tensor_nil_uint8, tensor0_uint8(uint8), tensor1_uint8(Tensor[(?), uint8]), tensor2_uint8(Tensor[(?, ?), uint8]), tensor3_uint8(Tensor[(?, ?, ?), uint8]), tensor4_uint8(Tensor[(?, ?, ?, ?), uint8]), tensor5_uint8(Tensor[(?, ?, ?, ?, ?), uint8]), tensor6_uint8(Tensor[(?, ?, ?, ?, ?, ?), uint8]), } type Option[A] { Some(A), None, } type tensor_int32_t { tensor_nil_int32, tensor0_int32(int32), tensor1_int32(Tensor[(?), int32]), tensor2_int32(Tensor[(?, ?), int32]), tensor3_int32(Tensor[(?, ?, ?), int32]), tensor4_int32(Tensor[(?, ?, ?, ?), int32]), tensor5_int32(Tensor[(?, ?, ?, ?, ?), int32]), tensor6_int32(Tensor[(?, ?, ?, ?, ?, ?), int32]), } type tensor_int64_t { tensor_nil_int64, tensor0_int64(int64), tensor1_int64(Tensor[(?), int64]), tensor2_int64(Tensor[(?, ?), int64]), tensor3_int64(Tensor[(?, ?, ?), int64]), tensor4_int64(Tensor[(?, ?, ?, ?), int64]), tensor5_int64(Tensor[(?, ?, ?, ?, ?), int64]), tensor6_int64(Tensor[(?, ?, ?, ?, ?, ?), int64]), } type List[A] { Cons(A, List[A]), Nil, } type tensor_float64_t { tensor_nil_float64, tensor0_float64(float64), tensor1_float64(Tensor[(?), float64]), tensor2_float64(Tensor[(?, ?), float64]), tensor3_float64(Tensor[(?, ?, ?), float64]), tensor4_float64(Tensor[(?, ?, ?, ?), float64]), tensor5_float64(Tensor[(?, ?, ?, ?, ?), float64]), tensor6_float64(Tensor[(?, ?, ?, ?, ?, ?), float64]), } type tensor_uint16_t { tensor_nil_uint16, tensor0_uint16(uint16), tensor1_uint16(Tensor[(?), uint16]), tensor2_uint16(Tensor[(?, ?), uint16]), tensor3_uint16(Tensor[(?, ?, ?), uint16]), tensor4_uint16(Tensor[(?, ?, ?, ?), uint16]), tensor5_uint16(Tensor[(?, ?, ?, ?, ?), uint16]), tensor6_uint16(Tensor[(?, ?, ?, ?, ?, ?), uint16]), } type tensor_int16_t { tensor_nil_int16, tensor0_int16(int16), tensor1_int16(Tensor[(?), int16]), tensor2_int16(Tensor[(?, ?), int16]), tensor3_int16(Tensor[(?, ?, ?), int16]), tensor4_int16(Tensor[(?, ?, ?, ?), int16]), tensor5_int16(Tensor[(?, ?, ?, ?, ?), int16]), tensor6_int16(Tensor[(?, ?, ?, ?, ?, ?), int16]), } def @tidl_tvm_0(%nqlib_compute_graph_input_0_out_c7x: Tensor[(150000, 5), float32], %nqlib_compute_graph_input_1_out_c7x: Tensor[(150000, 1), float32], global_symbol="tidl_tvm_0", Primitive=1, Compiler="tidl", Inline=1) -> Tensor[(1, 48, 150000, 1), float32] { %0 = ailice.PrepareBevCoordsAndFeatures(%nqlib_compute_graph_input_0_out_c7x, %nqlib_compute_graph_input_1_out_c7x, out_dtype="", x_min=-70f, y_min=-60f, voxel_size_x=0.15625f, voxel_size_y=0.15625f, bev_size_x=896, bev_size_y=768, z_offset=1.55f) /* ty=(Tensor[(150000, 3), int32], Tensor[(150000, 8), float32]) */; %1 = %0.1; %2 = expand_dims(%1, axis=0) /* ty=Tensor[(1, 150000, 8), float32] */; %3 = expand_dims(%2, axis=-2) /* ty=Tensor[(1, 150000, 1, 8), float32] */; %4 = transpose(%3, axes=[0, 3, 1, 2]) /* ty=Tensor[(1, 8, 150000, 1), float32] */; %5 = nn.conv2d(%4, meta[relay.Constant][0] /* ty=Tensor[(48, 8, 1, 1), float32] */, padding=[0, 0, 0, 0], channels=48, kernel_size=[1, 1]) /* ty=Tensor[(1, 48, 150000, 1), float32] */; %6 = nn.batch_norm(%5, meta[relay.Constant][1] /* ty=Tensor[(48), float32] */, meta[relay.Constant][2] /* ty=Tensor[(48), float32] */, meta[relay.Constant][3] /* ty=Tensor[(48), float32] */, meta[relay.Constant][4] /* ty=Tensor[(48), float32] */, epsilon=0.001f) /* ty=(Tensor[(1, 48, 150000, 1), float32], Tensor[(48), float32], Tensor[(48), float32]) */; %7 = %6.0; %8 = nn.relu(%7) /* ty=Tensor[(1, 48, 150000, 1), float32] */; %9 = nn.conv2d(%8, meta[relay.Constant][5] /* ty=Tensor[(48, 48, 1, 1), float32] */, padding=[0, 0, 0, 0], channels=48, kernel_size=[1, 1]) /* ty=Tensor[(1, 48, 150000, 1), float32] */; %10 = nn.batch_norm(%9, meta[relay.Constant][6] /* ty=Tensor[(48), float32] */, meta[relay.Constant][7] /* ty=Tensor[(48), float32] */, meta[relay.Constant][8] /* ty=Tensor[(48), float32] */, meta[relay.Constant][9] /* ty=Tensor[(48), float32] */, epsilon=0.001f) /* ty=(Tensor[(1, 48, 150000, 1), float32], Tensor[(48), float32], Tensor[(48), float32]) */; %11 = %10.0; nn.relu(%11) /* ty=Tensor[(1, 48, 150000, 1), float32] */ } def @main(%nqlib_compute_graph_input_0_out: Tensor[(150000, 5), float32], %nqlib_compute_graph_input_1_out: Tensor[(150000, 1), float32]) -> Tensor[(1, 48, 150000, 1), float32] { @tidl_tvm_0(%nqlib_compute_graph_input_0_out, %nqlib_compute_graph_input_1_out) /* ty=Tensor[(1, 48, 150000, 1), float32] */ } 1 { "nodes": [ { "op": "null", "name": "nqlib_compute_graph_input_0_out", "inputs": [] }, { "op": "null", "name": "nqlib_compute_graph_input_1_out", "inputs": [] }, { "op": "tvm_op", "name": "tidl_tvm_0", "attrs": { "num_outputs": "1", "num_inputs": "2", "flatten_data": "0", "Compiler": "tidl", "global_symbol": "tidl_tvm_0", "func_name": "tidl_tvm_0", "hash": "12b811e4a65d6284", "out_layout": "", "data_layout": "NCHW", "kernel_layout": "OIHW" }, "inputs": [ [ 0, 0, 0 ], [ 1, 0, 0 ] ] } ], "arg_nodes": [0, 1], "heads": [ [ 2, 0, 0 ] ], "attrs": { "dltype": [ "list_str", [ "float32", "float32", "float32" ] ], "shape": [ "list_shape", [ [150000, 5], [150000, 1], [1, 48, 150000, 1] ] ], "storage_id": [ "list_int", [0, 1, 2] ] }, "node_row_ptr": [0, 1, 2, 3] } 1 | 57600000 | 3|TIDL_ConvolutionLayer |tidl_0_o0_netFormat | 0| 1| 1| 2 x x x x x x x | 3 | 1 48 150000 1 | 1 48 150000 1 | 345600000 | 4|TIDL_DataConvertLayer |tidl_0_o0 | 0| 1| 1| 3 x x x x x x x | 4 | 1 48 150000 1 | 1 48 150000 1 | 7200000 | 5|TIDL_DataLayer |tidl_0_o0 | 0| 1| -1| 4 x x x x x x x | 0 | 1 48 150000 1 | 0 0 0 0 | 0 | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total Giga Macs : 0.4116 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cp /workspace/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/tidl_j721e_08_02_00_11/ti_dl/test/testvecs/config/import/device_configs/j721e_config.cfg /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.perf_sim_config.txt cd /workspace/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/tidl_j721e_08_02_00_11/ti_dl/utils/perfsim && ./ti_cnnperfsim.out /workspace/TI/awesome_tda4/tvm_import/panolidar_export/tempDir/tidl_import_subgraph0.txt.perf_sim_config.txt 1601 0 2 substitute string tidl_net_ not found /workspace/ti-processor-sdk-rtos-j721e-evm-08_02_00_05/tidl_j721e_08_02_00_11/ti_dl/utils/tidlModelGraphviz/out/tidl_graphVisualiser.out ./panolidar_export/tempDir//subgraph0_net.bin **************************************************** ** TIDL Model Checker ** **************************************************** **************************************************** ** ALL MODEL CHECK PASSED ** ****************************************************

0 Anshu Jain over 3 years ago in reply to Yongchao Ding

TI__Guru 56820 points

Hi,
Thanks for sharing this. Can you also confirm the SDK release version which you are using?

Regards,
Anshu

0 Yongchao Ding over 3 years ago in reply to Anshu Jain

Prodigy 40 points

ti-processor-sdk-rtos-j721e-evm-08_02_00_05.tar.gz

0 Anshu Jain over 3 years ago in reply to Yongchao Ding

TI__Guru 56820 points

Hi,

This release contains optimized implementation for float to int8/uint8 conversion so I would expect it to be not so bad. Will it be possible for you to share the model with us so that we can debug this further?

Regards,
Anshu

0 Yongchao Ding over 3 years ago in reply to Anshu Jain

Prodigy 40 points

5466.model.zip

I have update model as attached.

I use two ways for transform model into TIDL, first is from pth model into TIDL using TVM, second is onnx model into TIDL using onnxImporter. And the graph using TVM will generate dataConverter. However, the graph using ONNXImporter can input float and output float without dataConverter.

0 Kangjia Dong over 3 years ago in reply to Yongchao Ding

TI__Intellectual 2285 points

Datacovnerter layer performance on tensor sizes 1x8x150000x1, 1x48x150000x1 => 12.8ms, 4.8ms

(TODO) We need to reproduce the issue, understand the performance and relationship with tensor size and dimensions
ONNXRT import didn’t automatically insert the data convert layers, but it will happen in the runtime. There should be an option to ONNX import to insert the data convert layers. In our experience, data converter layers in TIDL graph should be faster than code in runtime.

0 Kangjia Dong over 3 years ago in reply to Kangjia Dong

TI__Intellectual 2285 points

This issue has been fixed now. The latest performance update show as below

# define graph in relay
# def @main(%i0: Tensor[(1, 8, 150000, 1), float32], %w: Tensor[(48, 8, 1, 1), float32]) -> Tensor[(1, 48, 150000, 1), float32] {
# nn.conv2d(%i0, %w, Tensor[(1, 8, 150000, 1), float32], Tensor[(48, 8, 1, 1), float32], padding=[0, 0, 0, 0], kernel_size=[1, 1]) /* ty=Tensor[(1, 48, 150000, 1), float32] */
# }
# performance of same tensor size, different dimension sizes, c7x cycles
# data convert layer cycles for converting input tensor
# 1x8x150000x1 => 8471107 == after fix ==> 333395
# 1x8x75000x2 => 4270291 == after fix ==> 331165
# 1x8x50000x3 => 2870182 == after fix ==> 338064
# 1x8x30000x5 => 1752769
# 1x8x15000x10 => 913246
# 1x8x1500x100 => 355347
# 1x8x150x1000 => 362558
#
# data convert layer cycle for converting output tensor
# 1x48x150000x1 => 14331686 == after fix ==> 2397351
# 1x48x75000x2 => 7132379 == after fix ==> 2409835
# 1x48x50000x3 => 4904740 == after fix ==> 2399583
# 1x48x30000x5 => 3066402
# 1x48x15000x10 => 2440451
# 1x48x1500x100 => 2536086
# 1x48x150x1000 => 2430688

Processors

Processors forum

TDA4VM: DataConvert Layer in TIDL subgraph cost too much time in inference