• State Resolved
  • Locked Locked
  • Replies 20 replies
  • Answers 1 answer
  • Subscribers 100 subscribers
  • Views 1311 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VMXEVM: How to make the TIDL Datalayer with fixed point parameters?

lance
Intellectual 500 points
Part Number: TDA4VMXEVM


Hi all,

I have one questions about TIDL model input.

If I have 2 models：ModelA and ModelB, if the input of ModelB is the output of ModelA. 

When I use 8-bit quantization, the output layer of ModelA is uint8 with a scale and a pair of Min-Max value. 

How can I set the scale and the pair of Min-Max value of the input datalayer of ModelB.

This is the output layer of ModeA:

And this is the input layer of ModelB:

It should be the same with ModelB.

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    Thank you sooooo much. I ignored this configuration earlier. XD

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    Hi Kumar

    This doesn't completely solve my problem. I have 2 input in ModelB. The scale of input_1 is s_1 and the scale of input_2 is s_2.

    I have set that:

    inQuantFactor = s1 s2

    inDataNameList = "input_1, input_2"

    But in the trace log I can see the scales of input_1 and input_2 are both s1.

    Can I set the scale for the two inputs separately?

  • 0 in reply to lance
    TI__Mastermind 22145 points

    Yes, the user can set a scale for each input separately. Please refer below lines in "tidl_import_common.cpp" of import tool for reference

    Can you check this import again?

        
    int32_t tidl_fillInDataLayerShape(sTIDL_OrgNetwork_t  &pOrgTIDLNetStructure, tidl_import_config * params, int32_t layerIndex)
    {
             ......
           orgTIDLNetStructure.TIDLPCLayers[i].outData[0].tensorScale = params->inQuantFactor[inDataIdx];
          inDataIdx++;

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    Yes.

    I have refered these codes and step into these lines. 

    This param "orgTIDLNetStructure.TIDLPCLayers[i].outData[0].tensorScale" of different inputs are indeed set to different values.

    But in trace log the scales printed for the different inputs are the same.

    I can't see the code of TIDL inference lib, so I can't figure out where is wrong.

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    I looked into the net file generated by TIDL import tool.

    "TIDLLayer[0].outData[0].tensorScale" of Input_1 is s1(1.0) and  "TIDLLayer[1].outData[0].tensorScale" of Input_2 is s2(15.0).

    That is correct.

    But in the following trace log:

    The scales are the same.  s2 should be 15.0.

  • 0 in reply to lance
    TI__Mastermind 22145 points

    This is not expected.

    Can you share the complete console log for import and inference with debugTraceLevel = 2. We will check this get back.

    Also, share the import config file

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    OK 

    console log:

    Fullscreen 27748.log.txt Download 
    ONNX Model (Proto) File  : ../../test/testvecs/sy/models/tda4-head.onnx  
TIDL Network File      : ../../test/testvecs/sy/models/tidl/tidl_net_euc_.bin  
TIDL IO Info File      : ../../test/testvecs/sy/models/tidl/tidl_io_euc__mp_  
Num of Layer Detected :  21 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Num|TIDL Layer Name               |Out Data Name                                     |Group |#Ins  |#Outs |Inbuf Ids                       |Outbuf Id |In NCHW                             |Out NCHW                            |MACS       |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    0|TIDL_DataLayer                |input_data_original                               |     0|    -1|     1|  x   x   x   x   x   x   x   x |  0       |       0        0        0        0 |       1        3     1088     1920 |         0 |
    1|TIDL_DataLayer                |relu5_6_sep                                       |     0|    -1|     1|  x   x   x   x   x   x   x   x |  1       |       0        0        0        0 |       1       32       68      120 |         0 |
    2|TIDL_ResizeLayer              |69                                                |     0|     1|     1|  1   x   x   x   x   x   x   x |  2       |       1       32       68      120 |       1       32      136      240 |   4177920 |
    3|TIDL_BatchNormLayer           |input_data                                        |     0|     1|     1|  0   x   x   x   x   x   x   x |  3       |       1        3     1088     1920 |       1        3     1088     1920 |  25067520 |
(confidetial....)


Processing config file #0 : /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/utils/tidlModelImport/tempDir/qunat_stats_config.txt 
      223518,      0.213 0x7f220259f010
lance tidl_tb_algCreate createParams.net->TIDLLayers[i].outData[0].tensorScale: layer[0]=1.000000 layer1[1]=11.661250
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Num|TIDL Layer Name               |Group |#Ins  |#Outs |Inbuf Ids                       |Outbuf Id |In NCHW                             |Out NCHW                            |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    0|TIDL_DataLayer                |     0|    -1|     1|  x   x   x   x   x   x   x   x |  0       |       0        0        0        0 |       1        3     1088     1920 |
    1|TIDL_DataLayer                |     0|    -1|     1|  x   x   x   x   x   x   x   x |  1       |       0        0        0        0 |       1       32       68      120 |
    2|TIDL_ResizeLayer              |     1|     1|     1|  1   x   x   x   x   x   x   x |  2       |       1       32       68      120 |       1       32      136      240 |
    3|TIDL_BatchNormLayer           |     1|     1|     1|  0   x   x   x   x   x   x   x |  3       |       1        3     1088     1920 |       1        3     1088     1920 |
(confidetial....)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Alg Alloc for Layer # -    0
Alg Alloc for Layer # -    1
Alg Alloc for Layer # -    2
Alg Alloc for Layer # -    3
Alg Alloc for Layer # -    4
Alg Alloc for Layer # -    5
Alg Alloc for Layer # -    6
Alg Alloc for Layer # -    7
Alg Alloc for Layer # -    8
Alg Alloc for Layer # -    9
Alg Alloc for Layer # -   10
Alg Alloc for Layer # -   11
Alg Alloc for Layer # -   12
Alg Alloc for Layer # -   13
Alg Alloc for Layer # -   14
Alg Alloc for Layer # -   15
Alg Alloc for Layer # -   16
Alg Alloc for Layer # -   17
Alg Alloc for Layer # -   18
Alg Alloc for Layer # -   19
Alg Alloc for Layer # -   20
Num,    Space,     SizeinBytes,   SineInMB
   0,    17,         2824,      0.003 0x55649ff49810
   1,    17,          128,      0.000 0x55649ff274e0
   2,    17,        16384,      0.016 0x55649ff4a980
   3,    17,       458752,      0.438 0x7f220252e010
   4,    17,      8159232,      7.781 0x7f219eca5010
   5,    17,         9352,      0.009 0x55649ff4e990
   6,    17,     74255651,     70.816 0x7f219a5d4010
   7,    17,          256,      0.000 0x55649ff4a320
   8,    17,    201197984,    191.877 0x7f218e5f3010
   9,    17,    251500392,    239.849 0x7f217f619010
  10,    17,      4788480,      4.567 0x7f217f187010
  11,    17,       294416,      0.281 0x7f217f13f010
  12,    17,      2095939,      1.999 0x7f217ef3f010
Total External Memory (DDR) Size =    542779790,    517.635 
Alg Init for Layer # -    0
Alg Init for Layer # -    1
Alg Init for Layer # -    2
Alg Init for Layer # -    3
Alg Init for Layer # -    4
Alg Init for Layer # -    5
Alg Init for Layer # -    6
Alg Init for Layer # -    7
Alg Init for Layer # -    8
Alg Init for Layer # -    9
Alg Init for Layer # -   10
Alg Init for Layer # -   11
Alg Init for Layer # -   12
Alg Init for Layer # -   13
Alg Init for Layer # -   14
Alg Init for Layer # -   15
Alg Init for Layer # -   16
Alg Init for Layer # -   17
Alg Init for Layer # -   18
Alg Init for Layer # -   19
Alg Init for Layer # -   20

 Instance created for  /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/utils/tidlModelImport/tempDir/qunat_stats_config.txt

Processing Cnt :    0, InstCnt :    0 /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/test/testvecs/sy/models/tidl/tidl_net_euc_.bin!
    12574866,     11.992 0x7f217e340010
      549824,      0.524 0x7f217e2b9010
     4191622,      3.997 0x7f217deb9010
 ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

#    0 . ..
lance tidl_tb_algProcess TIDL_InArgs scale[1225716580] :0.000000 585526.000000 0.000000 585525.750000 
Starting Layer # -    2 
   1    1.00000    0.00000  255.00000 0
Processing Layer # -    2
   2    1.00000    0.00000  255.00000 0
End of Layer # -    2
Starting Layer # -    3 
   0    1.00000    0.00000  255.00000 0
Processing Layer # -    3
   3   51.49005   -0.99048    1.46638 1
End of Layer # -    3
Starting Layer # -    4 
Processing Layer # -    4
   4   24.69281    0.00000    8.00000 0
End of Layer # -    4
Starting Layer # -    5 
Processing Layer # -    5
   5   18.37357    0.00000    8.00000 0
End of Layer # -    5
Starting Layer # -    6 
Processing Layer # -    6
   6    8.78616    0.00000   16.00000 0
End of Layer # -    6
Starting Layer # -    7 
Processing Layer # -    7
   7   16.59457    0.00000    8.00000 0
End of Layer # -    7
Starting Layer # -    8 
Processing Layer # -    8
   8   55.24134    0.00000    4.00000 0
End of Layer # -    8
Starting Layer # -    9 
Processing Layer # -    9
   9   39.21532    0.00000    4.00000 0
End of Layer # -    9
Starting Layer # -   10 
Processing Layer # -   10
  10   24.07987    0.00000    8.00000 0
End of Layer # -   10
Starting Layer # -   11 
Processing Layer # -   11
  11   29.62162    0.00000    8.00000 0
End of Layer # -   11
Starting Layer # -   12 
Processing Layer # -   12
  12   17.06432    0.00000    8.00000 0
End of Layer # -   12
Starting Layer # -   13 
Processing Layer # -   13
  13   17.68302    0.00000    8.00000 0
End of Layer # -   13
Starting Layer # -   14 
Processing Layer # -   14
  14    0.50000    0.00000  259.50000 0
End of Layer # -   14
Starting Layer # -   15 
Processing Layer # -   15
  15   10.01480    0.00000   16.00000 0
End of Layer # -   15
Starting Layer # -   16 
Processing Layer # -   16
  16    2.19248  -32.00000   32.00000 1
End of Layer # -   16
Starting Layer # -   17 
Processing Layer # -   17
  17    2.19248  -32.00000   32.00000 1
End of Layer # -   17
Starting Layer # -   18 
Processing Layer # -   18
  18    2.19248  -32.00000   32.00000 1
End of Layer # -   18
Starting Layer # -   19 
Processing Layer # -   19
  19    1.00000    0.00000   14.00000 0
End of Layer # -   19
lance tidl_tb_algProcess TIDL_InArgs scale[1225716580] :0.000000 585526.000000 0.000000 585525.750000 
lance tidl_tb_algProcess TIDL_outArgs scale[1] :1.000000 0.000000 585527.000000 0.000000 
 T   27776.33  ... .... .....
------------------ Network Compiler Traces -----------------------------
Main iteration numer: 0....

 Life time alive buffers are with ID: 40001 ( 2), 20020 ( 1), 40002 ( 2), 10020 ( 0), 19 ( 2), 
Preparing for memory allocation : internal iteration number: 0
successful Memory allocation


~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
cd /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/test && ./PC_dsp_test_dl_algo.out /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/utils/tidlModelImport/tempDir/configFilesList.txtcp /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/test/testvecs/config/import/perfsim_base.cfg tempSimDir/perf_sim_config.txt
cd /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/utils/perfsim && ./ti_cnnperfsim.out /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/utils/tidlModelImport/tempSimDir/perf_sim_config.txt 1 0 2
/home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/utils/tidlModelGraphviz/out/tidl_graphVisualiser.out ../../test/testvecs/sy/models/tidl/tidl_net_euc_.bin
****************************************************
**               TIDL Model Checker              **
****************************************************
****************************************************
**                ALL MODEL CHECK PASSED          **
****************************************************

    import config file:

    Thank you very much for your help

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    And I think the max and min values of ModelA's output should also be passed into ModelB together with scale(inQuantFactor).

    I used QAT in my ModelA and ModelB.

  • 0 in reply to lance
    TI__Mastermind 22145 points

    Itried to re-produce this issue with release REL.TIDL.J7.01.01.00.10" using a test network with below import configuration. In my case data ID 0 and 1 are the adat layers .

    modelType = 2
    inputNetFile = "../../test/testvecs/models/ixx.onnx"
    outputNetFile = "../../test/testvecs/config/tidl_models/onnx/net.bin"
    outputParamsFile = "../../test/testvecs/config/tidl_models/onnx/io_"
    inDataNorm = 1
    inMean = 128.0 128.0 128.0 128.0 128.0 128.0
    inScale = 0.015625 0.015625 0.015625 0.015625 0.015625 0.015625
    inElementType = 0 0
    inQuantFactor = 1.0 11.32
    inWidth = 768 768
    inHeight = 384 384
    inNumChannels = 3 3
    inData = "../../test/testvecs/config/tiad_bmp_dof.txt"
    postProcType = 3
    numFrames = 1
    numParamBits = 8
    numFeatureBits = 8
    postProcDataId = 0
    perfSimConfig = ../../test/testvecs/config/import/perfsim_base.cfg
    reserved = 3
    #compileConstraintsFlag = 3

    Below the infer console log and scale values are used as expected.

    ----------------------- TIDL Process with REF_ONLY FLOW ------------------------

    # 0 . .. 0 1.00000 0.00000 255.00000 0
    2 58.51429 0.00000 4.00000 0
    4 117.02857 0.00000 2.00000 0
    6 21.77276 0.00000 8.00000 0
    8 10.97210 -8.00000 8.00000 1
    10 64.57143 0.00000 2.00000 0
    12 35.62561 0.00000 4.00000 0
    14 29.61090 -4.00000 4.00000 1
    16 207.68193 0.00000 1.00000 0
    18 73.03101 0.00000 2.00000 0
    20 21.05399 -4.00000 4.00000 1
    22 10.52699 -8.00000 8.00000 1
    1 11.32000 0.00000 255.00000 0
    3 47.12065 0.00000 4.00000 0
    5 80.41924 0.00000 2.00000 0

    Can you confirm the release version that you are using? If it is the same. Did you make any changes to the import tool?

    For QAT model also, only scale value needs to be set for data layers.Min and Max values are not required.

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    My release version is the same with yours.

    The first layer of my model is resizelayer,  Will this have an impact?

  • 0 in reply to lance
    TI__Mastermind 22145 points

    We don't expect it to change based on Layer type.

    Did you make any change in the TIDL tools?

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    And The next layer of resize layer is eltwise layer.

    The min-max values and scales of Resizelayer and eltwiseLayer are all calculated from the parameters of the previous layer.( Do I understand correctly ？)

    So I think the min-max values should also be passed.

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    The only change I make in the TIDL tools is that I add a resize layer( as shown following). Because in onnx opset10 the upsample layer is defalut named as "resize" rather than "upsample".

    sTIDL_onnxOpParamMap_t tidl_onnxOpParamMapTable[] =
    {
      { "Conv",                          TIDL_onnxMapConvParams },
      { "MaxPool",                       TIDL_onnxMapMaxPoolParams },
      { "Relu",                          TIDL_onnxMapReluParams },
      { "Clip",                          TIDL_onnxMapClipParams },
      { "PRelu",                         TIDL_onnxMapPReluParams },
      { "Concat",                        TIDL_onnxMapConcatParams },
      { "AveragePool",                   TIDL_onnxMapAvgPoolParams },
      { "GlobalAveragePool",             TIDL_onnxMapGlobalAvgPoolParams },
      { "Upsample",                      TIDL_onnxMapUpsampleParams },
      { "ConvTranspose",                 TIDL_onnxMapConvTransposeParams },
      { "Reshape",                       TIDL_onnxMapReshapeParams },
      { "Transpose",                     TIDL_onnxMapTransposeParams },
      { "Add",                           TIDL_onnxMapAddParams },
      { "Sum",                           TIDL_onnxMapAddParams },
      { "ArgMax",                        TIDL_onnxMapArgmaxParams },
      { "BatchNormalization",            TIDL_onnxMapBNParams },
      { "Gemm",                          TIDL_onnxGemmParams },
      { "Softmax",                       TIDL_onnxSoftmaxParams },
      { "Split",                         TIDL_onnxSplitParams },
      { "Flatten",                       TIDL_onnxMapFlattenParams },
      { "Pad",                           TIDL_onnxMapPadParams },
      { "Mul",                           TIDL_onnxMapMulParams },
     //{ "Constant",                    TIDL_onnxMapConstParams },
      { "Dropout",                       TIDL_onnxMapDropoutParams },
      { "Resize",                      TIDL_onnxMapUpsampleParams },
    }
  • 0 in reply to kumar.desappan
    Intellectual 500 points

    This error still occurs when I import my onnx opset9 model with tidl without any changes.

  • 0 in reply to lance
    TI__Mastermind 22145 points

    Can you make the first compute layer as convolution instead of resize and try. 

    I could not reproduce this issue at my end yet.

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    You can see the console log I posted yesterday.

    Processing config file #0 : /home//psdk_rtos_auto_j7_06_02_00_21/tidl_j7_01_01_00_10/ti_dl/utils/tidlModelImport/tempDir/qunat_stats_config.txt
          223518,      0.213 0x7f220259f010
    lance tidl_tb_algCreate createParams.net->TIDLLayers[i].outData[0].tensorScale: layer[0]=1.000000 layer1[1]=11.661250

     

    tensorScale is correct in tidl_tb_algCreate()

     

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    I tried but still failed.

  • 0 in reply to kumar.desappan
    Intellectual 500 points

    I dumped the params stored in the net.bin, and found the scales are correct.

    But in the console log, the params of layer12(aka.datalayer2) are totally different. And the final result is wrong.

    And the scale of the next layer's indata is also correct.

  • 0 in reply to lance
    TI__Intellectual 2235 points

    There is some issue inside TIDL(version 1.1.0.10, in PSDKRA 6.2).

    You can set "quantRangeUpdateFactor = 0.0" in import config to avoid this issue.