• State Resolved
  • Locked Locked
  • Replies 9 replies
  • Subscribers 97 subscribers
  • Views 676 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: FPS of resnet50 on TIDL too low

Sai Niranjan Reddy Kukkala
Prodigy 185 points
Part Number: TDA4VM

Hi, 

Problem: FPS of resnet50 on TIDL too low

Environment used: 

Edgeai-tidl-tools and torchvision-ti

The above git repos were installed in separate docker environments. 

Reproduction steps: 

1) We have used torchvision-ti environment to generate the onnx format. We got the resnet model using the following line of code

resnet_model = xvision.models.__dict__['resnet50_x1'](model_config)

This was used to convert to onnx format. 

2) We used code from edgeai-tidl-tools/examples/jupyter_notebooks/custom-model-onnx.ipynb to convert the model from onnx to TIDL. 

Noticing the fps stats from that code, it shows that the resnet50 takes around 0.5 fps. 

I believe that this needs to be higher. Could you let me know if there are any steps I am missing? Maybe there are specific steps or version that I need to use?

Thank You

Niranjan

  • 0
    TI__Genius 9065 points

    Hi Niranjan,

    Can you please share the generated model? I am suspecting some nodes may be getting delegated to ARM instead of C7x-MMA which could be a possible cause of the lower fps.

    Regards,

    Anand

  • 0 in reply to Anand Pathak
    Prodigy 185 points

    Hi Anand, 

    I am attaching the onnx model here.(link)

    I have seen the output logs and they are as follows. It shows that all the layers are TIDL supported

    **************************************Start of Output Log********************************************

    tidl_tools_path = /resources/edgeai-tidl-tools-08_02_00_05/tidl_tools
    artifacts_folder = ./resnet_50_top_artifacts/
    tidl_tensor_bits = 8
    debug_level = 1
    num_tidl_subgraphs = 16
    tidl_denylist =
    tidl_calibration_accuracy_level = 7
    tidl_calibration_options:num_frames_calibration = 8
    tidl_calibration_options:bias_calibration_iterations = 3
    power_of_2_quantization = 2
    enable_high_resolution_optimization = 0
    pre_batchnorm_fold = 1
    add_data_convert_ops = 0
    output_feature_16bit_names_list =
    m_params_16bit_names_list =
    reserved_compile_constraints_flag = 1601
    ti_internal_reserved_1 =

    ****** WARNING : Network not identified as Object Detection network - Ignore if network is not OD *****

    Supported TIDL layer type --- Conv -- Conv_0
    Supported TIDL layer type --- Relu -- Relu_1
    Supported TIDL layer type --- MaxPool -- MaxPool_2
    Supported TIDL layer type --- Conv -- Conv_3
    Supported TIDL layer type --- Relu -- Relu_4
    Supported TIDL layer type --- Conv -- Conv_5
    Supported TIDL layer type --- Relu -- Relu_6
    Supported TIDL layer type --- Conv -- Conv_7
    Supported TIDL layer type --- Conv -- Conv_8
    Supported TIDL layer type --- Add -- Add_9
    Supported TIDL layer type --- Relu -- Relu_10
    Supported TIDL layer type --- Conv -- Conv_11
    Supported TIDL layer type --- Relu -- Relu_12
    Supported TIDL layer type --- Conv -- Conv_13
    Supported TIDL layer type --- Relu -- Relu_14
    Supported TIDL layer type --- Conv -- Conv_15
    Supported TIDL layer type --- Add -- Add_16
    Supported TIDL layer type --- Relu -- Relu_17
    Supported TIDL layer type --- Conv -- Conv_18
    Supported TIDL layer type --- Relu -- Relu_19
    Supported TIDL layer type --- Conv -- Conv_20
    Supported TIDL layer type --- Relu -- Relu_21
    Supported TIDL layer type --- Conv -- Conv_22
    Supported TIDL layer type --- Add -- Add_23
    Supported TIDL layer type --- Relu -- Relu_24
    Supported TIDL layer type --- Conv -- Conv_25
    Supported TIDL layer type --- Relu -- Relu_26
    Supported TIDL layer type --- Conv -- Conv_27
    Supported TIDL layer type --- Relu -- Relu_28
    Supported TIDL layer type --- Conv -- Conv_29
    Supported TIDL layer type --- Conv -- Conv_30
    Supported TIDL layer type --- Add -- Add_31
    Supported TIDL layer type --- Relu -- Relu_32
    Supported TIDL layer type --- Conv -- Conv_33
    Supported TIDL layer type --- Relu -- Relu_34
    Supported TIDL layer type --- Conv -- Conv_35
    Supported TIDL layer type --- Relu -- Relu_36
    Supported TIDL layer type --- Conv -- Conv_37
    Supported TIDL layer type --- Add -- Add_38
    Supported TIDL layer type --- Relu -- Relu_39
    Supported TIDL layer type --- Conv -- Conv_40
    Supported TIDL layer type --- Relu -- Relu_41
    Supported TIDL layer type --- Conv -- Conv_42
    Supported TIDL layer type --- Relu -- Relu_43
    Supported TIDL layer type --- Conv -- Conv_44
    Supported TIDL layer type --- Add -- Add_45
    Supported TIDL layer type --- Relu -- Relu_46
    Supported TIDL layer type --- Conv -- Conv_47
    Supported TIDL layer type --- Relu -- Relu_48
    Supported TIDL layer type --- Conv -- Conv_49
    Supported TIDL layer type --- Relu -- Relu_50
    Supported TIDL layer type --- Conv -- Conv_51
    Supported TIDL layer type --- Add -- Add_52
    Supported TIDL layer type --- Relu -- Relu_53
    Supported TIDL layer type --- Conv -- Conv_54
    Supported TIDL layer type --- Relu -- Relu_55
    Supported TIDL layer type --- Conv -- Conv_56
    Supported TIDL layer type --- Relu -- Relu_57
    Supported TIDL layer type --- Conv -- Conv_58
    Supported TIDL layer type --- Conv -- Conv_59
    Supported TIDL layer type --- Add -- Add_60
    Supported TIDL layer type --- Relu -- Relu_61
    Supported TIDL layer type --- Conv -- Conv_62
    Supported TIDL layer type --- Relu -- Relu_63
    Supported TIDL layer type --- Conv -- Conv_64
    Supported TIDL layer type --- Relu -- Relu_65
    Supported TIDL layer type --- Conv -- Conv_66
    Supported TIDL layer type --- Add -- Add_67
    Supported TIDL layer type --- Relu -- Relu_68
    Supported TIDL layer type --- Conv -- Conv_69
    Supported TIDL layer type --- Relu -- Relu_70
    Supported TIDL layer type --- Conv -- Conv_71
    Supported TIDL layer type --- Relu -- Relu_72
    Supported TIDL layer type --- Conv -- Conv_73
    Supported TIDL layer type --- Add -- Add_74
    Supported TIDL layer type --- Relu -- Relu_75
    Supported TIDL layer type --- Conv -- Conv_76
    Supported TIDL layer type --- Relu -- Relu_77
    Supported TIDL layer type --- Conv -- Conv_78
    Supported TIDL layer type --- Relu -- Relu_79
    Supported TIDL layer type --- Conv -- Conv_80
    Supported TIDL layer type --- Add -- Add_81
    Supported TIDL layer type --- Relu -- Relu_82
    Supported TIDL layer type --- Conv -- Conv_83
    Supported TIDL layer type --- Relu -- Relu_84
    Supported TIDL layer type --- Conv -- Conv_85
    Supported TIDL layer type --- Relu -- Relu_86
    Supported TIDL layer type --- Conv -- Conv_87
    Supported TIDL layer type --- Add -- Add_88
    Supported TIDL layer type --- Relu -- Relu_89
    Supported TIDL layer type --- Conv -- Conv_90
    Supported TIDL layer type --- Relu -- Relu_91
    Supported TIDL layer type --- Conv -- Conv_92
    Supported TIDL layer type --- Relu -- Relu_93
    Supported TIDL layer type --- Conv -- Conv_94
    Supported TIDL layer type --- Add -- Add_95
    Supported TIDL layer type --- Relu -- Relu_96
    Supported TIDL layer type --- Conv -- Conv_97
    Supported TIDL layer type --- Relu -- Relu_98
    Supported TIDL layer type --- Conv -- Conv_99
    Supported TIDL layer type --- Relu -- Relu_100
    Supported TIDL layer type --- Conv -- Conv_101
    Supported TIDL layer type --- Conv -- Conv_102
    Supported TIDL layer type --- Add -- Add_103
    Supported TIDL layer type --- Relu -- Relu_104
    Supported TIDL layer type --- Conv -- Conv_105
    Supported TIDL layer type --- Relu -- Relu_106
    Supported TIDL layer type --- Conv -- Conv_107
    Supported TIDL layer type --- Relu -- Relu_108
    Supported TIDL layer type --- Conv -- Conv_109
    Supported TIDL layer type --- Add -- Add_110
    Supported TIDL layer type --- Relu -- Relu_111
    Supported TIDL layer type --- Conv -- Conv_112
    Supported TIDL layer type --- Relu -- Relu_113
    Supported TIDL layer type --- Conv -- Conv_114
    Supported TIDL layer type --- Relu -- Relu_115
    Supported TIDL layer type --- Conv -- Conv_116
    Supported TIDL layer type --- Add -- Add_117
    Supported TIDL layer type --- Relu -- Relu_118

    Preliminary subgraphs created = 1
    Final number of subgraphs created are : 1, - Offloaded Nodes - 119, Total Nodes - 119
    Running runtimes graphviz - /resources/edgeai-tidl-tools-08_02_00_05/tidl_tools/tidl_graphVisualiser_runtimes.out ./resnet_50_top_artifacts//allowedNode.txt ./resnet_50_top_artifacts//tempDir/graphvizInfo.txt ./resnet_50_top_artifacts//tempDir/runtimes_visualization.svg
    *** In TIDL_createStateImportFunc ***
    Compute on node : TIDLExecutionProvider_TIDL_0_0
    0, Conv, 3, 1, input.1, 491
    1, Relu, 1, 1, 491, 321
    2, MaxPool, 1, 1, 321, 322
    3, Conv, 3, 1, 322, 494
    4, Relu, 1, 1, 494, 325
    5, Conv, 3, 1, 325, 497
    6, Relu, 1, 1, 497, 328
    7, Conv, 3, 1, 328, 500
    8, Conv, 3, 1, 322, 503
    9, Add, 2, 1, 500, 333
    10, Relu, 1, 1, 333, 334
    11, Conv, 3, 1, 334, 506
    12, Relu, 1, 1, 506, 337
    13, Conv, 3, 1, 337, 509
    14, Relu, 1, 1, 509, 340
    15, Conv, 3, 1, 340, 512
    16, Add, 2, 1, 512, 343
    17, Relu, 1, 1, 343, 344
    18, Conv, 3, 1, 344, 515
    19, Relu, 1, 1, 515, 347
    20, Conv, 3, 1, 347, 518
    21, Relu, 1, 1, 518, 350
    22, Conv, 3, 1, 350, 521
    23, Add, 2, 1, 521, 353
    24, Relu, 1, 1, 353, 354
    25, Conv, 3, 1, 354, 524
    26, Relu, 1, 1, 524, 357
    27, Conv, 3, 1, 357, 527
    28, Relu, 1, 1, 527, 360
    29, Conv, 3, 1, 360, 530
    30, Conv, 3, 1, 354, 533
    31, Add, 2, 1, 530, 365
    32, Relu, 1, 1, 365, 366
    33, Conv, 3, 1, 366, 536
    34, Relu, 1, 1, 536, 369
    35, Conv, 3, 1, 369, 539
    36, Relu, 1, 1, 539, 372
    37, Conv, 3, 1, 372, 542
    38, Add, 2, 1, 542, 375
    39, Relu, 1, 1, 375, 376
    40, Conv, 3, 1, 376, 545
    41, Relu, 1, 1, 545, 379
    42, Conv, 3, 1, 379, 548
    43, Relu, 1, 1, 548, 382
    44, Conv, 3, 1, 382, 551
    45, Add, 2, 1, 551, 385
    46, Relu, 1, 1, 385, 386
    47, Conv, 3, 1, 386, 554
    48, Relu, 1, 1, 554, 389
    49, Conv, 3, 1, 389, 557
    50, Relu, 1, 1, 557, 392
    51, Conv, 3, 1, 392, 560
    52, Add, 2, 1, 560, 395
    53, Relu, 1, 1, 395, 396
    54, Conv, 3, 1, 396, 563
    55, Relu, 1, 1, 563, 399
    56, Conv, 3, 1, 399, 566
    57, Relu, 1, 1, 566, 402
    58, Conv, 3, 1, 402, 569
    59, Conv, 3, 1, 396, 572
    60, Add, 2, 1, 569, 407
    61, Relu, 1, 1, 407, 408
    62, Conv, 3, 1, 408, 575
    63, Relu, 1, 1, 575, 411
    64, Conv, 3, 1, 411, 578
    65, Relu, 1, 1, 578, 414
    66, Conv, 3, 1, 414, 581
    67, Add, 2, 1, 581, 417
    68, Relu, 1, 1, 417, 418
    69, Conv, 3, 1, 418, 584
    70, Relu, 1, 1, 584, 421
    71, Conv, 3, 1, 421, 587
    72, Relu, 1, 1, 587, 424
    73, Conv, 3, 1, 424, 590
    74, Add, 2, 1, 590, 427
    75, Relu, 1, 1, 427, 428
    76, Conv, 3, 1, 428, 593
    77, Relu, 1, 1, 593, 431
    78, Conv, 3, 1, 431, 596
    79, Relu, 1, 1, 596, 434
    80, Conv, 3, 1, 434, 599
    81, Add, 2, 1, 599, 437
    82, Relu, 1, 1, 437, 438
    83, Conv, 3, 1, 438, 602
    84, Relu, 1, 1, 602, 441
    85, Conv, 3, 1, 441, 605
    86, Relu, 1, 1, 605, 444
    87, Conv, 3, 1, 444, 608
    88, Add, 2, 1, 608, 447
    89, Relu, 1, 1, 447, 448
    90, Conv, 3, 1, 448, 611
    91, Relu, 1, 1, 611, 451
    92, Conv, 3, 1, 451, 614
    93, Relu, 1, 1, 614, 454
    94, Conv, 3, 1, 454, 617
    95, Add, 2, 1, 617, 457
    96, Relu, 1, 1, 457, 458
    97, Conv, 3, 1, 458, 620
    98, Relu, 1, 1, 620, 461
    99, Conv, 3, 1, 461, 623
    100, Relu, 1, 1, 623, 464
    101, Conv, 3, 1, 464, 626
    102, Conv, 3, 1, 458, 629
    103, came here second
    Add, 2, 1, 626, 469
    104, Relu, 1, 1, 469, 470
    105, Conv, 3, 1, 470, 632
    106, Relu, 1, 1, 632, 473
    107, Conv, 3, 1, 473, 635
    108, Relu, 1, 1, 635, 476
    109, Conv, 3, 1, 476, 638
    110, Add, 2, 1, 638, 479
    111, Relu, 1, 1, 479, 480
    112, Conv, 3, 1, 480, 641
    113, Relu, 1, 1, 641, 483
    114, Conv, 3, 1, 483, 644
    115, Relu, 1, 1, 644, 486
    116, Conv, 3, 1, 486, 647
    117, Add, 2, 1, 647, 489
    118, Relu, 1, 1, 489, 490

    Input tensor name - input.1
    Output tensor name - 490
    In TIDL_onnxRtImportInit subgraph_name=490
    Layer 0, subgraph id 490, name=490
    Layer 1, subgraph id 490, name=input.1
    In TIDL_runtimesOptimizeNet: LayerIndex = 121, dataIndex = 120
    **************************************End of Output Log********************************************
    The Inference was as follows - 
    Inferences Per Second : 0.52 fps
    Inference Time Per Image : 1906.41 ms
    DDR BW Per Image : 0.00 MB
    Also, the graph shows that there were no CPU executions. 
    I am not sure what is missing exactly? 

    Thank You

    Niranjan

  • 0 in reply to Sai Niranjan Reddy Kukkala
    TI__Genius 9065 points

    Hi Niranjan,

    Can you set enable_high_resolution_optimization = 1 and check?

    Regards,

    Anand

  • 0 in reply to Anand Pathak
    Prodigy 185 points

    Hi Anand,

    I have tried the setting but I don't see any difference. 

    Here is the complete output log -> link

    Thank You

    Niranjan

  • 0 in reply to Sai Niranjan Reddy Kukkala
    TI__Genius 9065 points

    Hi Niranjan,

    Are you getting 0.5 fps even with debug_leve = 0? Just checking since setting debug_level > 0 results in higher inference times than expected.

    Regards,

    Anand

  • 0 in reply to Anand Pathak
    Prodigy 185 points

    Hi Anand,

    I tried the option of debug_level=0 and it didn't really change much. 

    We did feel that the simulation could be an issue so we did try it on the tda4vm hardware. We were able to get an FPS of around 13 for resnet50 with the resolution of 224*224. 

    However, I do see the benchmark is higher => around 40 fps. 

    Second question -> does the benchmar stats of TI, from the repo https://github.com/TexasInstruments/edgeai-tidl-tools, does not always represent the hardware? 

    Thank You

    Niranjan

  • 0 in reply to Sai Niranjan Reddy Kukkala
    TI__Genius 9065 points

    Hi Niranjan,

    Second question -> does the benchmark stats of TI, from the repo https://github.com/TexasInstruments/edgeai-tidl-tools, does not always represent the hardware?

    => No, it does not always represent the hardware, if you run on PC, it will give corresponding benchmarking value for PC and for hardware it will give the hardware value. So 0.5 fps was PC execution time.

    Regarding the benchmark of 40 fps, can you please specify from where you are quoting this benchmark?

    Regards,

    Anand

  • 0 in reply to Anand Pathak
    TI__Genius 9065 points

    Hi Niranjan,

    I tried running the above attached 299*299 resolution model on my end and I am getting around 120 fps for the same.

    Inferences Per Second : 121.06 fps
    Inference Time Per Image : 8.26 ms

    Regards,

    Anand

  • 0 in reply to Anand Pathak
    Prodigy 185 points

    Hi Anand, 

    I see that the TIDL graph execution is around 8.616 ms. 

    However, including time from other operations like file io and draw operations, it comes around 32 ms(31 fps). 

    This is currently enough for our use case. I believe the main issue was with the simulator. 

    Thank You

    Niranjan