SK-AM62A-LP: Compilation of ONNX drops accuracy to zero

Hello Akash,

For the second approach with modelmaker, could you please supply your YAML config file?

I notice a few strange things about your logs, and it makes me wonder if you are using a 'lite' model version like "yolox_nano_lite". These have a few layer changes from the typical mmdetection version of a model, one of which is replacing large maxpool's with a series of 3x3's that feed into each other. This is why you see multiple subgraphs and lines like: 

-------------------------------------------------------------------------------
|          Core           |      No. of Nodes       |   Number of Subgraphs   |
-------------------------------------------------------------------------------
| C7x                     |                     265 |                       2 |
| CPU                     |                       5 |                       x |
-------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
|   Node    | Node Name |                                            Reason                                            |
------------------------------------------------------------------------------------------------------------------------
| Reshape   | 5         | Subgraph does not have any compute node                                                      |
| Transpose | 6         | For firmware version < 10_01_04_00, only permutes are suported when number of dimensions > 4 |
| MaxPool   | 79        | Kernel size (13x13) with stride (1x1) not supported                                          |
| MaxPool   | 78        | Kernel size (9x9) with stride (1x1) not supported                                            |
| MaxPool   | 77        | Kernel size (5x5) with stride (1x1) not supported                                            |

Have you changed any of the source for mmdetection that relates to this model? This should not occur ordinarily 

The lines like: 

Invalid Layer Name  /multi_level_conv_obj.2/Conv_output_0
Invalid Layer Name  /multi_level_conv_reg.2/Conv_output_0
Invalid Layer Name  /multi_level_conv_cls.2/Conv_output_0
Invalid Layer Name  /multi_level_conv_obj.1/Conv_output_0
Invalid Layer Name  /multi_level_conv_reg.1/Conv_output_0
Invalid Layer Name  /multi_level_conv_cls.1/Conv_output_0
Invalid Layer Name  /multi_level_conv_obj.0/Conv_output_0
Invalid Layer Name  /multi_level_conv_reg.0/Conv_output_0
Invalid Layer Name  /multi_level_conv_cls.0/Conv_output_0

Are coming because the last Conv layers in the model are designated to run in 16-bit mode to increase accuracy (and reducing box location jitter), but those layer names do not exist in the ONNX model. The layer names are specified in edgeai-modelmaker in a few places [1]. I have seen that recent releases for tensorlab / related dependencies produce a model.onnx that does not use there same layer names (often replaced with a single numerical identifier)

• It's strange that this portion of the parser even runs if tensor_bits is set to 32.. I would expect it to not reference that setting if not in 8-bit mode.

When the model runs, can you print the outputs? Are they valid in any way? Another reason you can hit 0 mAP is from the validation dataset not including representative data or being so small that missing a couple detections results in worst-case accuracy. 

[1] https://github.com/search?q=repo%3ATexasInstruments%2Fedgeai-tensorlab+multi_level_conv_obj.2&type=code

• https://github.com/TexasInstruments/edgeai-tensorlab/blob/21ab9c1df91ab8984ed65f786989ceca3eabb089/edgeai-modelmaker/data/descriptions/description_vision.yaml#L1490 

BR,
Reese

Hello Reese,

Appreciate the prompt response. For the Approach 1 i have not changed anything apart from specifying my custom dataset and changing the input resolution from 416x416 to 640x640 for the yolox_nano_lite model, which I suppose should not have any influence on the layers towards the end of the model. I have created my own config based on the default yolox_nano_lite.py config provided in the edgeai-mmdetection/config_edgeailite/yolox folder. Following is my model config that I used for the edgeai-mmdetection

auto_scale_lr = dict(base_batch_size=64, enable=False)
backend_args = None
base_lr = 0.002
batch_size = 64
classes = 'vehicle'
coco_img_dir = 'images/'
convert_to_lite_model = dict(model_surgery=1)
custom_hooks = [
    dict(num_last_epochs=15, priority=48, type='YOLOXModeSwitchHook'),
    dict(priority=48, type='SyncNormHook'),
    dict(
        ema_type='ExpMomentumEMA',
        momentum=0.0001,
        priority=49,
        type='EMAHook',
        update_buffers=True),
]
data_root = 'data/survue/ec45d7a0_dataset_fix/'
dataset_type = 'CocoDataset'
default_hooks = dict(
    checkpoint=dict(interval=5, max_keep_ckpts=3, type='CheckpointHook'),
    logger=dict(interval=50, type='LoggerHook'),
    param_scheduler=dict(type='ParamSchedulerHook'),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    timer=dict(type='IterTimerHook'),
    visualization=dict(draw=True, type='DetVisualizationHook'))
default_scope = 'mmdet'
env_cfg = dict(
    cudnn_benchmark=False,
    dist_cfg=dict(backend='nccl'),
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
img_scale = (
    640,
    640,
)
img_scales = [
    (
        640,
        640,
    ),
    (
        320,
        320,
    ),
    (
        960,
        960,
    ),
]
interval = 5
launcher = 'none'
load_from = None
log_level = 'INFO'
log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)
max_epochs = 300
metainfo = dict(classes='vehicle')
model = dict(
    backbone=dict(
        act_cfg=dict(type='ReLU'),
        deepen_factor=0.33,
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        out_indices=(
            2,
            3,
            4,
        ),
        spp_kernal_sizes=(
            5,
            9,
            13,
        ),
        type='CSPDarknet',
        use_depthwise=False,
        widen_factor=0.25),
    bbox_head=dict(
        act_cfg=dict(type='ReLU'),
        feat_channels=64,
        in_channels=64,
        loss_bbox=dict(
            eps=1e-16,
            loss_weight=5.0,
            mode='square',
            reduction='sum',
            type='IoULoss'),
        loss_cls=dict(
            loss_weight=1.0,
            reduction='sum',
            type='CrossEntropyLoss',
            use_sigmoid=True),
        loss_l1=dict(loss_weight=1.0, reduction='sum', type='L1Loss'),
        loss_obj=dict(
            loss_weight=1.0,
            reduction='sum',
            type='CrossEntropyLoss',
            use_sigmoid=True),
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        num_classes=1,
        stacked_convs=2,
        strides=(
            8,
            16,
            32,
        ),
        type='YOLOXHead',
        use_depthwise=False),
    data_preprocessor=dict(
        batch_augments=[
            dict(
                interval=10,
                random_size_range=(
                    320,
                    640,
                ),
                size_divisor=32,
                type='BatchSyncRandomResize'),
        ],
        pad_size_divisor=32,
        type='DetDataPreprocessor'),
    neck=dict(
        act_cfg=dict(type='ReLU'),
        in_channels=[
            64,
            128,
            256,
        ],
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        num_csp_blocks=1,
        out_channels=64,
        type='YOLOXPAFPN',
        upsample_cfg=dict(mode='nearest', scale_factor=2),
        use_depthwise=False),
    test_cfg=dict(nms=dict(iou_threshold=0.65, type='nms'), score_thr=0.01),
    train_cfg=dict(assigner=dict(center_radius=2.5, type='SimOTAAssigner')),
    type='YOLOX')
num_classes = 1
num_last_epochs = 15
num_workers = 8
optim_wrapper = dict(
    optimizer=dict(
        lr=0.002, momentum=0.9, nesterov=True, type='SGD',
        weight_decay=0.0005),
    paramwise_cfg=dict(bias_decay_mult=0.0, norm_decay_mult=0.0),
    type='OptimWrapper')
param_scheduler = [
    dict(
        begin=0,
        by_epoch=True,
        convert_to_iter_based=True,
        end=5,
        type='mmdet.QuadraticWarmupLR'),
    dict(
        T_max=285,
        begin=5,
        by_epoch=True,
        convert_to_iter_based=True,
        end=285,
        eta_min=5e-05,
        type='CosineAnnealingLR'),
    dict(begin=285, by_epoch=True, end=300, factor=1, type='ConstantLR'),
]
quantization = 0
resume = False
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
    batch_size=64,
    dataset=dict(
        ann_file='annotations/instances_val.json',
        backend_args=None,
        data_prefix=dict(img='images/'),
        data_root='data/survue/ec45d7a0_dataset_fix/',
        metainfo=dict(classes='vehicle'),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=True, scale=(
                640,
                640,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
    ann_file=
    'data/survue/ec45d7a0_dataset_fix/annotations/instances_train.json',
    backend_args=None,
    classwise=True,
    metric='bbox',
    type='CocoMetric')
test_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(keep_ratio=True, scale=(
        640,
        640,
    ), type='Resize'),
    dict(
        pad_to_square=True,
        pad_val=dict(img=(
            114.0,
            114.0,
            114.0,
        )),
        type='Pad'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(
        meta_keys=(
            'img_id',
            'img_path',
            'ori_shape',
            'img_shape',
            'scale_factor',
        ),
        type='PackDetInputs'),
]
train_ann_file = 'annotations/instances_train.json'
train_cfg = dict(max_epochs=300, type='EpochBasedTrainLoop', val_interval=5)
train_dataloader = dict(
    batch_size=64,
    dataset=dict(
        dataset=dict(
            ann_file='annotations/instances_train.json',
            backend_args=None,
            data_prefix=dict(img='images/'),
            data_root='data/survue/ec45d7a0_dataset_fix/',
            filter_cfg=dict(filter_empty_gt=False, min_size=32),
            metainfo=dict(classes='vehicle'),
            pipeline=[
                dict(backend_args=None, type='LoadImageFromFile'),
                dict(type='LoadAnnotations', with_bbox=True),
            ],
            type='CocoDataset'),
        pipeline=[
            dict(img_scale=(
                640,
                640,
            ), pad_val=114.0, type='Mosaic'),
            dict(
                border=(
                    -320,
                    -320,
                ),
                scaling_ratio_range=(
                    0.5,
                    1.5,
                ),
                type='RandomAffine'),
            dict(type='YOLOXHSVRandomAug'),
            dict(prob=0.5, type='RandomFlip'),
            dict(keep_ratio=True, scale=(
                640,
                640,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(
                keep_empty=False,
                min_gt_bbox_wh=(
                    1,
                    1,
                ),
                type='FilterAnnotations'),
            dict(type='PackDetInputs'),
        ],
        type='MultiImageMixDataset'),
    num_workers=8,
    persistent_workers=True,
    sampler=dict(shuffle=True, type='DefaultSampler'))
train_dataset = dict(
    dataset=dict(
        ann_file='annotations/instances_train.json',
        backend_args=None,
        data_prefix=dict(img='images/'),
        data_root='data/survue/ec45d7a0_dataset_fix/',
        filter_cfg=dict(filter_empty_gt=False, min_size=32),
        metainfo=dict(classes='vehicle'),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(type='LoadAnnotations', with_bbox=True),
        ],
        type='CocoDataset'),
    pipeline=[
        dict(img_scale=(
            640,
            640,
        ), pad_val=114.0, type='Mosaic'),
        dict(
            border=(
                -320,
                -320,
            ),
            scaling_ratio_range=(
                0.5,
                1.5,
            ),
            type='RandomAffine'),
        dict(type='YOLOXHSVRandomAug'),
        dict(prob=0.5, type='RandomFlip'),
        dict(keep_ratio=True, scale=(
            640,
            640,
        ), type='Resize'),
        dict(
            pad_to_square=True,
            pad_val=dict(img=(
                114.0,
                114.0,
                114.0,
            )),
            type='Pad'),
        dict(
            keep_empty=False,
            min_gt_bbox_wh=(
                1,
                1,
            ),
            type='FilterAnnotations'),
        dict(type='PackDetInputs'),
    ],
    type='MultiImageMixDataset')
train_pipeline = [
    dict(img_scale=(
        640,
        640,
    ), pad_val=114.0, type='Mosaic'),
    dict(
        border=(
            -320,
            -320,
        ),
        scaling_ratio_range=(
            0.5,
            1.5,
        ),
        type='RandomAffine'),
    dict(type='YOLOXHSVRandomAug'),
    dict(prob=0.5, type='RandomFlip'),
    dict(keep_ratio=True, scale=(
        640,
        640,
    ), type='Resize'),
    dict(
        pad_to_square=True,
        pad_val=dict(img=(
            114.0,
            114.0,
            114.0,
        )),
        type='Pad'),
    dict(keep_empty=False, min_gt_bbox_wh=(
        1,
        1,
    ), type='FilterAnnotations'),
    dict(type='PackDetInputs'),
]
tta_model = dict(
    tta_cfg=dict(max_per_img=100, nms=dict(iou_threshold=0.65, type='nms')),
    type='DetTTAModel')
tta_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(
        transforms=[
            [
                dict(keep_ratio=True, scale=(
                    640,
                    640,
                ), type='Resize'),
                dict(keep_ratio=True, scale=(
                    320,
                    320,
                ), type='Resize'),
                dict(keep_ratio=True, scale=(
                    960,
                    960,
                ), type='Resize'),
            ],
            [
                dict(prob=1.0, type='RandomFlip'),
                dict(prob=0.0, type='RandomFlip'),
            ],
            [
                dict(
                    pad_to_square=True,
                    pad_val=dict(img=(
                        114.0,
                        114.0,
                        114.0,
                    )),
                    type='Pad'),
            ],
            [
                dict(type='LoadAnnotations', with_bbox=True),
            ],
            [
                dict(
                    meta_keys=(
                        'img_id',
                        'img_path',
                        'ori_shape',
                        'img_shape',
                        'scale_factor',
                        'flip',
                        'flip_direction',
                    ),
                    type='PackDetInputs'),
            ],
        ],
        type='TestTimeAug'),
]
val_ann_file = 'annotations/instances_val.json'
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
    batch_size=64,
    dataset=dict(
        ann_file='annotations/instances_val.json',
        backend_args=None,
        data_prefix=dict(img='images/'),
        data_root='data/survue/ec45d7a0_dataset_fix/',
        metainfo=dict(classes='vehicle'),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=True, scale=(
                640,
                640,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
    ann_file='data/survue/ec45d7a0_dataset_fix/annotations/instances_val.json',
    backend_args=None,
    metric='bbox',
    type='CocoMetric')
vis_backends = [
    dict(type='LocalVisBackend'),
    dict(type='TensorboardVisBackend'),
]
visualizer = dict(
    name='visualizer',
    type='DetLocalVisualizer',
    vis_backends=[
        dict(type='LocalVisBackend'),
        dict(type='TensorboardVisBackend'),
    ])
work_dir = 'work_dirs/2025_07_09_11_31_38_yolox_nano_lite_dump_ep300_lr0.002_b64'

For the approach 2 using edgeai-modelmaker following is the config_detection.yaml. I have used -

common:
    target_module: 'vision'
    task_type: 'detection'
    target_device: 'AM62A'
    # run_name can be any string, but there are some special cases:
    # {date-time} will be replaced with datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    # {model_name} will be replaced with the name of the model
    # run_name: '{date-time}/{model_name}'
    run_name: '20250725-114808/yolox_nano_lite'

dataset:
    # enable/disable dataset loading
    enable: True #False
    # max_num_files: [750, 250] #None

    # Object Detection Dataset Examples:
    # -------------------------------------
    # Example 1, (known datasets): 'widerface_detection', 'pascal_voc0712', 'coco_detection', 'udacity_selfdriving', 'tomato_detection', 'tiscapes2017_driving'
    # dataset_name: widerface_detection
    # -------------------------------------
    # Example 2, give a dataset name and input_data_path.
    # input_data_path could be a path to zip file, tar file, folder OR http, https link to zip or tar files
    # for input_data_path these are provided with this repository as examples:
    #    'http://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/datasets/tiscapes2017_driving.zip'
    #    'http://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/datasets/animal_detection.zip'
    # -------------------------------------
    # Example 3, give image folders with annotation files (require list with values for both train and val splits)
    # dataset_name: coco_detection
    # input_data_path: ["./data/projects/coco_detection/dataset/train2017",
    #                        "./data/projects/coco_detection/dataset/val2017"]
    # input_annotation_path: ["./data/projects/coco_detection/dataset/annotations/instances_train2017.json",
    #                        "./data/projects/coco_detection/dataset/annotations/instances_val2017.json"]
    # -------------------------------------
    # dataset_name: tiscapes2017_driving
    # input_data_path: 'http://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/datasets/tiscapes2017_driving.zip'
    dataset_name: survue
    input_data_path: ["./data/projects/ec45d7a0_dataset_fix",]
    input_annotation_path: ["./data/projects/ec45d7a0_dataset_fix/annotations/instances_train.json",
                           "./data/projects/ec45d7a0_dataset_fix/annotations/instances_val.json"]

training:
    # enable/disable training
    enable: True #False

    # Object Detection model chosen can be changed here if needed
    # options are: 'yolox_s_lite', 'yolox_tiny_lite', 'yolox_nano_lite', 'yolox_pico_lite', 'yolox_femto_lite', 'yolov7_l_lite'
    model_name: 'yolox_nano_lite'

    training_epochs: 30 #30
    batch_size: 128 #32
    learning_rate: 0.002
    num_gpus: 1 #1 #4

compilation:
    # enable/disable compilation
    enable: True #False
    tensor_bits: 32

I have occasionally set the enable = False for "training" and just tried compiling the trained model with the specified config, which generated the logs from my original post for the modelmaker approach.

Akash

Hi Akash,

We had some fixes related to this, a week back in the branch r10.1.
Just for the confirmation, can you check if you are already up to date with r10.1 branch.

Hi Akash,

I will echo Rekib's note to make sure you pull latest 10.1 branch commits

These configurations look ordinary to me. You've also stated that the trained ONNX model on CPU is behaving ordinarily, and I see from the compilation logs that about 100 samples were found for benchmarking.

• I am curious if this validation set was read and processed correctly, and if the labels are good.
• I'd be interested to know what the output detections look like as well. Are they all invalid/null detections or do they actually match something in the image?

It still stands that the maxpool layers were not correctly replaced with cascaded 3x3 maxpools. This seems like it may be related to the issue

• Like other 'lite' models, you correctly have the "convert_to_lite_model = dict(model_surgery=1)" set
• Are we reaching this part, where we recognize the setting for model-optimization, and produce a replacement dictionary [1][2]?  This would happen before compilation, and should be prior to training 

[1] https://github.com/TexasInstruments/edgeai-tensorlab/blob/c7a97e43b760522f3d4e443780a19a895a1ce1b7/edgeai-mmdetection/mmdet/utils/model_optimization.py#L94C21-L94C42 

[2] https://github.com/TexasInstruments/edgeai-tensorlab/blob/c7a97e43b760522f3d4e443780a19a895a1ce1b7/edgeai-mmdetection/tools/train.py#L182 

BR,
Reese

Hi Reese,

I started off with trying Rekib's suggestions and included the latest fixes made to the branch r10.1 of edgeai-tensorlab. I was able to train a model and compile with tensor_bits =32, and get an accuracy figure in the compilation logs post compilation using edgeai-modelmaker. That said the logs still include the "Invalid Layer Name /multi_level_conv_cls.0/Conv_output_0" messages. After the compilation I am able to get inference outputs in the outputs folder as shown in the screenshot below that I assume are from post compilation model which do successfully demonstrate that the model is able to perform the detection task.

modelmaker_compilation_run.log

I also tried performing inference on the SOC using the compiled model and was unsuccessful with the following logs. 

/usr/lib/python3.12/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/usr/lib/python3.12/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/usr/lib/python3.12/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/usr/lib/python3.12/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/usr/lib/python3.12/site-packages/filterpy-1.4.5-py3.12.egg/filterpy/common/helpers.py:364: SyntaxWarning: invalid escape sequence '\S'
/usr/lib/python3.12/site-packages/filterpy-1.4.5-py3.12.egg/filterpy/common/helpers.py:364: SyntaxWarning: invalid escape sequence '\S'
libtidl_onnxrt_EP loaded 0x3318d330
Final number of subgraphs created are : 1, - Offloaded Nodes - 271, Total Nodes - 271
APP: Init ... !!!
  4453.042435 s: MEM: Init ... !!!
  4453.042511 s: MEM: Initialized DMA HEAP (fd=5) !!!
  4453.042702 s: MEM: Init ... Done !!!
  4453.042755 s: IPC: Init ... !!!
  4453.060100 s: IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  4453.065066 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  4453.065255 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_ERROR
  4453.065288 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_WARNING
  4453.065298 s:  VX_ZONE_INFO: Globally Enabled VX_ZONE_INFO
  4453.066472 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-0
  4453.067202 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-1
  4453.067479 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-2
  4453.067778 s:  VX_ZONE_INFO: [tivxPlatformCreateTargetId:134] Added target MPU-3
  4453.067812 s:  VX_ZONE_INFO: [tivxInitLocal:126] Initialization Done !!!
  4453.067885 s:  VX_ZONE_INFO: Globally Disabled VX_ZONE_INFO
  4453.095605 s:  VX_ZONE_ERROR: [ownContextSendCmd:912] Command ack message returned failure cmd_status: -1
  4453.095649 s:  VX_ZONE_ERROR: [ownNodeKernelInit:604] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  4453.095666 s:  VX_ZONE_ERROR: [ownNodeKernelInit:605] Please be sure the target callbacks have been registered for this core
  4453.095678 s:  VX_ZONE_ERROR: [ownNodeKernelInit:606] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  4453.095693 s:  VX_ZONE_ERROR: [ownGraphNodeKernelInit:690] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  4453.095749 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Node kernel init failed
  4453.095763 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
2025-08-01 18:30:11,895 [INFO]: survue.py(main:64) >> Starting GStreamer Pipeline
  4453.336873 s:  VX_ZONE_ERROR: [ownContextSendCmd:912] Command ack message returned failure cmd_status: -1
  4453.336916 s:  VX_ZONE_ERROR: [ownNodeKernelInit:604] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  4453.336933 s:  VX_ZONE_ERROR: [ownNodeKernelInit:605] Please be sure the target callbacks have been registered for this core
  4453.336944 s:  VX_ZONE_ERROR: [ownNodeKernelInit:606] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  4453.336959 s:  VX_ZONE_ERROR: [ownGraphNodeKernelInit:690] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  4453.336982 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Node kernel init failed
  4453.336994 s:  VX_ZONE_ERROR: [ TIDL subgraph dets ] Graph verify failed
  4453.337122 s:  VX_ZONE_ERROR: [ownGraphScheduleGraphWrapper:944] graph is not in a state required to be scheduled
  4453.337135 s:  VX_ZONE_ERROR: [vxProcessGraph:868] schedule graph failed
  4453.337146 s:  VX_ZONE_ERROR: [vxProcessGraph:873] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
2025-08-01 18:30:12.002199148 [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TIDL Compute Invoke Failed.
Exception in thread Thread-1 (pipeline):
Traceback (most recent call last):
  File "/usr/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/project-survue/source/infer_pipe.py", line 124, in pipeline
    result = self.run_time(input_img)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/project-survue/source/edgeai_dl_inferer.py", line 198, in __call__
    return self.interpreter.run(None, {self.input_name: input_img})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 217, in run
    return self._sess.run(output_names, input_feed, run_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running TIDL_0 node. Name:'TIDLExecutionProvider_TIDL_0_0' Status Message: TIDL Compute Invoke Failed.

The SOC used for inference is configured as follows - 

EVM - SK-AM62A-LP

SDK version - 10.01.00.05(Flashed using Balena Etcher)

SDK Release Date - Jan 15, 2025

SDK md5sum - dea8a075d8cabd0f207c23200672a9e5

Please let me know if there is a newer version of the SDK available as well, since the fixes in the r10.1 branch are less than a month old, whereas the SDK I am currently using is from Jan 2025. Successful inference and compilation in my dev environment leads me to believe that the compilation was successful and the problem may now potentially lie in the SDK for performing inference on the SOC. Looking forward to hearing your thoughts!

Akash

Hi Akash,

This is forward progress! Let's address what's still happening:

Akash Dhande said:

That said the logs still include the "Invalid Layer Name /multi_level_conv_cls.0/Conv_output_0" messages.

I believe this is similar to some of what was covered in another thread [0]. There is a seting called "advanced_options:output_feature_16bit_names_list" that sets a layer to run in 16-bit mode for improved accuracy. The error you see happens when a layer is included in this list, but was not found in the model. If the layer wasn't found, it will instead use the default tensor_bits setting. This specific compilation setting is specified within modelmaker -- for you model, I believe this is at [1]. 

• I will note that for tensor_bits=32, this 16-bit setting will not make any meaningful change, and you can safely ignore the error.  Once you go to tensor_bits=8, it will matter more. Main effect will be that box coordinates are more jittery.

I have noticed that newer torch versions will export an ONNX model  that uses different layer names than what we've validated in the past. The model architecture is the same, but layer names differ such that the preset 16-bit layer-names don't match your export ONNX model. Those specific layers are the last Conv layers that feed into the single-shot detection head [2]

Akash Dhande said:

I also tried performing inference on the SOC using the compiled model and was unsuccessful with the following logs. 

This is because you are compiling 32-bit mode, which is used as a reference for accuracy evaluation on PC. It is not supported on the target SoC [3]. You will need to compile your model with tensor_bits=8 or 16 for it to function on the SoC. 

[0]  SK-AM62A-LP: [edgeai-modelmaker] strange compilation run.log printout 

[1] https://github.com/TexasInstruments/edgeai-tensorlab/blob/main/edgeai-modelmaker/edgeai_modelmaker/ai_modules/vision/training/edgeai_mmdetection/detection.py#L285

[2]https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/tidl_fsg_od_meta_arch.md#onnx-ssd 

[3] https://github.com/TexasInstruments/edgeai-tidl-tools/blob/master/docs/tidl_osr_debug.md#steps-to-debug-functional-mismatch-and-accuracy-in-host-emulation 

BR,
Reese

Hi Reese,

I was able to compile my model with tensor_bits=8/16, in both cases I observed very bad accuracy with the inference on the SoC, when compared with the visualisations generated in the outputs folder while compiling. The accuracy is noticeably worse as I see next to zero predictions when performing inference on the SoC, even with low confidence thresholds for e.g. 0.2.

I noticed the lates r11.0 release of the edgeai-tensorlab repository and I have already tested my model using the same. In the latest r11 release the modelmaker completes training but fails to compile the model with the following error codes and logs.

root@instance-20250630-202809:/opt/code/edgeai-modelmaker# python3 scripts/run_modelmaker.py config_detection.yaml --target_device AM62A                                             [2/2]
argv: ['scripts/run_modelmaker.py', 'config_detection.yaml', '--target_device', 'AM62A']
---------------------------------------------------------------------
INFO: ModelMaker - task_type:detection model_name:yolox_nano_lite dataset_name:survue run_name:20250808-142402/yolox_nano_lite
- Model: yolox_nano_lite
- TargetDevices & Estimated Inference Times (ms): {'TDA4VM': 3.74, 'AM62A': 8.87, 'AM67A': '8.87 (with 1/2 device capability)', 'AM68A': 3.73, 'AM69A': '3.64 (with 1/4th device capability)', 'AM62': 516.15}
- This model can be compiled for the above device(s).
---------------------------------------------------------------------
downloading from https://software-dl.ti.com/jacinto7/esd/modelzoo/10_01_00/models/vision/detection/coco/edgeai-mmdet/yolox_nano_lite_416x416_20220214_checkpoint.pth to ./data/downloads/pretrained/yolox_nano_lite/yolox_nano_lite_416x416_20220214_checkpoint.pth
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 27605950/27605950 [00:00<00:00, 55344220.37B/s]
assuming the given download_url is a valid path: /opt/code/edgeai-modelmaker/data/projects/ec45d7a0_dataset_fix
INFO: ModelMaker - max_num_files is set to: 10000
INFO: ModelMaker - dataset split sizes are limited to: {'train': 1721, 'val': 443}
INFO: ModelMaker - dataset loading OK
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.11s)
creating index...
index created!
WARNING:20250808-142404: model_shortlist=100 - this will cause only a subset of models to be selected for run
WARNING:20250808-142404: if the model that you wish is not being selected for run, then remove this model_shortlist -
WARNING:20250808-142404: this model_shortlist could be being set in settings_base.yaml or passed inside run_benchmarks_pc.sh -
INFO: ModelMaker - run params is at: /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/run.yaml
INFO: ModelMaker - running training - for detailed info see the log file: /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training/run.log
TASKS TOTAL=1, NUM_RUNNING=0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [3:24:34<00:00, 12274.27s/it, postfix={'RUNNING': [], 'COMPLETED': ['yolox_nano_lite']}]
Trained model is at: /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training

SUCCESS: ModelMaker - Training completed.
INFO: ModelMaker - running compilation - for detailed info see the log file: /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/work/od-8200/run.log
WARNING:20250808-174839: model_shortlist=100 - this will cause only a subset of models to be selected for run
WARNING:20250808-174839: if the model that you wish is not being selected for run, then remove this model_shortlist -
WARNING:20250808-174839: this model_shortlist could be being set in settings_base.yaml or passed inside run_benchmarks_pc.sh -

INFO:20250808-174839: number of configs - 1
TASKS TOTAL=1, NUM_RUNNING=1:   0%|                                                                                                                           | 0/1 [00:02<?, ?it/s, postfix={'RUNNING': ['od-8200:import'], 'COMPLETED': []}]
ERROR:20250808-174844: Error occurred: od-8200:import - Error Code: -11 at /opt/edgeai/code/edgeai-benchmark/edgeai_benchmark/utils/parallel_runner.py
TASKS TOTAL=1, NUM_RUNNING=1:   0%|                                                                                                                            | 0/1 [00:06<?, ?it/s, postfix={'RUNNING': ['od-8200:infer'], 'COMPLETED': []}]
ERROR:20250808-174849: Error occurred: od-8200:infer - Error Code: -11 at /opt/edgeai/code/edgeai-benchmark/edgeai_benchmark/utils/parallel_runner.py
TASKS TOTAL=1, NUM_RUNNING=0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:10<00:00,  3.58s/it, postfix={'RUNNING': [], 'COMPLETED': ['od-8200']}]
WARNING: Benchmark - completed: 0/1
TASKS TOTAL=1, NUM_RUNNING=0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:10<00:00, 10.68s/it, postfix={'RUNNING': [], 'COMPLETED': ['od-8200']}]
INFO: packaging artifacts to /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/pkg please wait...
WARNING:20250808-174849: could not package - /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/work/od-8200
Traceback (most recent call last):
  File "/opt/code/edgeai-modelmaker/scripts/run_modelmaker.py", line 153, in <module>
    main(config)
  File "/opt/code/edgeai-modelmaker/scripts/run_modelmaker.py", line 88, in main
    model_runner.run()
  File "/opt/code/edgeai-modelmaker/edgeai_modelmaker/ai_modules/vision/runner.py", line 222, in run
    self.model_compilation.run()
  File "/opt/code/edgeai-modelmaker/edgeai_modelmaker/ai_modules/vision/compilation/edgeai_benchmark.py", line 164, in run
    edgeai_benchmark.interfaces.package_artifacts(self.settings, self.work_dir, out_dir=self.package_dir, custom_model=True)
  File "/opt/edgeai/code/edgeai-benchmark/edgeai_benchmark/interfaces/run_package.py", line 271, in package_artifacts
    with open(os.path.join(out_dir,'artifacts.yaml'), 'w') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/pkg/artifacts.yaml'

Upon looking deeper I found that the compilation folder was indeed missing some files and not just the artifacts.yaml. The run.log in the compilation folder reveals the following - 

INFO:20250811-123507: starting - od-8200

INFO:20250811-123507: running - od-8200

INFO:20250811-123507: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7f64f7046860>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7f64f70467d0>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7f64f6a2bd30>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7f64f6a2bd90>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7f64f6a380d0>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-nano-lite-mmdet-coco-416x416', 'shortlisted': True, 'recommended': True}}

INFO:20250811-123507: import  - od-8200 - this may take some time...
INFO:20250811-123507: model_path - /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training/model.onnx
INFO:20250811-123507: model_file - /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/work/od-8200/model/model.onnx
INFO:20250811-123507: quant_file - /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/work/od-8200/model/model_qparams.prototxt
Downloading 1/1: /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training/model.onnx
Download done for /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training/model.onnx
Downloading 1/1: /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training/model.onnx
Download done for /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training/model.onnx
Error -   /opt/edgeai/code/edgeai-benchmark/tools/tidl_tools_package/AM62A/tidl_tools/tidl_model_import_onnx.so: cannot open shared object file: No such file or directory 

INFO:20250811-123513: starting - od-8200

INFO:20250811-123513: running - od-8200

INFO:20250811-123513: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7f64f7046860>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7f64f70467d0>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7f64f6a2bd30>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7f64f6a2bd90>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7f64f6a380d0>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-nano-lite-mmdet-coco-416x416', 'shortlisted': True, 'recommended': True}}

INFO:20250811-123513: infer  - od-8200 - this may take some time...
INFO:20250811-123513: model_path - /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/training/model.onnx
INFO:20250811-123513: model_file - /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/work/od-8200/model/model.onnx
INFO:20250811-123513: quant_file - /opt/code/edgeai-modelmaker/data/projects/survue/run/20250808-142402/yolox_nano_lite/compilation/work/od-8200/model/model_qparams.prototxt
Error -   libtidl_onnxrt_EP.so: cannot open shared object file: No such file or directory 
libtidl_onnxrt_EP loaded (nil) 

While setting up the docker environment I am using for my development, the edgeai-benchmark fails to download tidl_tools for AM62A SoC specifically, while it works for other SoCs. The url used for the download in the setup scripts provided is - 

http://software-dl.ti.com/jacinto7/esd/tidl-tools/11_00_08_00/TIDL_TOOLS/AM62A/tidl_tools_gpu.tar.gz --> This URL seems to be not working.

I would appreciate a prompt response as it seems like a quick fix to me. Any public URL which contains the tidl_tools_gpu.tar.gz for AM62A v11_00_08_00 would be perfect.

Knowing your team is still developing and bugfixing on the edgeai-tensorlab repository, I would also suggest addition of some test cases to make sure all the provided URLs are functioning.

Best,

Akash

Hello Akash,

AM62A did not have an 11.0 SDK release. As a result, the automated code for pulling/applying the tidl_tools path download did not find tools for this specific SoC. 

Akash Dhande said:

http://software-dl.ti.com/jacinto7/esd/tidl-tools/11_00_08_00/TIDL_TOOLS/AM62A/tidl_tools_gpu.tar.gz --> This URL seems to be not working.

Correct; it unfortunately does not exist. 

Instead, there would be an 11_00_07_00 release, which produces model-artifacts that are backwards compatible to the 10.1 SDK. 

The way to apply the changes for 11_00_07_00 into the 10.1 SDK are covered within edgeai-tidl-tools (update_target.md) [0]. You can download the tidl-tools for this version and place it in the same tools/AM62A/tidl_tools location, and try compiling from there. You will need to follow the documentation [0] for how to update the target before using the resulting artifacts. 

[0] https://github.com/TexasInstruments/edgeai-tidl-tools/blob/11_00_07_00/docs/update_target.md 

Hello Reese,

I am seeing a SDK release for AM62A, which seems to be a 11.0 SDK release as shown in the screenshot below - 

Please do clarify what versions/branches of edgeai-tensorlab should I be using for the above latest AM62A SDK to train my models with.

Hi Akash,

This is 11.01 SDK.  Our version strings follow convention of two characters per version component, so 11.1 and 11.01 mean the same. Period is the important delimiter. 

edgeai-tensorlab typically lags behind SDK release by several weeks. AM62A had an earlier release than other Edge AI / ADAS processors. The tensorlab update will not come out until all devices in the Edge AI category are ready. 11.1 TIDL tools are also not available for AM62A.

My suggestion would be to train with r11.0 branch. However, you should first replace tools/AM62A/tidl_tools (which failed to setup before) with 11.00.07 tidl tools or 10.01.04 tools, and use these with the 10.1 SDK for now.

BR,
Reese