This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: pytorch-mmdetection with pytorch-jacinto-ai setup failed

Part Number: TDA4VM

I installed pytorch-jacinto-ai mmcv and pytorch-mmdetection. but when i run ./run_detection_train.sh, get ImportError. 

# hy @ hy-Z390-GAMING-X in ~/lyw/pytorch-mmdetection on git:master x [15:53:42] 
$ ./run_detection_train.sh
Training with: ./configs/ssd/ssd-lite_regnet_fpn_bgr.py @ 2021-01-14_15-59-49
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmcv/utils/registry.py:64: UserWarning: The old API of register_module(module, force=False) is deprecated and will be removed, please use the new API register_module(name=None, force=False, module=None) instead.
  'The old API of register_module(module, force=False) '
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmcv/utils/registry.py:64: UserWarning: The old API of register_module(module, force=False) is deprecated and will be removed, please use the new API register_module(name=None, force=False, module=None) instead.
  'The old API of register_module(module, force=False) '
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmcv/utils/registry.py:64: UserWarning: The old API of register_module(module, force=False) is deprecated and will be removed, please use the new API register_module(name=None, force=False, module=None) instead.
  'The old API of register_module(module, force=False) '
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmcv/utils/registry.py:64: UserWarning: The old API of register_module(module, force=False) is deprecated and will be removed, please use the new API register_module(name=None, force=False, module=None) instead.
  'The old API of register_module(module, force=False) '
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/fp16/deprecated_fp16_utils.py:28: UserWarning: Importing auto_fp16 from "mmdet.core" will be deprecated in the future. Please import them from "mmcv.runner" instead
  'Importing auto_fp16 from "mmdet.core" will be '
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/fp16/deprecated_fp16_utils.py:28: UserWarning: Importing auto_fp16 from "mmdet.core" will be deprecated in the future. Please import them from "mmcv.runner" instead
  'Importing auto_fp16 from "mmdet.core" will be '
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/fp16/deprecated_fp16_utils.py:28: UserWarning: Importing auto_fp16 from "mmdet.core" will be deprecated in the future. Please import them from "mmcv.runner" instead
  'Importing auto_fp16 from "mmdet.core" will be '
/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/fp16/deprecated_fp16_utils.py:28: UserWarning: Importing auto_fp16 from "mmdet.core" will be deprecated in the future. Please import them from "mmcv.runner" instead
  'Importing auto_fp16 from "mmdet.core" will be '
Traceback (most recent call last):
  File "./xmmdet/tools/train.py", line 16, in <module>
Traceback (most recent call last):
  File "./xmmdet/tools/train.py", line 16, in <module>
Traceback (most recent call last):
  File "./xmmdet/tools/train.py", line 16, in <module>
Traceback (most recent call last):
    from xmmdet.apis import set_random_seed, train_detector  File "./xmmdet/tools/train.py", line 16, in <module>

  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/__init__.py", line 7, in <module>
    from xmmdet.apis import set_random_seed, train_detector
  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/__init__.py", line 7, in <module>
    from .apis import *
    from xmmdet.apis import set_random_seed, train_detector  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/__init__.py", line 4, in <module>

    from .apis import *  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/__init__.py", line 7, in <module>

  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/__init__.py", line 4, in <module>
    from xmmdet.apis import set_random_seed, train_detector
      File "/home/hy/lyw/pytorch-mmdetection/xmmdet/__init__.py", line 7, in <module>
from .test import multi_gpu_test, single_gpu_test
  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/test.py", line 12, in <module>
        from .apis import *from .test import multi_gpu_test, single_gpu_test

  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/__init__.py", line 4, in <module>
  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/test.py", line 12, in <module>
        from mmdet.core import encode_mask_results, tensor2imgsfrom .apis import *

  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/__init__.py", line 4, in <module>
ImportError: cannot import name 'tensor2imgs' from 'mmdet.core' (/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/__init__.py)
        from .test import multi_gpu_test, single_gpu_testfrom mmdet.core import encode_mask_results, tensor2imgs

  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/test.py", line 12, in <module>
ImportError: cannot import name 'tensor2imgs' from 'mmdet.core' (/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/__init__.py)
    from .test import multi_gpu_test, single_gpu_test
  File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/test.py", line 12, in <module>
    from mmdet.core import encode_mask_results, tensor2imgs
ImportError: cannot import name 'tensor2imgs' from 'mmdet.core' (/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/__init__.py)
    from mmdet.core import encode_mask_results, tensor2imgs
ImportError: cannot import name 'tensor2imgs' from 'mmdet.core' (/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/__init__.py)
Traceback (most recent call last):
  File "./scripts/train_detection_main.py", line 30, in <module>
    distributed_launch.main()
  File "/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/hy/anaconda3/envs/lyw/bin/python', '-u', './xmmdet/tools/train.py', '--local_rank=3', '--launcher=pytorch', './configs/ssd/ssd-lite_regnet_fpn_bgr.py']' returned non-zero exit status 1.
Done.

  • The import line in the error message is: ImportError: cannot import name 'tensor2imgs' from 'mmdet.core' (/home/hy/anaconda3/envs/lyw/lib/python3.7/site-packages/mmdet/core/__init__.py)

    This is error is because mmdetection has changed, but we have not updated our extension to keep up with the changes in mmdetection (we shall do that soon). For now, can you try an older version of mmdetection (and the corresponding version of mmcv).

    I am not able to recollect the exact version of mmdetection that matches with our extension try, but you can try the versions from 2.4.0 onwards.

    Let us know if this helps or not.

    Best regards,

    Manu.

  • hi,I have soled the issue  by changing the code in /xmmdet/apis/test.py 

    from mmdet.core import encode_mask_results#, tensor2imgs
    from mmcv import tensor2imgs

    but it get an anther Error:

    subprocess.CalledProcessError: Command '['/home/hy/miniconda3/envs/lyw/bin/python', '-u', './xmmdet/tools/train.py', '--local_rank=1', '--launcher=pytorch', './configs/ssd/ssd-lite_regnet_fpn_bgr.py']' returned non-zero exit status 1.
    Done.
    (lyw) hy@hy:~/lyw/pytorch-mmdetection$ Traceback (most recent call last):
      File "./xmmdet/tools/train.py", line 201, in <module>
        main()
      File "./xmmdet/tools/train.py", line 197, in main
        meta=meta)
      File "/home/hy/lyw/pytorch-mmdetection/xmmdet/apis/train.py", line 146, in train_detector
        runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
        epoch_runner(data_loaders[i], **kwargs)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
        self.run_iter(data_batch, train_mode=True)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter
        **kwargs)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmcv/parallel/distributed.py", line 46, in train_step
        output = self.module.train_step(*inputs[0], **kwargs[0])
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmdet/models/detectors/base.py", line 247, in train_step
        losses = self(**data)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
        result = self.forward(*input, **kwargs)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
        return old_func(*args, **kwargs)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmdet/models/detectors/base.py", line 181, in forward
        return self.forward_train(img, img_metas, **kwargs)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmdet/models/detectors/single_stage.py", line 95, in forward_train
        gt_labels, gt_bboxes_ignore)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmdet/models/dense_heads/base_dense_head.py", line 54, in forward_train
        losses = self.loss(*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 164, in new_func
        return old_func(*args, **kwargs)
      File "/home/hy/miniconda3/envs/lyw/lib/python3.7/site-packages/mmdet/models/dense_heads/ssd_head.py", line 245, in loss
        'classification scores become infinite or NaN!'
    AssertionError: classification scores become infinite or NaN!

    my training log:

    2021-01-19 17:19:49,972 - mmdet - INFO - Environment info:
    ------------------------------------------------------------
    sys.platform: linux
    Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0]
    CUDA available: True
    GPU 0,1: GeForce RTX 3090
    CUDA_HOME: /usr/local/cuda
    NVCC: Build cuda_11.1.TC455_06.29069683_0
    GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
    PyTorch: 1.7.1
    PyTorch compiling details: PyTorch built with:
      - GCC 7.3
      - C++ Version: 201402
      - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
      - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
      - OpenMP 201511 (a.k.a. OpenMP 4.5)
      - NNPACK is enabled
      - CPU capability usage: AVX2
      - CUDA Runtime 11.0
      - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_37,code=compute_37
      - CuDNN 8.0.5
      - Magma 2.5.2
      - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 
    
    TorchVision: 0.8.2
    OpenCV: 4.5.1
    MMCV: 1.2.5
    MMCV Compiler: GCC 7.5
    MMCV CUDA Compiler: 11.1
    MMDetection: 2.8.0+d754380
    ------------------------------------------------------------
    
    2021-01-19 17:19:51,197 - mmdet - INFO - Distributed training: True
    2021-01-19 17:19:52,397 - mmdet - INFO - Config:
    TBD = None
    img_norm_cfg = dict(
        mean=[103.53, 116.28, 123.675], std=[57.375, 57.12, 58.395], to_rgb=False)
    train_pipeline = [
        dict(type='LoadImageFromFile', to_float32=True),
        dict(type='LoadAnnotations', with_bbox=True),
        dict(
            type='PhotoMetricDistortion',
            brightness_delta=32,
            contrast_range=(0.5, 1.5),
            saturation_range=(0.5, 1.5),
            hue_delta=18),
        dict(
            type='Expand',
            mean=[103.53, 116.28, 123.675],
            to_rgb=False,
            ratio_range=(1, 4)),
        dict(
            type='MinIoURandomCrop',
            min_ious=(0.1, 0.3, 0.5, 0.7, 0.9),
            min_crop_size=0.3),
        dict(type='Resize', img_scale=(512, 512), keep_ratio=False),
        dict(
            type='Normalize',
            mean=[103.53, 116.28, 123.675],
            std=[57.375, 57.12, 58.395],
            to_rgb=False),
        dict(type='RandomFlip', flip_ratio=0.5),
        dict(type='Pad', size_divisor=32),
        dict(type='DefaultFormatBundle'),
        dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
    ]
    test_pipeline = [
        dict(type='LoadImageFromFile'),
        dict(
            type='MultiScaleFlipAug',
            img_scale=(512, 512),
            flip=False,
            transforms=[
                dict(type='Resize', keep_ratio=False),
                dict(type='RandomFlip'),
                dict(
                    type='Normalize',
                    mean=[103.53, 116.28, 123.675],
                    std=[57.375, 57.12, 58.395],
                    to_rgb=False),
                dict(type='Pad', size_divisor=32),
                dict(type='ImageToTensor', keys=['img']),
                dict(type='Collect', keys=['img'])
            ])
    ]
    dataset_type = 'VOCDataset'
    data_root = 'data/sjt_highway_hycoco_dataet_mix_input_analyze'
    dataset_repeats = 10
    data = dict(
        samples_per_gpu=68,
        workers_per_gpu=0,
        train=dict(
            type='RepeatDataset',
            times=10,
            dataset=dict(
                type='VOCDataset',
                ann_file=[
                    'data/sjt_highway_hycoco_dataet_mix_input_analyze/ImageSets/Main/train0.txt'
                ],
                img_prefix=['data/sjt_highway_hycoco_dataet_mix_input_analyze'],
                pipeline=[
                    dict(type='LoadImageFromFile', to_float32=True),
                    dict(type='LoadAnnotations', with_bbox=True),
                    dict(
                        type='PhotoMetricDistortion',
                        brightness_delta=32,
                        contrast_range=(0.5, 1.5),
                        saturation_range=(0.5, 1.5),
                        hue_delta=18),
                    dict(
                        type='Expand',
                        mean=[103.53, 116.28, 123.675],
                        to_rgb=False,
                        ratio_range=(1, 4)),
                    dict(
                        type='MinIoURandomCrop',
                        min_ious=(0.1, 0.3, 0.5, 0.7, 0.9),
                        min_crop_size=0.3),
                    dict(type='Resize', img_scale=(512, 512), keep_ratio=False),
                    dict(
                        type='Normalize',
                        mean=[103.53, 116.28, 123.675],
                        std=[57.375, 57.12, 58.395],
                        to_rgb=False),
                    dict(type='RandomFlip', flip_ratio=0.5),
                    dict(type='Pad', size_divisor=32),
                    dict(type='DefaultFormatBundle'),
                    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
                ])),
        val=dict(
            type='VOCDataset',
            ann_file=
            'data/sjt_highway_hycoco_dataet_mix_input_analyze/ImageSets/Main/test0.txt',
            img_prefix='data/sjt_highway_hycoco_dataet_mix_input_analyze',
            pipeline=[
                dict(type='LoadImageFromFile'),
                dict(
                    type='MultiScaleFlipAug',
                    img_scale=(512, 512),
                    flip=False,
                    transforms=[
                        dict(type='Resize', keep_ratio=False),
                        dict(type='RandomFlip'),
                        dict(
                            type='Normalize',
                            mean=[103.53, 116.28, 123.675],
                            std=[57.375, 57.12, 58.395],
                            to_rgb=False),
                        dict(type='Pad', size_divisor=32),
                        dict(type='ImageToTensor', keys=['img']),
                        dict(type='Collect', keys=['img'])
                    ])
            ]),
        test=dict(
            type='VOCDataset',
            ann_file=
            'data/sjt_highway_hycoco_dataet_mix_input_analyze/ImageSets/Main/val.txt',
            img_prefix='data/sjt_highway_hycoco_dataet_mix_input_analyze',
            pipeline=[
                dict(type='LoadImageFromFile'),
                dict(
                    type='MultiScaleFlipAug',
                    img_scale=(512, 512),
                    flip=False,
                    transforms=[
                        dict(type='Resize', keep_ratio=False),
                        dict(type='RandomFlip'),
                        dict(
                            type='Normalize',
                            mean=[103.53, 116.28, 123.675],
                            std=[57.375, 57.12, 58.395],
                            to_rgb=False),
                        dict(type='Pad', size_divisor=32),
                        dict(type='ImageToTensor', keys=['img']),
                        dict(type='Collect', keys=['img'])
                    ])
            ]))
    evaluation = dict(interval=1, metric='mAP')
    cudnn_benchmark = True
    checkpoint_config = dict(interval=1)
    log_config = dict(interval=100, hooks=[dict(type='TextLoggerHook')])
    dist_params = dict(backend='nccl')
    log_level = 'INFO'
    load_from = None
    resume_from = None
    workflow = [('train', 1)]
    print_model_complexity = True
    train_cfg = dict(
        assigner=dict(
            type='MaxIoUAssigner',
            pos_iou_thr=0.5,
            neg_iou_thr=0.5,
            min_pos_iou=0.0,
            ignore_iof_thr=-1,
            gt_max_assign_all=False),
        smoothl1_beta=1.0,
        allowed_border=-1,
        pos_weight=-1,
        neg_pos_ratio=3,
        debug=False)
    test_cfg = dict(
        nms=dict(type='nms', iou_thr=0.45),
        min_bbox_size=0,
        score_thr=0.02,
        max_per_img=200)
    optimizer = dict(type='SGD', lr=0.001, momentum=0.9, weight_decay=4e-05)
    optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
    warmup_cfg = dict(warmup='linear', warmup_iters=1000, warmup_ratio=0.001)
    lr_config = dict(
        policy='CosineAnnealing',
        min_lr_ratio=0.0001,
        warmup='linear',
        warmup_iters=1000,
        warmup_ratio=0.001)
    total_epochs = 120
    input_size = (512, 512)
    num_classes_dict = dict(CocoDataset=80, VOCDataset=13, CityscapesDataset=8)
    num_classes = 13
    quantize = False
    initial_learning_rate = 0.001
    samples_per_gpu = 68
    backbone_type = 'RegNet'
    backbone_arch = 'regnetx_800mf'
    to_rgb = False
    decoder_fpn_type = 'FPNLite'
    decoder_conv_type = 'ConvDWSep'
    fpn_width_fact = 4
    decoder_width_fact = 4
    decoder_depth_fact = 4
    regnet_settings = dict({
        'regnetx_200mf':
        dict(
            bacbone_out_channels=[32, 56, 152, 368],
            group_size_dw=8,
            fpn_intermediate_channels=256,
            fpn_out_channels=256,
            fpn_num_blocks=4,
            pretrained='open-mmlab://regnetx_200mf'),
        'regnetx_400mf':
        dict(
            bacbone_out_channels=[32, 64, 160, 384],
            group_size_dw=16,
            fpn_intermediate_channels=256,
            fpn_out_channels=256,
            fpn_num_blocks=4,
            pretrained='open-mmlab://regnetx_400mf'),
        'regnetx_800mf':
        dict(
            bacbone_out_channels=[64, 128, 288, 672],
            group_size_dw=16,
            fpn_intermediate_channels=256,
            fpn_out_channels=256,
            fpn_num_blocks=4,
            pretrained='open-mmlab://regnetx_800mf'),
        'regnetx_1.6gf':
        dict(
            bacbone_out_channels=[72, 168, 408, 912],
            group_size_dw=24,
            fpn_intermediate_channels=264,
            fpn_out_channels=264,
            fpn_num_blocks=4,
            pretrained='open-mmlab://regnetx_1.6gf'),
        'regnetx_3.2gf':
        dict(
            bacbone_out_channels=[96, 192, 432, 1008],
            group_size_dw=48,
            fpn_intermediate_channels=288,
            fpn_out_channels=288,
            fpn_num_blocks=4,
            pretrained='open-mmlab://regnetx_3.2gf')
    })
    regnet_cfg = dict(
        bacbone_out_channels=[64, 128, 288, 672],
        group_size_dw=16,
        fpn_intermediate_channels=256,
        fpn_out_channels=256,
        fpn_num_blocks=4,
        pretrained='open-mmlab://regnetx_800mf')
    pretrained = 'open-mmlab://regnetx_800mf'
    bacbone_out_channels = [64, 128, 288, 672]
    backbone_out_indices = (0, 1, 2, 3)
    fpn_in_channels = [64, 128, 288, 672]
    fpn_out_channels = 256
    fpn_start_level = 1
    fpn_num_outs = 6
    fpn_upsample_mode = 'bilinear'
    fpn_upsample_cfg = dict(scale_factor=2, mode='bilinear')
    fpn_num_blocks = 4
    fpn_intermediate_channels = 256
    fpn_bifpn_cfg = dict()
    fpn_add_extra_convs = 'on_input'
    basesize_ratio_range = (0.1, 0.9)
    input_size_divisor = 32
    conv_cfg = dict(type='ConvDWSep', group_size_dw=16)
    norm_cfg = dict(type='BN')
    model = dict(
        type='SingleStageDetector',
        pretrained='open-mmlab://regnetx_800mf',
        backbone=dict(
            type='RegNet',
            arch='regnetx_800mf',
            out_indices=(0, 1, 2, 3),
            norm_eval=False,
            style='pytorch'),
        neck=dict(
            type='FPNLite',
            in_channels=[64, 128, 288, 672],
            out_channels=256,
            start_level=1,
            num_outs=6,
            add_extra_convs='on_input',
            upsample_cfg=dict(scale_factor=2, mode='bilinear'),
            conv_cfg=dict(type='ConvDWSep', group_size_dw=16),
            norm_cfg=dict(type='BN')),
        bbox_head=dict(
            type='SSDLiteHead',
            in_channels=[256, 256, 256, 256, 256, 256],
            num_classes=13,
            conv_cfg=dict(type='ConvDWSep', group_size_dw=16),
            anchor_generator=dict(
                type='SSDAnchorGenerator',
                scale_major=False,
                input_size=(512, 512),
                basesize_ratio_range=(0.1, 0.9),
                strides=[8, 16, 32, 64, 128, 256],
                ratios=[[2], [2, 3], [2, 3], [2, 3], [2, 3], [2]]),
            bbox_coder=dict(
                type='DeltaXYWHBBoxCoder',
                target_means=[0.0, 0.0, 0.0, 0.0],
                target_stds=[0.1, 0.1, 0.2, 0.2])))
    work_dir = './work_dirs/ssd-lite_regnet_fpn_bgr'
    gpu_ids = range(0, 1)
    
    2021-01-19 17:19:52,695 - mmdet - INFO - load model from: open-mmlab://regnetx_800mf
    2021-01-19 17:19:52,731 - mmdet - WARNING - The model and loaded state dict do not match exactly
    
    unexpected key in source state_dict: fc.weight, fc.bias
    
    2021-01-19 17:19:53,196 - mmdet - INFO - SingleStageDetector(
      5.611 GMac, 100.000% MACs, 
      (backbone): RegNet(
        4.253 GMac, 75.800% MACs, 
        (conv1): Conv2d(0.057 GMac, 1.009% MACs, 3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn1): BatchNorm2d(0.004 GMac, 0.075% MACs, 32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(0.002 GMac, 0.037% MACs, inplace=True)
        (layer1): ResLayer(
          0.407 GMac, 7.251% MACs, 
          (0): Bottleneck(
            0.407 GMac, 7.251% MACs, 
            (conv1): Conv2d(0.134 GMac, 2.392% MACs, 32, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.008 GMac, 0.150% MACs, 64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.151 GMac, 2.691% MACs, 64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=4, bias=False)
            (bn2): BatchNorm2d(0.002 GMac, 0.037% MACs, 64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.067 GMac, 1.196% MACs, 64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.002 GMac, 0.037% MACs, 64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.004 GMac, 0.075% MACs, inplace=True)
            (relu2): ReLU(0.001 GMac, 0.019% MACs, inplace=True)
            (relu3): ReLU(0.001 GMac, 0.019% MACs, inplace=True)
            (downsample): Sequential(
              0.036 GMac, 0.635% MACs, 
              (0): Conv2d(0.034 GMac, 0.598% MACs, 32, 64, kernel_size=(1, 1), stride=(2, 2), bias=False)
              (1): BatchNorm2d(0.002 GMac, 0.037% MACs, 64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (add): AddBlock(inplace=False, signed=True)
          )
        )
        (layer2): ResLayer(
          0.75 GMac, 13.362% MACs, 
          (0): Bottleneck(
            0.321 GMac, 5.718% MACs, 
            (conv1): Conv2d(0.134 GMac, 2.392% MACs, 64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.004 GMac, 0.075% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.075 GMac, 1.346% MACs, 128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=8, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.067 GMac, 1.196% MACs, 128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.002 GMac, 0.037% MACs, inplace=True)
            (relu2): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (relu3): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (downsample): Sequential(
              0.035 GMac, 0.617% MACs, 
              (0): Conv2d(0.034 GMac, 0.598% MACs, 64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
              (1): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (add): AddBlock(inplace=False, signed=True)
          )
          (1): Bottleneck(
            0.214 GMac, 3.822% MACs, 
            (conv1): Conv2d(0.067 GMac, 1.196% MACs, 128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.075 GMac, 1.346% MACs, 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=8, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.067 GMac, 1.196% MACs, 128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (relu2): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (relu3): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (2): Bottleneck(
            0.214 GMac, 3.822% MACs, 
            (conv1): Conv2d(0.067 GMac, 1.196% MACs, 128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.075 GMac, 1.346% MACs, 128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=8, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.067 GMac, 1.196% MACs, 128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.019% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (relu2): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (relu3): ReLU(0.001 GMac, 0.009% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
        )
        (layer3): ResLayer(
          1.612 GMac, 28.729% MACs, 
          (0): Bottleneck(
            0.322 GMac, 5.739% MACs, 
            (conv1): Conv2d(0.151 GMac, 2.691% MACs, 128, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.002 GMac, 0.042% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.042 GMac, 0.757% MACs, 288, 288, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=18, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.001 GMac, 0.021% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (downsample): Sequential(
              0.038 GMac, 0.683% MACs, 
              (0): Conv2d(0.038 GMac, 0.673% MACs, 128, 288, kernel_size=(1, 1), stride=(2, 2), bias=False)
              (1): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (add): AddBlock(inplace=False, signed=True)
          )
          (1): Bottleneck(
            0.215 GMac, 3.832% MACs, 
            (conv1): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.042 GMac, 0.757% MACs, 288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=18, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (2): Bottleneck(
            0.215 GMac, 3.832% MACs, 
            (conv1): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.042 GMac, 0.757% MACs, 288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=18, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (3): Bottleneck(
            0.215 GMac, 3.832% MACs, 
            (conv1): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.042 GMac, 0.757% MACs, 288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=18, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (4): Bottleneck(
            0.215 GMac, 3.832% MACs, 
            (conv1): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.042 GMac, 0.757% MACs, 288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=18, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (5): Bottleneck(
            0.215 GMac, 3.832% MACs, 
            (conv1): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.042 GMac, 0.757% MACs, 288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=18, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (6): Bottleneck(
            0.215 GMac, 3.832% MACs, 
            (conv1): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.042 GMac, 0.757% MACs, 288, 288, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=18, bias=False)
            (bn2): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.085 GMac, 1.514% MACs, 288, 288, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.001 GMac, 0.011% MACs, 288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.005% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
        )
        (layer4): ResLayer(
          1.422 GMac, 25.337% MACs, 
          (0): Bottleneck(
            0.392 GMac, 6.978% MACs, 
            (conv1): Conv2d(0.198 GMac, 3.532% MACs, 288, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.001 GMac, 0.025% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.025 GMac, 0.441% MACs, 672, 672, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=42, bias=False)
            (bn2): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.001 GMac, 0.012% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (downsample): Sequential(
              0.05 GMac, 0.889% MACs, 
              (0): Conv2d(0.05 GMac, 0.883% MACs, 288, 672, kernel_size=(1, 1), stride=(2, 2), bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
            (add): AddBlock(inplace=False, signed=True)
          )
          (1): Bottleneck(
            0.258 GMac, 4.590% MACs, 
            (conv1): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.025 GMac, 0.441% MACs, 672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=42, bias=False)
            (bn2): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (2): Bottleneck(
            0.258 GMac, 4.590% MACs, 
            (conv1): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.025 GMac, 0.441% MACs, 672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=42, bias=False)
            (bn2): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (3): Bottleneck(
            0.258 GMac, 4.590% MACs, 
            (conv1): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.025 GMac, 0.441% MACs, 672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=42, bias=False)
            (bn2): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
          (4): Bottleneck(
            0.258 GMac, 4.590% MACs, 
            (conv1): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn1): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv2): Conv2d(0.025 GMac, 0.441% MACs, 672, 672, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=42, bias=False)
            (bn2): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (conv3): Conv2d(0.116 GMac, 2.060% MACs, 672, 672, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn3): BatchNorm2d(0.0 GMac, 0.006% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (relu1): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu2): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (relu3): ReLU(0.0 GMac, 0.003% MACs, inplace=True)
            (add): AddBlock(inplace=False, signed=True)
          )
        )
      )
      (neck): FPNLite(
        0.833 GMac, 14.852% MACs, 
        (lateral_convs): ModuleList(
          0.257 GMac, 4.571% MACs, 
          (0): ConvNormAct2d(
            0.136 GMac, 2.429% MACs, 
            (0): Conv2d(0.134 GMac, 2.392% MACs, 128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (1): BatchNorm2d(0.002 GMac, 0.037% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (1): ConvNormAct2d(
            0.076 GMac, 1.355% MACs, 
            (0): Conv2d(0.075 GMac, 1.346% MACs, 288, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (1): BatchNorm2d(0.001 GMac, 0.009% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (2): ConvNormAct2d(
            0.044 GMac, 0.787% MACs, 
            (0): Conv2d(0.044 GMac, 0.785% MACs, 672, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (1): BatchNorm2d(0.0 GMac, 0.002% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (fpn_convs): ModuleList(
          0.577 GMac, 10.280% MACs, 
          (0): ConvDWSep2d(
            0.425 GMac, 7.569% MACs, 
            (0): Sequential(
              0.154 GMac, 2.747% MACs, 
              (0): Conv2d(0.151 GMac, 2.691% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.002 GMac, 0.037% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.001 GMac, 0.019% MACs, )
            )
            (1): Sequential(
              0.271 GMac, 4.821% MACs, 
              (0): Conv2d(0.268 GMac, 4.784% MACs, 256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(0.002 GMac, 0.037% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
          )
          (1): ConvDWSep2d(
            0.106 GMac, 1.892% MACs, 
            (0): Sequential(
              0.039 GMac, 0.687% MACs, 
              (0): Conv2d(0.038 GMac, 0.673% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.001 GMac, 0.009% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.005% MACs, )
            )
            (1): Sequential(
              0.068 GMac, 1.205% MACs, 
              (0): Conv2d(0.067 GMac, 1.196% MACs, 256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(0.001 GMac, 0.009% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
          )
          (2): ConvDWSep2d(
            0.027 GMac, 0.473% MACs, 
            (0): Sequential(
              0.01 GMac, 0.172% MACs, 
              (0): Conv2d(0.009 GMac, 0.168% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.002% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.001% MACs, )
            )
            (1): Sequential(
              0.017 GMac, 0.301% MACs, 
              (0): Conv2d(0.017 GMac, 0.299% MACs, 256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.002% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
          )
          (3): ConvDWSep2d(
            0.017 GMac, 0.309% MACs, 
            (0): Sequential(
              0.006 GMac, 0.113% MACs, 
              (0): Conv2d(0.006 GMac, 0.110% MACs, 672, 672, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=42, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.002% MACs, 672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.001% MACs, )
            )
            (1): Sequential(
              0.011 GMac, 0.197% MACs, 
              (0): Conv2d(0.011 GMac, 0.196% MACs, 672, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.001% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
          )
          (4): ConvDWSep2d(
            0.002 GMac, 0.030% MACs, 
            (0): Sequential(
              0.001 GMac, 0.011% MACs, 
              (0): Conv2d(0.001 GMac, 0.011% MACs, 256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.001 GMac, 0.019% MACs, 
              (0): Conv2d(0.001 GMac, 0.019% MACs, 256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
          )
          (5): ConvDWSep2d(
            0.0 GMac, 0.007% MACs, 
            (0): Sequential(
              0.0 GMac, 0.003% MACs, 
              (0): Conv2d(0.0 GMac, 0.003% MACs, 256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.0 GMac, 0.005% MACs, 
              (0): Conv2d(0.0 GMac, 0.005% MACs, 256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            )
          )
        )
        (adds): ModuleList(
          0.0 GMac, 0.000% MACs, 
          (0): AddBlock(inplace=False, signed=True)
          (1): AddBlock(inplace=False, signed=True)
          (2): AddBlock(inplace=False, signed=True)
        )
      )
      (bbox_head): SSDLiteHead(
        0.525 GMac, 9.349% MACs, 
        (reg_convs): ModuleList(
          0.231 GMac, 4.112% MACs, 
          (0): ConvDWSep2d(
            0.171 GMac, 3.047% MACs, 
            (0): Sequential(
              0.154 GMac, 2.747% MACs, 
              (0): Conv2d(0.151 GMac, 2.691% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.002 GMac, 0.037% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.001 GMac, 0.019% MACs, )
            )
            (1): Sequential(
              0.017 GMac, 0.300% MACs, 
              (0): Conv2d(0.017 GMac, 0.300% MACs, 256, 16, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (1): ConvDWSep2d(
            0.045 GMac, 0.799% MACs, 
            (0): Sequential(
              0.039 GMac, 0.687% MACs, 
              (0): Conv2d(0.038 GMac, 0.673% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.001 GMac, 0.009% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.005% MACs, )
            )
            (1): Sequential(
              0.006 GMac, 0.113% MACs, 
              (0): Conv2d(0.006 GMac, 0.113% MACs, 256, 24, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (2): ConvDWSep2d(
            0.011 GMac, 0.200% MACs, 
            (0): Sequential(
              0.01 GMac, 0.172% MACs, 
              (0): Conv2d(0.009 GMac, 0.168% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.002% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.001% MACs, )
            )
            (1): Sequential(
              0.002 GMac, 0.028% MACs, 
              (0): Conv2d(0.002 GMac, 0.028% MACs, 256, 24, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (3): ConvDWSep2d(
            0.003 GMac, 0.050% MACs, 
            (0): Sequential(
              0.002 GMac, 0.043% MACs, 
              (0): Conv2d(0.002 GMac, 0.042% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.001% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.0 GMac, 0.007% MACs, 
              (0): Conv2d(0.0 GMac, 0.007% MACs, 256, 24, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (4): ConvDWSep2d(
            0.001 GMac, 0.012% MACs, 
            (0): Sequential(
              0.001 GMac, 0.011% MACs, 
              (0): Conv2d(0.001 GMac, 0.011% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.0 GMac, 0.002% MACs, 
              (0): Conv2d(0.0 GMac, 0.002% MACs, 256, 24, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (5): ConvDWSep2d(
            0.0 GMac, 0.003% MACs, 
            (0): Sequential(
              0.0 GMac, 0.003% MACs, 
              (0): Conv2d(0.0 GMac, 0.003% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.0 GMac, 0.000% MACs, 
              (0): Conv2d(0.0 GMac, 0.000% MACs, 256, 16, kernel_size=(1, 1), stride=(1, 1))
            )
          )
        )
        (cls_convs): ModuleList(
          0.294 GMac, 5.237% MACs, 
          (0): ConvDWSep2d(
            0.213 GMac, 3.798% MACs, 
            (0): Sequential(
              0.154 GMac, 2.747% MACs, 
              (0): Conv2d(0.151 GMac, 2.691% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.002 GMac, 0.037% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.001 GMac, 0.019% MACs, )
            )
            (1): Sequential(
              0.059 GMac, 1.051% MACs, 
              (0): Conv2d(0.059 GMac, 1.051% MACs, 256, 56, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (1): ConvDWSep2d(
            0.061 GMac, 1.081% MACs, 
            (0): Sequential(
              0.039 GMac, 0.687% MACs, 
              (0): Conv2d(0.038 GMac, 0.673% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.001 GMac, 0.009% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.005% MACs, )
            )
            (1): Sequential(
              0.022 GMac, 0.394% MACs, 
              (0): Conv2d(0.022 GMac, 0.394% MACs, 256, 84, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (2): ConvDWSep2d(
            0.015 GMac, 0.270% MACs, 
            (0): Sequential(
              0.01 GMac, 0.172% MACs, 
              (0): Conv2d(0.009 GMac, 0.168% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.002% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.001% MACs, )
            )
            (1): Sequential(
              0.006 GMac, 0.098% MACs, 
              (0): Conv2d(0.006 GMac, 0.098% MACs, 256, 84, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (3): ConvDWSep2d(
            0.004 GMac, 0.068% MACs, 
            (0): Sequential(
              0.002 GMac, 0.043% MACs, 
              (0): Conv2d(0.002 GMac, 0.042% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.001% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.001 GMac, 0.025% MACs, 
              (0): Conv2d(0.001 GMac, 0.025% MACs, 256, 84, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (4): ConvDWSep2d(
            0.001 GMac, 0.017% MACs, 
            (0): Sequential(
              0.001 GMac, 0.011% MACs, 
              (0): Conv2d(0.001 GMac, 0.011% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.0 GMac, 0.006% MACs, 
              (0): Conv2d(0.0 GMac, 0.006% MACs, 256, 84, kernel_size=(1, 1), stride=(1, 1))
            )
          )
          (5): ConvDWSep2d(
            0.0 GMac, 0.004% MACs, 
            (0): Sequential(
              0.0 GMac, 0.003% MACs, 
              (0): Conv2d(0.0 GMac, 0.003% MACs, 256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=16, bias=False)
              (1): BatchNorm2d(0.0 GMac, 0.000% MACs, 256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (2): ReLU(0.0 GMac, 0.000% MACs, )
            )
            (1): Sequential(
              0.0 GMac, 0.001% MACs, 
              (0): Conv2d(0.0 GMac, 0.001% MACs, 256, 56, kernel_size=(1, 1), stride=(1, 1))
            )
          )
        )
      )
    )
    2021-01-19 17:19:53,197 - mmdet - INFO - 
    
    2021-01-19 17:19:53,205 - mmdet - INFO - Compute   : 5.61 GMac
    2021-01-19 17:19:53,205 - mmdet - INFO - Parameters: 8.25 M
    2021-01-19 17:20:13,861 - mmdet - INFO - Start running, host: hy@hy, work_dir: /home/hy/lyw/pytorch-mmdetection/work_dirs/ssd-lite_regnet_fpn_bgr
    2021-01-19 17:20:13,862 - mmdet - INFO - workflow: [('train', 1)], max: 120 epochs
    2021-01-19 17:35:19,849 - mmdet - INFO - Epoch [1][100/4315]	lr: 9.990e-05, eta: 54 days, 6:51:20, time: 9.060, data_time: 7.266, memory: 20318, loss_cls: 19.7739, loss_bbox: 5.1733, loss: 24.9473, grad_norm: 19.1240
    2021-01-19 17:50:23,180 - mmdet - INFO - Epoch [1][200/4315]	lr: 1.998e-04, eta: 54 days, 4:41:45, time: 9.033, data_time: 7.359, memory: 20318, loss_cls: 14.4417, loss_bbox: 4.2256, loss: 18.6673, grad_norm: 8.6438
    2021-01-19 18:05:39,966 - mmdet - INFO - Epoch [1][300/4315]	lr: 2.997e-04, eta: 54 days, 10:15:24, time: 9.168, data_time: 7.358, memory: 20318, loss_cls: 11.0594, loss_bbox: 3.6649, loss: 14.7243, grad_norm: 6.7019
    2021-01-19 18:21:00,675 - mmdet - INFO - Epoch [1][400/4315]	lr: 3.996e-04, eta: 54 days, 14:19:08, time: 9.207, data_time: 7.441, memory: 20318, loss_cls: 8.2737, loss_bbox: 3.2740, loss: 11.5477, grad_norm: 5.1419
    2021-01-19 18:36:21,984 - mmdet - INFO - Epoch [1][500/4315]	lr: 4.995e-04, eta: 54 days, 16:49:33, time: 9.213, data_time: 7.552, memory: 20318, loss_cls: 6.1987, loss_bbox: 2.9949, loss: 9.1935, grad_norm: 4.2124
    2021-01-19 18:51:49,888 - mmdet - INFO - Epoch [1][600/4315]	lr: 5.994e-04, eta: 54 days, 19:59:29, time: 9.279, data_time: 7.577, memory: 20318, loss_cls: 4.8481, loss_bbox: 2.7756, loss: 7.6236, grad_norm: 3.5941
    2021-01-19 19:07:16,428 - mmdet - INFO - Epoch [1][700/4315]	lr: 6.993e-04, eta: 54 days, 21:53:57, time: 9.265, data_time: 7.409, memory: 20318, loss_cls: 4.1782, loss_bbox: 2.6133, loss: 6.7914, grad_norm: 3.2475
    2021-01-19 19:23:26,937 - mmdet - INFO - Epoch [1][800/4315]	lr: 7.992e-04, eta: 55 days, 7:09:30, time: 9.705, data_time: 7.988, memory: 20318, loss_cls: 3.7640, loss_bbox: 2.4390, loss: 6.2030, grad_norm: 2.9417
    2021-01-19 19:38:59,401 - mmdet - INFO - Epoch [1][900/4315]	lr: 8.991e-04, eta: 55 days, 8:13:51, time: 9.325, data_time: 7.442, memory: 20318, loss_cls: 3.5266, loss_bbox: 2.3001, loss: 5.8267, grad_norm: 2.8921
    2021-01-19 19:54:25,051 - mmdet - INFO - Epoch [1][1000/4315]	lr: 9.990e-04, eta: 55 days, 8:03:31, time: 9.256, data_time: 7.338, memory: 20318, loss_cls: 3.4318, loss_bbox: 2.2215, loss: 5.6533, grad_norm: 3.0163
    2021-01-19 20:10:00,285 - mmdet - INFO - Epoch [1][1100/4315]	lr: 1.000e-03, eta: 55 days, 9:07:17, time: 9.352, data_time: 7.362, memory: 20318, loss_cls: 3.3668, loss_bbox: 2.1431, loss: 5.5099, grad_norm: 2.9501

  • Hi yewei li,

    I have attached the an update source code of our extension pytorch-mmdetection to be used with the latest versions of mmdetection (V2.8.0) and mmcv (1.2.5). Please try it and let us know. We shall publish the official update on our git repository later.

    pytorch-mmdetection-release-9d2b33c.zip

  • Hi Manu Mathew,

    Thank you.

    I have tried the update source code of your extension pytorch-mmdetection.A But the same error occurs. 

  • It is working for me and I am able to train. 

    I am using GTX1080Ti (vs RTX 3090 that you have). Also, I am using cuda10.2 (vs cuda11 that you are using). Other than that I couldn't find a major difference in your environment. 

    So I am not able to suggest anything. May be you can try the original mmdetection to see whether you get the same issue there or not.

    Best regards,.