SK-AM68: How to change trainning and compilation parameters in Model Maker

These two parameters that you mentioned are parameters are passed from the edgeai-modelmaker to edgeai-benchmark, so you have to modify in edgeai-modelmaker itself

You can specify under the compilation field in the modelmaker config yaml file https://github.com/TexasInstruments/edgeai-tensorlab/blob/main/edgeai-modelmaker/config_detection.yaml

compilation:

   # enable/disable compilation
   enable: True #False
   # tensor_bits: 8 #16 #32
   detection_top_k: 200
   detection_threshold: 0.1

ok, thanks.

have changed the trainning and compilation parameters as Model Composer in Model Maker, 

seems the average procision is a bit low,  in addition to these parameters, what other ways can be used to improve procision?  thanks.

Default parameters are optimized for minimizing the run time. Changing settings for accuracy measurement may increase the run time. 

You can try setting the following parameters for accuracy measurement:

detection_top_k: 500
detection_threshold: 0.05

There is the following error when i run it on the SK-AM68 EVM,

root@am68a-sk:/opt/edgeai-gst-apps/apps_cpp/bin/Release# ./app_edgeai ~/obj_detection.yaml
libtidl_onnxrt_EP loaded 0xb9196b0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
   212.665547 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
   212.665655 s:  VX_ZONE_INIT:Enabled
   212.665668 s:  VX_ZONE_ERROR:Enabled
   212.665675 s:  VX_ZONE_WARNING:Enabled
   212.666675 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
   212.667486 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
TIDL_RT_OVX: ERROR: Config file size (93976 bytes) does not match size of sTIDL_IOBufDesc_t (37912 bytes)
   212.668415 s:  VX_ZONE_ERROR:[tivxAddKernelTIDL:269] invalid values for num_input_tensors or num_output_tensors 
   212.700588 s:  VX_ZONE_ERROR:[vxQueryKernel:140] Invalid kernel reference
   212.700624 s:  VX_ZONE_ERROR:[vxMapUserDataObject:384] Invalid user data object reference
   212.700635 s:  VX_ZONE_ERROR:[vxUnmapUserDataObject:469] Invalid user data object reference
   212.700642 s:  VX_ZONE_ERROR:[tivxTIDLNode:113] Exceeded max parameters for a kernel
   212.700650 s:  VX_ZONE_ERROR:[vxSetReferenceName:960] Invalid reference
   212.700658 s:  VX_ZONE_ERROR:[vxSetReferenceName:960] Invalid reference
   212.700663 s:  VX_ZONE_ERROR:[vxSetReferenceName:960] Invalid reference
graph
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

i want to know, if i used SDK version 9.2,  then i must use  edgeai-tensor 9.2 , or main branch is ok?

I remembered that it can run SDK version 9.2 and  edgeai-tensor main branch a few weeks ago,

Now i rebase edgeai-tensor to latest  or 9.2 branch, it can not run the model.

9.2 and branch and main branch of edgeai-tensorlab has the same contents as of now. Once 10.0 branch is released main branch will be in sync with that branch.

We have not added any major changes changes in last few weeks - only couple of bugfixes. you can see the commits in the history. So if it was running few weeks ago, it should run now also.

May be you can try rebooting the EVM to see if the model works. 

then now there is no matched Edgeai-tensorlab version with PROCESSOR-SDK-LINUX-AM68A  Version: 10.00.00.08 since the latest tensorlab version  is r9.2？ 

We are planning to push the r10.0 branch for edgeai-tensorlab in a few days.

root@am68a-sk:/opt/edgeai-gst-apps/apps_cpp/bin# cd Release/
root@am68a-sk:/opt/edgeai-gst-apps/apps_cpp/bin/Release# ls
app_config_checker  app_edgeai
root@am68a-sk:/opt/edgeai-gst-apps/apps_cpp/bin/Release# ./app_edgeai /opt/edgeai-gst-apps/configs/imx219_cam_example.yaml
libtidl_onnxrt_EP loaded 0x3571a750 
Final number of subgraphs created are : 1, - Offloaded Nodes - 283, Total Nodes - 283 
APP: Init ... !!!
MEM: Init ... !!!
MEM: Initialized DMA HEAP (fd=5) !!!
MEM: Init ... Done !!!
IPC: Init ... !!!
IPC: Init ... Done !!!
REMOTE_SERVICE: Init ... !!!
REMOTE_SERVICE: Init ... Done !!!
  1433.134006 s: GTC Frequency = 200 MHz
APP: Init ... Done !!!
  1433.134120 s:  VX_ZONE_INIT:Enabled
  1433.134129 s:  VX_ZONE_ERROR:Enabled
  1433.134136 s:  VX_ZONE_WARNING:Enabled
  1433.134856 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-0 
  1433.135009 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-1 
  1433.135174 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-2 
  1433.135279 s:  VX_ZONE_INIT:[tivxPlatformCreateTargetId:116] Added target MPU-3 
  1433.135291 s:  VX_ZONE_INIT:[tivxInitLocal:136] Initialization Done !!!
  1433.135783 s:  VX_ZONE_INIT:[tivxHostInitLocal:101] Initialization Done for HOST !!!
  1433.154097 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  1433.154125 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  1433.154132 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  1433.154139 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  1433.154147 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  1433.154172 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  1433.154179 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
TIDL_RT_OVX: ERROR: Verifying TIDL graph ... Failed !!!
TIDL_RT_OVX: ERROR: Verify OpenVX graph failed
graph
==========[INPUT PIPELINE(S)]==========

[PIPE-0]

v4l2src device=/dev/video-imx219-cam0 io-mode=5 ! queue leaky=2 ! capsfilter caps="video/x-bayer, width=(int)1920, height=(int)1080, format=(string)rggb;" ! tiovxisp dcc-isp-file=/opt/imaging/imx219/linear/dcc_viss.bin sensor-name=SENSOR_SONY_IMX219_RPI ! capsfilter caps="video/x-raw, format=(string)NV12;" ! tiovxmultiscaler name=multiscaler_split_00
multiscaler_split_00. ! queue ! capsfilter caps="video/x-raw, width=(int)480, height=(int)416;" ! tiovxmultiscaler target=1 ! capsfilter caps="video/x-raw, width=(int)416, height=(int)416;" ! tiovxdlpreproc out-pool-size=4 data-type=3 tensor-format=1 ! capsfilter caps="application/x-tensor-tiovx;" ! appsink max-buffers=2 drop=true name=flow0_pre_proc0
multiscaler_split_00. ! queue ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720;" ! tiovxdlcolorconvert out-pool-size=4 ! capsfilter caps="video/x-raw, format=(string)RGB;" ! appsink max-buffers=2 drop=true name=flow0_sensor0

==========[OUTPUT PIPELINE]==========

appsrc do-timestamp=true format=3 block=true name=flow0_post_proc0 ! tiovxdlcolorconvert ! capsfilter caps="video/x-raw, width=(int)1280, height=(int)720, format=(string)NV12;" ! queue ! mosaic0.sink0

tiovxmosaic target=1 background=/tmp/background0 name=mosaic0 src::pool-size=4
sink_0::startx="<320>" sink_0::starty="<150>" sink_0::widths="<1280>" sink_0::heights="<720>"
! capsfilter caps="video/x-raw, format=(string)NV12, width=(int)1920, height=(int)1080;" ! queue ! tiperfoverlay title=IMX219 Camera ! kmssink sync=false max-lateness=5000000 qos=true processing-deadline=15000000 driver-name=tidss connector-id=40 plane-id=31 force-modesetting=true

 +-----------------------------------------------------------------+ | IMX219 Camera| +-----------------------------------------------------------------+ +-----------------------------------------------------------------+ | Input Source: /dev/video-imx219-cam0| | Model Name:   object_detection_test| | Model Type:   detection| +-----------------------------------------------------------------+ | dl-inference:     0.00 ms  from     0 samples  | | total time:     0.00 ms  from     0 samples  | | framerate:     0.00 ms  from     0 samples  | +-----------------------------------------------------------------+  1433.954422 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  1433.954463 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  1433.954472 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  1433.954479 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  1433.954490 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  1433.954509 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  1433.954517 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  1433.954582 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  1433.954591 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  1433.954599 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  1433.962006 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  1433.962047 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  1433.962055 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  1433.962062 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  1433.962071 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  1433.962083 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  1433.962089 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  1433.962153 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  1433.962162 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  1433.962168 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!
  1433.985935 s:  VX_ZONE_ERROR:[ownContextSendCmd:875] Command ack message returned failure cmd_status: -1
  1433.985965 s:  VX_ZONE_ERROR:[ownNodeKernelInit:590] Target kernel, TIVX_CMD_NODE_CREATE failed for node TIDLNode
  1433.985973 s:  VX_ZONE_ERROR:[ownNodeKernelInit:591] Please be sure the target callbacks have been registered for this core
  1433.985980 s:  VX_ZONE_ERROR:[ownNodeKernelInit:592] If the target callbacks have been registered, please ensure no errors are occurring within the create callback of this kernel
  1433.985990 s:  VX_ZONE_ERROR:[ownGraphNodeKernelInit:608] kernel init for node 0, kernel com.ti.tidl:1:2 ... failed !!!
  1433.986002 s:  VX_ZONE_ERROR:[vxVerifyGraph:2159] Node kernel init failed
  1433.986009 s:  VX_ZONE_ERROR:[vxVerifyGraph:2213] Graph verify failed
  1433.986066 s:  VX_ZONE_ERROR:[ownGraphScheduleGraphWrapper:885] graph is not in a state required to be scheduled
  1433.986073 s:  VX_ZONE_ERROR:[vxProcessGraph:813] schedule graph failed
  1433.986079 s:  VX_ZONE_ERROR:[vxProcessGraph:818] wait graph failed
ERROR: Running TIDL graph ... Failed !!!

could you help check this error?   i used SDK 9.2 and edgeai-tensorlab 9.2,

i re-trained and compiled the sample model,

./run_modelmaker AM68A  config_detection.yaml

 Download the recompiled model to the target board and run, will get the above error,
Is it a version matching problem of tidl-tools? i have checked the version is r9.2

edgeai-tensorlab/edgeai-benchmark/tools/AM68A/tidl_tools/version.yaml

target_device: AM68A
version: 9.2
release_id: 09_02_09_00
release_name: r9.2

another question:

i used these parameters,

training:
    # enable/disable training
    enable: True #False

    # Object Detection model chosen can be changed here if needed
    # options are: 'yolox_s_lite', 'yolox_tiny_lite', 'yolox_nano_lite', 'yolox_pico_lite', 'yolox_femto_lite'
    model_name: 'yolox_s_lite'

    training_epochs: 60 #30
    batch_size: 8 #32
    learning_rate: 0.002
    #num_gpus: 0 #1 #4

compilation:
    # enable/disable compilation
    enable: True #False
    tensor_bits: 16 #16 #32
    detection_top_k: 1000
    detection_threshold: 0.01
    calibration_iterations: 10
    Calibration Frames: 10  

the training and compilation logs

2024-09-20 01:32:51,374 - mmdet - INFO - Exp name: yolox_s_lite.py
2024-09-20 01:32:51,374 - mmdet - INFO - Epoch(val) [57][36]	bbox_mAP: 0.5470, bbox_mAP_50: 0.9370, bbox_mAP_75: 0.5420, bbox_mAP_s: -1.0000, bbox_mAP_m: 0.5240, bbox_mAP_l: 0.5720, bbox_mAP_copypaste: 0.547 0.937 0.542 -1.000 0.524 0.572
2024-09-20 01:34:43,889 - mmdet - INFO - Evaluating bbox...
2024-09-20 01:34:43,921 - mmdet - INFO - 
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.538
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.956
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.513
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.504
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.581
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.592
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.592
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.592
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.521
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.632

2024-09-20 01:34:43,922 - mmdet - INFO - Exp name: yolox_s_lite.py
2024-09-20 01:34:43,922 - mmdet - INFO - Epoch(val) [58][36]	bbox_mAP: 0.5380, bbox_mAP_50: 0.9560, bbox_mAP_75: 0.5130, bbox_mAP_s: -1.0000, bbox_mAP_m: 0.5040, bbox_mAP_l: 0.5810, bbox_mAP_copypaste: 0.538 0.956 0.513 -1.000 0.504 0.581
2024-09-20 01:36:18,758 - mmdet - INFO - Evaluating bbox...
2024-09-20 01:36:18,788 - mmdet - INFO - 
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.492
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.983
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.400
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.483
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.500
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.550
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.550
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.550
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.509
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.559

2024-09-20 01:36:18,789 - mmdet - INFO - Exp name: yolox_s_lite.py
2024-09-20 01:36:18,789 - mmdet - INFO - Epoch(val) [59][36]	bbox_mAP: 0.4920, bbox_mAP_50: 0.9830, bbox_mAP_75: 0.4000, bbox_mAP_s: -1.0000, bbox_mAP_m: 0.4830, bbox_mAP_l: 0.5000, bbox_mAP_copypaste: 0.492 0.983 0.400 -1.000 0.483 0.500
2024-09-20 01:37:05,410 - mmdet - INFO - Saving checkpoint at 60 epochs
2024-09-20 01:37:11,637 - mmdet - INFO - Evaluating bbox...
2024-09-20 01:37:11,667 - mmdet - INFO - 
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.459
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.965
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.346
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.481
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.443
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.524
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.524
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.524
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.506
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.511

2024-09-20 01:37:11,669 - mmdet - INFO - Exp name: yolox_s_lite.py
2024-09-20 01:37:11,670 - mmdet - INFO - Epoch(val) [60][36]	bbox_mAP: 0.4590, bbox_mAP_50: 0.9650, bbox_mAP_75: 0.3460, bbox_mAP_s: -1.0000, bbox_mAP_m: 0.4810, bbox_mAP_l: 0.4430, bbox_mAP_copypaste: 0.459 0.965 0.346 -1.000 0.481 0.443

SUCCESS: ModelMaker - Training completed.

SUCCESS:20240920-014630: benchmark results - {}

INFO:20240920-014631: running - od-8220
INFO:20240920-014631: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7b8d4d0fb910>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7b8d4dde3e80>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7b8cf4d321d0>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7b8cf4d32230>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7b8cf4d324d0>, 'metric': {'label_offset_pred': 0}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10}}
INFO:20240920-014631: infer  - od-8220 - this may take some time...
infer : od-8220                                             |   0%|          || 0/36 [00:00<?, ?it/s]
infer : od-8220                                             |          |     0% 0/36| [< ]
infer : od-8220                                             |   3%|2         || 1/36 [00:13<07:46, 13.32s/it]
infer : od-8220                                             |   6%|5         || 2/36 [00:26<07:34, 13.36s/it]
infer : od-8220                                             |   8%|8         || 3/36 [00:40<07:21, 13.37s/it]
infer : od-8220                                             |  11%|#1        || 4/36 [00:53<07:10, 13.44s/it]
infer : od-8220                                             |  14%|#3        || 5/36 [01:06<06:55, 13.40s/it]
infer : od-8220                                             |  17%|#6        || 6/36 [01:20<06:41, 13.38s/it]
infer : od-8220                                             |  19%|#9        || 7/36 [01:33<06:29, 13.43s/it]
infer : od-8220                                             |  22%|##2       || 8/36 [01:47<06:15, 13.41s/it]
infer : od-8220                                             |  25%|##5       || 9/36 [02:00<06:01, 13.40s/it]
infer : od-8220                                             |  28%|##7       || 10/36 [02:13<05:47, 13.38s/it]
infer : od-8220                                             |  31%|###       || 11/36 [02:27<05:34, 13.37s/it]
infer : od-8220                                             |  33%|###3      || 12/36 [02:40<05:20, 13.34s/it]
infer : od-8220                                             |  36%|###6      || 13/36 [02:53<05:06, 13.35s/it]
infer : od-8220                                             |  39%|###8      || 14/36 [03:07<04:54, 13.38s/it]
infer : od-8220                                             |  42%|####1     || 15/36 [03:20<04:40, 13.34s/it]
infer : od-8220                                             |  44%|####4     || 16/36 [03:34<04:27, 13.37s/it]
infer : od-8220                                             |  47%|####7     || 17/36 [03:47<04:13, 13.36s/it]
infer : od-8220                                             |  50%|#####     || 18/36 [04:00<04:00, 13.38s/it]
infer : od-8220                                             |  53%|#####2    || 19/36 [04:14<03:47, 13.36s/it]
infer : od-8220                                             |  56%|#####5    || 20/36 [04:27<03:34, 13.38s/it]
infer : od-8220                                             |  58%|#####8    || 21/36 [04:41<03:21, 13.41s/it]
infer : od-8220                                             |  61%|######1   || 22/36 [04:54<03:07, 13.37s/it]
infer : od-8220                                             |  64%|######3   || 23/36 [05:07<02:53, 13.36s/it]
infer : od-8220                                             |  67%|######6   || 24/36 [05:20<02:40, 13.33s/it]
infer : od-8220                                             |  69%|######9   || 25/36 [05:34<02:26, 13.30s/it]
infer : od-8220                                             |  72%|#######2  || 26/36 [05:47<02:12, 13.28s/it]
infer : od-8220                                             |  75%|#######5  || 27/36 [06:00<01:59, 13.28s/it]
infer : od-8220                                             |  78%|#######7  || 28/36 [06:14<01:46, 13.36s/it]
infer : od-8220                                             |  81%|########  || 29/36 [06:27<01:33, 13.33s/it]
infer : od-8220                                             |  83%|########3 || 30/36 [06:40<01:19, 13.33s/it]
infer : od-8220                                             |  86%|########6 || 31/36 [06:54<01:06, 13.30s/it]
infer : od-8220                                             |  89%|########8 || 32/36 [07:07<00:53, 13.32s/it]
infer : od-8220                                             |  92%|#########1|| 33/36 [07:20<00:39, 13.33s/it]
infer : od-8220                                             |  94%|#########4|| 34/36 [07:34<00:26, 13.38s/it]
infer : od-8220                                             |  97%|#########7|| 35/36 [07:47<00:13, 13.34s/it]
infer : od-8220                                             | 100%|##########|| 36/36 [08:00<00:00, 13.33s/it]
infer : od-8220                                             | 100%|##########|| 36/36 [08:00<00:00, 13.36s/it]

INFO:20240920-015432: infer completed  - od-8220 - 481 sec

SUCCESS:20240920-015432: benchmark results - {'infer_path': 'od-8220', 'accuracy_ap[.5:.95]%': 28.925218, 'accuracy_ap50%': 66.666667, 'num_subgraphs': 1, 'infer_time_core_ms': 13305.751348, 'infer_time_subgraph_ms': 13305.659418, 'ddr_transfer_mb': 0.0, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0}

The compilation accuracy reported is lower than training accuracy and there a variety of reasons for it.

1. detection_top_k and detection_threshold parameters that we discussed above certainly influences the accuracy. You can try to adjust the parameters as I suggested above to see the accuracy improves. 

2. Quantization certainly influences the accuracy - you try run compilation in float simulation mode (under compilation, set tensor_bits: 32) to measure accuracy with quantization. If quantization is indeed the reason for accuracy drop, then you need to put some of the layers (especially the last convolution layers) into 16 bit to recover the accuracy. 

another issue:

now i used the same dataset with Model Composer and Model Maker,

and used the same trainning and compilation parameters,

Model Composer:

Model Maker:

When I download the trained model to the target board, the recognition accuracy of the model trained with model composer is OK, but the accuracy of the model trained with model maker is much lower. 

As you know, model composer latest supports sdk9.1. We need to use SDK10.0 but the recognition accuracy of the model trained using model maker is very low even using the same data set. How should we solve this problem? How to improve the accuracy using model maker? thanks.

Can you pull the latest edgeai-tensorlab r10.0 and try again. There was a problem with accuracy was due to a change in the model and it has been corrected. 

  I have pulled the latest edgeai-tensorlab r10.0, Looks like nothing has improved.

annother issue:
I use yolox_s_lite Model, and Training uses static images, For some dynamic targets, the recognition rate is too low. Which parameters can be adjusted?

Which dataset is used for the training above where you are comparing the 9.1 ModelComposer and 10.0 ModelMaker? I shall try to reproduce the same scenario and will help to solve if there is any issue.

Also, can you clarify the following statement: "I use yolox_s_lite Model, and Training uses static images, For some dynamic targets, the recognition rate is too low."  I did not quite understand it - what does it mean?

1）I'm using my own dataset, which is captured by a device's camera.

and the config file

common:
    target_module: 'vision'
    task_type: 'detection'
    target_device: 'AM68A'
    # run_name can be any string, but there are some special cases:
    # {date-time} will be replaced with datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    # {model_name} will be replaced with the name of the model
    run_name: '{date-time}/{model_name}'

dataset:
    # enable/disable dataset loading
    enable: True #False
    # max_num_files: [750, 250] #None

    # Object Detection Dataset Examples:
    # -------------------------------------
    # Example 1, (known datasets): 'widerface_detection', 'pascal_voc0712', 'coco_detection', 'udacity_selfdriving', 'tomato_detection', 'tiscapes2017_driving'
    # dataset_name: widerface_detection
    # -------------------------------------
    # Example 2, give a dataset name and input_data_path.
    # input_data_path could be a path to zip file, tar file, folder OR http, https link to zip or tar files
    # for input_data_path these are provided with this repository as examples:
    #    'http://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/datasets/tiscapes2017_driving.zip'
    #    'http://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/datasets/animal_detection.zip'
    # -------------------------------------
    # Example 3, give image folders with annotation files (require list with values for both train and val splits)
    # dataset_name: coco_detection
    # input_data_path: ["./data/projects/coco_detection/dataset/train2017",
    #                        "./data/projects/coco_detection/dataset/val2017"]
    # input_annotation_path: ["./data/projects/coco_detection/dataset/annotations/instances_train2017.json",
    #                        "./data/projects/coco_detection/dataset/annotations/instances_val2017.json"]
    # -------------------------------------
    dataset_name: fork_20241014
    #input_data_path: ["./data/datasets/obj_detection/images/"]
    #input_annotation_path: ["./data/datasets/obj_detection/annotations/instances.json"]
    input_data_path: './data/datasets/fork_20241014'
    input_annotation_path: './data/datasets/fork_20241014/annotations/instances.json'
training:
    # enable/disable training
    enable: True #False

    # Object Detection model chosen can be changed here if needed
    # options are: 'yolox_s_lite', 'yolox_tiny_lite', 'yolox_nano_lite', 'yolox_pico_lite', 'yolox_femto_lite'
    model_name: 'yolox_s_lite'

    training_epochs: 45 #30
    batch_size: 8 #32
    learning_rate: 0.002
    #num_gpus: 0 #1 #4

compilation:
    # enable/disable compilation
    enable: True #False
    tensor_bits: 16 #16 #32
    detection_top_k: 500
    detection_threshold: 0.05
    calibration_iterations: 10
    Calibration Frames: 10  

py310) mchi@ubuntu22:~/work/edgeai-tensorlab/edgeai-modelmaker$ ./run_modelmaker.sh  AM68A config_only_fork.yaml 
Number of AVX cores detected in PC: 16
AVX compilation speedup in PC     : 1
Target device                     : AM68A
PYTHONPATH                        : .:
TIDL_TOOLS_PATH                   : ../edgeai-benchmark/tools/AM68A/tidl_tools
LD_LIBRARY_PATH                   : ../edgeai-benchmark/tools/AM68A/tidl_tools
argv: ['./scripts/run_modelmaker.py', 'config_only_fork.yaml', '--target_device', 'AM68A']
{'common': {'verbose_mode': True, 'download_path': './data/downloads', 'projects_path': './data/projects', 'project_path': None, 'project_run_path': None, 'task_type': 'detection', 'target_machine': 'evm', 'target_device': 'AM68A', 'run_name': '{date-time}/{model_name}', 'target_module': 'vision'}, 'download': [{'download_url': 'https://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/models/vision/detection/coco/edgeai-mmdet/yolox_s_lite_640x640_20220221_checkpoint.pth', 'download_path': '{download_path}/pretrained/yolox_s_lite'}], 'dataset': {'enable': True, 'dataset_name': 'only_fork', 'dataset_path': None, 'extract_path': None, 'split_factor': 0.8, 'split_names': ('train', 'val'), 'max_num_files': 10000, 'input_data_path': './data/datasets/only_fork', 'input_annotation_path': './data/datasets/only_fork/annotations/instances.json', 'data_path_splits': None, 'data_dir': 'images', 'annotation_path_splits': None, 'annotation_dir': 'annotations', 'annotation_prefix': 'instances', 'annotation_format': 'coco_json', 'dataset_download': True, 'dataset_reload': False}, 'training': {'enable': True, 'model_name': 'yolox_s_lite', 'model_training_id': 'yolox_s_lite', 'training_backend': 'edgeai_mmdetection', 'pretrained_checkpoint_path': {'download_url': 'https://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/models/vision/detection/coco/edgeai-mmdet/yolox_s_lite_640x640_20220221_checkpoint.pth', 'download_path': '{download_path}/pretrained/yolox_s_lite'}, 'pretrained_weight_state_dict_name': None, 'target_devices': {'TDA4VM': {'performance_fps': None, 'performance_infer_time_ms': 10.14, 'accuracy_factor': 56.9, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 38.3, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM62A': {'performance_fps': None, 'performance_infer_time_ms': 43.94, 'accuracy_factor': 56.9, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 38.3, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM67A': {'performance_fps': None, 'performance_infer_time_ms': '43.94 (with 1/2 device capability)', 'accuracy_factor': 56.9, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 38.3, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM68A': {'performance_fps': None, 'performance_infer_time_ms': 10.22, 'accuracy_factor': 56.9, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 38.3, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM69A': {'performance_fps': None, 'performance_infer_time_ms': '9.82 (with 1/4th device capability)', 'accuracy_factor': 56.9, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 38.3, 'accuracy_unit2': 'AP[.5:.95]%'}}, 'project_path': None, 'dataset_path': None, 'training_path': None, 'log_file_path': None, 'log_summary_regex': None, 'summary_file_path': None, 'model_checkpoint_path': None, 'model_export_path': None, 'model_proto_path': None, 'model_packaged_path': None, 'training_epochs': 30, 'warmup_epochs': 1, 'num_last_epochs': 5, 'batch_size': 8, 'learning_rate': 0.002, 'num_classes': None, 'weight_decay': 0.0001, 'input_resize': 640, 'input_cropsize': 640, 'training_device': None, 'num_gpus': 0, 'distributed': True, 'training_master_port': 29500, 'with_background_class': None, 'model_architecture': 'yolox', 'training_devices': {'cpu': True, 'cuda': True}}, 'compilation': {'enable': True, 'preset_name': None, 'model_compilation_id': 'od-8220', 'compilation_path': None, 'model_compiled_path': None, 'log_file_path': None, 'log_summary_regex': None, 'summary_file_path': None, 'output_tensors_path': None, 'model_packaged_path': None, 'model_visualization_path': None, 'tensor_bits': 16, 'calibration_frames': 10, 'calibration_iterations': 10, 'num_frames': None, 'num_output_frames': 50, 'detection_threshold': 0.05, 'detection_top_k': 500, 'save_output': True, 'tidl_offload': True, 'input_optimization': False, 'capture_log': True, 'runtime_options': {'advanced_options:output_feature_16bit_names_list': '/multi_level_conv_obj.2/Conv_output_0, /multi_level_conv_reg.2/Conv_output_0, /multi_level_conv_cls.2/Conv_output_0, /multi_level_conv_obj.1/Conv_output_0, /multi_level_conv_reg.1/Conv_output_0, /multi_level_conv_cls.1/Conv_output_0, /multi_level_conv_obj.0/Conv_output_0, /multi_level_conv_reg.0/Conv_output_0, /multi_level_conv_cls.0/Conv_output_0'}, 'metric': {'label_offset_pred': 0}, 'Calibration Frames': 10}}
---------------------------------------------------------------------
Run Name: 20241014-170809/yolox_s_lite
- Model: yolox_s_lite
- TargetDevices & Estimated Inference Times (ms): {'TDA4VM': 10.14, 'AM62A': 43.94, 'AM67A': '43.94 (with 1/2 device capability)', 'AM68A': 10.22, 'AM69A': '9.82 (with 1/4th device capability)'}
- This model can be compiled for the above device(s).
---------------------------------------------------------------------
assuming the given download_url is a valid path: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/datasets/only_fork
dataset split sizes {'train': 180, 'val': 44}
max_num_files is set to: 10000
dataset split sizes are limited to: {'train': 180, 'val': 44}
dataset loading OK
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
Selecting model configs from Python module: ./configs
Run params is at: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/run.yaml
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/mmengine/optim/optimizer/zero_optimizer.py:11: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
  from torch.distributed.optim import \
10/14 17:08:11 - mmengine - INFO - 
------------------------------------------------------------
System environment:
    sys.platform: linux
    Python: 3.10.15 (main, Oct  8 2024, 10:32:26) [GCC 11.4.0]
    CUDA available: False
    MUSA available: False
    numpy_random_seed: 1280278782
    GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
    PyTorch: 2.4.0+cpu
    PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=0, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

    TorchVision: 0.19.0+cpu
    OpenCV: 4.10.0
    MMEngine: 0.10.5

Runtime environment:
    cudnn_benchmark: False
    mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
    dist_cfg: {'backend': 'nccl'}
    seed: 1280278782
    Distributed launcher: none
    Distributed training: False
    GPU number: 1
------------------------------------------------------------

10/14 17:08:13 - mmengine - INFO - Config:
auto_scale_lr = dict(base_batch_size=64, enable=False)
backend_args = None
base_lr = 0.01
classes = (
    'tree',
    'split',
    'lack',
)
convert_to_lite_model = dict(model_surgery=1)
custom_hooks = [
    dict(num_last_epochs=15, priority=48, type='YOLOXModeSwitchHook'),
    dict(priority=48, type='SyncNormHook'),
    dict(
        ema_type='ExpMomentumEMA',
        momentum=0.0001,
        priority=49,
        type='EMAHook',
        update_buffers=True),
]
data_root = '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset'
dataset_type = 'CocoDataset'
default_hooks = dict(
    checkpoint=dict(interval=1, max_keep_ckpts=3, type='CheckpointHook'),
    logger=dict(interval=50, type='LoggerHook'),
    param_scheduler=dict(type='ParamSchedulerHook'),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    timer=dict(type='IterTimerHook'),
    visualization=dict(type='DetVisualizationHook'))
default_scope = 'mmdet'
env_cfg = dict(
    cudnn_benchmark=False,
    dist_cfg=dict(backend='nccl'),
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
export_onnx_model = True
img_scale = (
    640,
    640,
)
img_scales = [
    (
        640,
        640,
    ),
    (
        320,
        320,
    ),
    (
        960,
        960,
    ),
]
interval = 1
launcher = 'none'
load_from = './data/downloads/pretrained/yolox_s_lite/yolox_s_lite_640x640_20220221_checkpoint.pth'
log_level = 'INFO'
log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)
max_epochs = 30
model = dict(
    backbone=dict(
        act_cfg=dict(type='ReLU'),
        deepen_factor=0.33,
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        out_indices=(
            2,
            3,
            4,
        ),
        spp_kernal_sizes=(
            5,
            9,
            13,
        ),
        type='CSPDarknet',
        use_depthwise=False,
        widen_factor=0.5),
    bbox_head=dict(
        act_cfg=dict(type='ReLU'),
        feat_channels=128,
        in_channels=128,
        loss_bbox=dict(
            eps=1e-16,
            loss_weight=5.0,
            mode='square',
            reduction='sum',
            type='IoULoss'),
        loss_cls=dict(
            loss_weight=1.0,
            reduction='sum',
            type='CrossEntropyLoss',
            use_sigmoid=True),
        loss_l1=dict(loss_weight=1.0, reduction='sum', type='L1Loss'),
        loss_obj=dict(
            loss_weight=1.0,
            reduction='sum',
            type='CrossEntropyLoss',
            use_sigmoid=True),
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        num_classes=3,
        stacked_convs=2,
        strides=(
            8,
            16,
            32,
        ),
        type='YOLOXHead',
        use_depthwise=False),
    data_preprocessor=dict(
        batch_augments=[
            dict(
                interval=10,
                random_size_range=(
                    480,
                    800,
                ),
                size_divisor=32,
                type='BatchSyncRandomResize'),
        ],
        pad_size_divisor=32,
        type='DetDataPreprocessor'),
    neck=dict(
        act_cfg=dict(type='ReLU'),
        in_channels=[
            128,
            256,
            512,
        ],
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        num_csp_blocks=1,
        out_channels=128,
        type='YOLOXPAFPN',
        upsample_cfg=dict(mode='nearest', scale_factor=2),
        use_depthwise=False),
    test_cfg=dict(nms=dict(iou_threshold=0.65, type='nms'), score_thr=0.01),
    train_cfg=dict(assigner=dict(center_radius=2.5, type='SimOTAAssigner')),
    type='YOLOX')
num_last_epochs = 15
optim_wrapper = dict(
    optimizer=dict(
        lr=0.002, momentum=0.9, nesterov=True, type='SGD',
        weight_decay=0.0005),
    paramwise_cfg=dict(bias_decay_mult=0.0, norm_decay_mult=0.0),
    type='OptimWrapper')
param_scheduler = [
    dict(
        begin=0,
        by_epoch=True,
        convert_to_iter_based=True,
        end=5,
        type='mmdet.QuadraticWarmupLR'),
    dict(
        T_max=15,
        begin=5,
        by_epoch=True,
        convert_to_iter_based=True,
        end=15,
        eta_min=0.0005,
        type='CosineAnnealingLR'),
    dict(begin=15, by_epoch=True, end=30, factor=1, type='ConstantLR'),
]
quantization = 0
resume = False
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
    batch_size=8,
    dataset=dict(
        ann_file=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset/annotations/instances_val.json',
        backend_args=None,
        data_prefix=dict(img='val/'),
        data_root=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset',
        metainfo=dict(classes=(
            'tree',
            'split',
            'lack',
        )),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=True, scale=(
                640,
                640,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=4,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
    ann_file=
    '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset/annotations/instances_val.json',
    backend_args=None,
    metric='bbox',
    type='CocoMetric')
test_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(keep_ratio=True, scale=(
        640,
        640,
    ), type='Resize'),
    dict(
        pad_to_square=True,
        pad_val=dict(img=(
            114.0,
            114.0,
            114.0,
        )),
        type='Pad'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(
        meta_keys=(
            'img_id',
            'img_path',
            'ori_shape',
            'img_shape',
            'scale_factor',
        ),
        type='PackDetInputs'),
]
train_cfg = dict(max_epochs=30, type='EpochBasedTrainLoop', val_interval=1)
train_dataloader = dict(
    batch_size=8,
    dataset=dict(
        dataset=dict(
            ann_file=
            '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset/annotations/instances_train.json',
            backend_args=None,
            data_prefix=dict(img='train/'),
            data_root=
            '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset',
            filter_cfg=dict(filter_empty_gt=False, min_size=32),
            metainfo=dict(classes=(
                'tree',
                'split',
                'lack',
            )),
            pipeline=[
                dict(backend_args=None, type='LoadImageFromFile'),
                dict(type='LoadAnnotations', with_bbox=True),
            ],
            type='CocoDataset'),
        pipeline=[
            dict(img_scale=(
                640,
                640,
            ), pad_val=114.0, type='Mosaic'),
            dict(
                border=(
                    -320,
                    -320,
                ),
                scaling_ratio_range=(
                    0.1,
                    2,
                ),
                type='RandomAffine'),
            dict(
                img_scale=(
                    640,
                    640,
                ),
                pad_val=114.0,
                ratio_range=(
                    0.8,
                    1.6,
                ),
                type='MixUp'),
            dict(type='YOLOXHSVRandomAug'),
            dict(prob=0.5, type='RandomFlip'),
            dict(keep_ratio=True, scale=(
                640,
                640,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(
                keep_empty=False,
                min_gt_bbox_wh=(
                    1,
                    1,
                ),
                type='FilterAnnotations'),
            dict(type='PackDetInputs'),
        ],
        type='MultiImageMixDataset'),
    num_workers=4,
    persistent_workers=True,
    sampler=dict(shuffle=True, type='DefaultSampler'))
train_dataset = dict(
    dataset=dict(
        ann_file=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset/annotations/instances_train.json',
        backend_args=None,
        data_prefix=dict(img='train/'),
        data_root=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset',
        filter_cfg=dict(filter_empty_gt=False, min_size=32),
        metainfo=dict(classes=(
            'tree',
            'split',
            'lack',
        )),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(type='LoadAnnotations', with_bbox=True),
        ],
        type='CocoDataset'),
    pipeline=[
        dict(img_scale=(
            640,
            640,
        ), pad_val=114.0, type='Mosaic'),
        dict(
            border=(
                -320,
                -320,
            ),
            scaling_ratio_range=(
                0.1,
                2,
            ),
            type='RandomAffine'),
        dict(
            img_scale=(
                640,
                640,
            ),
            pad_val=114.0,
            ratio_range=(
                0.8,
                1.6,
            ),
            type='MixUp'),
        dict(type='YOLOXHSVRandomAug'),
        dict(prob=0.5, type='RandomFlip'),
        dict(keep_ratio=True, scale=(
            640,
            640,
        ), type='Resize'),
        dict(
            pad_to_square=True,
            pad_val=dict(img=(
                114.0,
                114.0,
                114.0,
            )),
            type='Pad'),
        dict(
            keep_empty=False,
            min_gt_bbox_wh=(
                1,
                1,
            ),
            type='FilterAnnotations'),
        dict(type='PackDetInputs'),
    ],
    type='MultiImageMixDataset')
train_pipeline = [
    dict(img_scale=(
        640,
        640,
    ), pad_val=114.0, type='Mosaic'),
    dict(
        border=(
            -320,
            -320,
        ),
        scaling_ratio_range=(
            0.1,
            2,
        ),
        type='RandomAffine'),
    dict(
        img_scale=(
            640,
            640,
        ),
        pad_val=114.0,
        ratio_range=(
            0.8,
            1.6,
        ),
        type='MixUp'),
    dict(type='YOLOXHSVRandomAug'),
    dict(prob=0.5, type='RandomFlip'),
    dict(keep_ratio=True, scale=(
        640,
        640,
    ), type='Resize'),
    dict(
        pad_to_square=True,
        pad_val=dict(img=(
            114.0,
            114.0,
            114.0,
        )),
        type='Pad'),
    dict(keep_empty=False, min_gt_bbox_wh=(
        1,
        1,
    ), type='FilterAnnotations'),
    dict(type='PackDetInputs'),
]
tta_model = dict(
    tta_cfg=dict(max_per_img=100, nms=dict(iou_threshold=0.65, type='nms')),
    type='DetTTAModel')
tta_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(
        transforms=[
            [
                dict(keep_ratio=True, scale=(
                    640,
                    640,
                ), type='Resize'),
                dict(keep_ratio=True, scale=(
                    320,
                    320,
                ), type='Resize'),
                dict(keep_ratio=True, scale=(
                    960,
                    960,
                ), type='Resize'),
            ],
            [
                dict(prob=1.0, type='RandomFlip'),
                dict(prob=0.0, type='RandomFlip'),
            ],
            [
                dict(
                    pad_to_square=True,
                    pad_val=dict(img=(
                        114.0,
                        114.0,
                        114.0,
                    )),
                    type='Pad'),
            ],
            [
                dict(type='LoadAnnotations', with_bbox=True),
            ],
            [
                dict(
                    meta_keys=(
                        'img_id',
                        'img_path',
                        'ori_shape',
                        'img_shape',
                        'scale_factor',
                        'flip',
                        'flip_direction',
                    ),
                    type='PackDetInputs'),
            ],
        ],
        type='TestTimeAug'),
]
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
    batch_size=8,
    dataset=dict(
        ann_file=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset/annotations/instances_val.json',
        backend_args=None,
        data_prefix=dict(img='val/'),
        data_root=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset',
        metainfo=dict(classes=(
            'tree',
            'split',
            'lack',
        )),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=True, scale=(
                640,
                640,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=4,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
    ann_file=
    '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/dataset/annotations/instances_val.json',
    backend_args=None,
    metric='bbox',
    type='CocoMetric')
vis_backends = [
    dict(type='LocalVisBackend'),
]
visualizer = dict(
    name='visualizer',
    type='DetLocalVisualizer',
    vis_backends=[
        dict(type='LocalVisBackend'),
    ])
work_dir = '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training'

10/14 17:08:15 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
10/14 17:08:15 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_load_checkpoint:
(49          ) EMAHook                            
 -------------------- 
before_train:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_train_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(48          ) YOLOXModeSwitchHook                
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DistSamplerSeedHook                
 -------------------- 
before_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
after_train_epoch:
(NORMAL      ) IterTimerHook                      
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_val:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_val_epoch:
(48          ) SyncNormHook                       
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
 -------------------- 
before_val_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_val_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DetVisualizationHook               
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_val_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
after_val:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_save_checkpoint:
(49          ) EMAHook                            
 -------------------- 
after_train:
(VERY_HIGH   ) RuntimeInfoHook                    
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_test:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_test_epoch:
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
 -------------------- 
before_test_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_test_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DetVisualizationHook               
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_test_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_test:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
after_run:
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
/home/mchi/work/edgeai-tensorlab/edgeai-modeloptimization/torchmodelopt/edgeai_torchmodelopt/xmodelopt/surgery/v1/__init__.py:68: UserWarning: WARNING - xmodelopt.v1.surgery can only replace modules. To replace functions or operators, please use the torch.fx based xmodelopt.v2.surgery instead
  warnings.warn("WARNING - xmodelopt.v1.surgery can only replace modules. To replace functions or operators, please use the torch.fx based xmodelopt.v2.surgery instead")
model surgery done
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stem.conv_in.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stem.conv_in.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stem.conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stem.conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.reduce_layers.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.reduce_layers.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.reduce_layers.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.reduce_layers.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.downsamples.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.downsamples.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.downsamples.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.downsamples.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.main_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.main_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.short_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.short_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.final_conv.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.final_conv.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.out_convs.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.out_convs.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.out_convs.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.out_convs.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.out_convs.2.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- neck.out_convs.2.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.0.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.0.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.1.bn.weight:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.1.bn.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_cls.0.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_cls.1.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_cls.2.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_reg.0.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_reg.1.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_reg.2.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_obj.0.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_obj.1.bias:weight_decay=0.0
10/14 17:08:17 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_obj.2.bias:weight_decay=0.0
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
10/14 17:08:19 - mmengine - WARNING - init_weights of YOLOX has been called more than once.
Loads checkpoint by local backend from path: ./data/downloads/pretrained/yolox_s_lite/yolox_s_lite_640x640_20220221_checkpoint.pth
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/mmengine/runner/checkpoint.py:347: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(filename, map_location=map_location)
The model and loaded state dict do not match exactly

size mismatch for bbox_head.multi_level_conv_cls.0.weight: copying a param with shape torch.Size([80, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.0.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.1.weight: copying a param with shape torch.Size([80, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.1.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.2.weight: copying a param with shape torch.Size([80, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.2.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
unexpected key in source state_dict: ema_backbone_stem_conv_in_conv_weight, ema_backbone_stem_conv_in_bn_weight, ema_backbone_stem_conv_in_bn_bias, ema_backbone_stem_conv_in_bn_running_mean, ema_backbone_stem_conv_in_bn_running_var, ema_backbone_stem_conv_in_bn_num_batches_tracked, ema_backbone_stem_conv_conv_weight, ema_backbone_stem_conv_bn_weight, ema_backbone_stem_conv_bn_bias, ema_backbone_stem_conv_bn_running_mean, ema_backbone_stem_conv_bn_running_var, ema_backbone_stem_conv_bn_num_batches_tracked, ema_backbone_stage1_0_conv_weight, ema_backbone_stage1_0_bn_weight, ema_backbone_stage1_0_bn_bias, ema_backbone_stage1_0_bn_running_mean, ema_backbone_stage1_0_bn_running_var, ema_backbone_stage1_0_bn_num_batches_tracked, ema_backbone_stage1_1_main_conv_conv_weight, ema_backbone_stage1_1_main_conv_bn_weight, ema_backbone_stage1_1_main_conv_bn_bias, ema_backbone_stage1_1_main_conv_bn_running_mean, ema_backbone_stage1_1_main_conv_bn_running_var, ema_backbone_stage1_1_main_conv_bn_num_batches_tracked, ema_backbone_stage1_1_short_conv_conv_weight, ema_backbone_stage1_1_short_conv_bn_weight, ema_backbone_stage1_1_short_conv_bn_bias, ema_backbone_stage1_1_short_conv_bn_running_mean, ema_backbone_stage1_1_short_conv_bn_running_var, ema_backbone_stage1_1_short_conv_bn_num_batches_tracked, ema_backbone_stage1_1_final_conv_conv_weight, ema_backbone_stage1_1_final_conv_bn_weight, ema_backbone_stage1_1_final_conv_bn_bias, ema_backbone_stage1_1_final_conv_bn_running_mean, ema_backbone_stage1_1_final_conv_bn_running_var, ema_backbone_stage1_1_final_conv_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv1_conv_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_bias, ema_backbone_stage1_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv1_bn_running_var, ema_backbone_stage1_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv2_conv_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_bias, ema_backbone_stage1_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv2_bn_running_var, ema_backbone_stage1_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_0_conv_weight, ema_backbone_stage2_0_bn_weight, ema_backbone_stage2_0_bn_bias, ema_backbone_stage2_0_bn_running_mean, ema_backbone_stage2_0_bn_running_var, ema_backbone_stage2_0_bn_num_batches_tracked, ema_backbone_stage2_1_main_conv_conv_weight, ema_backbone_stage2_1_main_conv_bn_weight, ema_backbone_stage2_1_main_conv_bn_bias, ema_backbone_stage2_1_main_conv_bn_running_mean, ema_backbone_stage2_1_main_conv_bn_running_var, ema_backbone_stage2_1_main_conv_bn_num_batches_tracked, ema_backbone_stage2_1_short_conv_conv_weight, ema_backbone_stage2_1_short_conv_bn_weight, ema_backbone_stage2_1_short_conv_bn_bias, ema_backbone_stage2_1_short_conv_bn_running_mean, ema_backbone_stage2_1_short_conv_bn_running_var, ema_backbone_stage2_1_short_conv_bn_num_batches_tracked, ema_backbone_stage2_1_final_conv_conv_weight, ema_backbone_stage2_1_final_conv_bn_weight, ema_backbone_stage2_1_final_conv_bn_bias, ema_backbone_stage2_1_final_conv_bn_running_mean, ema_backbone_stage2_1_final_conv_bn_running_var, ema_backbone_stage2_1_final_conv_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv1_conv_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_bias, ema_backbone_stage2_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv1_bn_running_var, ema_backbone_stage2_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv2_conv_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_bias, ema_backbone_stage2_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv2_bn_running_var, ema_backbone_stage2_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv1_conv_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_bias, ema_backbone_stage2_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv1_bn_running_var, ema_backbone_stage2_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv2_conv_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_bias, ema_backbone_stage2_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv2_bn_running_var, ema_backbone_stage2_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv1_conv_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_bias, ema_backbone_stage2_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv1_bn_running_var, ema_backbone_stage2_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv2_conv_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_bias, ema_backbone_stage2_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv2_bn_running_var, ema_backbone_stage2_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage3_0_conv_weight, ema_backbone_stage3_0_bn_weight, ema_backbone_stage3_0_bn_bias, ema_backbone_stage3_0_bn_running_mean, ema_backbone_stage3_0_bn_running_var, ema_backbone_stage3_0_bn_num_batches_tracked, ema_backbone_stage3_1_main_conv_conv_weight, ema_backbone_stage3_1_main_conv_bn_weight, ema_backbone_stage3_1_main_conv_bn_bias, ema_backbone_stage3_1_main_conv_bn_running_mean, ema_backbone_stage3_1_main_conv_bn_running_var, ema_backbone_stage3_1_main_conv_bn_num_batches_tracked, ema_backbone_stage3_1_short_conv_conv_weight, ema_backbone_stage3_1_short_conv_bn_weight, ema_backbone_stage3_1_short_conv_bn_bias, ema_backbone_stage3_1_short_conv_bn_running_mean, ema_backbone_stage3_1_short_conv_bn_running_var, ema_backbone_stage3_1_short_conv_bn_num_batches_tracked, ema_backbone_stage3_1_final_conv_conv_weight, ema_backbone_stage3_1_final_conv_bn_weight, ema_backbone_stage3_1_final_conv_bn_bias, ema_backbone_stage3_1_final_conv_bn_running_mean, ema_backbone_stage3_1_final_conv_bn_running_var, ema_backbone_stage3_1_final_conv_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv1_conv_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_bias, ema_backbone_stage3_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv1_bn_running_var, ema_backbone_stage3_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv2_conv_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_bias, ema_backbone_stage3_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv2_bn_running_var, ema_backbone_stage3_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv1_conv_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_bias, ema_backbone_stage3_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv1_bn_running_var, ema_backbone_stage3_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv2_conv_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_bias, ema_backbone_stage3_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv2_bn_running_var, ema_backbone_stage3_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv1_conv_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_bias, ema_backbone_stage3_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv1_bn_running_var, ema_backbone_stage3_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv2_conv_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_bias, ema_backbone_stage3_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv2_bn_running_var, ema_backbone_stage3_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage4_0_conv_weight, ema_backbone_stage4_0_bn_weight, ema_backbone_stage4_0_bn_bias, ema_backbone_stage4_0_bn_running_mean, ema_backbone_stage4_0_bn_running_var, ema_backbone_stage4_0_bn_num_batches_tracked, ema_backbone_stage4_1_conv1_conv_weight, ema_backbone_stage4_1_conv1_bn_weight, ema_backbone_stage4_1_conv1_bn_bias, ema_backbone_stage4_1_conv1_bn_running_mean, ema_backbone_stage4_1_conv1_bn_running_var, ema_backbone_stage4_1_conv1_bn_num_batches_tracked, ema_backbone_stage4_1_conv2_conv_weight, ema_backbone_stage4_1_conv2_bn_weight, ema_backbone_stage4_1_conv2_bn_bias, ema_backbone_stage4_1_conv2_bn_running_mean, ema_backbone_stage4_1_conv2_bn_running_var, ema_backbone_stage4_1_conv2_bn_num_batches_tracked, ema_backbone_stage4_2_main_conv_conv_weight, ema_backbone_stage4_2_main_conv_bn_weight, ema_backbone_stage4_2_main_conv_bn_bias, ema_backbone_stage4_2_main_conv_bn_running_mean, ema_backbone_stage4_2_main_conv_bn_running_var, ema_backbone_stage4_2_main_conv_bn_num_batches_tracked, ema_backbone_stage4_2_short_conv_conv_weight, ema_backbone_stage4_2_short_conv_bn_weight, ema_backbone_stage4_2_short_conv_bn_bias, ema_backbone_stage4_2_short_conv_bn_running_mean, ema_backbone_stage4_2_short_conv_bn_running_var, ema_backbone_stage4_2_short_conv_bn_num_batches_tracked, ema_backbone_stage4_2_final_conv_conv_weight, ema_backbone_stage4_2_final_conv_bn_weight, ema_backbone_stage4_2_final_conv_bn_bias, ema_backbone_stage4_2_final_conv_bn_running_mean, ema_backbone_stage4_2_final_conv_bn_running_var, ema_backbone_stage4_2_final_conv_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv1_conv_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_bias, ema_backbone_stage4_2_blocks_0_conv1_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv1_bn_running_var, ema_backbone_stage4_2_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv2_conv_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_bias, ema_backbone_stage4_2_blocks_0_conv2_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv2_bn_running_var, ema_backbone_stage4_2_blocks_0_conv2_bn_num_batches_tracked, ema_neck_reduce_layers_0_conv_weight, ema_neck_reduce_layers_0_bn_weight, ema_neck_reduce_layers_0_bn_bias, ema_neck_reduce_layers_0_bn_running_mean, ema_neck_reduce_layers_0_bn_running_var, ema_neck_reduce_layers_0_bn_num_batches_tracked, ema_neck_reduce_layers_1_conv_weight, ema_neck_reduce_layers_1_bn_weight, ema_neck_reduce_layers_1_bn_bias, ema_neck_reduce_layers_1_bn_running_mean, ema_neck_reduce_layers_1_bn_running_var, ema_neck_reduce_layers_1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_main_conv_conv_weight, ema_neck_top_down_blocks_0_main_conv_bn_weight, ema_neck_top_down_blocks_0_main_conv_bn_bias, ema_neck_top_down_blocks_0_main_conv_bn_running_mean, ema_neck_top_down_blocks_0_main_conv_bn_running_var, ema_neck_top_down_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_short_conv_conv_weight, ema_neck_top_down_blocks_0_short_conv_bn_weight, ema_neck_top_down_blocks_0_short_conv_bn_bias, ema_neck_top_down_blocks_0_short_conv_bn_running_mean, ema_neck_top_down_blocks_0_short_conv_bn_running_var, ema_neck_top_down_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_final_conv_conv_weight, ema_neck_top_down_blocks_0_final_conv_bn_weight, ema_neck_top_down_blocks_0_final_conv_bn_bias, ema_neck_top_down_blocks_0_final_conv_bn_running_mean, ema_neck_top_down_blocks_0_final_conv_bn_running_var, ema_neck_top_down_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_1_main_conv_conv_weight, ema_neck_top_down_blocks_1_main_conv_bn_weight, ema_neck_top_down_blocks_1_main_conv_bn_bias, ema_neck_top_down_blocks_1_main_conv_bn_running_mean, ema_neck_top_down_blocks_1_main_conv_bn_running_var, ema_neck_top_down_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_short_conv_conv_weight, ema_neck_top_down_blocks_1_short_conv_bn_weight, ema_neck_top_down_blocks_1_short_conv_bn_bias, ema_neck_top_down_blocks_1_short_conv_bn_running_mean, ema_neck_top_down_blocks_1_short_conv_bn_running_var, ema_neck_top_down_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_final_conv_conv_weight, ema_neck_top_down_blocks_1_final_conv_bn_weight, ema_neck_top_down_blocks_1_final_conv_bn_bias, ema_neck_top_down_blocks_1_final_conv_bn_running_mean, ema_neck_top_down_blocks_1_final_conv_bn_running_var, ema_neck_top_down_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_downsamples_0_conv_weight, ema_neck_downsamples_0_bn_weight, ema_neck_downsamples_0_bn_bias, ema_neck_downsamples_0_bn_running_mean, ema_neck_downsamples_0_bn_running_var, ema_neck_downsamples_0_bn_num_batches_tracked, ema_neck_downsamples_1_conv_weight, ema_neck_downsamples_1_bn_weight, ema_neck_downsamples_1_bn_bias, ema_neck_downsamples_1_bn_running_mean, ema_neck_downsamples_1_bn_running_var, ema_neck_downsamples_1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_main_conv_conv_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_bias, ema_neck_bottom_up_blocks_0_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_main_conv_bn_running_var, ema_neck_bottom_up_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_short_conv_conv_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_bias, ema_neck_bottom_up_blocks_0_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_short_conv_bn_running_var, ema_neck_bottom_up_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_final_conv_conv_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_bias, ema_neck_bottom_up_blocks_0_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_final_conv_bn_running_var, ema_neck_bottom_up_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_main_conv_conv_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_bias, ema_neck_bottom_up_blocks_1_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_main_conv_bn_running_var, ema_neck_bottom_up_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_short_conv_conv_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_bias, ema_neck_bottom_up_blocks_1_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_short_conv_bn_running_var, ema_neck_bottom_up_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_final_conv_conv_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_bias, ema_neck_bottom_up_blocks_1_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_final_conv_bn_running_var, ema_neck_bottom_up_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_out_convs_0_conv_weight, ema_neck_out_convs_0_bn_weight, ema_neck_out_convs_0_bn_bias, ema_neck_out_convs_0_bn_running_mean, ema_neck_out_convs_0_bn_running_var, ema_neck_out_convs_0_bn_num_batches_tracked, ema_neck_out_convs_1_conv_weight, ema_neck_out_convs_1_bn_weight, ema_neck_out_convs_1_bn_bias, ema_neck_out_convs_1_bn_running_mean, ema_neck_out_convs_1_bn_running_var, ema_neck_out_convs_1_bn_num_batches_tracked, ema_neck_out_convs_2_conv_weight, ema_neck_out_convs_2_bn_weight, ema_neck_out_convs_2_bn_bias, ema_neck_out_convs_2_bn_running_mean, ema_neck_out_convs_2_bn_running_var, ema_neck_out_convs_2_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_0_conv_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_bias, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_1_conv_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_bias, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_0_conv_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_bias, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_1_conv_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_bias, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_0_conv_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_bias, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_1_conv_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_bias, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_0_conv_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_bias, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_1_conv_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_bias, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_0_conv_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_bias, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_1_conv_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_bias, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_0_conv_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_bias, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_1_conv_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_bias, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_conv_cls_0_weight, ema_bbox_head_multi_level_conv_cls_0_bias, ema_bbox_head_multi_level_conv_cls_1_weight, ema_bbox_head_multi_level_conv_cls_1_bias, ema_bbox_head_multi_level_conv_cls_2_weight, ema_bbox_head_multi_level_conv_cls_2_bias, ema_bbox_head_multi_level_conv_reg_0_weight, ema_bbox_head_multi_level_conv_reg_0_bias, ema_bbox_head_multi_level_conv_reg_1_weight, ema_bbox_head_multi_level_conv_reg_1_bias, ema_bbox_head_multi_level_conv_reg_2_weight, ema_bbox_head_multi_level_conv_reg_2_bias, ema_bbox_head_multi_level_conv_obj_0_weight, ema_bbox_head_multi_level_conv_obj_0_bias, ema_bbox_head_multi_level_conv_obj_1_weight, ema_bbox_head_multi_level_conv_obj_1_bias, ema_bbox_head_multi_level_conv_obj_2_weight, ema_bbox_head_multi_level_conv_obj_2_bias

The model and loaded state dict do not match exactly

size mismatch for bbox_head.multi_level_conv_cls.0.weight: copying a param with shape torch.Size([80, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.0.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.1.weight: copying a param with shape torch.Size([80, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.1.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.2.weight: copying a param with shape torch.Size([80, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.2.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
unexpected key in source state_dict: ema_backbone_stem_conv_in_conv_weight, ema_backbone_stem_conv_in_bn_weight, ema_backbone_stem_conv_in_bn_bias, ema_backbone_stem_conv_in_bn_running_mean, ema_backbone_stem_conv_in_bn_running_var, ema_backbone_stem_conv_in_bn_num_batches_tracked, ema_backbone_stem_conv_conv_weight, ema_backbone_stem_conv_bn_weight, ema_backbone_stem_conv_bn_bias, ema_backbone_stem_conv_bn_running_mean, ema_backbone_stem_conv_bn_running_var, ema_backbone_stem_conv_bn_num_batches_tracked, ema_backbone_stage1_0_conv_weight, ema_backbone_stage1_0_bn_weight, ema_backbone_stage1_0_bn_bias, ema_backbone_stage1_0_bn_running_mean, ema_backbone_stage1_0_bn_running_var, ema_backbone_stage1_0_bn_num_batches_tracked, ema_backbone_stage1_1_main_conv_conv_weight, ema_backbone_stage1_1_main_conv_bn_weight, ema_backbone_stage1_1_main_conv_bn_bias, ema_backbone_stage1_1_main_conv_bn_running_mean, ema_backbone_stage1_1_main_conv_bn_running_var, ema_backbone_stage1_1_main_conv_bn_num_batches_tracked, ema_backbone_stage1_1_short_conv_conv_weight, ema_backbone_stage1_1_short_conv_bn_weight, ema_backbone_stage1_1_short_conv_bn_bias, ema_backbone_stage1_1_short_conv_bn_running_mean, ema_backbone_stage1_1_short_conv_bn_running_var, ema_backbone_stage1_1_short_conv_bn_num_batches_tracked, ema_backbone_stage1_1_final_conv_conv_weight, ema_backbone_stage1_1_final_conv_bn_weight, ema_backbone_stage1_1_final_conv_bn_bias, ema_backbone_stage1_1_final_conv_bn_running_mean, ema_backbone_stage1_1_final_conv_bn_running_var, ema_backbone_stage1_1_final_conv_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv1_conv_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_bias, ema_backbone_stage1_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv1_bn_running_var, ema_backbone_stage1_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv2_conv_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_bias, ema_backbone_stage1_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv2_bn_running_var, ema_backbone_stage1_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_0_conv_weight, ema_backbone_stage2_0_bn_weight, ema_backbone_stage2_0_bn_bias, ema_backbone_stage2_0_bn_running_mean, ema_backbone_stage2_0_bn_running_var, ema_backbone_stage2_0_bn_num_batches_tracked, ema_backbone_stage2_1_main_conv_conv_weight, ema_backbone_stage2_1_main_conv_bn_weight, ema_backbone_stage2_1_main_conv_bn_bias, ema_backbone_stage2_1_main_conv_bn_running_mean, ema_backbone_stage2_1_main_conv_bn_running_var, ema_backbone_stage2_1_main_conv_bn_num_batches_tracked, ema_backbone_stage2_1_short_conv_conv_weight, ema_backbone_stage2_1_short_conv_bn_weight, ema_backbone_stage2_1_short_conv_bn_bias, ema_backbone_stage2_1_short_conv_bn_running_mean, ema_backbone_stage2_1_short_conv_bn_running_var, ema_backbone_stage2_1_short_conv_bn_num_batches_tracked, ema_backbone_stage2_1_final_conv_conv_weight, ema_backbone_stage2_1_final_conv_bn_weight, ema_backbone_stage2_1_final_conv_bn_bias, ema_backbone_stage2_1_final_conv_bn_running_mean, ema_backbone_stage2_1_final_conv_bn_running_var, ema_backbone_stage2_1_final_conv_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv1_conv_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_bias, ema_backbone_stage2_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv1_bn_running_var, ema_backbone_stage2_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv2_conv_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_bias, ema_backbone_stage2_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv2_bn_running_var, ema_backbone_stage2_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv1_conv_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_bias, ema_backbone_stage2_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv1_bn_running_var, ema_backbone_stage2_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv2_conv_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_bias, ema_backbone_stage2_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv2_bn_running_var, ema_backbone_stage2_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv1_conv_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_bias, ema_backbone_stage2_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv1_bn_running_var, ema_backbone_stage2_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv2_conv_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_bias, ema_backbone_stage2_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv2_bn_running_var, ema_backbone_stage2_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage3_0_conv_weight, ema_backbone_stage3_0_bn_weight, ema_backbone_stage3_0_bn_bias, ema_backbone_stage3_0_bn_running_mean, ema_backbone_stage3_0_bn_running_var, ema_backbone_stage3_0_bn_num_batches_tracked, ema_backbone_stage3_1_main_conv_conv_weight, ema_backbone_stage3_1_main_conv_bn_weight, ema_backbone_stage3_1_main_conv_bn_bias, ema_backbone_stage3_1_main_conv_bn_running_mean, ema_backbone_stage3_1_main_conv_bn_running_var, ema_backbone_stage3_1_main_conv_bn_num_batches_tracked, ema_backbone_stage3_1_short_conv_conv_weight, ema_backbone_stage3_1_short_conv_bn_weight, ema_backbone_stage3_1_short_conv_bn_bias, ema_backbone_stage3_1_short_conv_bn_running_mean, ema_backbone_stage3_1_short_conv_bn_running_var, ema_backbone_stage3_1_short_conv_bn_num_batches_tracked, ema_backbone_stage3_1_final_conv_conv_weight, ema_backbone_stage3_1_final_conv_bn_weight, ema_backbone_stage3_1_final_conv_bn_bias, ema_backbone_stage3_1_final_conv_bn_running_mean, ema_backbone_stage3_1_final_conv_bn_running_var, ema_backbone_stage3_1_final_conv_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv1_conv_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_bias, ema_backbone_stage3_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv1_bn_running_var, ema_backbone_stage3_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv2_conv_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_bias, ema_backbone_stage3_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv2_bn_running_var, ema_backbone_stage3_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv1_conv_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_bias, ema_backbone_stage3_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv1_bn_running_var, ema_backbone_stage3_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv2_conv_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_bias, ema_backbone_stage3_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv2_bn_running_var, ema_backbone_stage3_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv1_conv_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_bias, ema_backbone_stage3_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv1_bn_running_var, ema_backbone_stage3_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv2_conv_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_bias, ema_backbone_stage3_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv2_bn_running_var, ema_backbone_stage3_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage4_0_conv_weight, ema_backbone_stage4_0_bn_weight, ema_backbone_stage4_0_bn_bias, ema_backbone_stage4_0_bn_running_mean, ema_backbone_stage4_0_bn_running_var, ema_backbone_stage4_0_bn_num_batches_tracked, ema_backbone_stage4_1_conv1_conv_weight, ema_backbone_stage4_1_conv1_bn_weight, ema_backbone_stage4_1_conv1_bn_bias, ema_backbone_stage4_1_conv1_bn_running_mean, ema_backbone_stage4_1_conv1_bn_running_var, ema_backbone_stage4_1_conv1_bn_num_batches_tracked, ema_backbone_stage4_1_conv2_conv_weight, ema_backbone_stage4_1_conv2_bn_weight, ema_backbone_stage4_1_conv2_bn_bias, ema_backbone_stage4_1_conv2_bn_running_mean, ema_backbone_stage4_1_conv2_bn_running_var, ema_backbone_stage4_1_conv2_bn_num_batches_tracked, ema_backbone_stage4_2_main_conv_conv_weight, ema_backbone_stage4_2_main_conv_bn_weight, ema_backbone_stage4_2_main_conv_bn_bias, ema_backbone_stage4_2_main_conv_bn_running_mean, ema_backbone_stage4_2_main_conv_bn_running_var, ema_backbone_stage4_2_main_conv_bn_num_batches_tracked, ema_backbone_stage4_2_short_conv_conv_weight, ema_backbone_stage4_2_short_conv_bn_weight, ema_backbone_stage4_2_short_conv_bn_bias, ema_backbone_stage4_2_short_conv_bn_running_mean, ema_backbone_stage4_2_short_conv_bn_running_var, ema_backbone_stage4_2_short_conv_bn_num_batches_tracked, ema_backbone_stage4_2_final_conv_conv_weight, ema_backbone_stage4_2_final_conv_bn_weight, ema_backbone_stage4_2_final_conv_bn_bias, ema_backbone_stage4_2_final_conv_bn_running_mean, ema_backbone_stage4_2_final_conv_bn_running_var, ema_backbone_stage4_2_final_conv_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv1_conv_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_bias, ema_backbone_stage4_2_blocks_0_conv1_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv1_bn_running_var, ema_backbone_stage4_2_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv2_conv_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_bias, ema_backbone_stage4_2_blocks_0_conv2_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv2_bn_running_var, ema_backbone_stage4_2_blocks_0_conv2_bn_num_batches_tracked, ema_neck_reduce_layers_0_conv_weight, ema_neck_reduce_layers_0_bn_weight, ema_neck_reduce_layers_0_bn_bias, ema_neck_reduce_layers_0_bn_running_mean, ema_neck_reduce_layers_0_bn_running_var, ema_neck_reduce_layers_0_bn_num_batches_tracked, ema_neck_reduce_layers_1_conv_weight, ema_neck_reduce_layers_1_bn_weight, ema_neck_reduce_layers_1_bn_bias, ema_neck_reduce_layers_1_bn_running_mean, ema_neck_reduce_layers_1_bn_running_var, ema_neck_reduce_layers_1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_main_conv_conv_weight, ema_neck_top_down_blocks_0_main_conv_bn_weight, ema_neck_top_down_blocks_0_main_conv_bn_bias, ema_neck_top_down_blocks_0_main_conv_bn_running_mean, ema_neck_top_down_blocks_0_main_conv_bn_running_var, ema_neck_top_down_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_short_conv_conv_weight, ema_neck_top_down_blocks_0_short_conv_bn_weight, ema_neck_top_down_blocks_0_short_conv_bn_bias, ema_neck_top_down_blocks_0_short_conv_bn_running_mean, ema_neck_top_down_blocks_0_short_conv_bn_running_var, ema_neck_top_down_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_final_conv_conv_weight, ema_neck_top_down_blocks_0_final_conv_bn_weight, ema_neck_top_down_blocks_0_final_conv_bn_bias, ema_neck_top_down_blocks_0_final_conv_bn_running_mean, ema_neck_top_down_blocks_0_final_conv_bn_running_var, ema_neck_top_down_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_1_main_conv_conv_weight, ema_neck_top_down_blocks_1_main_conv_bn_weight, ema_neck_top_down_blocks_1_main_conv_bn_bias, ema_neck_top_down_blocks_1_main_conv_bn_running_mean, ema_neck_top_down_blocks_1_main_conv_bn_running_var, ema_neck_top_down_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_short_conv_conv_weight, ema_neck_top_down_blocks_1_short_conv_bn_weight, ema_neck_top_down_blocks_1_short_conv_bn_bias, ema_neck_top_down_blocks_1_short_conv_bn_running_mean, ema_neck_top_down_blocks_1_short_conv_bn_running_var, ema_neck_top_down_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_final_conv_conv_weight, ema_neck_top_down_blocks_1_final_conv_bn_weight, ema_neck_top_down_blocks_1_final_conv_bn_bias, ema_neck_top_down_blocks_1_final_conv_bn_running_mean, ema_neck_top_down_blocks_1_final_conv_bn_running_var, ema_neck_top_down_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_downsamples_0_conv_weight, ema_neck_downsamples_0_bn_weight, ema_neck_downsamples_0_bn_bias, ema_neck_downsamples_0_bn_running_mean, ema_neck_downsamples_0_bn_running_var, ema_neck_downsamples_0_bn_num_batches_tracked, ema_neck_downsamples_1_conv_weight, ema_neck_downsamples_1_bn_weight, ema_neck_downsamples_1_bn_bias, ema_neck_downsamples_1_bn_running_mean, ema_neck_downsamples_1_bn_running_var, ema_neck_downsamples_1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_main_conv_conv_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_bias, ema_neck_bottom_up_blocks_0_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_main_conv_bn_running_var, ema_neck_bottom_up_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_short_conv_conv_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_bias, ema_neck_bottom_up_blocks_0_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_short_conv_bn_running_var, ema_neck_bottom_up_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_final_conv_conv_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_bias, ema_neck_bottom_up_blocks_0_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_final_conv_bn_running_var, ema_neck_bottom_up_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_main_conv_conv_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_bias, ema_neck_bottom_up_blocks_1_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_main_conv_bn_running_var, ema_neck_bottom_up_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_short_conv_conv_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_bias, ema_neck_bottom_up_blocks_1_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_short_conv_bn_running_var, ema_neck_bottom_up_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_final_conv_conv_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_bias, ema_neck_bottom_up_blocks_1_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_final_conv_bn_running_var, ema_neck_bottom_up_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_out_convs_0_conv_weight, ema_neck_out_convs_0_bn_weight, ema_neck_out_convs_0_bn_bias, ema_neck_out_convs_0_bn_running_mean, ema_neck_out_convs_0_bn_running_var, ema_neck_out_convs_0_bn_num_batches_tracked, ema_neck_out_convs_1_conv_weight, ema_neck_out_convs_1_bn_weight, ema_neck_out_convs_1_bn_bias, ema_neck_out_convs_1_bn_running_mean, ema_neck_out_convs_1_bn_running_var, ema_neck_out_convs_1_bn_num_batches_tracked, ema_neck_out_convs_2_conv_weight, ema_neck_out_convs_2_bn_weight, ema_neck_out_convs_2_bn_bias, ema_neck_out_convs_2_bn_running_mean, ema_neck_out_convs_2_bn_running_var, ema_neck_out_convs_2_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_0_conv_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_bias, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_1_conv_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_bias, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_0_conv_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_bias, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_1_conv_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_bias, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_0_conv_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_bias, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_1_conv_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_bias, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_0_conv_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_bias, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_1_conv_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_bias, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_0_conv_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_bias, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_1_conv_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_bias, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_0_conv_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_bias, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_1_conv_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_bias, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_conv_cls_0_weight, ema_bbox_head_multi_level_conv_cls_0_bias, ema_bbox_head_multi_level_conv_cls_1_weight, ema_bbox_head_multi_level_conv_cls_1_bias, ema_bbox_head_multi_level_conv_cls_2_weight, ema_bbox_head_multi_level_conv_cls_2_bias, ema_bbox_head_multi_level_conv_reg_0_weight, ema_bbox_head_multi_level_conv_reg_0_bias, ema_bbox_head_multi_level_conv_reg_1_weight, ema_bbox_head_multi_level_conv_reg_1_bias, ema_bbox_head_multi_level_conv_reg_2_weight, ema_bbox_head_multi_level_conv_reg_2_bias, ema_bbox_head_multi_level_conv_obj_0_weight, ema_bbox_head_multi_level_conv_obj_0_bias, ema_bbox_head_multi_level_conv_obj_1_weight, ema_bbox_head_multi_level_conv_obj_1_bias, ema_bbox_head_multi_level_conv_obj_2_weight, ema_bbox_head_multi_level_conv_obj_2_bias

10/14 17:08:19 - mmengine - INFO - Load checkpoint from ./data/downloads/pretrained/yolox_s_lite/yolox_s_lite_640x640_20220221_checkpoint.pth
10/14 17:08:19 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
10/14 17:08:19 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
10/14 17:08:19 - mmengine - INFO - Checkpoints will be saved to /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training.
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/functional.py:513: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3609.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:09:49 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:09:49 - mmengine - INFO - Epoch(train)  [1][23/23]  base_lr: 8.0000e-05 lr: 8.0000e-05  eta: 0:43:27  time: 3.9097  data_time: 0.0825  loss: 14.1700  loss_cls: 1.7932  loss_bbox: 3.9160  loss_obj: 8.4608
10/14 17:09:49 - mmengine - INFO - Saving checkpoint at 1 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:09:56 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.040
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.061
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.055
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.018
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.085
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.052
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.052
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.052
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.035
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.092
10/14 17:09:57 - mmengine - INFO - bbox_mAP_copypaste: 0.040 0.061 0.055 -1.000 0.018 0.085
10/14 17:09:57 - mmengine - INFO - Epoch(val) [1][6/6]    coco/bbox_mAP: 0.0400  coco/bbox_mAP_50: 0.0610  coco/bbox_mAP_75: 0.0550  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.0180  coco/bbox_mAP_l: 0.0850  data_time: 0.0524  time: 0.9184
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:10:41 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:10:41 - mmengine - INFO - Epoch(train)  [2][23/23]  base_lr: 3.2000e-04 lr: 3.2000e-04  eta: 0:31:15  time: 2.9130  data_time: 0.0626  loss: 11.7472  loss_cls: 1.4722  loss_bbox: 3.6515  loss_obj: 6.6235
10/14 17:10:41 - mmengine - INFO - Saving checkpoint at 2 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:10:48 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.07s).
Accumulating evaluation results...
DONE (t=0.06s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.096
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.269
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.038
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.109
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.058
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.292
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.292
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.292
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.210
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.263
10/14 17:10:48 - mmengine - INFO - bbox_mAP_copypaste: 0.096 0.269 0.038 -1.000 0.109 0.058
10/14 17:10:48 - mmengine - INFO - Epoch(val) [2][6/6]    coco/bbox_mAP: 0.0960  coco/bbox_mAP_50: 0.2690  coco/bbox_mAP_75: 0.0380  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.1090  coco/bbox_mAP_l: 0.0580  data_time: 0.0187  time: 0.8073
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:11:39 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:11:39 - mmengine - INFO - Epoch(train)  [3][23/23]  base_lr: 7.2000e-04 lr: 7.2000e-04  eta: 0:27:47  time: 2.1225  data_time: 0.0406  loss: 8.7936  loss_cls: 1.1047  loss_bbox: 3.3019  loss_obj: 4.3869
10/14 17:11:39 - mmengine - INFO - Saving checkpoint at 3 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:11:46 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.047
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.168
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.011
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.064
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.101
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.153
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.153
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.153
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.170
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.156
10/14 17:11:46 - mmengine - INFO - bbox_mAP_copypaste: 0.047 0.168 0.011 -1.000 0.064 0.101
10/14 17:11:46 - mmengine - INFO - Epoch(val) [3][6/6]    coco/bbox_mAP: 0.0470  coco/bbox_mAP_50: 0.1680  coco/bbox_mAP_75: 0.0110  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.0640  coco/bbox_mAP_l: 0.1010  data_time: 0.0159  time: 0.7620
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:12:51 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:12:51 - mmengine - INFO - Epoch(train)  [4][23/23]  base_lr: 1.2800e-03 lr: 1.2800e-03  eta: 0:27:07  time: 2.4393  data_time: 0.0407  loss: 7.7786  loss_cls: 0.9528  loss_bbox: 3.2344  loss_obj: 3.5913
10/14 17:12:51 - mmengine - INFO - Saving checkpoint at 4 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:12:59 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.03s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.002
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.010
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.002
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.076
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.076
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.076
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.010
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.076
10/14 17:12:59 - mmengine - INFO - bbox_mAP_copypaste: 0.002 0.010 0.000 -1.000 0.000 0.002
10/14 17:12:59 - mmengine - INFO - Epoch(val) [4][6/6]    coco/bbox_mAP: 0.0020  coco/bbox_mAP_50: 0.0100  coco/bbox_mAP_75: 0.0000  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.0000  coco/bbox_mAP_l: 0.0020  data_time: 0.0192  time: 0.7781
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:13:59 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:13:59 - mmengine - INFO - Epoch(train)  [5][23/23]  base_lr: 2.0000e-03 lr: 2.0000e-03  eta: 0:25:55  time: 2.6272  data_time: 0.0398  loss: 7.5545  loss_cls: 0.8969  loss_bbox: 3.3021  loss_obj: 3.3554
10/14 17:13:59 - mmengine - INFO - Saving checkpoint at 5 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:14:09 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.23s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.61s).
Accumulating evaluation results...
DONE (t=0.16s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.002
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.018
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.003
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.065
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.073
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.075
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.075
10/14 17:14:10 - mmengine - INFO - bbox_mAP_copypaste: 0.002 0.018 0.000 -1.000 0.000 0.003
10/14 17:14:10 - mmengine - INFO - Epoch(val) [5][6/6]    coco/bbox_mAP: 0.0020  coco/bbox_mAP_50: 0.0180  coco/bbox_mAP_75: 0.0000  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.0000  coco/bbox_mAP_l: 0.0030  data_time: 0.0193  time: 0.8709
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:15:40 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:15:40 - mmengine - INFO - Epoch(train)  [6][23/23]  base_lr: 1.9850e-03 lr: 1.9850e-03  eta: 0:26:45  time: 3.2615  data_time: 0.0386  loss: 7.4526  loss_cls: 0.8566  loss_bbox: 3.2784  loss_obj: 3.3176
10/14 17:15:40 - mmengine - INFO - Saving checkpoint at 6 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:15:47 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.05s).
Accumulating evaluation results...
DONE (t=0.05s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.031
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.102
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.002
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.108
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.008
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.153
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.153
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.153
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.203
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.116
10/14 17:15:47 - mmengine - INFO - bbox_mAP_copypaste: 0.031 0.102 0.002 -1.000 0.108 0.008
10/14 17:15:47 - mmengine - INFO - Epoch(val) [6][6/6]    coco/bbox_mAP: 0.0310  coco/bbox_mAP_50: 0.1020  coco/bbox_mAP_75: 0.0020  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.1080  coco/bbox_mAP_l: 0.0080  data_time: 0.0156  time: 0.7666
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:16:55 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:16:55 - mmengine - INFO - Epoch(train)  [7][23/23]  base_lr: 1.9379e-03 lr: 1.9379e-03  eta: 0:25:41  time: 3.4087  data_time: 0.0383  loss: 7.3860  loss_cls: 0.8140  loss_bbox: 3.2512  loss_obj: 3.3208
10/14 17:16:55 - mmengine - INFO - Saving checkpoint at 7 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:17:02 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.041
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.138
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.010
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.039
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.054
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.188
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.188
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.188
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.133
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.187
10/14 17:17:02 - mmengine - INFO - bbox_mAP_copypaste: 0.041 0.138 0.010 -1.000 0.039 0.054
10/14 17:17:02 - mmengine - INFO - Epoch(val) [7][6/6]    coco/bbox_mAP: 0.0410  coco/bbox_mAP_50: 0.1380  coco/bbox_mAP_75: 0.0100  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.0390  coco/bbox_mAP_l: 0.0540  data_time: 0.0155  time: 0.7560
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:18:03 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:18:03 - mmengine - INFO - Epoch(train)  [8][23/23]  base_lr: 1.8608e-03 lr: 1.8608e-03  eta: 0:24:18  time: 2.7711  data_time: 0.0394  loss: 7.0824  loss_cls: 0.7829  loss_bbox: 3.1976  loss_obj: 3.1019
10/14 17:18:03 - mmengine - INFO - Saving checkpoint at 8 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:18:11 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.145
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.378
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.053
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.158
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.137
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.295
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.295
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.295
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.248
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.294
10/14 17:18:11 - mmengine - INFO - bbox_mAP_copypaste: 0.145 0.378 0.053 -1.000 0.158 0.137
10/14 17:18:11 - mmengine - INFO - Epoch(val) [8][6/6]    coco/bbox_mAP: 0.1450  coco/bbox_mAP_50: 0.3780  coco/bbox_mAP_75: 0.0530  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.1580  coco/bbox_mAP_l: 0.1370  data_time: 0.0191  time: 0.7843
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:19:04 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:19:04 - mmengine - INFO - Epoch(train)  [9][23/23]  base_lr: 1.7569e-03 lr: 1.7569e-03  eta: 0:22:42  time: 2.5368  data_time: 0.0391  loss: 6.6590  loss_cls: 0.7643  loss_bbox: 3.1074  loss_obj: 2.7873
10/14 17:19:04 - mmengine - INFO - Saving checkpoint at 9 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:19:11 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.193
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.569
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.027
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.245
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.198
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.352
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.352
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.352
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.523
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.307
10/14 17:19:11 - mmengine - INFO - bbox_mAP_copypaste: 0.193 0.569 0.027 -1.000 0.245 0.198
10/14 17:19:11 - mmengine - INFO - Epoch(val) [9][6/6]    coco/bbox_mAP: 0.1930  coco/bbox_mAP_50: 0.5690  coco/bbox_mAP_75: 0.0270  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2450  coco/bbox_mAP_l: 0.1980  data_time: 0.0181  time: 0.7707
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:20:00 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:20:00 - mmengine - INFO - Epoch(train) [10][23/23]  base_lr: 1.6309e-03 lr: 1.6309e-03  eta: 0:21:05  time: 2.2253  data_time: 0.0389  loss: 6.2811  loss_cls: 0.7287  loss_bbox: 3.0205  loss_obj: 2.5319
10/14 17:20:00 - mmengine - INFO - Saving checkpoint at 10 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:20:07 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.225
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.702
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.119
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.213
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.245
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.403
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.405
10/14 17:20:07 - mmengine - INFO - bbox_mAP_copypaste: 0.225 0.702 0.119 -1.000 0.213 0.245
10/14 17:20:07 - mmengine - INFO - Epoch(val) [10][6/6]    coco/bbox_mAP: 0.2250  coco/bbox_mAP_50: 0.7020  coco/bbox_mAP_75: 0.1190  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2130  coco/bbox_mAP_l: 0.2450  data_time: 0.0177  time: 0.7796
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:21:03 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:21:03 - mmengine - INFO - Epoch(train) [11][23/23]  base_lr: 1.4882e-03 lr: 1.4882e-03  eta: 0:19:49  time: 2.2829  data_time: 0.0380  loss: 6.0295  loss_cls: 0.6970  loss_bbox: 2.9536  loss_obj: 2.3789
10/14 17:21:03 - mmengine - INFO - Saving checkpoint at 11 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:21:09 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.196
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.537
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.068
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.235
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.198
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.298
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.298
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.298
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.415
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.249
10/14 17:21:09 - mmengine - INFO - bbox_mAP_copypaste: 0.196 0.537 0.068 -1.000 0.235 0.198
10/14 17:21:09 - mmengine - INFO - Epoch(val) [11][6/6]    coco/bbox_mAP: 0.1960  coco/bbox_mAP_50: 0.5370  coco/bbox_mAP_75: 0.0680  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2350  coco/bbox_mAP_l: 0.1980  data_time: 0.0147  time: 0.7562
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:22:07 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:22:07 - mmengine - INFO - Epoch(train) [12][23/23]  base_lr: 1.3352e-03 lr: 1.3352e-03  eta: 0:18:39  time: 2.4092  data_time: 0.0380  loss: 5.8786  loss_cls: 0.6867  loss_bbox: 2.9341  loss_obj: 2.2578
10/14 17:22:07 - mmengine - INFO - Saving checkpoint at 12 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:22:15 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.275
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.649
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.146
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.208
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.305
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.403
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.403
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.403
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.310
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.426
10/14 17:22:15 - mmengine - INFO - bbox_mAP_copypaste: 0.275 0.649 0.146 -1.000 0.208 0.305
10/14 17:22:15 - mmengine - INFO - Epoch(val) [12][6/6]    coco/bbox_mAP: 0.2750  coco/bbox_mAP_50: 0.6490  coco/bbox_mAP_75: 0.1460  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2080  coco/bbox_mAP_l: 0.3050  data_time: 0.0177  time: 0.7846
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:23:36 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:23:36 - mmengine - INFO - Epoch(train) [13][23/23]  base_lr: 1.1784e-03 lr: 1.1784e-03  eta: 0:18:02  time: 2.9279  data_time: 0.0401  loss: 5.7397  loss_cls: 0.6928  loss_bbox: 2.9028  loss_obj: 2.1441
10/14 17:23:36 - mmengine - INFO - Saving checkpoint at 13 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:23:43 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.321
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.698
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.221
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.252
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.351
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.431
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.431
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.431
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.345
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.438
10/14 17:23:43 - mmengine - INFO - bbox_mAP_copypaste: 0.321 0.698 0.221 -1.000 0.252 0.351
10/14 17:23:43 - mmengine - INFO - Epoch(val) [13][6/6]    coco/bbox_mAP: 0.3210  coco/bbox_mAP_50: 0.6980  coco/bbox_mAP_75: 0.2210  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2520  coco/bbox_mAP_l: 0.3510  data_time: 0.0165  time: 0.7822
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:25:09 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:25:09 - mmengine - INFO - Epoch(train) [14][23/23]  base_lr: 1.0247e-03 lr: 1.0247e-03  eta: 0:17:23  time: 3.4991  data_time: 0.0406  loss: 5.6920  loss_cls: 0.6942  loss_bbox: 2.8824  loss_obj: 2.1154
10/14 17:25:09 - mmengine - INFO - Saving checkpoint at 14 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:25:16 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.323
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.787
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.182
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.265
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.353
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.465
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.465
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.465
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.473
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.460
10/14 17:25:16 - mmengine - INFO - bbox_mAP_copypaste: 0.323 0.787 0.182 -1.000 0.265 0.353
10/14 17:25:16 - mmengine - INFO - Epoch(val) [14][6/6]    coco/bbox_mAP: 0.3230  coco/bbox_mAP_50: 0.7870  coco/bbox_mAP_75: 0.1820  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2650  coco/bbox_mAP_l: 0.3530  data_time: 0.0190  time: 0.8488
10/14 17:25:16 - mmengine - INFO - No mosaic and mixup aug now!
10/14 17:25:16 - mmengine - INFO - Add additional L1 loss now!
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:26:10 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:26:10 - mmengine - INFO - Epoch(train) [15][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:16:07  time: 2.9382  data_time: 0.0389  loss: 5.4552  loss_cls: 0.6597  loss_bbox: 2.7127  loss_obj: 1.7169  loss_l1: 0.7955
10/14 17:26:10 - mmengine - INFO - Saving checkpoint at 15 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:26:18 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.291
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.854
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.053
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.247
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.306
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.410
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.410
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.410
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.411
10/14 17:26:18 - mmengine - INFO - bbox_mAP_copypaste: 0.291 0.854 0.053 -1.000 0.247 0.306
10/14 17:26:18 - mmengine - INFO - Epoch(val) [15][6/6]    coco/bbox_mAP: 0.2910  coco/bbox_mAP_50: 0.8540  coco/bbox_mAP_75: 0.0530  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2470  coco/bbox_mAP_l: 0.3060  data_time: 0.0169  time: 0.8015
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:26:59 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:26:59 - mmengine - INFO - Epoch(train) [16][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:14:42  time: 2.0695  data_time: 0.0323  loss: 5.1913  loss_cls: 0.6280  loss_bbox: 2.5622  loss_obj: 1.2960  loss_l1: 0.7665
10/14 17:26:59 - mmengine - INFO - Saving checkpoint at 16 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:27:06 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.367
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.836
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.234
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.326
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.399
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.470
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.470
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.470
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.405
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.487
10/14 17:27:06 - mmengine - INFO - bbox_mAP_copypaste: 0.367 0.836 0.234 -1.000 0.326 0.399
10/14 17:27:06 - mmengine - INFO - Epoch(val) [16][6/6]    coco/bbox_mAP: 0.3670  coco/bbox_mAP_50: 0.8360  coco/bbox_mAP_75: 0.2340  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.3260  coco/bbox_mAP_l: 0.3990  data_time: 0.0153  time: 0.7815
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:28:32 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:28:32 - mmengine - INFO - Epoch(train) [17][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:13:56  time: 2.6791  data_time: 0.0265  loss: 5.0533  loss_cls: 0.6247  loss_bbox: 2.4882  loss_obj: 1.1418  loss_l1: 0.7986
10/14 17:28:32 - mmengine - INFO - Saving checkpoint at 17 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:28:39 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.320
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.831
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.145
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.293
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.337
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.479
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.479
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.479
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.475
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.481
10/14 17:28:39 - mmengine - INFO - bbox_mAP_copypaste: 0.320 0.831 0.145 -1.000 0.293 0.337
10/14 17:28:39 - mmengine - INFO - Epoch(val) [17][6/6]    coco/bbox_mAP: 0.3200  coco/bbox_mAP_50: 0.8310  coco/bbox_mAP_75: 0.1450  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2930  coco/bbox_mAP_l: 0.3370  data_time: 0.0181  time: 0.8011
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:29:48 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:29:48 - mmengine - INFO - Epoch(train) [18][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:12:55  time: 3.1996  data_time: 0.0262  loss: 4.9602  loss_cls: 0.6132  loss_bbox: 2.4011  loss_obj: 1.1398  loss_l1: 0.8061
10/14 17:29:48 - mmengine - INFO - Saving checkpoint at 18 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:29:55 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.374
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.822
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.268
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.324
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.400
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.499
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.499
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.499
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.525
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.492
10/14 17:29:55 - mmengine - INFO - bbox_mAP_copypaste: 0.374 0.822 0.268 -1.000 0.324 0.400
10/14 17:29:55 - mmengine - INFO - Epoch(val) [18][6/6]    coco/bbox_mAP: 0.3740  coco/bbox_mAP_50: 0.8220  coco/bbox_mAP_75: 0.2680  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.3240  coco/bbox_mAP_l: 0.4000  data_time: 0.0209  time: 0.8063
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:31:01 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:31:01 - mmengine - INFO - Epoch(train) [19][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:11:51  time: 3.0501  data_time: 0.0258  loss: 4.8937  loss_cls: 0.6047  loss_bbox: 2.3746  loss_obj: 1.1199  loss_l1: 0.7945
10/14 17:31:01 - mmengine - INFO - Saving checkpoint at 19 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:31:08 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.297
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.709
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.233
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.262
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.338
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.424
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.424
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.424
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.473
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.427
10/14 17:31:08 - mmengine - INFO - bbox_mAP_copypaste: 0.297 0.709 0.233 -1.000 0.262 0.338
10/14 17:31:08 - mmengine - INFO - Epoch(val) [19][6/6]    coco/bbox_mAP: 0.2970  coco/bbox_mAP_50: 0.7090  coco/bbox_mAP_75: 0.2330  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2620  coco/bbox_mAP_l: 0.3380  data_time: 0.0148  time: 0.7591
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:31:54 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:31:54 - mmengine - INFO - Epoch(train) [20][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:10:37  time: 2.4172  data_time: 0.0246  loss: 4.8478  loss_cls: 0.6099  loss_bbox: 2.4079  loss_obj: 1.0558  loss_l1: 0.7742
10/14 17:31:54 - mmengine - INFO - Saving checkpoint at 20 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:32:01 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.385
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.834
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.268
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.290
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.420
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.490
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.490
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.490
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.410
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.517
10/14 17:32:01 - mmengine - INFO - bbox_mAP_copypaste: 0.385 0.834 0.268 -1.000 0.290 0.420
10/14 17:32:01 - mmengine - INFO - Epoch(val) [20][6/6]    coco/bbox_mAP: 0.3850  coco/bbox_mAP_50: 0.8340  coco/bbox_mAP_75: 0.2680  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2900  coco/bbox_mAP_l: 0.4200  data_time: 0.0191  time: 0.7859
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:33:03 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:33:03 - mmengine - INFO - Epoch(train) [21][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:09:33  time: 2.3233  data_time: 0.0255  loss: 4.7330  loss_cls: 0.6053  loss_bbox: 2.3748  loss_obj: 1.0183  loss_l1: 0.7345
10/14 17:33:03 - mmengine - INFO - Saving checkpoint at 21 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:33:10 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.315
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.827
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.173
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.229
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.356
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.373
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.437
10/14 17:33:10 - mmengine - INFO - bbox_mAP_copypaste: 0.315 0.827 0.173 -1.000 0.229 0.356
10/14 17:33:10 - mmengine - INFO - Epoch(val) [21][6/6]    coco/bbox_mAP: 0.3150  coco/bbox_mAP_50: 0.8270  coco/bbox_mAP_75: 0.1730  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2290  coco/bbox_mAP_l: 0.3560  data_time: 0.0178  time: 0.7776
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:34:15 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:34:15 - mmengine - INFO - Epoch(train) [22][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:08:29  time: 2.7059  data_time: 0.0266  loss: 4.7032  loss_cls: 0.6013  loss_bbox: 2.3459  loss_obj: 1.0096  loss_l1: 0.7464
10/14 17:34:15 - mmengine - INFO - Saving checkpoint at 22 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:34:23 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.284
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.774
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.139
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.185
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.314
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.362
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.362
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.362
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.337
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.363
10/14 17:34:23 - mmengine - INFO - bbox_mAP_copypaste: 0.284 0.774 0.139 -1.000 0.185 0.314
10/14 17:34:23 - mmengine - INFO - Epoch(val) [22][6/6]    coco/bbox_mAP: 0.2840  coco/bbox_mAP_50: 0.7740  coco/bbox_mAP_75: 0.1390  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.1850  coco/bbox_mAP_l: 0.3140  data_time: 0.0172  time: 0.7872
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:35:29 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:35:29 - mmengine - INFO - Epoch(train) [23][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:07:26  time: 2.8734  data_time: 0.0259  loss: 4.7343  loss_cls: 0.5980  loss_bbox: 2.3273  loss_obj: 1.0424  loss_l1: 0.7666
10/14 17:35:29 - mmengine - INFO - Saving checkpoint at 23 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:35:36 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.370
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.822
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.302
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.326
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.389
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.508
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.508
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.508
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.550
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.499
10/14 17:35:36 - mmengine - INFO - bbox_mAP_copypaste: 0.370 0.822 0.302 -1.000 0.326 0.389
10/14 17:35:36 - mmengine - INFO - Epoch(val) [23][6/6]    coco/bbox_mAP: 0.3700  coco/bbox_mAP_50: 0.8220  coco/bbox_mAP_75: 0.3020  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.3260  coco/bbox_mAP_l: 0.3890  data_time: 0.0164  time: 0.7707
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:36:49 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:36:49 - mmengine - INFO - Epoch(train) [24][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:06:25  time: 3.0689  data_time: 0.0252  loss: 4.5019  loss_cls: 0.5856  loss_bbox: 2.2552  loss_obj: 0.9185  loss_l1: 0.7427
10/14 17:36:49 - mmengine - INFO - Saving checkpoint at 24 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:36:56 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.432
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.855
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.331
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.274
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.478
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.510
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.510
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.510
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.382
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.547
10/14 17:36:56 - mmengine - INFO - bbox_mAP_copypaste: 0.432 0.855 0.331 -1.000 0.274 0.478
10/14 17:36:56 - mmengine - INFO - Epoch(val) [24][6/6]    coco/bbox_mAP: 0.4320  coco/bbox_mAP_50: 0.8550  coco/bbox_mAP_75: 0.3310  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2740  coco/bbox_mAP_l: 0.4780  data_time: 0.0196  time: 0.7864
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:38:04 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:38:04 - mmengine - INFO - Epoch(train) [25][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:05:21  time: 3.0250  data_time: 0.0256  loss: 4.2019  loss_cls: 0.5724  loss_bbox: 2.1639  loss_obj: 0.7871  loss_l1: 0.6784
10/14 17:38:04 - mmengine - INFO - Saving checkpoint at 25 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:38:11 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.426
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.882
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.285
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.437
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.449
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.493
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.488
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.511
10/14 17:38:11 - mmengine - INFO - bbox_mAP_copypaste: 0.426 0.882 0.285 -1.000 0.437 0.449
10/14 17:38:11 - mmengine - INFO - Epoch(val) [25][6/6]    coco/bbox_mAP: 0.4260  coco/bbox_mAP_50: 0.8820  coco/bbox_mAP_75: 0.2850  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.4370  coco/bbox_mAP_l: 0.4490  data_time: 0.0179  time: 0.7870
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:39:27 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:39:27 - mmengine - INFO - Epoch(train) [26][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:04:19  time: 3.0975  data_time: 0.0255  loss: 4.1790  loss_cls: 0.5728  loss_bbox: 2.1480  loss_obj: 0.7855  loss_l1: 0.6727
10/14 17:39:27 - mmengine - INFO - Saving checkpoint at 26 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:39:34 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.366
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.878
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.137
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.300
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.431
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.451
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.451
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.451
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.425
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.491
10/14 17:39:34 - mmengine - INFO - bbox_mAP_copypaste: 0.366 0.878 0.137 -1.000 0.300 0.431
10/14 17:39:34 - mmengine - INFO - Epoch(val) [26][6/6]    coco/bbox_mAP: 0.3660  coco/bbox_mAP_50: 0.8780  coco/bbox_mAP_75: 0.1370  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.3000  coco/bbox_mAP_l: 0.4310  data_time: 0.0172  time: 0.7640
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:40:53 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:40:53 - mmengine - INFO - Epoch(train) [27][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:03:16  time: 3.2962  data_time: 0.0249  loss: 4.5491  loss_cls: 0.5917  loss_bbox: 2.2908  loss_obj: 0.9215  loss_l1: 0.7451
10/14 17:40:53 - mmengine - INFO - Saving checkpoint at 27 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:41:00 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.353
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.829
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.223
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.261
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.408
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.456
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.456
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.456
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.370
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.497
10/14 17:41:00 - mmengine - INFO - bbox_mAP_copypaste: 0.353 0.829 0.223 -1.000 0.261 0.408
10/14 17:41:00 - mmengine - INFO - Epoch(val) [27][6/6]    coco/bbox_mAP: 0.3530  coco/bbox_mAP_50: 0.8290  coco/bbox_mAP_75: 0.2230  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2610  coco/bbox_mAP_l: 0.4080  data_time: 0.0179  time: 0.7858
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:42:33 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:42:33 - mmengine - INFO - Epoch(train) [28][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:02:12  time: 3.7828  data_time: 0.0252  loss: 4.6422  loss_cls: 0.5966  loss_bbox: 2.3284  loss_obj: 0.9524  loss_l1: 0.7648
10/14 17:42:33 - mmengine - INFO - Saving checkpoint at 28 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:42:40 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.03s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.389
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.873
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.315
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.361
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.425
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.508
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.508
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.508
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.505
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.530
10/14 17:42:40 - mmengine - INFO - bbox_mAP_copypaste: 0.389 0.873 0.315 -1.000 0.361 0.425
10/14 17:42:40 - mmengine - INFO - Epoch(val) [28][6/6]    coco/bbox_mAP: 0.3890  coco/bbox_mAP_50: 0.8730  coco/bbox_mAP_75: 0.3150  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.3610  coco/bbox_mAP_l: 0.4250  data_time: 0.0185  time: 0.7901
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:43:34 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:43:34 - mmengine - INFO - Epoch(train) [29][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:01:05  time: 3.2356  data_time: 0.0249  loss: 4.4983  loss_cls: 0.5885  loss_bbox: 2.2865  loss_obj: 0.8952  loss_l1: 0.7281
10/14 17:43:34 - mmengine - INFO - Saving checkpoint at 29 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:43:41 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.383
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.878
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.233
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.285
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.423
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.473
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.473
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.473
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.437
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.494
10/14 17:43:41 - mmengine - INFO - bbox_mAP_copypaste: 0.383 0.878 0.233 -1.000 0.285 0.423
10/14 17:43:41 - mmengine - INFO - Epoch(val) [29][6/6]    coco/bbox_mAP: 0.3830  coco/bbox_mAP_50: 0.8780  coco/bbox_mAP_75: 0.2330  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2850  coco/bbox_mAP_l: 0.4230  data_time: 0.0182  time: 0.7672
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:45:14 - mmengine - INFO - Exp name: yolox_s_lite_20241014_170811
10/14 17:45:14 - mmengine - INFO - Epoch(train) [30][23/23]  base_lr: 8.8093e-04 lr: 8.8093e-04  eta: 0:00:00  time: 3.1104  data_time: 0.0256  loss: 4.4741  loss_cls: 0.5845  loss_bbox: 2.2724  loss_obj: 0.8956  loss_l1: 0.7216
10/14 17:45:14 - mmengine - INFO - Saving checkpoint at 30 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/14 17:45:21 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.02s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.266
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.661
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.098
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.248
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.262
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.383
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.383
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.383
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.407
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.378
10/14 17:45:21 - mmengine - INFO - bbox_mAP_copypaste: 0.266 0.661 0.098 -1.000 0.248 0.262
10/14 17:45:21 - mmengine - INFO - Epoch(val) [30][6/6]    coco/bbox_mAP: 0.2660  coco/bbox_mAP_50: 0.6610  coco/bbox_mAP_75: 0.0980  coco/bbox_mAP_s: -1.0000  coco/bbox_mAP_m: 0.2480  coco/bbox_mAP_l: 0.2620  data_time: 0.0187  time: 0.7885
10/14 17:45:21 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/14 17:45:21 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/14 17:45:21 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
10/14 17:45:21 - mmengine - INFO - Export PyTorch model to ONNX: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training/model.onnx.
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/onnx/symbolic_opset9.py:5715: UserWarning: Exporting aten::index operator of advanced indexing in opset 17 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
10/14 17:45:23 - mmengine - INFO - Execute onnx optimize passes.
Converted model is valid!
Trained model is at: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training

SUCCESS: ModelMaker - Training completed.

INFO:20241014-174527: model import is in progress - please see the log file for status.
configs to run: ['od-8220']
number of configs: 1

INFO:20241014-174527: parallel_run - parallel_processes:1 parallel_devices=[0]
TASKS                                                       |          |     0% 0/1| [< ]
INFO:20241014-174527: starting process on parallel_device - 0   0%|          || 0/1 [00:00<?, ?it/s]

INFO:20241014-174527: starting - od-8220
INFO:20241014-174527: model_path - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training/model.onnx
INFO:20241014-174527: model_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/compilation/AM68A/work/od-8220/model/model.onnx
INFO:20241014-174527: quant_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/compilation/AM68A/work/od-8220/model/model_qparams.prototxt
Downloading 1/1: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training/model.onnx
Download done for /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training/model.onnx
Downloading 1/1: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training/model.onnx
Download done for /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training/model.onnx

INFO:20241014-174529: running - od-8220
INFO:20241014-174529: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7b00ae8c31f0>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7b012b0ca740>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7b00ae305030>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7b00ae305090>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7b00ae305330>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-s-lite-mmdet-coco-640x640', 'shortlisted': True, 'recommended': True}}
INFO:20241014-174529: import  - od-8220 - this may take some time...
INFO:20241014-174627: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0

INFO:20241014-174727: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0

INFO:20241014-174827: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0

INFO:20241014-174927: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0

INFO:20241014-175027: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7b00fff36050>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7b00fff36050>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process

INFO:20241014-175051: import completed  - od-8220 - 322 sec

SUCCESS:20241014-175051: benchmark results - {}
TASKS                                                       | 100%|██████████|| 1/1 [05:24<00:00, 324.05s/it]
TASKS                                                       | 100%|██████████|| 1/1 [05:23<00:00, 323.95s/it]


INFO:20241014-175051: model inference is in progress - please see the log file for status.
configs to run: ['od-8220']
number of configs: 1

INFO:20241014-175051: parallel_run - parallel_processes:1 parallel_devices=[0]
TASKS                                                       |          |     0% 0/1| [< ]
INFO:20241014-175051: starting process on parallel_device - 0   0%|          || 0/1 [00:00<?, ?it/s]

INFO:20241014-175051: starting - od-8220
INFO:20241014-175051: model_path - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/training/model.onnx
INFO:20241014-175051: model_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/compilation/AM68A/work/od-8220/model/model.onnx
INFO:20241014-175051: quant_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/compilation/AM68A/work/od-8220/model/model_qparams.prototxt

INFO:20241014-175051: running - od-8220
INFO:20241014-175051: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7b00ae8c31f0>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x7b012b0ca740>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7b00ae305030>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x7b00ae305090>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7b00ae305330>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-s-lite-mmdet-coco-640x640', 'shortlisted': True, 'recommended': True}}
INFO:20241014-175051: infer  - od-8220 - this may take some time...libtidl_onnxrt_EP loaded 0x59d9c1d89df0 
Final number of subgraphs created are : 1, - Offloaded Nodes - 271, Total Nodes - 271 
The soft limit is 10240
The hard limit is 10240
MEM: Init ... !!!
MEM: Init ... Done !!!
 0.0s:  VX_ZONE_INIT:Enabled
 0.3s:  VX_ZONE_ERROR:Enabled
 0.4s:  VX_ZONE_WARNING:Enabled
 0.1585s:  VX_ZONE_INIT:[tivxInit:190] Initialization Done !!!
infer : od-8220                                             |  11%|#1        || 5/44 [00:51<06:44, 10.38s/it]
INFO:20241014-175151: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8220                                             |  25%|##5       || 11/44 [01:54<05:43, 10.42s/it]
INFO:20241014-175251: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8220                                             |  39%|###8      || 17/44 [02:56<04:41, 10.41s/it]
INFO:20241014-175351: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8220                                             |  52%|#####2    || 23/44 [03:59<03:38, 10.38s/it]
INFO:20241014-175451: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8220                                             |  64%|######3   || 28/44 [04:51<02:46, 10.38s/it]
INFO:20241014-175551: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8220                                             |  77%|#######7  || 34/44 [05:53<01:44, 10.40s/it]
INFO:20241014-175651: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8220                                             |  91%|######### || 40/44 [06:56<00:41, 10.45s/it]
INFO:20241014-175751: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8220                                             | 100%|##########|| 44/44 [07:38<00:00, 10.41s/it]
MEM: Deinit ... !!!
MEM: Alloc's: 27 alloc's of 179056888 bytes 
MEM: Free's : 27 free's  of 179056888 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7b00fff36050>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x7b00fff36050>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process

INFO:20241014-175830: infer completed  - od-8220 - 459 secLoading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.01s).
Accumulating evaluation results...
DONE (t=0.01s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.257
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.663
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.153
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.214
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.261
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.308
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.311
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.311
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.253
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.324


SUCCESS:20241014-175830: benchmark results - {'infer_path': 'od-8220', 'accuracy_ap[.5:.95]%': 25.728576, 'accuracy_ap50%': 66.30363, 'num_subgraphs': 0, 'infer_time_core_ms': 10367.334968, 'infer_time_subgraph_ms': 0.0, 'ddr_transfer_mb': 0.0, 'perfsim_time_ms': 32.76587, 'perfsim_ddr_transfer_mb': 118.47, 'perfsim_gmacs': 13.316}
TASKS                                                       | 100%|██████████|| 1/1 [07:38<00:00, 458.90s/it]
TASKS                                                       | 100%|██████████|| 1/1 [07:38<00:00, 458.79s/it]

packaging artifacts to /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/compilation/AM68A/pkg please wait...
SUCCESS:20241014-175832: finished packaging - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/compilation/AM68A/work/od-8220
Compiled model is at: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/only_fork/run/20241014-170809/yolox_s_lite/compilation/AM68A/pkg/20241014-170809_yolox_s_lite_onnxrt_AM68A.tar.gz
(py310) mchi@ubuntu22:~/work/edgeai-tensorlab/edgeai-modelmaker$ 

Just to add，even if I train and compile the official routines provided by TI, there are still warnings and assertions，does this have any impact?

10/15 16:57:44 - mmengine - INFO - bbox_mAP_copypaste: 0.127 0.243 0.116 0.002 0.095 0.458
10/15 16:57:44 - mmengine - INFO - Epoch(val) [1][14/14] coco/bbox_mAP: 0.1270 coco/bbox_mAP_50: 0.2430 coco/bbox_mAP_75: 0.1160 coco/bbox_mAP_s: 0.0020 coco/bbox_mAP_m: 0.0950 coco/bbox_mAP_l: 0.4580 data_time: 0.0252 time: 0.1895
10/15 16:57:45 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/15 16:57:45 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/15 16:57:45 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
10/15 16:57:45 - mmengine - INFO - Export PyTorch model to ONNX: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx.
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
with torch.cuda.amp.autocast(enabled=False):
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/onnx/symbolic_opset9.py:5715: UserWarning: Exporting aten::index operator of advanced indexing in opset 17 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
warnings.warn(
10/15 16:57:46 - mmengine - INFO - Execute onnx optimize passes.
Converted model is valid!
Trained model is at: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training

SUCCESS: ModelMaker - Training completed.

Download done for /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx

INFO:20241015-165749: running - od-8200
INFO:20241015-165749: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x746ef39a2140>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x746ef39559c0>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x746e77420a00>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x746e77420a60>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x746e77420d00>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-nano-lite-mmdet-coco-416x416', 'shortlisted': True, 'recommended': True}}
INFO:20241015-165749: import - od-8200 - this may take some time...Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x746ecc22a0e0>
Traceback (most recent call last):
File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
self._shutdown_workers()
File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
if w.is_alive():
File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x746ecc22a0e0>
Traceback (most recent call last):
File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
self._shutdown_workers()
File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
if w.is_alive():
File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process

py310) mchi@ubuntu22:~/work/edgeai-tensorlab/edgeai-modelmaker$ ./run_modelmaker.sh AM68A config_detection.yaml 
Number of AVX cores detected in PC: 16
AVX compilation speedup in PC     : 1
Target device                     : AM68A
PYTHONPATH                        : .:
TIDL_TOOLS_PATH                   : ../edgeai-benchmark/tools/AM68A/tidl_tools
LD_LIBRARY_PATH                   : ../edgeai-benchmark/tools/AM68A/tidl_tools
argv: ['./scripts/run_modelmaker.py', 'config_detection.yaml', '--target_device', 'AM68A']
{'common': {'verbose_mode': True, 'download_path': './data/downloads', 'projects_path': './data/projects', 'project_path': None, 'project_run_path': None, 'task_type': 'detection', 'target_machine': 'evm', 'target_device': 'AM68A', 'run_name': '{date-time}/{model_name}', 'target_module': 'vision'}, 'download': [{'download_url': 'https://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/models/vision/detection/coco/edgeai-mmdet/yolox_nano_lite_416x416_20220214_checkpoint.pth', 'download_path': '{download_path}/pretrained/yolox_nano_lite'}], 'dataset': {'enable': True, 'dataset_name': 'tiscapes2017_driving', 'dataset_path': None, 'extract_path': None, 'split_factor': 0.8, 'split_names': ('train', 'val'), 'max_num_files': 10000, 'input_data_path': 'http://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/datasets/tiscapes2017_driving.zip', 'input_annotation_path': None, 'data_path_splits': None, 'data_dir': 'images', 'annotation_path_splits': None, 'annotation_dir': 'annotations', 'annotation_prefix': 'instances', 'annotation_format': 'coco_json', 'dataset_download': True, 'dataset_reload': False}, 'training': {'enable': True, 'model_name': 'yolox_nano_lite', 'model_training_id': 'yolox_nano_lite', 'training_backend': 'edgeai_mmdetection', 'pretrained_checkpoint_path': {'download_url': 'https://software-dl.ti.com/jacinto7/esd/modelzoo/08_06_00_01/models/vision/detection/coco/edgeai-mmdet/yolox_nano_lite_416x416_20220214_checkpoint.pth', 'download_path': '{download_path}/pretrained/yolox_nano_lite'}, 'pretrained_weight_state_dict_name': None, 'target_devices': {'TDA4VM': {'performance_fps': None, 'performance_infer_time_ms': 3.74, 'accuracy_factor': 40.1, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 24.8, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM62A': {'performance_fps': None, 'performance_infer_time_ms': 8.87, 'accuracy_factor': 40.1, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 24.8, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM67A': {'performance_fps': None, 'performance_infer_time_ms': '8.87 (with 1/2 device capability)', 'accuracy_factor': 40.1, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 24.8, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM68A': {'performance_fps': None, 'performance_infer_time_ms': 3.73, 'accuracy_factor': 40.1, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 24.8, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM69A': {'performance_fps': None, 'performance_infer_time_ms': '3.64 (with 1/4th device capability)', 'accuracy_factor': 40.1, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 24.8, 'accuracy_unit2': 'AP[.5:.95]%'}, 'AM62': {'performance_fps': None, 'performance_infer_time_ms': 516.15, 'accuracy_factor': 40.1, 'accuracy_unit': 'AP50%', 'accuracy_factor2': 24.8, 'accuracy_unit2': 'AP[.5:.95]%'}}, 'project_path': None, 'dataset_path': None, 'training_path': None, 'log_file_path': None, 'log_summary_regex': None, 'summary_file_path': None, 'model_checkpoint_path': None, 'model_export_path': None, 'model_proto_path': None, 'model_packaged_path': None, 'training_epochs': 1, 'warmup_epochs': 1, 'num_last_epochs': 5, 'batch_size': 8, 'learning_rate': 0.002, 'num_classes': None, 'weight_decay': 0.0001, 'input_resize': 416, 'input_cropsize': 416, 'training_device': None, 'num_gpus': 0, 'distributed': True, 'training_master_port': 29500, 'with_background_class': None, 'model_architecture': 'yolox', 'training_devices': {'cpu': True, 'cuda': True}}, 'compilation': {'enable': True, 'preset_name': None, 'model_compilation_id': 'od-8200', 'compilation_path': None, 'model_compiled_path': None, 'log_file_path': None, 'log_summary_regex': None, 'summary_file_path': None, 'output_tensors_path': None, 'model_packaged_path': None, 'model_visualization_path': None, 'tensor_bits': 8, 'calibration_frames': 10, 'calibration_iterations': 10, 'num_frames': None, 'num_output_frames': 50, 'detection_threshold': 0.6, 'detection_top_k': 200, 'save_output': True, 'tidl_offload': True, 'input_optimization': False, 'capture_log': True, 'runtime_options': {'advanced_options:output_feature_16bit_names_list': '/multi_level_conv_obj.2/Conv_output_0, /multi_level_conv_reg.2/Conv_output_0, /multi_level_conv_cls.2/Conv_output_0, /multi_level_conv_obj.1/Conv_output_0, /multi_level_conv_reg.1/Conv_output_0, /multi_level_conv_cls.1/Conv_output_0, /multi_level_conv_obj.0/Conv_output_0, /multi_level_conv_reg.0/Conv_output_0, /multi_level_conv_cls.0/Conv_output_0'}, 'metric': {'label_offset_pred': 0}}}
---------------------------------------------------------------------
Run Name: 20241015-165641/yolox_nano_lite
- Model: yolox_nano_lite
- TargetDevices & Estimated Inference Times (ms): {'TDA4VM': 3.74, 'AM62A': 8.87, 'AM67A': '8.87 (with 1/2 device capability)', 'AM68A': 3.73, 'AM69A': '3.64 (with 1/4th device capability)', 'AM62': 516.15}
- This model can be compiled for the above device(s).
---------------------------------------------------------------------
dataset split sizes {'train': 393, 'val': 107}
max_num_files is set to: 10000
dataset split sizes are limited to: {'train': 393, 'val': 107}
dataset loading OK
loading annotations into memory...
Done (t=0.05s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
Selecting model configs from Python module: ./configs
Run params is at: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/run.yaml
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/mmengine/optim/optimizer/zero_optimizer.py:11: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
  from torch.distributed.optim import \
10/15 16:56:52 - mmengine - INFO - 
------------------------------------------------------------
System environment:
    sys.platform: linux
    Python: 3.10.15 (main, Oct  8 2024, 10:32:26) [GCC 11.4.0]
    CUDA available: False
    MUSA available: False
    numpy_random_seed: 776552229
    GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
    PyTorch: 2.4.0+cpu
    PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.4.0, USE_CUDA=0, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

    TorchVision: 0.19.0+cpu
    OpenCV: 4.10.0
    MMEngine: 0.10.5

Runtime environment:
    cudnn_benchmark: False
    mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
    dist_cfg: {'backend': 'nccl'}
    seed: 776552229
    Distributed launcher: none
    Distributed training: False
    GPU number: 1
------------------------------------------------------------

10/15 16:56:53 - mmengine - INFO - Config:
auto_scale_lr = dict(base_batch_size=64, enable=False)
backend_args = None
base_lr = 0.01
classes = (
    'human',
    'trafficsign',
    'vehicle',
)
convert_to_lite_model = dict(model_surgery=1)
custom_hooks = [
    dict(num_last_epochs=15, priority=48, type='YOLOXModeSwitchHook'),
    dict(priority=48, type='SyncNormHook'),
    dict(
        ema_type='ExpMomentumEMA',
        momentum=0.0001,
        priority=49,
        type='EMAHook',
        update_buffers=True),
]
data_root = '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset'
dataset_type = 'CocoDataset'
default_hooks = dict(
    checkpoint=dict(interval=1, max_keep_ckpts=3, type='CheckpointHook'),
    logger=dict(interval=50, type='LoggerHook'),
    param_scheduler=dict(type='ParamSchedulerHook'),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    timer=dict(type='IterTimerHook'),
    visualization=dict(type='DetVisualizationHook'))
default_scope = 'mmdet'
env_cfg = dict(
    cudnn_benchmark=False,
    dist_cfg=dict(backend='nccl'),
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
export_onnx_model = True
img_scale = (
    640,
    640,
)
img_scales = [
    (
        640,
        640,
    ),
    (
        320,
        320,
    ),
    (
        960,
        960,
    ),
]
interval = 1
launcher = 'none'
load_from = './data/downloads/pretrained/yolox_nano_lite/yolox_nano_lite_416x416_20220214_checkpoint.pth'
log_level = 'INFO'
log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)
max_epochs = 1
model = dict(
    backbone=dict(
        act_cfg=dict(type='ReLU'),
        deepen_factor=0.33,
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        out_indices=(
            2,
            3,
            4,
        ),
        spp_kernal_sizes=(
            5,
            9,
            13,
        ),
        type='CSPDarknet',
        use_depthwise=False,
        widen_factor=0.25),
    bbox_head=dict(
        act_cfg=dict(type='ReLU'),
        feat_channels=64,
        in_channels=64,
        loss_bbox=dict(
            eps=1e-16,
            loss_weight=5.0,
            mode='square',
            reduction='sum',
            type='IoULoss'),
        loss_cls=dict(
            loss_weight=1.0,
            reduction='sum',
            type='CrossEntropyLoss',
            use_sigmoid=True),
        loss_l1=dict(loss_weight=1.0, reduction='sum', type='L1Loss'),
        loss_obj=dict(
            loss_weight=1.0,
            reduction='sum',
            type='CrossEntropyLoss',
            use_sigmoid=True),
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        num_classes=3,
        stacked_convs=2,
        strides=(
            8,
            16,
            32,
        ),
        type='YOLOXHead',
        use_depthwise=False),
    data_preprocessor=dict(
        batch_augments=[
            dict(
                interval=10,
                random_size_range=(
                    320,
                    640,
                ),
                size_divisor=32,
                type='BatchSyncRandomResize'),
        ],
        pad_size_divisor=32,
        type='DetDataPreprocessor'),
    neck=dict(
        act_cfg=dict(type='ReLU'),
        in_channels=[
            64,
            128,
            256,
        ],
        norm_cfg=dict(eps=0.001, momentum=0.03, type='BN'),
        num_csp_blocks=1,
        out_channels=64,
        type='YOLOXPAFPN',
        upsample_cfg=dict(mode='nearest', scale_factor=2),
        use_depthwise=False),
    test_cfg=dict(nms=dict(iou_threshold=0.65, type='nms'), score_thr=0.01),
    train_cfg=dict(assigner=dict(center_radius=2.5, type='SimOTAAssigner')),
    type='YOLOX')
num_last_epochs = 15
optim_wrapper = dict(
    optimizer=dict(
        lr=0.002, momentum=0.9, nesterov=True, type='SGD',
        weight_decay=0.0005),
    paramwise_cfg=dict(bias_decay_mult=0.0, norm_decay_mult=0.0),
    type='OptimWrapper')
param_scheduler = [
    dict(
        begin=0,
        by_epoch=True,
        convert_to_iter_based=True,
        end=5,
        type='mmdet.QuadraticWarmupLR'),
    dict(
        T_max=15,
        begin=5,
        by_epoch=True,
        convert_to_iter_based=True,
        end=15,
        eta_min=0.0005,
        type='CosineAnnealingLR'),
    dict(begin=15, by_epoch=True, end=30, factor=1, type='ConstantLR'),
]
quantization = 0
resume = False
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
    batch_size=8,
    dataset=dict(
        ann_file=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset/annotations/instances_val.json',
        backend_args=None,
        data_prefix=dict(img='val/'),
        data_root=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset',
        metainfo=dict(classes=(
            'human',
            'trafficsign',
            'vehicle',
        )),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=True, scale=(
                416,
                416,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=4,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
    ann_file=
    '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset/annotations/instances_val.json',
    backend_args=None,
    metric='bbox',
    type='CocoMetric')
test_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(keep_ratio=True, scale=(
        416,
        416,
    ), type='Resize'),
    dict(
        pad_to_square=True,
        pad_val=dict(img=(
            114.0,
            114.0,
            114.0,
        )),
        type='Pad'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(
        meta_keys=(
            'img_id',
            'img_path',
            'ori_shape',
            'img_shape',
            'scale_factor',
        ),
        type='PackDetInputs'),
]
train_cfg = dict(max_epochs=1, type='EpochBasedTrainLoop', val_interval=1)
train_dataloader = dict(
    batch_size=8,
    dataset=dict(
        dataset=dict(
            ann_file=
            '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset/annotations/instances_train.json',
            backend_args=None,
            data_prefix=dict(img='train/'),
            data_root=
            '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset',
            filter_cfg=dict(filter_empty_gt=False, min_size=32),
            metainfo=dict(classes=(
                'human',
                'trafficsign',
                'vehicle',
            )),
            pipeline=[
                dict(backend_args=None, type='LoadImageFromFile'),
                dict(type='LoadAnnotations', with_bbox=True),
            ],
            type='CocoDataset'),
        pipeline=[
            dict(img_scale=(
                640,
                640,
            ), pad_val=114.0, type='Mosaic'),
            dict(
                border=(
                    -320,
                    -320,
                ),
                scaling_ratio_range=(
                    0.5,
                    1.5,
                ),
                type='RandomAffine'),
            dict(type='YOLOXHSVRandomAug'),
            dict(prob=0.5, type='RandomFlip'),
            dict(keep_ratio=True, scale=(
                640,
                640,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(
                keep_empty=False,
                min_gt_bbox_wh=(
                    1,
                    1,
                ),
                type='FilterAnnotations'),
            dict(type='PackDetInputs'),
        ],
        type='MultiImageMixDataset'),
    num_workers=4,
    persistent_workers=True,
    sampler=dict(shuffle=True, type='DefaultSampler'))
train_dataset = dict(
    dataset=dict(
        ann_file=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset/annotations/instances_train.json',
        backend_args=None,
        data_prefix=dict(img='train/'),
        data_root=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset',
        filter_cfg=dict(filter_empty_gt=False, min_size=32),
        metainfo=dict(classes=(
            'human',
            'trafficsign',
            'vehicle',
        )),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(type='LoadAnnotations', with_bbox=True),
        ],
        type='CocoDataset'),
    pipeline=[
        dict(img_scale=(
            640,
            640,
        ), pad_val=114.0, type='Mosaic'),
        dict(
            border=(
                -320,
                -320,
            ),
            scaling_ratio_range=(
                0.1,
                2,
            ),
            type='RandomAffine'),
        dict(
            img_scale=(
                640,
                640,
            ),
            pad_val=114.0,
            ratio_range=(
                0.8,
                1.6,
            ),
            type='MixUp'),
        dict(type='YOLOXHSVRandomAug'),
        dict(prob=0.5, type='RandomFlip'),
        dict(keep_ratio=True, scale=(
            640,
            640,
        ), type='Resize'),
        dict(
            pad_to_square=True,
            pad_val=dict(img=(
                114.0,
                114.0,
                114.0,
            )),
            type='Pad'),
        dict(
            keep_empty=False,
            min_gt_bbox_wh=(
                1,
                1,
            ),
            type='FilterAnnotations'),
        dict(type='PackDetInputs'),
    ],
    type='MultiImageMixDataset')
train_pipeline = [
    dict(img_scale=(
        640,
        640,
    ), pad_val=114.0, type='Mosaic'),
    dict(
        border=(
            -320,
            -320,
        ),
        scaling_ratio_range=(
            0.5,
            1.5,
        ),
        type='RandomAffine'),
    dict(type='YOLOXHSVRandomAug'),
    dict(prob=0.5, type='RandomFlip'),
    dict(keep_ratio=True, scale=(
        640,
        640,
    ), type='Resize'),
    dict(
        pad_to_square=True,
        pad_val=dict(img=(
            114.0,
            114.0,
            114.0,
        )),
        type='Pad'),
    dict(keep_empty=False, min_gt_bbox_wh=(
        1,
        1,
    ), type='FilterAnnotations'),
    dict(type='PackDetInputs'),
]
tta_model = dict(
    tta_cfg=dict(max_per_img=100, nms=dict(iou_threshold=0.65, type='nms')),
    type='DetTTAModel')
tta_pipeline = [
    dict(backend_args=None, type='LoadImageFromFile'),
    dict(
        transforms=[
            [
                dict(keep_ratio=True, scale=(
                    640,
                    640,
                ), type='Resize'),
                dict(keep_ratio=True, scale=(
                    320,
                    320,
                ), type='Resize'),
                dict(keep_ratio=True, scale=(
                    960,
                    960,
                ), type='Resize'),
            ],
            [
                dict(prob=1.0, type='RandomFlip'),
                dict(prob=0.0, type='RandomFlip'),
            ],
            [
                dict(
                    pad_to_square=True,
                    pad_val=dict(img=(
                        114.0,
                        114.0,
                        114.0,
                    )),
                    type='Pad'),
            ],
            [
                dict(type='LoadAnnotations', with_bbox=True),
            ],
            [
                dict(
                    meta_keys=(
                        'img_id',
                        'img_path',
                        'ori_shape',
                        'img_shape',
                        'scale_factor',
                        'flip',
                        'flip_direction',
                    ),
                    type='PackDetInputs'),
            ],
        ],
        type='TestTimeAug'),
]
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
    batch_size=8,
    dataset=dict(
        ann_file=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset/annotations/instances_val.json',
        backend_args=None,
        data_prefix=dict(img='val/'),
        data_root=
        '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset',
        metainfo=dict(classes=(
            'human',
            'trafficsign',
            'vehicle',
        )),
        pipeline=[
            dict(backend_args=None, type='LoadImageFromFile'),
            dict(keep_ratio=True, scale=(
                416,
                416,
            ), type='Resize'),
            dict(
                pad_to_square=True,
                pad_val=dict(img=(
                    114.0,
                    114.0,
                    114.0,
                )),
                type='Pad'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(
                meta_keys=(
                    'img_id',
                    'img_path',
                    'ori_shape',
                    'img_shape',
                    'scale_factor',
                ),
                type='PackDetInputs'),
        ],
        test_mode=True,
        type='CocoDataset'),
    drop_last=False,
    num_workers=4,
    persistent_workers=True,
    sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
    ann_file=
    '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/dataset/annotations/instances_val.json',
    backend_args=None,
    metric='bbox',
    type='CocoMetric')
vis_backends = [
    dict(type='LocalVisBackend'),
]
visualizer = dict(
    name='visualizer',
    type='DetLocalVisualizer',
    vis_backends=[
        dict(type='LocalVisBackend'),
    ])
work_dir = '/home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training'

10/15 16:56:55 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
10/15 16:56:55 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_load_checkpoint:
(49          ) EMAHook                            
 -------------------- 
before_train:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_train_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(48          ) YOLOXModeSwitchHook                
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DistSamplerSeedHook                
 -------------------- 
before_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
after_train_epoch:
(NORMAL      ) IterTimerHook                      
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_val:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_val_epoch:
(48          ) SyncNormHook                       
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
 -------------------- 
before_val_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_val_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DetVisualizationHook               
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_val_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
after_val:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_save_checkpoint:
(49          ) EMAHook                            
 -------------------- 
after_train:
(VERY_HIGH   ) RuntimeInfoHook                    
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_test:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
before_test_epoch:
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
 -------------------- 
before_test_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_test_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DetVisualizationHook               
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_test_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(49          ) EMAHook                            
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_test:
(VERY_HIGH   ) RuntimeInfoHook                    
 -------------------- 
after_run:
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
/home/mchi/work/edgeai-tensorlab/edgeai-modeloptimization/torchmodelopt/edgeai_torchmodelopt/xmodelopt/surgery/v1/__init__.py:68: UserWarning: WARNING - xmodelopt.v1.surgery can only replace modules. To replace functions or operators, please use the torch.fx based xmodelopt.v2.surgery instead
  warnings.warn("WARNING - xmodelopt.v1.surgery can only replace modules. To replace functions or operators, please use the torch.fx based xmodelopt.v2.surgery instead")
model surgery done
loading annotations into memory...
Done (t=0.05s)
creating index...
index created!
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stem.conv_in.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stem.conv_in.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stem.conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stem.conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage1.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.1.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage2.1.blocks.2.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.1.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage3.1.blocks.2.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.1.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- backbone.stage4.2.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.reduce_layers.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.reduce_layers.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.reduce_layers.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.reduce_layers.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.0.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.top_down_blocks.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.downsamples.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.downsamples.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.downsamples.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.downsamples.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.0.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.main_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.main_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.short_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.short_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.final_conv.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.final_conv.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.bottom_up_blocks.1.blocks.0.conv2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.out_convs.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.out_convs.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.out_convs.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.out_convs.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.out_convs.2.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- neck.out_convs.2.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.0.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.1.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_cls_convs.2.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.0.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.1.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.0.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.0.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.1.bn.weight:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_reg_convs.2.1.bn.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_cls.0.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_cls.1.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_cls.2.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_reg.0.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_reg.1.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_reg.2.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_obj.0.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_obj.1.bias:weight_decay=0.0
10/15 16:56:58 - mmengine - INFO - paramwise_options -- bbox_head.multi_level_conv_obj.2.bias:weight_decay=0.0
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
10/15 16:57:00 - mmengine - WARNING - init_weights of YOLOX has been called more than once.
Loads checkpoint by local backend from path: ./data/downloads/pretrained/yolox_nano_lite/yolox_nano_lite_416x416_20220214_checkpoint.pth
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/mmengine/runner/checkpoint.py:347: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(filename, map_location=map_location)
The model and loaded state dict do not match exactly

size mismatch for bbox_head.multi_level_conv_cls.0.weight: copying a param with shape torch.Size([80, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.0.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.1.weight: copying a param with shape torch.Size([80, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.1.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.2.weight: copying a param with shape torch.Size([80, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.2.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
unexpected key in source state_dict: ema_backbone_stem_conv_in_conv_weight, ema_backbone_stem_conv_in_bn_weight, ema_backbone_stem_conv_in_bn_bias, ema_backbone_stem_conv_in_bn_running_mean, ema_backbone_stem_conv_in_bn_running_var, ema_backbone_stem_conv_in_bn_num_batches_tracked, ema_backbone_stem_conv_conv_weight, ema_backbone_stem_conv_bn_weight, ema_backbone_stem_conv_bn_bias, ema_backbone_stem_conv_bn_running_mean, ema_backbone_stem_conv_bn_running_var, ema_backbone_stem_conv_bn_num_batches_tracked, ema_backbone_stage1_0_conv_weight, ema_backbone_stage1_0_bn_weight, ema_backbone_stage1_0_bn_bias, ema_backbone_stage1_0_bn_running_mean, ema_backbone_stage1_0_bn_running_var, ema_backbone_stage1_0_bn_num_batches_tracked, ema_backbone_stage1_1_main_conv_conv_weight, ema_backbone_stage1_1_main_conv_bn_weight, ema_backbone_stage1_1_main_conv_bn_bias, ema_backbone_stage1_1_main_conv_bn_running_mean, ema_backbone_stage1_1_main_conv_bn_running_var, ema_backbone_stage1_1_main_conv_bn_num_batches_tracked, ema_backbone_stage1_1_short_conv_conv_weight, ema_backbone_stage1_1_short_conv_bn_weight, ema_backbone_stage1_1_short_conv_bn_bias, ema_backbone_stage1_1_short_conv_bn_running_mean, ema_backbone_stage1_1_short_conv_bn_running_var, ema_backbone_stage1_1_short_conv_bn_num_batches_tracked, ema_backbone_stage1_1_final_conv_conv_weight, ema_backbone_stage1_1_final_conv_bn_weight, ema_backbone_stage1_1_final_conv_bn_bias, ema_backbone_stage1_1_final_conv_bn_running_mean, ema_backbone_stage1_1_final_conv_bn_running_var, ema_backbone_stage1_1_final_conv_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv1_conv_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_bias, ema_backbone_stage1_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv1_bn_running_var, ema_backbone_stage1_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv2_conv_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_bias, ema_backbone_stage1_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv2_bn_running_var, ema_backbone_stage1_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_0_conv_weight, ema_backbone_stage2_0_bn_weight, ema_backbone_stage2_0_bn_bias, ema_backbone_stage2_0_bn_running_mean, ema_backbone_stage2_0_bn_running_var, ema_backbone_stage2_0_bn_num_batches_tracked, ema_backbone_stage2_1_main_conv_conv_weight, ema_backbone_stage2_1_main_conv_bn_weight, ema_backbone_stage2_1_main_conv_bn_bias, ema_backbone_stage2_1_main_conv_bn_running_mean, ema_backbone_stage2_1_main_conv_bn_running_var, ema_backbone_stage2_1_main_conv_bn_num_batches_tracked, ema_backbone_stage2_1_short_conv_conv_weight, ema_backbone_stage2_1_short_conv_bn_weight, ema_backbone_stage2_1_short_conv_bn_bias, ema_backbone_stage2_1_short_conv_bn_running_mean, ema_backbone_stage2_1_short_conv_bn_running_var, ema_backbone_stage2_1_short_conv_bn_num_batches_tracked, ema_backbone_stage2_1_final_conv_conv_weight, ema_backbone_stage2_1_final_conv_bn_weight, ema_backbone_stage2_1_final_conv_bn_bias, ema_backbone_stage2_1_final_conv_bn_running_mean, ema_backbone_stage2_1_final_conv_bn_running_var, ema_backbone_stage2_1_final_conv_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv1_conv_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_bias, ema_backbone_stage2_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv1_bn_running_var, ema_backbone_stage2_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv2_conv_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_bias, ema_backbone_stage2_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv2_bn_running_var, ema_backbone_stage2_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv1_conv_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_bias, ema_backbone_stage2_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv1_bn_running_var, ema_backbone_stage2_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv2_conv_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_bias, ema_backbone_stage2_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv2_bn_running_var, ema_backbone_stage2_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv1_conv_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_bias, ema_backbone_stage2_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv1_bn_running_var, ema_backbone_stage2_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv2_conv_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_bias, ema_backbone_stage2_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv2_bn_running_var, ema_backbone_stage2_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage3_0_conv_weight, ema_backbone_stage3_0_bn_weight, ema_backbone_stage3_0_bn_bias, ema_backbone_stage3_0_bn_running_mean, ema_backbone_stage3_0_bn_running_var, ema_backbone_stage3_0_bn_num_batches_tracked, ema_backbone_stage3_1_main_conv_conv_weight, ema_backbone_stage3_1_main_conv_bn_weight, ema_backbone_stage3_1_main_conv_bn_bias, ema_backbone_stage3_1_main_conv_bn_running_mean, ema_backbone_stage3_1_main_conv_bn_running_var, ema_backbone_stage3_1_main_conv_bn_num_batches_tracked, ema_backbone_stage3_1_short_conv_conv_weight, ema_backbone_stage3_1_short_conv_bn_weight, ema_backbone_stage3_1_short_conv_bn_bias, ema_backbone_stage3_1_short_conv_bn_running_mean, ema_backbone_stage3_1_short_conv_bn_running_var, ema_backbone_stage3_1_short_conv_bn_num_batches_tracked, ema_backbone_stage3_1_final_conv_conv_weight, ema_backbone_stage3_1_final_conv_bn_weight, ema_backbone_stage3_1_final_conv_bn_bias, ema_backbone_stage3_1_final_conv_bn_running_mean, ema_backbone_stage3_1_final_conv_bn_running_var, ema_backbone_stage3_1_final_conv_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv1_conv_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_bias, ema_backbone_stage3_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv1_bn_running_var, ema_backbone_stage3_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv2_conv_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_bias, ema_backbone_stage3_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv2_bn_running_var, ema_backbone_stage3_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv1_conv_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_bias, ema_backbone_stage3_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv1_bn_running_var, ema_backbone_stage3_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv2_conv_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_bias, ema_backbone_stage3_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv2_bn_running_var, ema_backbone_stage3_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv1_conv_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_bias, ema_backbone_stage3_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv1_bn_running_var, ema_backbone_stage3_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv2_conv_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_bias, ema_backbone_stage3_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv2_bn_running_var, ema_backbone_stage3_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage4_0_conv_weight, ema_backbone_stage4_0_bn_weight, ema_backbone_stage4_0_bn_bias, ema_backbone_stage4_0_bn_running_mean, ema_backbone_stage4_0_bn_running_var, ema_backbone_stage4_0_bn_num_batches_tracked, ema_backbone_stage4_1_conv1_conv_weight, ema_backbone_stage4_1_conv1_bn_weight, ema_backbone_stage4_1_conv1_bn_bias, ema_backbone_stage4_1_conv1_bn_running_mean, ema_backbone_stage4_1_conv1_bn_running_var, ema_backbone_stage4_1_conv1_bn_num_batches_tracked, ema_backbone_stage4_1_conv2_conv_weight, ema_backbone_stage4_1_conv2_bn_weight, ema_backbone_stage4_1_conv2_bn_bias, ema_backbone_stage4_1_conv2_bn_running_mean, ema_backbone_stage4_1_conv2_bn_running_var, ema_backbone_stage4_1_conv2_bn_num_batches_tracked, ema_backbone_stage4_2_main_conv_conv_weight, ema_backbone_stage4_2_main_conv_bn_weight, ema_backbone_stage4_2_main_conv_bn_bias, ema_backbone_stage4_2_main_conv_bn_running_mean, ema_backbone_stage4_2_main_conv_bn_running_var, ema_backbone_stage4_2_main_conv_bn_num_batches_tracked, ema_backbone_stage4_2_short_conv_conv_weight, ema_backbone_stage4_2_short_conv_bn_weight, ema_backbone_stage4_2_short_conv_bn_bias, ema_backbone_stage4_2_short_conv_bn_running_mean, ema_backbone_stage4_2_short_conv_bn_running_var, ema_backbone_stage4_2_short_conv_bn_num_batches_tracked, ema_backbone_stage4_2_final_conv_conv_weight, ema_backbone_stage4_2_final_conv_bn_weight, ema_backbone_stage4_2_final_conv_bn_bias, ema_backbone_stage4_2_final_conv_bn_running_mean, ema_backbone_stage4_2_final_conv_bn_running_var, ema_backbone_stage4_2_final_conv_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv1_conv_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_bias, ema_backbone_stage4_2_blocks_0_conv1_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv1_bn_running_var, ema_backbone_stage4_2_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv2_conv_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_bias, ema_backbone_stage4_2_blocks_0_conv2_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv2_bn_running_var, ema_backbone_stage4_2_blocks_0_conv2_bn_num_batches_tracked, ema_neck_reduce_layers_0_conv_weight, ema_neck_reduce_layers_0_bn_weight, ema_neck_reduce_layers_0_bn_bias, ema_neck_reduce_layers_0_bn_running_mean, ema_neck_reduce_layers_0_bn_running_var, ema_neck_reduce_layers_0_bn_num_batches_tracked, ema_neck_reduce_layers_1_conv_weight, ema_neck_reduce_layers_1_bn_weight, ema_neck_reduce_layers_1_bn_bias, ema_neck_reduce_layers_1_bn_running_mean, ema_neck_reduce_layers_1_bn_running_var, ema_neck_reduce_layers_1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_main_conv_conv_weight, ema_neck_top_down_blocks_0_main_conv_bn_weight, ema_neck_top_down_blocks_0_main_conv_bn_bias, ema_neck_top_down_blocks_0_main_conv_bn_running_mean, ema_neck_top_down_blocks_0_main_conv_bn_running_var, ema_neck_top_down_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_short_conv_conv_weight, ema_neck_top_down_blocks_0_short_conv_bn_weight, ema_neck_top_down_blocks_0_short_conv_bn_bias, ema_neck_top_down_blocks_0_short_conv_bn_running_mean, ema_neck_top_down_blocks_0_short_conv_bn_running_var, ema_neck_top_down_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_final_conv_conv_weight, ema_neck_top_down_blocks_0_final_conv_bn_weight, ema_neck_top_down_blocks_0_final_conv_bn_bias, ema_neck_top_down_blocks_0_final_conv_bn_running_mean, ema_neck_top_down_blocks_0_final_conv_bn_running_var, ema_neck_top_down_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_1_main_conv_conv_weight, ema_neck_top_down_blocks_1_main_conv_bn_weight, ema_neck_top_down_blocks_1_main_conv_bn_bias, ema_neck_top_down_blocks_1_main_conv_bn_running_mean, ema_neck_top_down_blocks_1_main_conv_bn_running_var, ema_neck_top_down_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_short_conv_conv_weight, ema_neck_top_down_blocks_1_short_conv_bn_weight, ema_neck_top_down_blocks_1_short_conv_bn_bias, ema_neck_top_down_blocks_1_short_conv_bn_running_mean, ema_neck_top_down_blocks_1_short_conv_bn_running_var, ema_neck_top_down_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_final_conv_conv_weight, ema_neck_top_down_blocks_1_final_conv_bn_weight, ema_neck_top_down_blocks_1_final_conv_bn_bias, ema_neck_top_down_blocks_1_final_conv_bn_running_mean, ema_neck_top_down_blocks_1_final_conv_bn_running_var, ema_neck_top_down_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_downsamples_0_conv_weight, ema_neck_downsamples_0_bn_weight, ema_neck_downsamples_0_bn_bias, ema_neck_downsamples_0_bn_running_mean, ema_neck_downsamples_0_bn_running_var, ema_neck_downsamples_0_bn_num_batches_tracked, ema_neck_downsamples_1_conv_weight, ema_neck_downsamples_1_bn_weight, ema_neck_downsamples_1_bn_bias, ema_neck_downsamples_1_bn_running_mean, ema_neck_downsamples_1_bn_running_var, ema_neck_downsamples_1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_main_conv_conv_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_bias, ema_neck_bottom_up_blocks_0_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_main_conv_bn_running_var, ema_neck_bottom_up_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_short_conv_conv_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_bias, ema_neck_bottom_up_blocks_0_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_short_conv_bn_running_var, ema_neck_bottom_up_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_final_conv_conv_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_bias, ema_neck_bottom_up_blocks_0_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_final_conv_bn_running_var, ema_neck_bottom_up_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_main_conv_conv_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_bias, ema_neck_bottom_up_blocks_1_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_main_conv_bn_running_var, ema_neck_bottom_up_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_short_conv_conv_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_bias, ema_neck_bottom_up_blocks_1_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_short_conv_bn_running_var, ema_neck_bottom_up_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_final_conv_conv_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_bias, ema_neck_bottom_up_blocks_1_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_final_conv_bn_running_var, ema_neck_bottom_up_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_out_convs_0_conv_weight, ema_neck_out_convs_0_bn_weight, ema_neck_out_convs_0_bn_bias, ema_neck_out_convs_0_bn_running_mean, ema_neck_out_convs_0_bn_running_var, ema_neck_out_convs_0_bn_num_batches_tracked, ema_neck_out_convs_1_conv_weight, ema_neck_out_convs_1_bn_weight, ema_neck_out_convs_1_bn_bias, ema_neck_out_convs_1_bn_running_mean, ema_neck_out_convs_1_bn_running_var, ema_neck_out_convs_1_bn_num_batches_tracked, ema_neck_out_convs_2_conv_weight, ema_neck_out_convs_2_bn_weight, ema_neck_out_convs_2_bn_bias, ema_neck_out_convs_2_bn_running_mean, ema_neck_out_convs_2_bn_running_var, ema_neck_out_convs_2_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_0_conv_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_bias, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_1_conv_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_bias, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_0_conv_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_bias, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_1_conv_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_bias, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_0_conv_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_bias, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_1_conv_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_bias, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_0_conv_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_bias, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_1_conv_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_bias, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_0_conv_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_bias, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_1_conv_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_bias, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_0_conv_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_bias, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_1_conv_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_bias, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_conv_cls_0_weight, ema_bbox_head_multi_level_conv_cls_0_bias, ema_bbox_head_multi_level_conv_cls_1_weight, ema_bbox_head_multi_level_conv_cls_1_bias, ema_bbox_head_multi_level_conv_cls_2_weight, ema_bbox_head_multi_level_conv_cls_2_bias, ema_bbox_head_multi_level_conv_reg_0_weight, ema_bbox_head_multi_level_conv_reg_0_bias, ema_bbox_head_multi_level_conv_reg_1_weight, ema_bbox_head_multi_level_conv_reg_1_bias, ema_bbox_head_multi_level_conv_reg_2_weight, ema_bbox_head_multi_level_conv_reg_2_bias, ema_bbox_head_multi_level_conv_obj_0_weight, ema_bbox_head_multi_level_conv_obj_0_bias, ema_bbox_head_multi_level_conv_obj_1_weight, ema_bbox_head_multi_level_conv_obj_1_bias, ema_bbox_head_multi_level_conv_obj_2_weight, ema_bbox_head_multi_level_conv_obj_2_bias

The model and loaded state dict do not match exactly

size mismatch for bbox_head.multi_level_conv_cls.0.weight: copying a param with shape torch.Size([80, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.0.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.1.weight: copying a param with shape torch.Size([80, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.1.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
size mismatch for bbox_head.multi_level_conv_cls.2.weight: copying a param with shape torch.Size([80, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1]).
size mismatch for bbox_head.multi_level_conv_cls.2.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([3]).
unexpected key in source state_dict: ema_backbone_stem_conv_in_conv_weight, ema_backbone_stem_conv_in_bn_weight, ema_backbone_stem_conv_in_bn_bias, ema_backbone_stem_conv_in_bn_running_mean, ema_backbone_stem_conv_in_bn_running_var, ema_backbone_stem_conv_in_bn_num_batches_tracked, ema_backbone_stem_conv_conv_weight, ema_backbone_stem_conv_bn_weight, ema_backbone_stem_conv_bn_bias, ema_backbone_stem_conv_bn_running_mean, ema_backbone_stem_conv_bn_running_var, ema_backbone_stem_conv_bn_num_batches_tracked, ema_backbone_stage1_0_conv_weight, ema_backbone_stage1_0_bn_weight, ema_backbone_stage1_0_bn_bias, ema_backbone_stage1_0_bn_running_mean, ema_backbone_stage1_0_bn_running_var, ema_backbone_stage1_0_bn_num_batches_tracked, ema_backbone_stage1_1_main_conv_conv_weight, ema_backbone_stage1_1_main_conv_bn_weight, ema_backbone_stage1_1_main_conv_bn_bias, ema_backbone_stage1_1_main_conv_bn_running_mean, ema_backbone_stage1_1_main_conv_bn_running_var, ema_backbone_stage1_1_main_conv_bn_num_batches_tracked, ema_backbone_stage1_1_short_conv_conv_weight, ema_backbone_stage1_1_short_conv_bn_weight, ema_backbone_stage1_1_short_conv_bn_bias, ema_backbone_stage1_1_short_conv_bn_running_mean, ema_backbone_stage1_1_short_conv_bn_running_var, ema_backbone_stage1_1_short_conv_bn_num_batches_tracked, ema_backbone_stage1_1_final_conv_conv_weight, ema_backbone_stage1_1_final_conv_bn_weight, ema_backbone_stage1_1_final_conv_bn_bias, ema_backbone_stage1_1_final_conv_bn_running_mean, ema_backbone_stage1_1_final_conv_bn_running_var, ema_backbone_stage1_1_final_conv_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv1_conv_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_weight, ema_backbone_stage1_1_blocks_0_conv1_bn_bias, ema_backbone_stage1_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv1_bn_running_var, ema_backbone_stage1_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage1_1_blocks_0_conv2_conv_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_weight, ema_backbone_stage1_1_blocks_0_conv2_bn_bias, ema_backbone_stage1_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage1_1_blocks_0_conv2_bn_running_var, ema_backbone_stage1_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_0_conv_weight, ema_backbone_stage2_0_bn_weight, ema_backbone_stage2_0_bn_bias, ema_backbone_stage2_0_bn_running_mean, ema_backbone_stage2_0_bn_running_var, ema_backbone_stage2_0_bn_num_batches_tracked, ema_backbone_stage2_1_main_conv_conv_weight, ema_backbone_stage2_1_main_conv_bn_weight, ema_backbone_stage2_1_main_conv_bn_bias, ema_backbone_stage2_1_main_conv_bn_running_mean, ema_backbone_stage2_1_main_conv_bn_running_var, ema_backbone_stage2_1_main_conv_bn_num_batches_tracked, ema_backbone_stage2_1_short_conv_conv_weight, ema_backbone_stage2_1_short_conv_bn_weight, ema_backbone_stage2_1_short_conv_bn_bias, ema_backbone_stage2_1_short_conv_bn_running_mean, ema_backbone_stage2_1_short_conv_bn_running_var, ema_backbone_stage2_1_short_conv_bn_num_batches_tracked, ema_backbone_stage2_1_final_conv_conv_weight, ema_backbone_stage2_1_final_conv_bn_weight, ema_backbone_stage2_1_final_conv_bn_bias, ema_backbone_stage2_1_final_conv_bn_running_mean, ema_backbone_stage2_1_final_conv_bn_running_var, ema_backbone_stage2_1_final_conv_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv1_conv_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_weight, ema_backbone_stage2_1_blocks_0_conv1_bn_bias, ema_backbone_stage2_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv1_bn_running_var, ema_backbone_stage2_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_0_conv2_conv_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_weight, ema_backbone_stage2_1_blocks_0_conv2_bn_bias, ema_backbone_stage2_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_0_conv2_bn_running_var, ema_backbone_stage2_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv1_conv_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_weight, ema_backbone_stage2_1_blocks_1_conv1_bn_bias, ema_backbone_stage2_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv1_bn_running_var, ema_backbone_stage2_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_1_conv2_conv_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_weight, ema_backbone_stage2_1_blocks_1_conv2_bn_bias, ema_backbone_stage2_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_1_conv2_bn_running_var, ema_backbone_stage2_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv1_conv_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_weight, ema_backbone_stage2_1_blocks_2_conv1_bn_bias, ema_backbone_stage2_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv1_bn_running_var, ema_backbone_stage2_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage2_1_blocks_2_conv2_conv_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_weight, ema_backbone_stage2_1_blocks_2_conv2_bn_bias, ema_backbone_stage2_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage2_1_blocks_2_conv2_bn_running_var, ema_backbone_stage2_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage3_0_conv_weight, ema_backbone_stage3_0_bn_weight, ema_backbone_stage3_0_bn_bias, ema_backbone_stage3_0_bn_running_mean, ema_backbone_stage3_0_bn_running_var, ema_backbone_stage3_0_bn_num_batches_tracked, ema_backbone_stage3_1_main_conv_conv_weight, ema_backbone_stage3_1_main_conv_bn_weight, ema_backbone_stage3_1_main_conv_bn_bias, ema_backbone_stage3_1_main_conv_bn_running_mean, ema_backbone_stage3_1_main_conv_bn_running_var, ema_backbone_stage3_1_main_conv_bn_num_batches_tracked, ema_backbone_stage3_1_short_conv_conv_weight, ema_backbone_stage3_1_short_conv_bn_weight, ema_backbone_stage3_1_short_conv_bn_bias, ema_backbone_stage3_1_short_conv_bn_running_mean, ema_backbone_stage3_1_short_conv_bn_running_var, ema_backbone_stage3_1_short_conv_bn_num_batches_tracked, ema_backbone_stage3_1_final_conv_conv_weight, ema_backbone_stage3_1_final_conv_bn_weight, ema_backbone_stage3_1_final_conv_bn_bias, ema_backbone_stage3_1_final_conv_bn_running_mean, ema_backbone_stage3_1_final_conv_bn_running_var, ema_backbone_stage3_1_final_conv_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv1_conv_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_weight, ema_backbone_stage3_1_blocks_0_conv1_bn_bias, ema_backbone_stage3_1_blocks_0_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv1_bn_running_var, ema_backbone_stage3_1_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_0_conv2_conv_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_weight, ema_backbone_stage3_1_blocks_0_conv2_bn_bias, ema_backbone_stage3_1_blocks_0_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_0_conv2_bn_running_var, ema_backbone_stage3_1_blocks_0_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv1_conv_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_weight, ema_backbone_stage3_1_blocks_1_conv1_bn_bias, ema_backbone_stage3_1_blocks_1_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv1_bn_running_var, ema_backbone_stage3_1_blocks_1_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_1_conv2_conv_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_weight, ema_backbone_stage3_1_blocks_1_conv2_bn_bias, ema_backbone_stage3_1_blocks_1_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_1_conv2_bn_running_var, ema_backbone_stage3_1_blocks_1_conv2_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv1_conv_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_weight, ema_backbone_stage3_1_blocks_2_conv1_bn_bias, ema_backbone_stage3_1_blocks_2_conv1_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv1_bn_running_var, ema_backbone_stage3_1_blocks_2_conv1_bn_num_batches_tracked, ema_backbone_stage3_1_blocks_2_conv2_conv_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_weight, ema_backbone_stage3_1_blocks_2_conv2_bn_bias, ema_backbone_stage3_1_blocks_2_conv2_bn_running_mean, ema_backbone_stage3_1_blocks_2_conv2_bn_running_var, ema_backbone_stage3_1_blocks_2_conv2_bn_num_batches_tracked, ema_backbone_stage4_0_conv_weight, ema_backbone_stage4_0_bn_weight, ema_backbone_stage4_0_bn_bias, ema_backbone_stage4_0_bn_running_mean, ema_backbone_stage4_0_bn_running_var, ema_backbone_stage4_0_bn_num_batches_tracked, ema_backbone_stage4_1_conv1_conv_weight, ema_backbone_stage4_1_conv1_bn_weight, ema_backbone_stage4_1_conv1_bn_bias, ema_backbone_stage4_1_conv1_bn_running_mean, ema_backbone_stage4_1_conv1_bn_running_var, ema_backbone_stage4_1_conv1_bn_num_batches_tracked, ema_backbone_stage4_1_conv2_conv_weight, ema_backbone_stage4_1_conv2_bn_weight, ema_backbone_stage4_1_conv2_bn_bias, ema_backbone_stage4_1_conv2_bn_running_mean, ema_backbone_stage4_1_conv2_bn_running_var, ema_backbone_stage4_1_conv2_bn_num_batches_tracked, ema_backbone_stage4_2_main_conv_conv_weight, ema_backbone_stage4_2_main_conv_bn_weight, ema_backbone_stage4_2_main_conv_bn_bias, ema_backbone_stage4_2_main_conv_bn_running_mean, ema_backbone_stage4_2_main_conv_bn_running_var, ema_backbone_stage4_2_main_conv_bn_num_batches_tracked, ema_backbone_stage4_2_short_conv_conv_weight, ema_backbone_stage4_2_short_conv_bn_weight, ema_backbone_stage4_2_short_conv_bn_bias, ema_backbone_stage4_2_short_conv_bn_running_mean, ema_backbone_stage4_2_short_conv_bn_running_var, ema_backbone_stage4_2_short_conv_bn_num_batches_tracked, ema_backbone_stage4_2_final_conv_conv_weight, ema_backbone_stage4_2_final_conv_bn_weight, ema_backbone_stage4_2_final_conv_bn_bias, ema_backbone_stage4_2_final_conv_bn_running_mean, ema_backbone_stage4_2_final_conv_bn_running_var, ema_backbone_stage4_2_final_conv_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv1_conv_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_weight, ema_backbone_stage4_2_blocks_0_conv1_bn_bias, ema_backbone_stage4_2_blocks_0_conv1_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv1_bn_running_var, ema_backbone_stage4_2_blocks_0_conv1_bn_num_batches_tracked, ema_backbone_stage4_2_blocks_0_conv2_conv_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_weight, ema_backbone_stage4_2_blocks_0_conv2_bn_bias, ema_backbone_stage4_2_blocks_0_conv2_bn_running_mean, ema_backbone_stage4_2_blocks_0_conv2_bn_running_var, ema_backbone_stage4_2_blocks_0_conv2_bn_num_batches_tracked, ema_neck_reduce_layers_0_conv_weight, ema_neck_reduce_layers_0_bn_weight, ema_neck_reduce_layers_0_bn_bias, ema_neck_reduce_layers_0_bn_running_mean, ema_neck_reduce_layers_0_bn_running_var, ema_neck_reduce_layers_0_bn_num_batches_tracked, ema_neck_reduce_layers_1_conv_weight, ema_neck_reduce_layers_1_bn_weight, ema_neck_reduce_layers_1_bn_bias, ema_neck_reduce_layers_1_bn_running_mean, ema_neck_reduce_layers_1_bn_running_var, ema_neck_reduce_layers_1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_main_conv_conv_weight, ema_neck_top_down_blocks_0_main_conv_bn_weight, ema_neck_top_down_blocks_0_main_conv_bn_bias, ema_neck_top_down_blocks_0_main_conv_bn_running_mean, ema_neck_top_down_blocks_0_main_conv_bn_running_var, ema_neck_top_down_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_short_conv_conv_weight, ema_neck_top_down_blocks_0_short_conv_bn_weight, ema_neck_top_down_blocks_0_short_conv_bn_bias, ema_neck_top_down_blocks_0_short_conv_bn_running_mean, ema_neck_top_down_blocks_0_short_conv_bn_running_var, ema_neck_top_down_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_final_conv_conv_weight, ema_neck_top_down_blocks_0_final_conv_bn_weight, ema_neck_top_down_blocks_0_final_conv_bn_bias, ema_neck_top_down_blocks_0_final_conv_bn_running_mean, ema_neck_top_down_blocks_0_final_conv_bn_running_var, ema_neck_top_down_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_0_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_top_down_blocks_1_main_conv_conv_weight, ema_neck_top_down_blocks_1_main_conv_bn_weight, ema_neck_top_down_blocks_1_main_conv_bn_bias, ema_neck_top_down_blocks_1_main_conv_bn_running_mean, ema_neck_top_down_blocks_1_main_conv_bn_running_var, ema_neck_top_down_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_short_conv_conv_weight, ema_neck_top_down_blocks_1_short_conv_bn_weight, ema_neck_top_down_blocks_1_short_conv_bn_bias, ema_neck_top_down_blocks_1_short_conv_bn_running_mean, ema_neck_top_down_blocks_1_short_conv_bn_running_var, ema_neck_top_down_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_final_conv_conv_weight, ema_neck_top_down_blocks_1_final_conv_bn_weight, ema_neck_top_down_blocks_1_final_conv_bn_bias, ema_neck_top_down_blocks_1_final_conv_bn_running_mean, ema_neck_top_down_blocks_1_final_conv_bn_running_var, ema_neck_top_down_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv1_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_top_down_blocks_1_blocks_0_conv2_conv_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_weight, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_bias, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_top_down_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_downsamples_0_conv_weight, ema_neck_downsamples_0_bn_weight, ema_neck_downsamples_0_bn_bias, ema_neck_downsamples_0_bn_running_mean, ema_neck_downsamples_0_bn_running_var, ema_neck_downsamples_0_bn_num_batches_tracked, ema_neck_downsamples_1_conv_weight, ema_neck_downsamples_1_bn_weight, ema_neck_downsamples_1_bn_bias, ema_neck_downsamples_1_bn_running_mean, ema_neck_downsamples_1_bn_running_var, ema_neck_downsamples_1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_main_conv_conv_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_weight, ema_neck_bottom_up_blocks_0_main_conv_bn_bias, ema_neck_bottom_up_blocks_0_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_main_conv_bn_running_var, ema_neck_bottom_up_blocks_0_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_short_conv_conv_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_weight, ema_neck_bottom_up_blocks_0_short_conv_bn_bias, ema_neck_bottom_up_blocks_0_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_short_conv_bn_running_var, ema_neck_bottom_up_blocks_0_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_final_conv_conv_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_weight, ema_neck_bottom_up_blocks_0_final_conv_bn_bias, ema_neck_bottom_up_blocks_0_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_0_final_conv_bn_running_var, ema_neck_bottom_up_blocks_0_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_0_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_0_blocks_0_conv2_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_main_conv_conv_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_weight, ema_neck_bottom_up_blocks_1_main_conv_bn_bias, ema_neck_bottom_up_blocks_1_main_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_main_conv_bn_running_var, ema_neck_bottom_up_blocks_1_main_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_short_conv_conv_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_weight, ema_neck_bottom_up_blocks_1_short_conv_bn_bias, ema_neck_bottom_up_blocks_1_short_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_short_conv_bn_running_var, ema_neck_bottom_up_blocks_1_short_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_final_conv_conv_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_weight, ema_neck_bottom_up_blocks_1_final_conv_bn_bias, ema_neck_bottom_up_blocks_1_final_conv_bn_running_mean, ema_neck_bottom_up_blocks_1_final_conv_bn_running_var, ema_neck_bottom_up_blocks_1_final_conv_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv1_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv1_bn_num_batches_tracked, ema_neck_bottom_up_blocks_1_blocks_0_conv2_conv_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_weight, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_bias, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_mean, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_running_var, ema_neck_bottom_up_blocks_1_blocks_0_conv2_bn_num_batches_tracked, ema_neck_out_convs_0_conv_weight, ema_neck_out_convs_0_bn_weight, ema_neck_out_convs_0_bn_bias, ema_neck_out_convs_0_bn_running_mean, ema_neck_out_convs_0_bn_running_var, ema_neck_out_convs_0_bn_num_batches_tracked, ema_neck_out_convs_1_conv_weight, ema_neck_out_convs_1_bn_weight, ema_neck_out_convs_1_bn_bias, ema_neck_out_convs_1_bn_running_mean, ema_neck_out_convs_1_bn_running_var, ema_neck_out_convs_1_bn_num_batches_tracked, ema_neck_out_convs_2_conv_weight, ema_neck_out_convs_2_bn_weight, ema_neck_out_convs_2_bn_bias, ema_neck_out_convs_2_bn_running_mean, ema_neck_out_convs_2_bn_running_var, ema_neck_out_convs_2_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_0_conv_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_weight, ema_bbox_head_multi_level_cls_convs_0_0_bn_bias, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_0_1_conv_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_weight, ema_bbox_head_multi_level_cls_convs_0_1_bn_bias, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_0_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_0_conv_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_weight, ema_bbox_head_multi_level_cls_convs_1_0_bn_bias, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_1_1_conv_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_weight, ema_bbox_head_multi_level_cls_convs_1_1_bn_bias, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_1_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_0_conv_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_weight, ema_bbox_head_multi_level_cls_convs_2_0_bn_bias, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_0_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_cls_convs_2_1_conv_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_weight, ema_bbox_head_multi_level_cls_convs_2_1_bn_bias, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_cls_convs_2_1_bn_running_var, ema_bbox_head_multi_level_cls_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_0_conv_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_weight, ema_bbox_head_multi_level_reg_convs_0_0_bn_bias, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_0_1_conv_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_weight, ema_bbox_head_multi_level_reg_convs_0_1_bn_bias, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_0_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_0_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_0_conv_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_weight, ema_bbox_head_multi_level_reg_convs_1_0_bn_bias, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_1_1_conv_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_weight, ema_bbox_head_multi_level_reg_convs_1_1_bn_bias, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_1_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_1_1_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_0_conv_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_weight, ema_bbox_head_multi_level_reg_convs_2_0_bn_bias, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_0_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_0_bn_num_batches_tracked, ema_bbox_head_multi_level_reg_convs_2_1_conv_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_weight, ema_bbox_head_multi_level_reg_convs_2_1_bn_bias, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_mean, ema_bbox_head_multi_level_reg_convs_2_1_bn_running_var, ema_bbox_head_multi_level_reg_convs_2_1_bn_num_batches_tracked, ema_bbox_head_multi_level_conv_cls_0_weight, ema_bbox_head_multi_level_conv_cls_0_bias, ema_bbox_head_multi_level_conv_cls_1_weight, ema_bbox_head_multi_level_conv_cls_1_bias, ema_bbox_head_multi_level_conv_cls_2_weight, ema_bbox_head_multi_level_conv_cls_2_bias, ema_bbox_head_multi_level_conv_reg_0_weight, ema_bbox_head_multi_level_conv_reg_0_bias, ema_bbox_head_multi_level_conv_reg_1_weight, ema_bbox_head_multi_level_conv_reg_1_bias, ema_bbox_head_multi_level_conv_reg_2_weight, ema_bbox_head_multi_level_conv_reg_2_bias, ema_bbox_head_multi_level_conv_obj_0_weight, ema_bbox_head_multi_level_conv_obj_0_bias, ema_bbox_head_multi_level_conv_obj_1_weight, ema_bbox_head_multi_level_conv_obj_1_bias, ema_bbox_head_multi_level_conv_obj_2_weight, ema_bbox_head_multi_level_conv_obj_2_bias

10/15 16:57:00 - mmengine - INFO - Load checkpoint from ./data/downloads/pretrained/yolox_nano_lite/yolox_nano_lite_416x416_20220214_checkpoint.pth
10/15 16:57:00 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
10/15 16:57:00 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
10/15 16:57:00 - mmengine - INFO - Checkpoints will be saved to /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training.
10/15 16:57:00 - mmengine - INFO - No mosaic and mixup aug now!
10/15 16:57:00 - mmengine - INFO - Add additional L1 loss now!
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/functional.py:513: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3609.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/task_modules/assigners/sim_ota_assigner.py:118: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/15 16:57:38 - mmengine - INFO - Exp name: yolox_nano_lite_20241015_165652
10/15 16:57:38 - mmengine - INFO - Epoch(train) [1][50/50]  base_lr: 8.0000e-05 lr: 8.0000e-05  eta: 0:00:00  time: 0.7575  data_time: 0.0209  loss: 7.1250  loss_cls: 1.5711  loss_bbox: 2.6173  loss_obj: 2.0568  loss_l1: 0.8798
10/15 16:57:38 - mmengine - INFO - Saving checkpoint at 1 epochs
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
10/15 16:57:43 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.01s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.73s).
Accumulating evaluation results...
DONE (t=0.11s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.127
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.243
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.116
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.002
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.095
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.458
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.180
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.180
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.180
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.014
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.193
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.530
10/15 16:57:44 - mmengine - INFO - bbox_mAP_copypaste: 0.127 0.243 0.116 0.002 0.095 0.458
10/15 16:57:44 - mmengine - INFO - Epoch(val) [1][14/14]    coco/bbox_mAP: 0.1270  coco/bbox_mAP_50: 0.2430  coco/bbox_mAP_75: 0.1160  coco/bbox_mAP_s: 0.0020  coco/bbox_mAP_m: 0.0950  coco/bbox_mAP_l: 0.4580  data_time: 0.0252  time: 0.1895
10/15 16:57:45 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/15 16:57:45 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/15 16:57:45 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
10/15 16:57:45 - mmengine - INFO - Export PyTorch model to ONNX: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx.
/home/mchi/work/edgeai-tensorlab/edgeai-mmdetection/mmdet/models/backbones/csp_darknet.py:244: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with torch.cuda.amp.autocast(enabled=False):
/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/onnx/symbolic_opset9.py:5715: UserWarning: Exporting aten::index operator of advanced indexing in opset 17 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
10/15 16:57:46 - mmengine - INFO - Execute onnx optimize passes.
Converted model is valid!
Trained model is at: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training

SUCCESS: ModelMaker - Training completed.

INFO:20241015-165748: model import is in progress - please see the log file for status.
configs to run: ['od-8200']
number of configs: 1

INFO:20241015-165748: parallel_run - parallel_processes:1 parallel_devices=[0]
TASKS                                                       |          |     0% 0/1| [< ]
INFO:20241015-165748: starting process on parallel_device - 0   0%|          || 0/1 [00:00<?, ?it/s]

INFO:20241015-165748: starting - od-8200
INFO:20241015-165748: model_path - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx
INFO:20241015-165748: model_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/compilation/AM68A/work/od-8200/model/model.onnx
INFO:20241015-165748: quant_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/compilation/AM68A/work/od-8200/model/model_qparams.prototxt
Downloading 1/1: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx
Download done for /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx
Downloading 1/1: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx
Download done for /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx

INFO:20241015-165749: running - od-8200
INFO:20241015-165749: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x746ef39a2140>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x746ef39559c0>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x746e77420a00>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x746e77420a60>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x746e77420d00>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-nano-lite-mmdet-coco-416x416', 'shortlisted': True, 'recommended': True}}
INFO:20241015-165749: import  - od-8200 - this may take some time...Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x746ecc22a0e0>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x746ecc22a0e0>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process

INFO:20241015-165822: import completed  - od-8200 - 34 sec

SUCCESS:20241015-165822: benchmark results - {}
TASKS                                                       | 100%|██████████|| 1/1 [00:34<00:00, 34.13s/it]
TASKS                                                       | 100%|██████████|| 1/1 [00:34<00:00, 34.02s/it]


INFO:20241015-165822: model inference is in progress - please see the log file for status.
configs to run: ['od-8200']
number of configs: 1

INFO:20241015-165822: parallel_run - parallel_processes:1 parallel_devices=[0]
TASKS                                                       |          |     0% 0/1| [< ]
INFO:20241015-165822: starting process on parallel_device - 0   0%|          || 0/1 [00:00<?, ?it/s]

INFO:20241015-165822: starting - od-8200
INFO:20241015-165822: model_path - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/training/model.onnx
INFO:20241015-165822: model_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/compilation/AM68A/work/od-8200/model/model.onnx
INFO:20241015-165822: quant_file - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/compilation/AM68A/work/od-8200/model/model_qparams.prototxt

INFO:20241015-165822: running - od-8200
INFO:20241015-165822: pipeline_config - {'task_type': 'detection', 'dataset_category': 'coco', 'calibration_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x746ef39a2140>, 'input_dataset': <edgeai_benchmark.datasets.modelmaker_datasets.ModelMakerDetectionDataset object at 0x746ef39559c0>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x746e77420a00>, 'session': <edgeai_benchmark.sessions.onnxrt_session.ONNXRTSession object at 0x746e77420a60>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x746e77420d00>, 'metric': {'label_offset_pred': 1}, 'model_info': {'metric_reference': {'accuracy_ap[.5:.95]%': None}, 'model_shortlist': 10, 'compact_name': 'yolox-nano-lite-mmdet-coco-416x416', 'shortlisted': True, 'recommended': True}}
INFO:20241015-165822: infer  - od-8200 - this may take some time...libtidl_onnxrt_EP loaded 0x5bd5f0a20b60 
Final number of subgraphs created are : 1, - Offloaded Nodes - 271, Total Nodes - 271 
The soft limit is 10240
The hard limit is 10240
MEM: Init ... !!!
MEM: Init ... Done !!!
 0.0s:  VX_ZONE_INIT:Enabled
 0.6s:  VX_ZONE_ERROR:Enabled
 0.9s:  VX_ZONE_WARNING:Enabled
 0.1694s:  VX_ZONE_INIT:[tivxInit:190] Initialization Done !!!
infer : od-8200                                             |  51%|#####1    || 55/107 [00:54<00:51,  1.01it/s]
INFO:20241015-165922: parallel_run - num_total_tasks:1 len(queued_tasks):0 len(process_dict):1 len(result_list):0
infer : od-8200                                             | 100%|##########|| 107/107 [01:45<00:00,  1.02it/s]
MEM: Deinit ... !!!
MEM: Alloc's: 27 alloc's of 29992297 bytes 
MEM: Free's : 27 free's  of 29992297 bytes 
MEM: Open's : 0 allocs  of 0 bytes 
MEM: Deinit ... Done !!!
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x746ecc22a0e0>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process
Exception ignored in: <function _MultiProcessingDataLoaderIter.__del__ at 0x746ecc22a0e0>
Traceback (most recent call last):
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1477, in __del__
    self._shutdown_workers()
  File "/home/mchi/.pyenv/versions/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1460, in _shutdown_workers
    if w.is_alive():
  File "/home/mchi/.pyenv/versions/3.10.15/lib/python3.10/multiprocessing/process.py", line 160, in is_alive
    assert self._parent_pid == os.getpid(), 'can only test a child process'
AssertionError: can only test a child process

INFO:20241015-170008: infer completed  - od-8200 - 105 secLoading and preparing results...
DONE (t=0.00s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.07s).
Accumulating evaluation results...
DONE (t=0.02s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.078
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.151
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.080
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.038
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.296
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.046
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.086
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.086
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.050
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.321


SUCCESS:20241015-170008: benchmark results - {'infer_path': 'od-8200', 'accuracy_ap[.5:.95]%': 7.798449, 'accuracy_ap50%': 15.078039, 'num_subgraphs': 0, 'infer_time_core_ms': 955.298973, 'infer_time_subgraph_ms': 0.0, 'ddr_transfer_mb': 0.0, 'perfsim_time_ms': 2.85721, 'perfsim_ddr_transfer_mb': 4.25, 'perfsim_gmacs': 1.458}
TASKS                                                       | 100%|██████████|| 1/1 [01:45<00:00, 105.74s/it]
TASKS                                                       | 100%|██████████|| 1/1 [01:45<00:00, 105.64s/it]

packaging artifacts to /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/compilation/AM68A/pkg please wait...
SUCCESS:20241015-170009: finished packaging - /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/compilation/AM68A/work/od-8200
Compiled model is at: /home/mchi/work/edgeai-tensorlab/edgeai-modelmaker/data/projects/tiscapes2017_driving/run/20241015-165641/yolox_nano_lite/compilation/AM68A/pkg/20241015-165641_yolox_nano_lite_onnxrt_AM68A.tar.gz

Hi, this assertion is due to multiprocessing (use of multiple process) and it happens in a __delete__ after the process has finished - it is harmless. And we are looking for ideas to avoid it and shall try to fix it as it is an annoyance. 

Could you help me with the accuracy issue?  thanks

10.0_xilutek-dataset_20241017_143225_learning_rate0.002.log

I did couple of trainings with the dataset that you shared - attached log files - both trained in 10.0

One of them used the default learning_rate, while the other one used half the default learning_rate 0.001

I observed that with the default learning rate, accuracy fluctuation is happening - accuracy suddenly goes down in an epoch. This can happen in small datasets like this scenario. 

One solution for this is to reduce the learning_rate. The second training used learning_rate 0.001 and gave even better results that what you had observed with 9.1

2555.0_xilutek-dataset_20241017_143937_learning_rate0.001.log

ok thanks , i will try  it.

In addition, the issue of dynamic object detection mentioned earlier, for example,

This is a training annotated picture， there is a very thin crack in the middle of the fork  

Copy the trained model to the target board ， cracks be identified when the fork is stationary?

When the fork is not stationary on the conveyor belt, the speed is very slow and almost unrecognizable.
For this case,  are there any good methods to solve this problem?

Does the training images including sufficient images captured moving fork? That should help to improve the accuracy during a real test using moving images.

If the accuracy of yolox_nano_lite model is not sufficient, then you can try yolox_s_lite model

  I have used yolox_s_lite model， seems this version，Is there a newer version available? How can I replace it?
www_modelzoo_path = 'software-dl.ti.com/.../08_06_00_01'
download_url': f'{www_modelzoo_path}/models/vision/detection/coco/edgeai-mmdet/yolox_s_lite_640x640_20220221_checkpoint.pth

Hi,

Is this issue still open?

Regards,

Brijesh

yolox_s_lite is the best object detection model that we have in edgeai-modelmaker as of now. If the detection quality is not expected, you can try adding more realistic images into the training set. i.e. if the failures are happening in the moving scenario, capture more images from the moving scenario and add to the training set.