Unexpectedly low AP after DEIMv2 fine-tuning can come from the training configuration rather than the model or learning rate. A pretrained-weight path may never reach the trainer, generated preprocessing may not match the backbone, or export may select different weights from validation. Training can continue through all of these mistakes, while the results become difficult to compare.
In one custom detection task, fine-tuning COCO weights produced 40.63% AP@0.5:0.95. A later configuration with Normalize produced 27.83%, initially suggesting a normalization regression. The logs showed that the second run had not actually fine-tuned the pretrained model: the weight argument had been lost during configuration transfer.
This article follows initialization, backbone preprocessing and evaluation weights through to ONNX export validation.
1. Check that the trainer loaded the supplied weights
The automatic training entry point accepted -t/--tuning. While merging the generated configuration with runtime arguments, it excluded tuning as though it were only a configuration-generation option. The model still built and trained, but the solver received tuning=None.
The log entry Training ViT-Tiny from scratch exposed the problem. A pretrained path in the outer command does not establish that the weight-loading branch ran. Configuration generation, overrides and initialization still have to pass the value through.
Inspect the command-line value, generated configuration, final cfg.tuning and solver loading report for the same run. Adding a path to YAML is insufficient if a later override clears it.
| Mode | Weights | Optimizer and scheduler |
|---|---|---|
| Training from scratch | No task fine-tuning starting point; declare backbone initialization separately | New |
| Fine-tuning | Pretrained parameters, adapting the head to the class count | Usually new |
| Resuming | Complete training checkpoint | Restored |
Partial loading often accommodates a changed class count. An expected head mismatch differs from many missing backbone parameters. The adaptation code matches parameter names and shapes and reports missed and unmatched; save these reports with the base-weight version.
The same tools also parsed YAML scientific notation such as 3e-4 as a string, failing only inside AdamW. Reparse generated configurations and validate types, ranges and mutually exclusive options. Reject simultaneous fine-tuning and resume options before launch.
2. Match preprocessing to the backbone
After repairing argument transfer, two runs that actually loaded COCO weights gave:
| Configuration | Recorded epoch | AP@0.5:0.95 |
|---|---|---|
| Without ImageNet Normalize | epoch 9 | 64.42% |
| With ImageNet Normalize | epoch 10 | 70.82% |
These runs did not reproduce the suspected large regression. The earlier 27.83% cannot support a claim against Normalize. The epochs also differ, so the entire 6.40 percentage-point gap cannot be attributed to normalization. A controlled comparison needs the same split, initialization, training budget and model-selection rule, plus complete validation curves.
In this adaptation, ViT/DINOv3 backbones use RGB, Resize, scaling to [0,1] and ImageNet Normalize. HGNetv2 does not add that Normalize step. Identify the model from its weight structure and configuration, rather than an s in its filename or its output size.
Check channel order, scaling order and geometry as well as normalization. For a model using this normalization:
Training, validation and deployment must follow the same channel rules. Do not divide by 255 twice or repeat Normalize when it is already fused into the graph or platform. Resize and Letterbox can produce the same shape with different pixels and coordinate relationships.
For historical weights trained with other preprocessing, first establish a reference using the original rules, then assess whether retraining is needed. Changing inference preprocessing alone mixes input-distribution changes with model changes.
3. Keep augmentation stages and evaluation criteria explicit
Strong augmentation early in training and reduced augmentation later create different validation regimes. Stopping after several unimproved epochs may end training before its final stage.
The adapted two-stage early-stopping logic monitors COCO bbox AP. Reaching patience in the first stage advances to the stage without augmentation; the second stage resets the counter and stops if improvement still fails. The archived default patience of 10 belongs to this entry point and is not an optimum for every DEIMv2 dataset. Log the minimum improvement, starting epoch, stage transition and selected best weights.
Distinguish AP@0.5:0.95, AP50 and visualizations at a chosen confidence threshold. Lowering the display threshold reveals more candidates without proving convergence. Fix class mappings, validation data and annotation policy when comparing runs. Changing boxes from object parts to whole objects, or splitting again after deduplication, invalidates direct comparison with old AP.
Separate data loading and augmentation, training iterations, validation and saving when measuring speed. Image caching can shorten an epoch by reducing decoding without making the model kernel faster. Keep validation precision, loader configuration and save frequency consistent.
4. Save the weights that validation selected
With EMA enabled, this implementation validates ema.module, rather than the ordinary model updated by the optimizer. Parameters from the same epoch are not necessarily identical. Selecting best by EMA AP and exporting ordinary model parameters changes the evaluated model.
One complete checkpoint occupied about 156 MB, including ordinary parameters, EMA and AdamW training state. The reduced best/last artifacts retain only the parameters used by validation: EMA if enabled, otherwise the ordinary model, under the model key. That artifact occupied about 39 MB.
The reduction removes duplicate parameters and training state; it does not reduce weight precision or quantize the model. These sizes describe one checkpoint and vary with model size.
| Artifact | Contents | Intended use |
|---|---|---|
| Complete training checkpoint | Model, EMA if enabled, optimizer, scheduler and progress | Resume through the original training entry point |
| Reduced best/last | One selected parameter set | Validate, infer, export or begin new fine-tuning |
| ONNX | Inference graph and parameters | Deployment reference, without training recovery |
A reduced file named last.pth still cannot necessarily resume training. The adapted resume entry point checks progress fields and rejects files containing only parameters. Save full state separately for recovery from long-running training failures.
Before export, rerun the fixed validation set with the reduced weights and check agreement with the selected best result. Save the weight hash, class mapping, preprocessing and ONNX reference outputs to distinguish training changes from export or compilation changes.
5. Export a verifiable inference graph
Use the parameters selected by validation, with matching model code in evaluation and deployment mode. Check loading paths for training checkpoints, EMA and plain state dictionaries individually. A .pth extension does not identify the parameter key. Retain model size, class count and class order from training.
This DEIMv2 export includes PostProcessor in ONNX. It accepts images and orig_target_sizes and returns labels, boxes and scores. The static baseline uses batch=1. Class filtering, TopK and coordinate restoration follow this exporter; no extra YOLO decode or NMS is applied.
| Interface | Contract for this export |
|---|---|
images |
float32 NCHW, with backbone-specific preprocessing |
orig_target_sizes |
int64 [1,2], ordered [W,H] |
labels, scores |
[1,300] |
boxes |
[1,300,4], original-image xyxy coordinates |
The [W,H] order comes from this PostProcessor's scaling implementation and is not universal among DETR exporters. A non-square test image exposes swapped width and height more easily than a square image. Preserve DINOv3/HGNetv2 normalization differences rather than changing training-time numerical rules to unify deployment.
Generate the official preprocessed input once, save it as NPZ and feed the same tensor to the PyTorch deployment model and CPU ONNX Runtime. Check names, shapes, dtypes and finite values before comparing scores and valid boxes. TopK can reorder candidates, so match boxes by class and IoU rather than row number. Cosine similarity between class IDs is not a task-accuracy metric.
Treat graph simplification, attention rewrites and sampling replacements as separate candidates and compare each with this reference. Passing the ONNX checker establishes structural validity, not agreement in preprocessing, coordinates or task results. Save weight and ONNX hashes, class definitions, preprocessing and fixed inputs/outputs. Include external weights when the ONNX uses external data.
Use the CPU reference to investigate platform errors. See DEIM numerical error and graph partitioning on Qualcomm HTP for forward/backward graph partitioning and fixed-feature probes, and DEIM sampling semantics and deployment boundaries on RKNN for sampling implementation constraints.
6. Lessons from the investigation
Checking the training conditions behind the two AP values was more useful than comparing the numbers alone. Normalize was the visible configuration change, but initialization had also changed. Final configuration and loading logs prevented further tuning around the wrong cause.
Automatic training entry points should reduce manual work while preserving a way to inspect their actual behavior. Explicit argument transfer, backbone preprocessing, evaluation weights and artifact purposes give later model comparisons and deployment investigations a traceable baseline.