Staged Qwen3-VL SFT: training strategy and limits

Domain fine-tuning of Qwen3-VL involves output formatting, task decisions and visual adaptation. The three-stage design starts with limited updates and expands trainable modules according to validation, aiming to reduce the impact of a small domain dataset on existing capabilities. Freezing visual parameters preserves their weights, but language-side updates can still change general answers.

Here, staged or hierarchical SFT means progressively changing the trainable scope. Supervision remains teacher-generated, reviewed answers. Without extra feature, distribution or logits losses, the stages cannot be described as feature or distribution distillation. When all stages use similar image-text tasks, formatting and task decisions are learned together; stage names describe emphasis.

1. What each stage is intended to do

The official Qwen3-VL-2B-Instruct configuration has 28 language layers and 24 visual blocks. Its last 10 language layers are indices 18–27; its last four visual blocks are 20–23. Other model sizes require their own configuration checks.

The proximity of upper language layers to outputs can motivate limited updates, but does not establish that those 10 layers specialize in JSON. Likewise, assigning edges and colors to the first 20 visual blocks and behavior semantics to the final four oversimplifies their roles.

2. Match the freeze plan to the framework implementation

In LLaMA-Factory, freeze is a partial-parameter fine-tuning mode. Other options containing the word freeze do not all disable training. The official argument definitions and adapter implementation distinguish:

Consequently, full with freeze_extra_modules=visual.blocks.0,... does not implement freezing the first 20 visual blocks. The inspected full branch does not read that list, while the freeze branch uses it to enable additional training. Without custom changes, disabling both visual-tower and projector freezing can update substantially more parameters than intended.

The second-stage plan for the final four visual blocks needs explicit parameter selection after the framework chooses its mode and before optimizer construction, using actual module names. Inspect the optimizer groups too. The archived material does not preserve a complete code revision and module-level training list, so launch commands alone cannot establish that historical training followed this design.

Qwen3-VL also has more than one terminal projector. In the 2B configuration, DeepStack takes features from visual blocks 5, 11 and 17 through separate mergers into the language model. Opening blocks 20–23 leaves those earlier feature extractors frozen; specify separately whether their mergers train. Inspect the main merger, DeepStack mergers, patch embedding and position embedding independently.

Run this check after the framework applies its freeze rules, using loss from an actual image-text batch:

trainable = [(n, p) for n, p in model.named_parameters() if p.requires_grad]
print("trainable params:", sum(p.numel() for _, p in trainable))

loss.backward()
with_grad = [n for n, p in trainable if p.grad is not None]
print("parameters with gradients:", len(with_grad))

Save module names as well as totals. Confirm that frozen blocks have no gradients and intended modules enter the optimizer. A gradient does not guarantee an update; compare parameters before and after an optimizer step when needed. Unfreezing after optimizer creation requires rebuilding or extending parameter groups.

If visual patch embedding is rewritten, validate forward and backward behavior for the actual trainable scope. See Equivalent visual patch embedding: Conv3D to Linear for an implementation preserving the original parameters.

3. Recheck training state at stage transitions

Loading a previous stage's model directory starts a new run from its weights; it does not restore optimizer, scheduler and step state. After changing trainable scope, build the optimizer for the new stage and set warmup explicitly. This differs from resume_from_checkpoint within an interrupted run.

A separate LoRA smoke test produces a different artifact from a freeze stage. If a stage saves only an adapter, subsequent full training must load it together with the base model or merge it first. An adapter directory is not a complete model. Inspect actual LoRA target matches rather than assuming names such as q_proj cover the full multimodal model.

The initial notes used learning rates 10^{-5}, 5\times10^{-6} and 10^{-6} for 3, 5 and 3 epochs. These are experimental configurations, not established optima or required budgets. Later records reduced the budget to 1, 2 and 1 epochs with rates 3\times10^{-6}, 8\times10^{-7} and 3\times10^{-7}, attempting fewer repeated updates on domain data.

With per-device batch 4, gradient accumulation 8 and two data-parallel workers, an update nominally accumulates 4\times8\times2=64 samples. Two visible NPUs do not establish two training workers; check the actual world size. Image count, resolution and text length also change the token budget for an equal sample batch. BF16 was the chosen precision in this environment, without establishing its optimality for every NPU task.

Adapters in compiled edge runtimes also depend on reserved branches, rank and cache state after loading. See Sophgo LoRA compilation reservations and dynamic loading for one implementation.

4. Fix multimodal inputs and validation settings together

image_max_pixels=65536 limits pixels per image. It is neither a fixed side length nor a total budget for a multi-image example. At approximately 256^2 pixels, check that small-object evidence survives resizing before trying to compensate by unfreezing more layers.

Check image count, order, placeholders and processor outputs for multi-image examples, including the actual visual grid and supervised tokens after truncation. Determine whether cutoff_len=4096 is sufficient from the processed sequence, rather than the raw JSON's character count.

The dataset registration name must match a key in dataset_info.json, and media paths must resolve within the training process. Copying train.json does not copy its images. Use matching multimodal templates, processors, thinking modes and stopping conditions for training and inference, then compare a fixed generation set. Changing a template name alone does not verify the resulting sequence or remove a learned JSON habit.

Setting eval_dataset does not ensure periodic validation. Configure evaluation strategy and frequency, saving and selection metrics, and check for eval logs. Training loss cannot replace validation curves; teacher-forced eval loss cannot replace generation tests.

5. Lower training loss and better task results

Archived curves show loss falling from roughly 2.5 to 0.5 in stage one, then to about 0.30 in stage two, and fluctuating around 0.28–0.34 in stage three. The model predicted training answers more easily. This does not establish separate improvements in formatting, visual decisions and robustness: fixed fields, JSON punctuation and explanatory text also contribute to loss.

Deployment tests later showed repetitive answers and ordinary questions forced into JSON. Reducing the training budget and weakening task prompts partly restored general answers, but domain performance declined. The existing domain model was ultimately retained for the dedicated task. The observations show a tradeoff between specialization and general answers, rather than simultaneous gains in all capabilities after full fine-tuning.

Most observations came from deployment tests. Compare floating-point checkpoints and quantized artifacts with identical inputs to separate template and deployment effects; learning rate alone cannot explain every anomaly. A dedicated model can have an explicit task scope. A model also serving general questions needs an independent regression set and tests of mixed data or smaller update scopes.

Full SFT is also distinct from quantization-aware training. Without simulated quantization or an appropriate loss, stage three cannot be called the most stable foundation for INT4. Reevaluate formatting, decisions and general capability against the same floating-point baseline after quantization. See Structured output and task evaluation for small Qwen models.

6. Lessons from the investigation

First establish that trainable scope changed as designed, then measure whether the change helped. Stage names, parameter names and falling loss cannot establish either result.

Useful baselines include single-stage LoRA with an equal budget, language-side fine-tuning with frozen vision, and staged training. Without these controls, describe the stages as an executed strategy rather than a proven optimum. Preserve design intent separately from experimental conclusions.

7. References

• Qwen3-VL-2B-Instruct configuration
• LLaMA-Factory fine-tuning arguments
• LLaMA-Factory full, freeze and LoRA implementation
• LLaMA-Factory visual freezing and module mappings
• Qwen3-VL DeepStack implementation
• Transformers training and evaluation arguments