Different first results from one-shot and iterative event localization on the same video do not necessarily come from model randomness. If one interface reads the original file while another transcodes it through FFmpeg first, timestamps and sampling grids may change even when the pictures look identical.
In the archived investigation, iterative localization used a temporary transcoded file in its first round, while one-shot localization used the original. Reusing the original video for the first iterative round restored agreement on the first result. Only subsequent rounds cropped the remaining interval.
1. The model sees a sampled sequence
A video VLM usually reads images sampled according to a time rule rather than processing every frame. Decoded pixels, frame order, time information and sampling parameters jointly define its input. Transcoding may alter frame rate, the PTS origin and variable-frame-rate structure, selecting different pictures as a result.
ffprobe -v error -select_streams v:0 \
-show_entries stream=r_frame_rate,avg_frame_rate,time_base,start_time,duration \
-of json video.mp4
Metadata can help explain differences. Ultimately, record actual sampling times and image hashes. Equal average frame rates do not guarantee equal sampled frames.
1.1 Frame indices are not timestamps
For constant-frame-rate video with a continuous timeline and known origin, a frame index can approximate time. For variable-frame-rate video, intervals vary, so frame_index / avg_fps accumulates error. time_base specifies a timestamp's unit; multiplying PTS by that unit gives the corresponding time. DTS schedules decoding and can differ from presentation time in videos with B-frames.
To check the time of the actual pictures, inspect decoded frames as well as the container's declared frame rate:
ffprobe -v error -select_streams v:0 -show_frames \
-show_entries frame=pts_time,best_effort_timestamp_time,key_frame \
-of json video.mp4 > frame_timestamps.json
For long videos, inspect a local segment first, but verify the actual starting point after seeking. best_effort_timestamp_time is presentation time inferred by the decoder. Record missing or abnormal source timestamps rather than treating inferred values as complete original records.
1.2 Which input conditions transcoding changes
Re-encoding can change pixels, frame rate and timeline. The fps filter drops or duplicates frames to construct a constant-frame-rate sequence. Its output times are not necessarily the original PTS values of the selected source frames. Output frame-synchronization settings also belong in the comparison. Even with -copyts, muxing and synchronization may adjust final timestamps; it is not a switch that makes inputs identical.
For example, a source frame might occur at 10.04 seconds while a processed sampling grid selects a neighboring picture around 10.00 seconds. The model may see a different phase of an action. Interpreting sampled frame 20 directly as original-video time 10.00 seconds would also confuse the sampling target time with source-frame presentation time. These numbers illustrate the mechanism; they were not measured in the archived experiment.
If a standardized encoding is required, use the same normalized media and sampling configuration for both interfaces. For a strict model-input comparison, reusing the same decoded frames is easier to audit. Save their indices, source times and the time information passed to the model together.
Define what an image hash compares. A file hash only establishes equality of file bytes. A frame hash requires consistent pixel format, dimensions and color conversion. Lossy transcoding can produce visually similar frames with different hashes. Inspect the visual difference before concluding that unequal hashes imply a temporal mismatch.
2. A baseline for iterative localization
| Round | Input | Time mapping |
|---|---|---|
| First | Original media | Original timeline |
| Later | Remaining interval or its crop | Local time plus an explicit offset |
| Final output | Intervals in the original media | Validate bounds and forward progress |
Preserve a mapping from local crop time to source time. A returned end time should advance the next search and avoid repeated hits or loops with no progress. Do not lose the origin offset introduced by transcoding.
2.1 Requested start versus actual first frame
A request to crop from 30 seconds does not mean that the output's first frame corresponds exactly to source time 30 seconds. FFmpeg input-side -ss generally seeks to an earlier seek point. With transcoding and default accurate seeking, it decodes and discards preceding pictures. Stream copy should not be assumed to provide the same crop precision.
trim selects frames without resetting timestamps. setpts=PTS-STARTPTS moves the first retained frame to zero. This example only illustrates the distinction:
ffmpeg -i video.mp4 -an \
-vf "trim=start=30:end=60,showinfo,setpts=PTS-STARTPTS" \
-c:v libx264 -fps_mode passthrough clip.mp4
Here showinfo appears before the timestamp reset, so the retained-frame times received by the filter can be inspected. FFmpeg input processing may also shift timestamps. Compare against ffprobe output from the original file to recover source time. If the video does not begin at zero, first establish whether 30 seconds means a stream timestamp or elapsed time from the video origin. This example re-encodes and omits audio; it is not a complete audio/video localization pipeline.
Let the source origin be t_0 and the first retained frame's source timestamp be p_0. If the crop only selects an interval and shifts time, a returned local time u maps as follows:
For a requested start of 30 seconds, an actual first retained frame at 30.08 seconds and a local result of 2 seconds, the source result is 32.08 seconds, not 32 seconds. Speed changes, segment concatenation or irregular retiming require piecewise mappings rather than one added offset.
2.2 Advancing to the next round
Maintain the search position on the original timeline and select each new segment from the original media. Repeatedly cropping the previous round's temporary video adds lossy encoding generations and makes the offset chain harder to audit.
Validate 0 <= start < end <= current window duration for every result, then map it to the original timeline and check progress. Empty results, malformed output and non-advancing end times should trigger a bounded retry or termination with a recorded reason. Do not resubmit the same window indefinitely.
Adjacent windows can overlap to preserve context, but interval-based deduplication is then necessary. Searching from the previous result's end also assumes that target events are ordered and do not overlap. Concurrent events may be skipped entirely. This procedure is not equivalent to complete enumeration.
3. Task definition and evaluation
One-shot localization returns one interval. Iterative localization enumerates intervals through repeated calls. Evaluate missed events, duplicates, interval IoU and accumulated offsets, rather than checking only the first answer.
An investigation can start with three controlled comparisons:
| Comparison | Fixed conditions | Difference to inspect |
|---|---|---|
| Both interfaces reuse identical frames and time information | Model, prompt, sampling and generation settings | Additional interface processing |
| Original versus transcoded file | Sampling policy and requested interval | Actual frames, source PTS and input pixels |
| Later window versus its corresponding source interval | Identical source frames and event labels | Residual offsets after local-to-source mapping |
If the first comparison still differs, inspect the video processor, maximum frame count, visual resolution, time prompts and generation settings. A seed cannot compensate for changed inputs. Agreement on one result does not establish agreement for all videos.
Place predictions and labels on the same original timeline before calculating temporal IoU:
Also inspect absolute errors at both endpoints. A small offset can sharply reduce IoU for a short event, while IoU may conceal the same offset in a long event. Match iterative outputs to labels one-to-one. Repeated hits on one label must not count as additional recall.
For distant behavior recognition, separate observable motion from interpretive labels. Sustained stillness and repeated head movements can be recorded, but inferring drowsiness or intent may exceed the visual evidence. Generating timed event descriptions and then matching rules is one option; its benefit still needs validation against independently labeled data.
If localization feeds persistence checks, triggering and recovery, see Triggering and recovery in CV video policies for media time, unknown observations and state resets.
4. Preserving temporal information
Multiframe training samples should retain actual frame positions, window duration and sampling policy. Uniform sampling, dense event-centered sampling and motion-triggered sampling produce different input distributions. Deduplication that ignores this can remove the turning point of an action.
Each sample should at least retain its source-media identifier, search window, actual source-frame PTS, model-input order and time information, and preprocessing version. If a model limits the frame count, save the sequence actually submitted after truncation or resampling, rather than only the originally planned list.
Sparse sampling limits the temporal detail available as evidence. Two frames per second can miss a short action between samples. An output with hundredth-of-a-second precision does not mean that the input supports that precision. After coarse localization, return to the corresponding original-video interval, sample more densely and refine the endpoints.
Semantically similar pictures are not always interchangeable. Reversing two nearly identical poses can describe the opposite event. Labels that depend on action direction or duration require preserving order and span.
Evaluation can include time reversal, removal of key frames and changed temporal spans. If conclusions remain completely unchanged, check whether the model is relying on static scene cues rather than temporal reasoning.
5. Lessons from the investigation
Media preprocessing is an experimental condition. A fixed prompt and seed do not make a fair comparison unless the actual frames and time information read by the model are also fixed.
Time is part of the output contract. A half-second offset can originate in sampling. Checking the media pipeline first helps avoid mistaking an engineering difference for a model-capability problem.