DEIM sampling semantics and deployment limits on RKNN

DEIMv2's deformable attention relies on sampling at continuous coordinates. Generating an RKNN file does not mean that sampling runs on the NPU. Registering CPU or GPU custom operators can complete execution, but adds graph partitions, synchronization and layout conversions. Deployment needs to account for executability, mathematical semantics and latency together.

This article covers an RK3576 retest and historical RK3588 comparisons using RKNN-Toolkit2/Runtime 2.3.2 and RKNPU Driver 0.9.8. The export baseline was DEIMv2-S with static batch=1 and 640×640 input. These results do not generalize to other model sizes or versions.

For Toolkit2, device Runtime and minimal-model setup, see Rockchip RKNN environment setup and device inference. For DEIM weights, preprocessing and export references, see DEIM fine-tuning: configuration, preprocessing and weight artifacts.

1. What GridSample does

torch.nn.functional.grid_sample(
    feature, grid,
    mode="bilinear",
    padding_mode="zeros",
    align_corners=False,
)

GridSample reads multiscale features at continuous coordinates predicted by the decoder, then aggregates them with attention weights. It is not a fixed Resize. Sampling positions vary with inputs and queries, so they cannot be folded into a fixed resizing table. The archived DEIMv2-S had four decoder layers and three feature levels, producing 12 GridSample calls. Removing them changes the model's semantics. Other DEIM variants do not necessarily have 12 calls.

With align_corners=False, normalized horizontal coordinate u maps to pixel coordinate:

The vertical coordinate uses feature height H in the same way. Bilinear interpolation reads four neighboring values and weights them by coordinate distance. For the feature [[0,10],[20,30]], sampling at pixel coordinates (x,y)=(0.25,0.5) gives 12.5. A value at one integer index cannot replace that result.

Under align_corners=False, u=-1 maps to x=-0.5, rather than the center of the first pixel. With zero padding, each of the four neighbors must be checked for bounds and padded separately. Clamping the whole coordinate into the image first implements different boundary behavior. Coordinate order (x,y), the half-pixel offset, interpolation and padding must all match. Equal output shapes are insufficient evidence of correctness.

2. The cost of fallback

With this version combination, the converter could produce an RKNN containing GridSample nodes, while the default device Runtime lacked their implementation and reported Unsupport CPU op: GridSample. ONNX export, RKNN construction, Runtime initialization and actual forward execution therefore need separate checks. File generation is an intermediate step.

Registering a CPU or GPU custom operator supplies a fallback implementation; it does not dynamically install a new NPU kernel. A GPU callback takes over the relevant nodes only. It does not turn the entire RKNN network into a Mali GPU graph. Comparing whole-graph GPU execution requires another GPU inference backend that supports the model.

On RK3576, GPU custom-operator registration failed after initialization. This is neither a zero-time result nor a successful model run. The RK3588 numbers come from earlier records; the RK3576 numbers are ranges from a small number of repeated calls. The table mixes averages and ranges. It is not a cross-chip ranking under uniform conditions and does not include a complete video pipeline.

Fusing CPU fallback reduced whole-graph time from about 961.3 ms to 856.5 ms, a reduction of roughly 10.9%. Reducing fallback scheduling and boundaries helped, but did not bring inference into the tens-of-milliseconds range. Fused GPU fallback still took about 952.5 ms, showing that moving a slow operator to a GPU does not by itself ensure an end-to-end speedup.

The official v2.5.0 changelog lists GPU support for GridSample. After upgrading, recheck the SoC, sampling mode, padding, layout and actual execution device rather than carrying forward the 2.3.2 failure conclusion. GPU support is also distinct from native NPU support. This article contains no whole-graph retest on the newer version.

Another practical partitioning approach is described in DEIM numerical errors and graph partitioning on Qualcomm HTP: use the three feature levels as the split and place the entire decoder on the CPU. That addressed HTP numerical differences and is not a verified deployment solution for this device.

3. How discrete sampling differs

Discrete sampling replaces bilinear interpolation with integer indexing and Gather. It avoids GridSample but changes the computation. The historical implementation used untransformed sampling locations r\in[0,1], converting r * [W,H] + 0.5 to integers. The default branch first mapped r to [-1,1] and then called GridSample. Equal output shapes do not make those coordinate calculations and samples equivalent.

The boundary behavior differs too. Integer indexing uses clamp to avoid out-of-bounds accesses, while the default path pads out-of-bounds neighbors with zero. The historical discrete implementation also clamped both coordinates to H-1. Rectangular features require checking horizontal W-1 and vertical H-1 separately. Correcting that indexing bound fixes that issue without restoring bilinear interpolation.

Switching a bilinearly trained model directly to discrete sampling should be treated as a separate model. Compare boxes, scores, labels and dataset AP again, particularly for small objects, boundaries and samples sensitive to sampling position. Training or fine-tuning on the discrete path also requires new training and export configurations. The original bilinear model's evaluation no longer establishes its performance.

Even after unsupported operators are removed, MatMul, Gather, Transpose and scheduling of small operators in a DETR graph may remain expensive. Convertibility does not establish real-time execution.

4. Measuring boundary costs

For k CPU/NPU boundaries in one inference, the end-to-end time can be decomposed as:

This decomposition describes a serial critical path. Where work overlaps, use measured end-to-end time. Here k counts actual execution boundaries, not simply the number of GridSample nodes in ONNX. Timing only the CPU kernel omits synchronization, layout conversion and transfer. DMA-BUF sharing can reduce explicit copies without eliminating synchronization, cache-coherence work or layout conversion.

Keep comparisons at three levels:

After warm-up, record forward-only and end-to-end P50/P95 separately, along with Toolkit, Runtime, driver, CPU thread count and NPU core configuration. Without component profiling, a slow discrete path does not identify one MatMul or Gather as the sole bottleneck.

Create dedicated interpolation tests for rectangular inputs and boundary coordinates: center, corners, out-of-bounds positions and half-pixel positions. A passing square-image inference does not establish correct coordinate scaling and clamp behavior at another shape.

The whole graph must also receive real original-image [W,H] through orig_target_sizes, using the dtype accepted by the platform model. Benchmarking with [0,0] may complete a forward pass while scaling all output boxes to zero. It cannot demonstrate valid detection. The exported PostProcessor already organizes labels/boxes/scores; do not apply a YOLO decoder again.

5. Lessons from the deployment

An operator-support table is only the first layer of deployment. Actual cost depends on how supported subgraphs connect and how data moves between devices and layouts. Fusing fallback calls can improve latency, but its limits need to remain explicit.

Run a minimal deployment on the target device before large-scale training. If the core architecture requires frequent heterogeneous fallback, a verified CNN detection baseline may meet the task's latency requirements more readily than further custom-operator work.

6. References

• RKNN-Toolkit2
• Official RKNN changelog
• PyTorch grid_sample
• ONNX GridSample semantics
• DEIMv2