NVIDIA AGX Orin system setup and TensorRT deployment

Deploying a Jetson AGX Orin involves the board-level system, compute libraries and application environment. Jetson Linux supplies BSP components such as the bootloader, kernel and drivers. JetPack adds compute components including CUDA and TensorRT. Flashing the system alone does not mean that Python bindings, model-building tools and container runtimes are complete.

This article follows the archived Jetson Linux R36.5, CUDA 12.6 and TensorRT 10.3 environment, from flashing to engine deployment. Paths and input names in commands are examples. Installation, flashing, network changes and Docker restarts are setup or maintenance steps. When investigating an existing device, run queries first and address only the missing components.

1. Establish the system and software stack

1.1 Distinguish the BSP, compute stack and Python environment

Record system information before deciding what to install:

cat /etc/nv_tegra_release
cat /etc/os-release
uname -r
uname -m
getconf GNU_LIBC_VERSION
dpkg-query -W 'nvidia-l4t-core' 'nvidia-jetpack'
dpkg-query -W 'cuda-cudart-*' 'libnvinfer*' 'python3-libnvinfer*'
readlink -f /usr/local/cuda

dpkg-query returns a nonzero status for missing packages. Interpret its output package by package. The absence of the nvidia-jetpack metapackage does not establish that CUDA or TensorRT is absent; they may have been installed individually.

The archived baseline was:

The official JetPack 6.2.2 combination includes Jetson Linux 36.5, CUDA 12.6 and TensorRT 10.3. This table defines the tutorial's scope; it does not instruct devices on other versions to downgrade. Compute components can also be upgraded through officially supported paths. One JetPack name does not establish every installed component version.

1.2 Check interpreters and build tools

command -v python3
python3 --version
/usr/bin/python3 --version
command -v nvcc
nvcc --version
command -v trtexec
ls -l /usr/src/tensorrt/bin/trtexec

A missing nvcc may mean only that CUDA development tools are not installed. It does not establish that GPU runtime libraries are unavailable. Conversely, nvcc --version identifies that compiler's version, without proving that the current Python process loads libraries from the same CUDA installation.

2. Flashing, login and storage

2.1 Identify the flashing target

Confirm the module, carrier board and boot medium before flashing. AGX Orin developer-kit configuration does not apply to every third-party carrier. Internal eMMC and external NVMe also use different flashing procedures. Back up target-disk data, then choose the matching SDK Manager or BSP tools.

With SDK Manager, select target hardware and JetPack version, connect Recovery USB, flash the system and complete first boot, then install required target SDK components. Use the host Ubuntu version required by that SDK Manager and BSP release.

The command-line approach requires a BSP and Sample Root Filesystem from the same release:

1. Extract the BSP to obtain Linux_for_Tegra.
2. Extract the matching Sample Root Filesystem into rootfs/, preserving permissions.
3. In the BSP directory, run sudo tools/l4t_flash_prerequisites.sh as required by that release, followed by sudo ./apply_binaries.sh.
4. Enter Force Recovery and check USB detection on the host.
5. Flash with the command for the actual board and medium.

# 在刷机主机执行，仅查询 USB 设备。
lsusb -d 0955:

The archived developer kit used this command for internal storage:

# 在已准备好的 BSP 目录执行；此命令会刷写目标设备。
sudo ./flash.sh jetson-agx-orin-devkit internal

External NVMe requires the corresponding BSP's l4t_initrd_flash.sh procedure and verification of the target disk and partition configuration. Do not simply replace internal with a drive name. USB detection in Recovery establishes a flashing connection only. After flashing, verify that the device boots from the intended medium.

2.2 Headless initialization and networking

Without a display, a developer kit's supported USB serial interface can complete initial user and network setup. Runtime USB networking and Recovery are different states: one provides access to the running system, while the other is for flashing.

Query actual connections rather than assuming an interface is named eth0:

ip -brief address
ip route
nmcli device status
nmcli connection show

For a static address, modify the existing connection and specify prefix, gateway and DNS. Changing the active interface remotely interrupts the session, so retain serial or another management path. The tutorial does not prescribe a private subnet; use the deployment network's assignment.

2.3 Check the root partition and data disks

findmnt /
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINTS
df -hT
docker info --format '{{.DockerRootDir}}'

Models, build artifacts and image layers can fill the root partition quickly. After mounting a data disk, verify write permissions and persistent mount configuration. Otherwise, after a reboot the service may write to an empty directory on the root partition instead.

Docker and an environment using the containerd image store may consume separate locations. Confirm the actual storage backend before migration. Stopping services, copying data, changing configuration and verifying after restart form a separate maintenance operation. Changing one path does not establish that all image data moved.

3. Complete the CUDA and TensorRT installation

Check that NVIDIA APT sources match the BSP:

grep -RhE '^[[:space:]]*deb .*repo.download.nvidia.com/jetson' \
  /etc/apt/sources.list /etc/apt/sources.list.d/
apt-cache policy nvidia-jetpack python3-libnvinfer libcudla-12-6
dpkg --audit

Do not mix another L4T generation's repositories merely to install a new package. On a fresh system with correct sources and dependencies, the full JetPack can be installed:

sudo apt-get update
sudo apt-get install nvidia-jetpack

For limited storage, select runtime and development packages from the official component list. TensorRT's native runtime, Python bindings, headers and trtexec are separate components. Lean or dispatch bindings alone do not provide a complete model-building environment.

Simulate dependency changes before adding packages, for example:

apt-get --simulate install python3-libnvinfer python3-libnvinfer-dev

Install only after confirming that the existing compute stack will not be replaced unexpectedly. Review CUDA packages marked “automatically removable” individually. Do not treat apt autoremove as an automatic final deployment step.

4. Diagnose Python and dynamic-library problems

4.1 Identify what Python actually imported

/usr/bin/python3 -B - <<'PY'
import sys
import tensorrt as trt

print("python:", sys.executable)
print("TensorRT:", getattr(trt, "__version__", "<missing>"))
print("module:", getattr(trt, "__file__", None))
PY

A normal environment should report both a version and a definite module location. If import succeeds without __version__, check for a same-named local file, empty directory or leftover namespace package. An error-free import alone does not establish complete bindings.

APT-installed bindings usually live in the system Python's dist-packages, which an isolated virtual environment may not see. To reuse system TensorRT, create an environment with the same Python minor version:

/usr/bin/python3 -m venv --system-site-packages .venv
. .venv/bin/activate
python -B -c 'import tensorrt as trt; print(trt.__version__, trt.__file__)'

This also exposes other system Python packages. For stronger isolation, use a complete environment matched to the platform. Do not copy a Python 3.10 binary extension into another minor version or overwrite JetPack dependencies with a blind pip install --upgrade tensorrt.

4.2 Missing libraries versus incorrect search paths

For an error such as:

ImportError: libcudla.so.1: cannot open shared object file

Inspect packages, files and the linker cache:

dpkg-query -W 'libcudla*'
ls -l /usr/local/cuda-12.6/targets/aarch64-linux/lib/libcudla.so*
ldconfig -p | grep -E 'libcudla|libnvinfer'
dpkg -L python3-libnvinfer | grep -E '\.so($|\.)'

Then run ldd against the actual Python extension found above. The archived path was:

ldd /usr/lib/python3.10/dist-packages/tensorrt/tensorrt.so

For missing runtime libraries in CUDA 12.6, install libcudla-12-6. Add the development package only when development requires it:

sudo apt-get install libcudla-12-6

Use the archived path repair only if the file exists and system search configuration is genuinely missing:

printf '%s\n' '/usr/local/cuda-12.6/targets/aarch64-linux/lib' \
  | sudo tee /etc/ld.so.conf.d/cuda-12-6-aarch64.conf
sudo ldconfig

Recheck ldconfig -p, ldd and Python import afterward. A same-named library in an old CUDA directory or Docker overlay is not necessarily suitable for the current host. Nor should a symbolic link disguise an incompatible SONAME as the required version.

4.3 Separate media dependencies from inference setup

Native media applications often depend on FFmpeg, OpenCV, Python and C++ runtime libraries together. Start with ldd on the actual binary and locate a build matching each missing SONAME. Upgrading an FFmpeg command-line tool broadly may not address those dependencies.

Interactive terminals, virtual environments and systemd services have different environments. Setting LD_LIBRARY_PATH in a personal shell does not automatically fix a service. Scope private libraries to that service's startup environment. Adding an entire conda library directory to system-wide search paths can replace the libstdc++, FFmpeg or OpenCV expected by media plugins.

5. Configure GPU containers

Query Docker and the NVIDIA runtime first:

docker version
docker info --format '{{json .Runtimes}}'
command -v nvidia-container-runtime
command -v nvidia-ctk
dpkg-query -W 'nvidia-container-toolkit*'

If the toolkit is missing, follow NVIDIA's installation guide to configure the corresponding APT repository and install it. Versions providing nvidia-ctk can configure Docker as follows:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

This updates Docker configuration and restarts the service, so schedule it as maintenance. Explicitly selecting the runtime is sufficient; one inference container does not require changing the default runtime for every container.

The image must satisfy aarch64, Jetson/iGPU support and compute-library compatibility together. An aarch64 server image does not necessarily support Jetson. Where image CUDA or TensorRT differs from the host, consult that image's support information.

For a compatible image that already includes Python TensorRT bindings, begin with an import check:

# jetson-trt:validated 代表已经核对兼容性的本地镜像。
docker run --rm --runtime=nvidia \
  -e NVIDIA_VISIBLE_DEVICES=all \
  jetson-trt:validated \
  python3 -B -c 'import tensorrt as trt; print(trt.__version__, trt.__file__)'

Then verify actual GPU computation separately. Container startup, library import and GPU-operator execution are three different checks. Mounting the host's entire /usr/lib is not an appropriate way to conceal incompatible image dependencies.

If the application also needs PyTorch, choose NVIDIA's Jetson wheel or a container explicitly supporting the current JetPack. Check Python, CUDA and framework-extension versions. A true result from torch.cuda.is_available() does not cover every extension. For no kernel image is available, inspect GPU-architecture support in the binary containing the failing operator.

6. Build TensorRT engines in the target environment

6.1 Fix the input contract

Before building, preserve the ONNX and external weights, and verify input names, dtypes, dimensions and output meanings. For detection, also fix RGB/BGR order, resizing or Letterbox, normalization, class order and the location of NMS. Establish PyTorch and ONNX numerical references first.

Find the actual trtexec and read its version's help:

/usr/src/tensorrt/bin/trtexec --help

A local symbolic link may expose it through PATH. Check the link target and dependencies so the shell does not call an older tool while Python imports a different runtime.

6.2 Establish a static FP16 baseline

This example uses TensorRT 10.3 and a static-input ONNX:

/usr/src/tensorrt/bin/trtexec \
  --onnx=model.onnx \
  --saveEngine=model-fp16.plan \
  --fp16 \
  --skipInference

--skipInference makes this step build and save the engine only. Building still performs computation and tactic timing, so use an environment with available resources.

A static ONNX does not need a dynamic profile. To increase batch size in a graph fixed at batch=1, re-export the appropriate static graph or export a genuinely dynamic dimension. Appending a batch number to the command does not change the graph contract.

--fp16 permits FP16 use; it does not require every layer or I/O tensor to be FP16. Verify the floating-point path before introducing INT8 and calibration, making export, compilation and quantization errors easier to separate.

If an FP16 engine builds and runs but fixed-sample outputs diverge materially, see TensorRT FP16 numerical troubleshooting through an equivalent QKV layout rewrite for investigation through intermediate tensors and graph expressions.

6.3 Dynamic batch and profiles

Suppose an input named images has only a dynamic first dimension, with channels and resolution fixed at 3×640×640:

/usr/src/tensorrt/bin/trtexec \
  --onnx=model-dynamic.onnx \
  --saveEngine=model-dynamic-fp16.plan \
  --fp16 \
  --minShapes=images:1x3x640x640 \
  --optShapes=images:4x3x640x640 \
  --maxShapes=images:8x3x640x640 \
  --skipInference

The values 1, 4 and 8 are examples. Min/max bound accepted inputs; opt is an important optimization target and should reflect common requests. Do not choose the range solely by the largest batch that fits in memory. Multi-input models must cover every dynamic input while preserving inter-input dimension relationships.

Check scripts separately for TensorRT 8.5 and 10.3:

A workspace limit is not the total memory limit for the build process or runtime. Orin shares physical memory among weights, activations, CPU applications and media buffers.

6.4 Engine compatibility

.plan and .engine are usually different extensions for serialized engines. An engine contains optimizations for target hardware and runtime. Successful generation on a server does not establish that it can be copied to Orin and run.

Prefer building on the target Orin or in a matching Jetson build environment. Version compatibility and hardware compatibility are separate options with platform limits. A hardware-compatibility mode unsupported on JetPack cannot provide general engine transfer from desktop GPUs to Orin.

6.5 Distinguish INT8 calibration from explicit QDQ

This discussion applies to TensorRT 10.3. That version marks implicit INT8 quantization and calibrator APIs deprecated, but maintaining older workflows still requires distinguishing the paths. Inspect ONNX for QuantizeLinear / DequantizeLinear before deciding where quantization ranges come from:

Calibration inputs must match production color handling, normalization, layout and shape. Recheck the cache after changes to the model, preprocessing or dimensions. trtexec does not automatically turn an image directory into correct calibration inputs; use a matching calibration program or compatible calibration table. INT8 performance tests without real calibration data cannot demonstrate acceptable accuracy.

A QDQ graph already expresses quantization semantics. A calibrator should not redefine those ranges. Removing Q/DQ is not ordinary graph cleanup, and quantization options should not be layered on without inspecting the graph. Preserve the original graph and logs after a build failure, and compare the floating-point baseline, QDQ graph and respective build settings separately. A graph that builds after QDQ removal does not by itself prove that “double quantization” caused the earlier failure.

Validate output branches separately. A detection graph may concatenate pixel-scale boxes with low-magnitude scores. If actual quantization granularity leaves scores with inadequate resolution, whole-output cosine similarity may hide losses near thresholds. Check real scales, layer precision and task results before considering mixed precision or output-branch changes. Lowering a detection threshold cannot recover numerical information already lost.

7. Validate execution and package deployment

7.1 Separate numerical checks from performance checks

Example loading and timing of a static engine:

/usr/src/tensorrt/bin/trtexec \
  --loadEngine=model-fp16.plan \
  --warmUp=1000 \
  --duration=10

For a dynamic engine, set a concrete runtime shape:

/usr/src/tensorrt/bin/trtexec \
  --loadEngine=model-dynamic-fp16.plan \
  --shapes=images:4x3x640x640 \
  --warmUp=1000 \
  --duration=10

These commands actually occupy the GPU. Default generated test inputs can check execution and timing, but do not establish real-image accuracy. Feed identical preprocessed tensors to PyTorch, ONNX and TensorRT, then compare raw outputs and final results. For dynamic models, include profile boundaries and common shapes.

Query power mode and frequency state with:

sudo nvpmodel -q
sudo jetson_clocks --show

jetson_clocks --show is a query. It is different from running the command without arguments to lock clocks. Record temperature, memory and other workloads during comparisons. Separate GPU compute, transfer and end-to-end latency. trtexec throughput is not automatically video-service throughput.

After integrating multistream video, budget decoding, inference, buffering and queuing separately. See Resource budgeting and capacity planning for multistream NVIDIA video AI.

7.2 Package reproducible deployment material

sha256sum model.onnx model-fp16.plan

Hash external weights and custom plugins too. Use configurable directories in deployment packages, and inject accounts, network addresses and service credentials through the runtime environment.

Narrow failures in the order system packages → dynamic libraries → Python bindings → engine → application. Establish working system Python and native TensorRT before adding virtual environments and containers, and enter the real media pipeline last. This reduces the cost of diagnosing several overlapping environment problems.

8. References

• JetPack 6.2.2 stack and installation
• Jetson Linux R36.5 flashing guide
• Jetson Linux packages and updates
• NVIDIA Container Toolkit installation and Docker configuration
• PyTorch on Jetson: installation and compatibility
• TensorRT trtexec options
• TensorRT engine compatibility
• TensorRT 10.x support matrix
• TensorRT 10.x quantization, calibration and QDQ