NVIDIA Releases DIN Deploy for Local AI Apps

NVIDIA has released Do Inference Now (DIN) Deploy, an open-source collection of C++ samples that demonstrate how to build local AI applications using ONNX Runtime with the TensorRT RTX execution provider for accelerated inference on Windows and Linux.

Architecture

The DIN Deploy samples follow a two-stage workflow that separates model preparation from runtime execution. Each sample includes a Python exporter that converts Hugging Face model checkpoints into ONNX format artifacts. The deployment stage then uses a native C++ command-line interface built on ONNX Runtime session and tensor APIs to run inference. This architecture allows developers to prepare models once using familiar Python tooling, then deploy them through lightweight C++ applications that leverage TensorRT RTX acceleration.

The repository currently supports four distinct AI capabilities. For automatic speech recognition, developers can choose between OpenAI Whisper, NVIDIA Parakeet TDT, or NVIDIA Nemotron ASR Streaming. The samples also include Meta SAM 2.1 for interactive segmentation tasks and FLUX for prompt-driven image generation. Each model follows the same export-then-deploy pattern, making it straightforward to add new models following the established structure.

Performance

According to measurements taken on DGX Spark hardware, GPU acceleration delivers substantial performance gains across the supported models. Parakeet TDT achieves 206x real-time performance, meaning it can process audio 206 times faster than the duration of the input. Nemotron ASR Streaming reaches 39x real-time performance, suitable for low-latency transcription applications.

The SAM 2.1 segmentation model demonstrates the impact of GPU acceleration on interactive workloads, running at 3 frames per second on GPU compared to 0.3 FPS on CPU - a 10x speedup that makes real-time segmentation practical. These benchmarks illustrate how the TensorRT RTX execution provider optimizes inference for NVIDIA GPUs while maintaining compatibility with the standard ONNX Runtime interface.

System Requirements

The samples target Windows and Linux operating systems. Developers need an NVIDIA GPU with TensorRT support to take advantage of the RTX execution provider, though the code can fall back to CPU execution through ONNX Runtime’s default providers. The repository assumes familiarity with C++ development environments and Python for the model export stage.

Each sample’s exporter handles the conversion from Hugging Face checkpoints to ONNX format, managing the graph transformations and optimization passes needed for TensorRT compatibility. The C++ runtime then loads these ONNX artifacts through the ONNX Runtime session API, which handles provider selection and tensor management.

Alternative Approaches

Developers building local AI applications have several framework options beyond ONNX Runtime with TensorRT. Native TensorRT APIs offer maximum control and performance but require more low-level code. PyTorch and TensorFlow provide higher-level interfaces with their own GPU acceleration paths, though they typically have larger runtime dependencies than ONNX Runtime.

For C++ developers specifically, the DIN Deploy samples demonstrate a practical middle ground - using ONNX as an interchange format lets teams prepare models in Python while keeping deployment code native. This approach works well for applications that need to minimize dependencies or integrate AI capabilities into existing C++ codebases.

The open-source nature of the samples makes them useful as reference implementations. Developers can examine the session configuration, tensor handling, and provider setup code to understand how to structure their own ONNX Runtime applications. The separation between export and deployment also supports workflows where model preparation happens on different systems than inference.