LabHub

Blog

NVIDIA Triton Inference Server Production Guide: GPU Model Serving Optimization Strategies

2026-03-08 · 29 min read · #ai-platform#triton#inference-server#gpu#model-serving#nvidia#2026-03

한국어English日本語中文

Triton Inference Server

Introduction

Training an ML model and serving it reliably in production are entirely different engineering challenges. In training, GPU utilization, batch size, and convergence speed are what matter; in serving, latency, throughput, GPU memory efficiency, running many models at once, and failure recovery are decisive. In a service that requires real-time inference in particular, you have to hold p99 latency under 10ms while handling thousands of requests per second, so simply putting a model on top of Flask runs into limits.

NVIDIA Triton Inference Server is an open-source inference server designed to solve these production model serving problems at the architectural level. It can serve the major frameworks - TensorFlow, PyTorch, ONNX, TensorRT, the Python backend and more - from a single server at the same time, and provides Dynamic Batching, Model Ensemble, Concurrent Model Execution, and GPU memory management. Since March 2025 it has been folded into the NVIDIA Dynamo platform, giving it an ecosystem that extends to LLM inference optimization.

This article comprehensively covers the architecture design, model configuration, performance optimization, Kubernetes deployment, and troubleshooting strategy you need to actually run Triton Inference Server in production. It aims to be a practical operations guide for teams serving dozens of models at once and handling tens of thousands of inference requests per second, rather than a simple tutorial.

Triton Inference Server Architecture

Core Component Structure

Triton Inference Server is made up of four core layers: Model Repository, Scheduler, Backend, and Inference Engine. When a client request arrives over the HTTP/gRPC endpoint, the scheduler groups requests into a suitable batch and the chosen backend performs the actual inference.

Client (HTTP/gRPC/C API)
        │
        ▼
┌─────────────────────────────┐
│   Request Handler           │
│   (HTTP: 8000, gRPC: 8001)  │
│   (Metrics: 8002)           │
├─────────────────────────────┤
│   Scheduler Layer           │
│   ┌───────────────────────┐ │
│   │ Dynamic Batcher       │ │
│   │ Sequence Batcher      │ │
│   │ Ensemble Scheduler    │ │
│   └───────────────────────┘ │
├─────────────────────────────┤
│   Backend Layer             │
│   ┌──────┬──────┬────────┐  │
│   │TRT   │ONNX  │PyTorch │  │
│   │TF    │Python│OpenVINO│  │
│   └──────┴──────┴────────┘  │
├─────────────────────────────┤
│   GPU/CPU Execution Engine  │
│   (CUDA Streams, MIG, MPS)  │
└─────────────────────────────┘

Model Repository Structure

Triton loads models from a file-system-based Model Repository. Each model has its own directory, made up of per-version subdirectories and a configuration file (config.pbtxt).

model_repository/
├── text_classifier/
│   ├── config.pbtxt
│   ├── 1/
│   │   └── model.onnx
│   └── 2/
│       └── model.onnx
├── image_detector/
│   ├── config.pbtxt
│   └── 1/
│       └── model.plan          # TensorRT engine
├── embedding_model/
│   ├── config.pbtxt
│   └── 1/
│       └── model.pt            # PyTorch TorchScript
└── preprocessing/
    ├── config.pbtxt
    └── 1/
        └── model.py            # Python backend

By default Triton serves the model with the highest version number, but you can customize the version policy in config.pbtxt. That lets you carry out canary deployments or A/B tests at the model level.

Supported Backends and Frameworks

The backends Triton supports are as follows.

BackendModel formatBest suited to
TensorRT.plan (TensorRT Engine)Top-performance GPU inference, CV/NLP
ONNX Runtime.onnxCross-framework compatibility, CPU/GPU
PyTorch (LibTorch).pt (TorchScript)Serving PyTorch models directly
TensorFlowSavedModelModels in the TF ecosystem
Python.pyCustom pre/post-processing, BLS
OpenVINOIR FormatIntel CPU optimization
DALIPipelineGPU-accelerated data preprocessing
FILXGBoost/LightGBMServing tree-based models

Installation and Model Deployment

Docker-Based Installation

The most standard way to run Triton in production is to use the NVIDIA NGC container.

# Run the NVIDIA Triton Inference Server container
# Pull the latest image from NGC (based on the 25.02 release)
docker pull nvcr.io/nvidia/tritonserver:25.02-py3

# Start the server together with the model repository
docker run --gpus all \
  --rm \
  -p 8000:8000 \
  -p 8001:8001 \
  -p 8002:8002 \
  -v $(pwd)/model_repository:/models \
  nvcr.io/nvidia/tritonserver:25.02-py3 \
  tritonserver \
    --model-repository=/models \
    --strict-model-config=false \
    --log-verbose=1

# Check the server status
curl -v localhost:8000/v2/health/ready

With the --strict-model-config=false option, Triton infers the input and output tensor information from the model file automatically. This auto-configuration works well for ONNX, TensorFlow SavedModel, and TensorRT Engine, but for the PyTorch and Python backends you must state config.pbtxt explicitly.

Writing the Model Configuration (config.pbtxt)

config.pbtxt is the core configuration file that determines how Triton loads and serves a model. Below is a typical configuration for an ONNX classification model.

# text_classifier/config.pbtxt
name: "text_classifier"
platform: "onnxruntime_onnx"
max_batch_size: 64

input [
  {
    name: "input_ids"
    data_type: TYPE_INT64
    dims: [ 512 ]
  },
  {
    name: "attention_mask"
    data_type: TYPE_INT64
    dims: [ 512 ]
  }
]

output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ 3 ]
  }
]

# Dynamic Batching configuration
dynamic_batching {
  preferred_batch_size: [ 8, 16, 32 ]
  max_queue_delay_microseconds: 100
}

# GPU instance configuration
instance_group [
  {
    count: 2
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

# Model version policy
version_policy: { latest { num_versions: 2 } }

# Optimization settings (ONNX Runtime)
optimization {
  execution_accelerators {
    gpu_execution_accelerator: [
      {
        name: "tensorrt"
        parameters {
          key: "precision_mode"
          value: "FP16"
        }
        parameters {
          key: "max_workspace_size_bytes"
          value: "1073741824"
        }
      }
    ]
  }
}

Here, a max_batch_size other than 0 means the model is capable of Dynamic Batching. The count in instance_group specifies how many model instances run concurrently on that GPU. Raising the instance count as far as GPU memory allows increases throughput, but exceeding memory produces an OOM error, so care is needed.

Dynamic Batching Optimization

How Dynamic Batching Works

Dynamic Batching is the feature that delivers the largest performance gain in Triton. Inference requests that arrive individually are grouped into a batch on the server side automatically and handed to the GPU in one go. Since GPUs are built for data-parallel processing, a larger batch means higher GPU utilization and a lower processing cost per request.

# Advanced Dynamic Batching configuration
dynamic_batching {
  # Preferred batch size: execute immediately once this size is reached
  preferred_batch_size: [ 4, 8, 16, 32 ]

  # Maximum queue wait time (microseconds)
  # Once this time passes, run the batch with whatever has gathered so far
  max_queue_delay_microseconds: 200

  # Request priority settings
  priority_levels: 3
  default_priority_level: 2

  # Queue policy: reject once the maximum queue size is exceeded
  default_queue_policy {
    timeout_action: REJECT
    default_timeout_microseconds: 5000000
    allow_timeout_override: true
    max_queue_size: 100
  }
}

max_queue_delay_microseconds is the key parameter governing the trade-off between latency and throughput. A small value lowers latency but leaves batches less full, so GPU utilization drops. A large value fills batches more and raises throughput, but the latency of an individual request goes up.

A Dynamic Batching Tuning Guide

To tune Dynamic Batching parameters in real production, follow this procedure.

  1. Measure the baseline: start with the defaults, without preferred_batch_size or max_queue_delay_microseconds.
  2. Explore batch sizes: with perf_analyzer, raise the batch size through 1, 2, 4, 8, 16, 32, and 64, measuring how throughput and latency change.
  3. Find the saturation point: identify the batch size at which throughput stops rising or latency climbs sharply. That becomes your optimal max_batch_size.
  4. Adjust the queue delay: tune max_queue_delay_microseconds to your SLA. For a real-time service 50~200us is appropriate; for batch processing 1000~5000us is.
# Measuring Dynamic Batching performance with perf_analyzer
# Measure while raising the concurrent request count from 1 to 32
perf_analyzer \
  -m text_classifier \
  -u localhost:8001 \
  -i grpc \
  --concurrency-range 1:32:4 \
  --measurement-interval 10000 \
  -b 1 \
  --percentile=99 \
  -f results_dynamic_batch.csv

# Check the results: how throughput (infer/sec) and p99 latency trend
# concurrency=1:  throughput=120 infer/sec, p99=8.3ms
# concurrency=4:  throughput=450 infer/sec, p99=9.1ms
# concurrency=8:  throughput=820 infer/sec, p99=10.2ms
# concurrency=16: throughput=1400 infer/sec, p99=12.5ms
# concurrency=32: throughput=1650 infer/sec, p99=22.7ms  <- saturation begins

At concurrency 32 the latency increase outweighs the throughput gain, so for this model a concurrency of around 16 is the optimal operating point.

Model Ensemble Pipelines

Ensemble Architecture

Model Ensemble is the feature that links several models into a single DAG (Directed Acyclic Graph) so preprocessing, inference, and post-processing run as a pipeline inside the server. It reduces network round trips between client and server and keeps intermediate tensors in GPU memory, removing the data transfer overhead.

# ensemble_pipeline/config.pbtxt
name: "ensemble_pipeline"
platform: "ensemble"
max_batch_size: 32

input [
  {
    name: "RAW_TEXT"
    data_type: TYPE_STRING
    dims: [ 1 ]
  }
]

output [
  {
    name: "PREDICTION"
    data_type: TYPE_FP32
    dims: [ 3 ]
  }
]

ensemble_scheduling {
  step [
    {
      model_name: "tokenizer"
      model_version: -1
      input_map {
        key: "TEXT_INPUT"
        value: "RAW_TEXT"
      }
      output_map {
        key: "INPUT_IDS"
        value: "tokenized_ids"
      }
      output_map {
        key: "ATTENTION_MASK"
        value: "tokenized_mask"
      }
    },
    {
      model_name: "text_classifier"
      model_version: -1
      input_map {
        key: "input_ids"
        value: "tokenized_ids"
      }
      input_map {
        key: "attention_mask"
        value: "tokenized_mask"
      }
      output_map {
        key: "logits"
        value: "raw_logits"
      }
    },
    {
      model_name: "postprocessor"
      model_version: -1
      input_map {
        key: "LOGITS"
        value: "raw_logits"
      }
      output_map {
        key: "RESULT"
        value: "PREDICTION"
      }
    }
  ]
}

In the configuration above, execution proceeds in the order tokenizer (Python backend) -> text_classifier (ONNX/TensorRT) -> postprocessor (Python backend). The input_map and output_map of each step define the tensor flow, and the intermediate tensor names (tokenized_ids, raw_logits and so on) connect the data between steps.

Pre/Post-Processing with the Python Backend

Here is an example of a Python backend model handling preprocessing and post-processing in an Ensemble.

# preprocessing/tokenizer/1/model.py
import triton_python_backend_utils as pb_utils
import numpy as np
from transformers import AutoTokenizer
import json


class TritonPythonModel:
    def initialize(self, args):
        """Called 1 time when the model is loaded. Initializes the tokenizer."""
        self.model_config = json.loads(args["model_config"])
        self.tokenizer = AutoTokenizer.from_pretrained(
            "bert-base-uncased",
            cache_dir="/models/cache"
        )
        self.max_length = 512

    def execute(self, requests):
        """Processes a batch of inference requests."""
        responses = []
        for request in requests:
            # Extract the input text
            text_input = pb_utils.get_input_tensor_by_name(
                request, "TEXT_INPUT"
            )
            texts = [
                t.decode("utf-8")
                for t in text_input.as_numpy().flatten()
            ]

            # Run tokenization
            encoded = self.tokenizer(
                texts,
                padding="max_length",
                truncation=True,
                max_length=self.max_length,
                return_tensors="np"
            )

            # Create the output tensor
            input_ids_tensor = pb_utils.Tensor(
                "INPUT_IDS",
                encoded["input_ids"].astype(np.int64)
            )
            attention_mask_tensor = pb_utils.Tensor(
                "ATTENTION_MASK",
                encoded["attention_mask"].astype(np.int64)
            )

            response = pb_utils.InferenceResponse(
                output_tensors=[input_ids_tensor, attention_mask_tensor]
            )
            responses.append(response)

        return responses

    def finalize(self):
        """Cleanup work when the model is unloaded."""
        print("Tokenizer model finalized.")

The Python backend's execute method is called per batch. Each request is one request grouped by Dynamic Batching, and you must return the same number of responses.

TensorRT Integration and Model Optimization

Converting to a TensorRT Engine

TensorRT is the deep learning optimization engine that delivers the highest inference performance on NVIDIA GPUs. Converting an ONNX model into a TensorRT engine can yield up to a 2x throughput gain at FP16 and up to 4x at INT8, compared with FP32.

# Convert the ONNX model into a TensorRT engine
# Note: convert in a container with the same TensorRT version as the Triton server
docker run --gpus all --rm \
  -v $(pwd):/workspace \
  nvcr.io/nvidia/tensorrt:25.02-py3 \
  trtexec \
    --onnx=/workspace/model.onnx \
    --saveEngine=/workspace/model.plan \
    --fp16 \
    --workspace=4096 \
    --minShapes=input_ids:1x512,attention_mask:1x512 \
    --optShapes=input_ids:16x512,attention_mask:16x512 \
    --maxShapes=input_ids:64x512,attention_mask:64x512 \
    --verbose

# Check the Dynamic Shape profile
# minShapes: minimum batch size (a single request)
# optShapes: optimal batch size (the most frequent case)
# maxShapes: maximum batch size (match this to the Dynamic Batching maximum)

The most common mistake when converting a TensorRT engine is a version mismatch. A TensorRT engine only works on a runtime that matches exactly the TensorRT library version used to convert it. If you load an engine converted with a trtexec whose TensorRT version differs from the one bundled in Triton container 25.02, you get an UNAVAILABLE: Internal: unable to create TensorRT engine error. You must use the same NGC container tag.

TensorRT Model Configuration

# image_detector/config.pbtxt
name: "image_detector"
platform: "tensorrt_plan"
max_batch_size: 32

input [
  {
    name: "images"
    data_type: TYPE_FP32
    dims: [ 3, 640, 640 ]
  }
]

output [
  {
    name: "detections"
    data_type: TYPE_FP32
    dims: [ 100, 6 ]
  }
]

dynamic_batching {
  preferred_batch_size: [ 4, 8, 16 ]
  max_queue_delay_microseconds: 100
}

instance_group [
  {
    count: 1
    kind: KIND_GPU
    gpus: [ 0 ]
  }
]

# TensorRT-specific optimization parameters
parameters {
  key: "TRT_ENGINE_CACHE_ENABLE"
  value: { string_value: "1" }
}
parameters {
  key: "TRT_ENGINE_CACHE_PATH"
  value: { string_value: "/models/cache/trt" }
}

Using TensorRT Acceleration from ONNX Runtime

Without explicitly converting an ONNX model into a TensorRT engine, you can apply TensorRT acceleration at runtime through the ONNX Runtime backend's TensorRT Execution Provider. The optimization.execution_accelerators block shown earlier in the text_classifier configuration is exactly this feature. Building the TensorRT engine takes time on the initial load, but enabling the cache skips the build on subsequent restarts.

GPU Memory Management and Multi-Model Serving

Instance Groups and GPU Allocation

The heart of GPU memory management in Triton is the instance_group setting. You can serve several models on one GPU at the same time, or run one model as several instances to raise throughput.

# Multi-GPU distribution configuration
instance_group [
  {
    # 2 instances on GPU 0
    count: 2
    kind: KIND_GPU
    gpus: [ 0 ]
  },
  {
    # 1 instance on GPU 1
    count: 1
    kind: KIND_GPU
    gpus: [ 1 ]
  },
  {
    # CPU fallback instance (for GPU failure)
    count: 1
    kind: KIND_CPU
  }
]

Calculating the GPU Memory Budget

When placing several models on one GPU in production, you have to work out the memory budget in advance.

# GPU memory profiling with Model Analyzer
# Measure how much GPU memory each model uses
model-analyzer profile \
  --model-repository=/models \
  --profile-models text_classifier,image_detector,embedding_model \
  --triton-launch-mode=docker \
  --triton-docker-image=nvcr.io/nvidia/tritonserver:25.02-py3 \
  --output-model-repository-path=/output/models \
  --export-path=/output/results \
  --run-config-search-max-concurrency 16 \
  --run-config-search-max-instance-count 4

# Example results (on an A100 80GB):
# text_classifier:  ~2.1GB per instance, optimal instance count: 4
# image_detector:   ~4.8GB per instance, optimal instance count: 2
# embedding_model:  ~1.5GB per instance, optimal instance count: 6
# Total usage: 2.1*4 + 4.8*2 + 1.5*6 = 8.4 + 9.6 + 9.0 = 27.0GB
# Free memory on an A100 80GB: 53GB (allowing for CUDA context and framework overhead)

Isolation with MIG (Multi-Instance GPU)

On A100, A30, and H100 GPUs, MIG lets you split one physical GPU into up to 7 independent GPU instances. Each instance has isolated memory, SMs (Streaming Multiprocessors), and L2 cache, so serving stays stable with no interference between models.

# Enable MIG on the A100 and create instances
sudo nvidia-smi -i 0 -mig 1

# Create 2 instances of the 3g.40gb profile (on an A100 80GB)
sudo nvidia-smi mig -i 0 -cgi 9,9 -C

# Check the MIG instances
nvidia-smi -L
# GPU 0: NVIDIA A100-SXM4-80GB
#   MIG 3g.40gb Device 0: (UUID: MIG-xxx-xxx)
#   MIG 3g.40gb Device 1: (UUID: MIG-xxx-xxx)

# Assign models per MIG instance in Triton
# Expose only a specific MIG instance when running Docker
docker run --gpus '"device=0:0"' \
  -v $(pwd)/models_group_a:/models \
  nvcr.io/nvidia/tritonserver:25.02-py3 \
  tritonserver --model-repository=/models

Model Loading Strategy

Triton offers three model loading modes.

# Start the server in EXPLICIT mode (manual model management)
tritonserver \
  --model-repository=/models \
  --model-control-mode=explicit

# Load a model through the Model Control API
curl -X POST localhost:8000/v2/repository/models/text_classifier/load

# Unload a model (frees GPU memory)
curl -X POST localhost:8000/v2/repository/models/text_classifier/unload

# List the currently loaded models
curl localhost:8000/v2/repository/index | python -m json.tool

EXPLICIT mode is useful for managing models dynamically in an environment with limited GPU memory. You can implement a strategy that unloads models used infrequently and loads the ones you need, according to the request pattern.

Client Code and Inference Requests

The Python Client (tritonclient)

Triton provides an official Python client library, tritonclient. It supports both the HTTP and gRPC protocols, and gRPC is recommended in production. gRPC uses binary serialization, so it carries less data transfer overhead than HTTP, and it supports streaming and bidirectional communication.

# Triton gRPC client example
import tritonclient.grpc as grpcclient
import numpy as np
from functools import partial
import queue


def run_inference():
    """Synchronous gRPC inference request"""
    # Create the gRPC client
    triton_client = grpcclient.InferenceServerClient(
        url="localhost:8001",
        verbose=False
    )

    # Check the server status
    if not triton_client.is_server_ready():
        raise RuntimeError("Triton server is not ready")

    # Fetch the model metadata
    metadata = triton_client.get_model_metadata("text_classifier")
    print(f"Model: {metadata.name}, Versions: {metadata.versions}")

    # Prepare the input data
    input_ids = np.random.randint(0, 30000, size=(1, 512)).astype(np.int64)
    attention_mask = np.ones((1, 512), dtype=np.int64)

    # Create the input tensor
    inputs = [
        grpcclient.InferInput("input_ids", input_ids.shape, "INT64"),
        grpcclient.InferInput("attention_mask", attention_mask.shape, "INT64"),
    ]
    inputs[0].set_data_from_numpy(input_ids)
    inputs[1].set_data_from_numpy(attention_mask)

    # Specify the output tensors
    outputs = [
        grpcclient.InferRequestedOutput("logits"),
    ]

    # Run inference (synchronous)
    result = triton_client.infer(
        model_name="text_classifier",
        inputs=inputs,
        outputs=outputs,
        client_timeout=5.0,  # 5 second timeout
        headers={"request-id": "req-001"}
    )

    # Extract the result
    logits = result.as_numpy("logits")
    print(f"Logits shape: {logits.shape}")
    print(f"Predictions: {np.argmax(logits, axis=-1)}")

    return logits


def run_async_inference():
    """Asynchronous gRPC inference request (high-throughput scenario)"""
    triton_client = grpcclient.InferenceServerClient(
        url="localhost:8001"
    )

    result_queue = queue.Queue()

    def callback(result, error):
        if error:
            result_queue.put(error)
        else:
            result_queue.put(result.as_numpy("logits"))

    # Send 100 requests asynchronously
    num_requests = 100
    for i in range(num_requests):
        input_ids = np.random.randint(0, 30000, size=(1, 512)).astype(np.int64)
        attention_mask = np.ones((1, 512), dtype=np.int64)

        inputs = [
            grpcclient.InferInput("input_ids", input_ids.shape, "INT64"),
            grpcclient.InferInput("attention_mask", attention_mask.shape, "INT64"),
        ]
        inputs[0].set_data_from_numpy(input_ids)
        inputs[1].set_data_from_numpy(attention_mask)

        outputs = [grpcclient.InferRequestedOutput("logits")]

        triton_client.async_infer(
            model_name="text_classifier",
            inputs=inputs,
            outputs=outputs,
            callback=partial(callback),
            request_id=f"async-req-{i}"
        )

    # Collect all the results
    results = []
    for _ in range(num_requests):
        res = result_queue.get()
        if isinstance(res, Exception):
            print(f"Error: {res}")
        else:
            results.append(res)

    print(f"Completed {len(results)}/{num_requests} requests")
    return results


if __name__ == "__main__":
    run_inference()
    run_async_inference()

Zero-Copy Transfer with Shared Memory

When sending large tensors (images, video frames and the like), using CUDA Shared Memory or System Shared Memory removes the data copy overhead.

import tritonclient.grpc as grpcclient
from tritonclient import utils
import tritonclient.utils.cuda_shared_memory as cudashm
import numpy as np


def inference_with_cuda_shared_memory():
    """Zero-copy inference using CUDA Shared Memory"""
    triton_client = grpcclient.InferenceServerClient(url="localhost:8001")

    # Input data
    image_data = np.random.rand(1, 3, 640, 640).astype(np.float32)
    input_byte_size = image_data.nbytes

    # Create the CUDA Shared Memory region
    shm_handle = cudashm.create_shared_memory_region(
        "input_images_shm", input_byte_size, 0  # GPU device 0
    )

    # Copy the data into CUDA Shared Memory
    cudashm.set_shared_memory_region(shm_handle, [image_data])

    # Register the Shared Memory region with the Triton server
    triton_client.register_cuda_shared_memory(
        "input_images_shm",
        cudashm.get_raw_handle(shm_handle),
        0,  # GPU device 0
        input_byte_size
    )

    # Point the input tensor at Shared Memory
    inputs = [grpcclient.InferInput("images", [1, 3, 640, 640], "FP32")]
    inputs[0].set_shared_memory("input_images_shm", input_byte_size)

    # Receive the output through CUDA Shared Memory too
    output_byte_size = 100 * 6 * 4  # 100 detections * 6 values * float32
    out_shm_handle = cudashm.create_shared_memory_region(
        "output_detections_shm", output_byte_size, 0
    )
    triton_client.register_cuda_shared_memory(
        "output_detections_shm",
        cudashm.get_raw_handle(out_shm_handle),
        0,
        output_byte_size
    )

    outputs = [grpcclient.InferRequestedOutput("detections")]
    outputs[0].set_shared_memory("output_detections_shm", output_byte_size)

    # Run inference (the data is passed straight from GPU memory)
    result = triton_client.infer(
        model_name="image_detector",
        inputs=inputs,
        outputs=outputs
    )

    # Read the result from Shared Memory
    detections = cudashm.get_contents_as_numpy(
        out_shm_handle,
        utils.triton_to_np_dtype("FP32"),
        [100, 6]
    )

    # Cleanup
    triton_client.unregister_cuda_shared_memory("input_images_shm")
    triton_client.unregister_cuda_shared_memory("output_detections_shm")
    cudashm.destroy_shared_memory_region(shm_handle)
    cudashm.destroy_shared_memory_region(out_shm_handle)

    return detections

Performance Profiling and Monitoring

Benchmarking with perf_analyzer

perf_analyzer is Triton's dedicated performance measurement tool. It generates synthetic load and measures a model's throughput and latency systematically.

# Basic performance measurement (concurrency based)
perf_analyzer \
  -m text_classifier \
  -u localhost:8001 \
  -i grpc \
  --concurrency-range 1:64:8 \
  --measurement-interval 10000 \
  --percentile=95 \
  --stability-percentage 10 \
  -v

# Request-rate-based measurement (simulating a real traffic pattern)
perf_analyzer \
  -m text_classifier \
  -u localhost:8001 \
  -i grpc \
  --request-rate-range 100:1000:100 \
  --measurement-interval 15000 \
  --percentile=99

# Performance measurement using real data
perf_analyzer \
  -m text_classifier \
  -u localhost:8001 \
  -i grpc \
  --input-data real_data.json \
  --concurrency-range 1:32 \
  --measurement-interval 10000

Collecting Prometheus Metrics

Triton exposes Prometheus-format metrics on port 8002. Let us look at the metrics that are essential for production monitoring.

# Check the Triton metrics endpoint
curl localhost:8002/metrics

# Key metrics:
# nv_inference_request_success     - number of successful inference requests
# nv_inference_request_failure     - number of failed inference requests
# nv_inference_count               - total inference executions
# nv_inference_exec_count          - number of batch executions
# nv_inference_request_duration_us - request processing time (microseconds)
# nv_inference_queue_duration_us   - queue wait time (microseconds)
# nv_inference_compute_infer_duration_us - actual inference time
# nv_gpu_utilization               - GPU utilization
# nv_gpu_memory_used_bytes         - GPU memory usage
# nv_gpu_power_usage               - GPU power consumption

# prometheus/triton-alerts.yaml
# Essential alert rules for running Triton
groups:
  - name: triton-inference-alerts
    rules:
      # Warn when the inference failure rate exceeds 1%
      - alert: TritonHighFailureRate
        expr: |
          rate(nv_inference_request_failure[5m])
          / (rate(nv_inference_request_success[5m]) + rate(nv_inference_request_failure[5m]))
          > 0.01
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: 'Triton inference failure rate above 1%'
          description: 'Model {{ $labels.model }} failure rate: {{ $value | humanizePercentage }}'

      # p99 latency exceeds the SLA
      - alert: TritonHighLatency
        expr: |
          histogram_quantile(0.99,
            rate(nv_inference_request_duration_us_bucket[5m])
          ) > 50000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Triton p99 latency above 50ms'

      # GPU memory utilization above 90%
      - alert: TritonGPUMemoryHigh
        expr: |
          nv_gpu_memory_used_bytes / nv_gpu_memory_total_bytes > 0.9
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: 'GPU memory utilization above 90% - OOM risk'

      # Queue wait time spiking (batching bottleneck)
      - alert: TritonQueueBacklog
        expr: |
          rate(nv_inference_queue_duration_us_sum[5m])
          / rate(nv_inference_queue_duration_us_count[5m])
          > 10000
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: 'Average inference queue wait above 10ms'

Kubernetes Deployment

Deploying with a Helm Chart

# triton-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: triton-inference-server
  namespace: ml-serving
  labels:
    app: triton
spec:
  replicas: 3
  selector:
    matchLabels:
      app: triton
  template:
    metadata:
      labels:
        app: triton
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8002'
        prometheus.io/path: '/metrics'
    spec:
      containers:
        - name: triton
          image: nvcr.io/nvidia/tritonserver:25.02-py3
          command: ['tritonserver']
          args:
            - '--model-repository=s3://ml-models/triton-repo'
            - '--model-control-mode=poll'
            - '--repository-poll-secs=30'
            - '--strict-model-config=false'
            - '--log-verbose=0'
            - '--exit-on-error=false'
            - '--rate-limiter=execution_count'
            - '--rate-limiter-resource=R1:4:0'
          ports:
            - containerPort: 8000
              name: http
            - containerPort: 8001
              name: grpc
            - containerPort: 8002
              name: metrics
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: '32Gi'
              cpu: '8'
            requests:
              nvidia.com/gpu: 1
              memory: '16Gi'
              cpu: '4'
          readinessProbe:
            httpGet:
              path: /v2/health/ready
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /v2/health/live
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 15
            failureThreshold: 5
          volumeMounts:
            - name: model-cache
              mountPath: /models/cache
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: model-cache
          emptyDir:
            sizeLimit: 50Gi
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 8Gi
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-tesla-a100
---
apiVersion: v1
kind: Service
metadata:
  name: triton-inference-svc
  namespace: ml-serving
spec:
  type: ClusterIP
  ports:
    - port: 8000
      targetPort: 8000
      name: http
    - port: 8001
      targetPort: 8001
      name: grpc
    - port: 8002
      targetPort: 8002
      name: metrics
  selector:
    app: triton
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: triton-hpa
  namespace: ml-serving
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: triton-inference-server
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Pods
      pods:
        metric:
          name: nv_inference_queue_duration_us
        target:
          type: AverageValue
          averageValue: '5000'
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120

Mounting the /dev/shm volume is mandatory for a Kubernetes deployment. The Python backend needs shared memory when it uses multiprocessing, and Docker's default 64MB limit is not enough. Mount a tmpfs with medium: Memory and allocate enough space (at least 2Gi).

When loading models directly from S3 or GCS, using model-control-mode=poll means new model versions are detected and loaded automatically on update, with no server restart. When a CI/CD pipeline uploads a model to S3, the serving model is swapped automatically within 30 seconds, giving you a zero-downtime deployment.

Comparing Model Serving Frameworks

Before adopting Triton, you should be clear about how it differs from the other major model serving solutions on the market.

Comparison itemTriton Inference ServervLLMTGI (Text Generation Inference)BentoML
Main useGeneral-purpose multi-framework servingLLM-specific inferenceLLM text generationML model packaging/serving
Frameworks supportedTensorRT, ONNX, PyTorch, TF, Python and more (8+)PyTorch (Transformer family)PyTorch (HF models)PyTorch, TF, ONNX, XGBoost, others
Core optimizationsDynamic Batching, Model Ensemble, Concurrent ExecutionPagedAttention, Continuous BatchingContinuous Batching, Flash AttentionAdaptive Batching, Runner
GPU memory managementInstance Group, MIG support, Rate LimiterPagedAttention removes memory fragmentationBasic CUDA memory managementUses framework defaults
Multi-model servingNative support (dozens to hundreds of models)Optimized for a single modelOptimized for a single modelModel management per Service
ProtocolsHTTP, gRPC, C APIOpenAI API compatibleOpenAI API compatible, gRPCHTTP REST, gRPC
Kubernetes integrationNative (Helm, KServe)Deployment configured directlyDeployment configured directlyBentoCloud, Yatai
MonitoringPrometheus nativeBasic metricsPrometheus supportedPrometheus, OpenTelemetry
LLM inference performanceNeeds the TensorRT-LLM backendBest (PagedAttention)Strong (Flash Attention)Depends on the backend
Configuration effortHigh (config.pbtxt required)Low (CLI flags)Low (environment variables)Medium (bentofile.yaml)
Learning curveSteepGentleGentleMedium
LicenseBSD 3-ClauseApache 2.0Apache 2.0Apache 2.0

When You Should Choose Triton

When You Should Choose Something Else

Production Troubleshooting

Failure Cases and Recovery Strategies

Case 1: GPU OOM (Out of Memory) Error

Symptom: inference failures on a specific model along with a CUDA error: out of memory error. Failed to allocate messages repeat in the server log.

Cause analysis: the count in instance_group was set beyond the GPU's memory capacity, or Dynamic Batching's max_batch_size was so large that intermediate tensors exceeded memory during batch processing.

Recovery procedure:

# 1. Check the current GPU memory state
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

# 2. Profile memory usage per model
model-analyzer profile \
  --model-repository=/models \
  --profile-models problematic_model \
  --run-config-search-max-instance-count 1

# 3. Edit config.pbtxt: lower the instance count, reduce max_batch_size
# instance_group count: 4 -> 2
# max_batch_size: 64 -> 32

# 4. Reload the model (without restarting the server)
curl -X POST localhost:8000/v2/repository/models/problematic_model/unload
curl -X POST localhost:8000/v2/repository/models/problematic_model/load

Case 2: TensorRT Engine Load Failure

Symptom: an UNAVAILABLE: Internal: unable to create TensorRT engine error at server startup.

Cause analysis: the TensorRT library version used to convert the engine does not match the TensorRT runtime version in the Triton container. Or the target GPU architecture used at engine build time (sm_80, sm_86 and so on) differs from the architecture of the GPU actually serving.

Recovery procedure:

# 1. Check the TensorRT version in the Triton container
docker run --rm nvcr.io/nvidia/tritonserver:25.02-py3 \
  dpkg -l | grep tensorrt

# 2. Rebuild the engine in a TensorRT container of the same version
docker run --gpus all --rm \
  -v $(pwd):/workspace \
  nvcr.io/nvidia/tritonserver:25.02-py3 \
  trtexec \
    --onnx=/workspace/model.onnx \
    --saveEngine=/workspace/model.plan \
    --fp16

# 3. Check the GPU architecture
nvidia-smi --query-gpu=gpu_name,compute_cap --format=csv
# A100: compute_cap 8.0 (sm_80)
# A10G: compute_cap 8.6 (sm_86)
# H100: compute_cap 9.0 (sm_90)

Case 3: Dynamic Batching Queue Timeout

Symptom: Request timeout expired errors surge under heavy traffic. Deadline Exceeded errors on the client.

Cause analysis: the Dynamic Batching queue is full and new requests are being refused. Requests are arriving faster than the model instances can process them.

Recovery procedure:

  1. Raise max_queue_size temporarily to stop requests being dropped.
  2. Raise the count in instance_group to secure more processing capacity.
  3. Scale out the number of Triton Pods through HPA.
  4. Fundamentally, convert the model to TensorRT to shorten the per-inference time.

Case 4: Python Backend Memory Leak

Symptom: host memory (RAM) usage climbs steadily the longer the server runs, and the Pod eventually restarts as OOMKilled.

Cause analysis: the Python backend's execute method creates objects per request but never releases them. Or results accumulate in a global list.

Recovery procedure:

# Wrong code (memory leak)
class TritonPythonModel:
    def initialize(self, args):
        self.results_cache = []  # grows without limit

    def execute(self, requests):
        for req in requests:
            result = process(req)
            self.results_cache.append(result)  # never cleaned up
        return responses

# Correct code (memory managed)
class TritonPythonModel:
    def initialize(self, args):
        self.model = load_model()  # loaded only 1 time, at initialization

    def execute(self, requests):
        responses = []
        for req in requests:
            result = self.model.predict(req)
            responses.append(build_response(result))
            # local variables are released automatically when the method returns
        return responses

Performance Debugging Checklist

When a performance problem arises, check the following items in order.

# 1. Check GPU utilization (if low, raise the batch size or instance count)
nvidia-smi dmon -s u -d 1

# 2. Check the queue wait time in the Triton metrics
curl -s localhost:8002/metrics | grep queue_duration

# 3. Check the inference time per model
curl -s localhost:8002/metrics | grep compute_infer_duration

# 4. Check batch efficiency (the ratio of inference_count to exec_count)
# inference_count / exec_count = average batch size
curl -s localhost:8002/metrics | grep -E "(nv_inference_count|nv_inference_exec_count)"

Operational Cautions and Checklists

Pre-Production Deployment Checklist

Model preparation:

Server configuration:

Infrastructure:

Monitoring:

Common Operational Mistakes

  1. Deploying without config.pbtxt: relying on auto-configuration in strict-model-config=false mode can serve the model with input/output shapes you did not expect, producing runtime errors. In production you must write an explicit config.pbtxt.

  2. Not cleaning up Shared Memory: if you do not call unregister/destroy after using CUDA Shared Memory, GPU memory leaks gradually. Always clean up with a try/finally pattern.

  3. Losing requests during a model hot reload: in POLL mode, requests to a model can fail during the few seconds it is reloading. Defend against this with client-side retry logic and a multi-Pod setup behind the load balancer.

  4. Not setting TensorRT Dynamic Shape: if you do not set minShapes/optShapes/maxShapes when building the TensorRT engine, it can only handle a fixed shape. To use it together with Dynamic Batching you must configure a Dynamic Shape profile.

  5. Initial delay from ONNX Runtime TRT acceleration: the first time you use ONNX Runtime's TensorRT Execution Provider, the TRT engine is built at runtime, so the first request can take several minutes. Store the engine cache on a persistent volume so the build is skipped on restart.

Advanced Optimization Strategies

Managing Resources with the Rate Limiter

When several models share one GPU, the Rate Limiter lets you control contention over GPU resources between them.

# Enable the Rate Limiter (at server startup)
tritonserver \
  --model-repository=/models \
  --rate-limiter=execution_count \
  --rate-limiter-resource=GPU_EXEC_SLOTS:8:0
  # Maximum concurrent slots on GPU 0: 8

# Configure heavy models to consume more slots
# heavy_model/config.pbtxt
name: "heavy_model"
rate_limiter {
  resources [
    {
      name: "GPU_EXEC_SLOTS"
      count: 4  # this model occupies 4 of the 8 slots
    }
  ]
}

# Light models consume fewer slots
# light_model/config.pbtxt
name: "light_model"
rate_limiter {
  resources [
    {
      name: "GPU_EXEC_SLOTS"
      count: 1  # occupies just 1 slot
    }
  ]
}

With this configuration, when 2 heavy_model instances run at once (4+4=8 slots) the other models wait, which prevents GPU memory overruns and performance degradation.

Response Cache

When many inference requests repeat over the same input, enabling the Response Cache lets you return the cached result without any GPU computation.

# Enable the Response Cache (64MB cache)
tritonserver \
  --model-repository=/models \
  --response-cache-byte-size=67108864

# Apply the cache to specific models only
# embedding_model/config.pbtxt
name: "embedding_model"
response_cache { enable: true }

A high cache hit rate cuts GPU load substantially, but the cache memory comes out of host RAM, so it needs to be sized sensibly. It has no effect on models whose input differs every time (real-time sensor data, for example).

Conclusion

NVIDIA Triton Inference Server is the open-source inference server offering the most comprehensive feature set for production GPU model serving. Dynamic Batching maximizes GPU utilization, Model Ensemble completes preprocessing, inference, and post-processing inside the server, and TensorRT integration reaches top-tier inference performance. It has everything enterprise operations needs: multi-model serving, version management, Prometheus metrics, and Kubernetes-native deployment.

That said, Triton's learning curve is steep. config.pbtxt-based model configuration, TensorRT engine builds and version management, checking compatibility across multiple backends, and managing the GPU memory budget all demand deep understanding from the operations team. If you serve only LLMs, vLLM may be the more efficient choice; if fast prototyping is the goal, BentoML or TGI may be.

What matters is not the tool itself but making the right choice for the characteristics of your serving workload. If you need to run multi-framework models efficiently on a single GPU cluster with enterprise-grade stability and observability, Triton Inference Server is the most mature choice available today.

References

Read the original: https://www.youngju.dev/blog/ai-platform/2026-03-08-ai-platform-triton-inference-server-gpu-serving-optimization.en

Comments

No comments yet.

Sign in to leave a comment