LLM Inference Serving Framework Comparison: TensorRT-LLM vs vLLM vs SGLang Production Deployment Strategy

• Introduction
• The Core Challenges of LLM Inference Serving
  • The KV Cache Memory Problem
  • The Evolution of Batching Strategies
• TensorRT-LLM in Depth
  • NVIDIA-Native Hardware Optimization
  • Building and Serving a TensorRT-LLM Model
  • Speculative Decoding in TensorRT-LLM
• vLLM in Depth
  • The PagedAttention Memory Management Architecture
  • The vLLM V1 Engine and Production Stack
  • Deploying vLLM to Kubernetes in Production
• SGLang in Depth
  • RadixAttention: A Breakthrough in KV Cache Reuse
  • Compressed Finite State Machine (Structured Generation)
  • Structured Output Generation in SGLang
• Benchmark Comparison of the 3 Frameworks
  • Feature Comparison Table
  • Throughput Benchmark on H100 (Llama-3.1-70B, TP=4)
  • Throughput by Model Size (H100 80GB, 64 concurrent requests, FP16)
  • Performance Across Concurrency Levels (Llama-3.1-70B, TP=4)
• Production Deployment Architecture
  • GPU Node Scheduling Strategy
  • Autoscaling and Monitoring
• Failure Cases and Troubleshooting
  • Case 1: A Serving Outage from OOM, and Memory Management
  • Case 2: Debugging an Accuracy Drop in a Quantized Model
  • Recovery Procedure Checklist
• Operational Caveats and a Selection Guide
  • Recommendation Matrix by Use Case
  • Cost Optimization Strategies
  • Vendor Lock-in Considerations
• Conclusion
• References

Introduction

Training an LLM and serving it in production are entirely different engineering problems. Training puts high throughput first, while serving has to hit three conflicting goals at once: throughput, latency, and memory efficiency. In user-facing services such as real-time chatbots and coding assistants in particular, once TTFT (Time To First Token) climbs past a few hundred milliseconds the user experience degrades sharply.

Between 2024 and 2026, three frameworks matured to production grade in response to this problem. TensorRT-LLM (NVIDIA) is strongest in the depth of its hardware optimization, vLLM (UC Berkeley) in memory efficiency and the breadth of its ecosystem, and SGLang (LMSYS) in KV Cache reuse and structured-generation performance.

This article digs into the internal architecture of each framework, compares them with benchmark data measured on H100, and then covers production deployment code and operational strategy.

The Core Challenges of LLM Inference Serving

LLM inference splits broadly into a Prefill stage (processing the whole prompt in one pass) and a Decode stage (generating tokens one at a time, autoregressively). Prefill is compute-bound and Decode is memory-bound, so the optimization strategy for the two stages is fundamentally different.

The KV Cache Memory Problem

Each layer of the Transformer decoder caches the Key/Value vectors of the preceding tokens. When Llama-3-70B is served in FP16, the KV Cache for a single request grows in proportion to sequence length and consumes roughly 2.5GB at 4096 tokens. Handling 32 concurrent requests therefore needs 80GB for the KV Cache alone.

The Evolution of Batching Strategies

With traditional Static Batching the GPU waits on the longest request until every request in the batch has finished, so GPU time is badly wasted on the short requests. Continuous Batching (vLLM, SGLang) and In-flight Batching (TensorRT-LLM) emerged to solve this. The core idea is the same in both: each request leaves the moment it completes, and a new request joins the batch immediately.

# Simulating the throughput difference between Static Batching and Continuous Batching
import numpy as np

def simulate_static_batching(requests, batch_size=8):
    """Static Batching: wait until the longest request in the batch finishes"""
    total_time = 0
    for i in range(0, len(requests), batch_size):
        batch = requests[i:i + batch_size]
        max_tokens = max(r["output_tokens"] for r in batch)
        total_time += max_tokens * 0.01  # assume 10ms per token
    return total_time

def simulate_continuous_batching(requests, batch_size=8):
    """Continuous Batching: insert a new request into a freed slot right away"""
    total_time = 0
    active_slots = []
    queue = list(requests)

    while queue or active_slots:
        # Insert new requests into the empty slots
        while len(active_slots) < batch_size and queue:
            active_slots.append(queue.pop(0))

        # Advance 1 step
        total_time += 0.01
        for slot in active_slots:
            slot["remaining"] = slot.get("remaining", slot["output_tokens"]) - 1

        # Remove completed requests
        active_slots = [s for s in active_slots if s["remaining"] > 0]

    return total_time

# 100 requests, output length random between 10 and 500 tokens
requests = [{"output_tokens": np.random.randint(10, 500)} for _ in range(100)]
static_time = simulate_static_batching(requests)
continuous_time = simulate_continuous_batching(
    [dict(r) for r in requests]
)

print(f"Static Batching total time: {static_time:.1f}s")
print(f"Continuous Batching total time: {continuous_time:.1f}s")
print(f"Throughput gain: {static_time / continuous_time:.1f}x")
# Example output:
# Static Batching total time: 62.5s
# Continuous Batching total time: 27.3s
# Throughput gain: 2.3x

TensorRT-LLM in Depth

NVIDIA-Native Hardware Optimization

TensorRT-LLM is the LLM inference engine NVIDIA developed for its own GPUs. It delivers a throughput gain of up to 8x or more over ordinary PyTorch inference, and on H100/H200/B200 GPUs in particular it makes full use of the FP8 Tensor Cores.

Key optimization techniques:

• Kernel Fusion: fuses Multi-Head Attention, LayerNorm, GELU and the like into a single CUDA kernel, removing memory-access overhead
• FP8/FP4 quantization: uses the FP8 Tensor Cores on H100 to reach 2x the throughput of FP16 while minimizing accuracy loss
• In-flight Batching: mixes Prefill and Decode work within a batch to maximize GPU utilization
• Paged KV Cache: inspired by vLLM, manages the KV Cache in non-contiguous memory blocks
• Tensor Parallelism / Pipeline Parallelism: shards the model automatically across a multi-GPU setup

Building and Serving a TensorRT-LLM Model

TensorRT-LLM follows a 2-step process: convert (build) the model into a TRT engine first, then serve it.

# 1. Install TensorRT-LLM (Docker recommended)
docker pull nvcr.io/nvidia/tritonserver:24.12-trtllm-python-py3

# 2. Convert the Hugging Face model into a TRT engine
# Llama-3-70B, FP8 quantization, Tensor Parallelism 4-way
python convert_checkpoint.py \
    --model_dir /models/Llama-3-70B \
    --output_dir /engines/llama-70b-ckpt \
    --dtype float16 \
    --tp_size 4

trtllm-build \
    --checkpoint_dir /engines/llama-70b-ckpt \
    --output_dir /engines/llama-70b-engine \
    --gemm_plugin float16 \
    --gpt_attention_plugin float16 \
    --max_batch_size 64 \
    --max_input_len 4096 \
    --max_seq_len 8192 \
    --use_paged_context_fmha enable \
    --use_fp8_context_fmha enable \
    --workers 4

# 3. Serve with Triton Inference Server
tritonserver \
    --model-repository=/engines/triton-repo \
    --http-port=8000 \
    --grpc-port=8001 \
    --metrics-port=8002

# Direct inference through the TensorRT-LLM Python API
import tensorrt_llm
from tensorrt_llm import LLM, SamplingParams

# Load the built engine
llm = LLM(
    model="/engines/llama-70b-engine",
    tensor_parallel_size=4,
    dtype="float16",
    kv_cache_config={
        "enable_block_reuse": True,
        "free_gpu_memory_fraction": 0.9,
    },
)

# Batch inference
prompts = [
    "Explain the concept of attention mechanism in transformers",
    "Write a Python function to implement binary search",
    "What are the key differences between TCP and UDP?",
]

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

outputs = llm.generate(prompts, sampling_params=sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt[:50]}...")
    print(f"Generated: {output.outputs[0].text[:100]}...")
    print(f"Tokens/sec: {output.outputs[0].token_ids.__len__() / output.metrics.generation_time:.1f}")
    print()

Speculative Decoding in TensorRT-LLM

TensorRT-LLM supports Speculative Decoding natively. A draft model generates several tokens quickly and the target model verifies them in one pass, improving decoding speed by 1.5-2.5x while preserving output quality.

# TensorRT-LLM Speculative Decoding configuration
from tensorrt_llm import LLM, SamplingParams

llm = LLM(
    model="/engines/llama-70b-engine",
    speculative_model="/engines/llama-8b-draft-engine",
    speculative_config={
        "num_draft_tokens": 5,
        "acceptance_method": "typical_acceptance",
    },
    tensor_parallel_size=4,
)

# Speculative Decoding is applied transparently
params = SamplingParams(temperature=0.0, max_tokens=1024)
output = llm.generate(["Explain quantum computing"], params)
# Internally the draft model generates 5 tokens at a time and the target model verifies them

vLLM in Depth

The PagedAttention Memory Management Architecture

vLLM is an inference engine published in 2023 by a research team at UC Berkeley, and it introduced an innovative KV Cache management technique called PagedAttention. It was designed with inspiration from the virtual-memory paging system of an operating system.

The conventional approach pre-allocates contiguous memory for the maximum sequence length of each request. When the actual generation is shorter, the remaining space is wasted — on average 60-80% of the KV Cache memory goes to waste.

PagedAttention splits the KV Cache into fixed-size blocks (16 tokens by default) and allocates a new block only when one is needed. A block table maps logical block numbers to physical block addresses, so contiguous memory is not required. This raises KV Cache memory utilization to 95% or higher.

The vLLM V1 Engine and Production Stack

vLLM introduced the V1 engine in late 2025, substantially reworking the architecture. The main changes:

• torch.compile integration: optimizes the model forward pass with the PyTorch 2 compiler
• Multi-process GPU execution: each GPU runs in a separate process, removing the GIL bottleneck
• Simplified scheduler: a single code path that unifies Prefix Caching, Chunked Prefill, and Speculative Decoding

# Running the vLLM server and calling the OpenAI-compatible API
# 1. Start the server
# vllm serve meta-llama/Llama-3.1-70B-Instruct \
#     --tensor-parallel-size 4 \
#     --max-model-len 8192 \
#     --gpu-memory-utilization 0.92 \
#     --enable-prefix-caching \
#     --enable-chunked-prefill \
#     --max-num-seqs 256 \
#     --port 8000

# 2. Call the OpenAI-compatible API from a Python client
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)

# Chat Completions API
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain how PagedAttention works in vLLM."},
    ],
    temperature=0.7,
    max_tokens=512,
    stream=True,
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

# 3. Batch inference (offline)
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    max_model_len=8192,
    gpu_memory_utilization=0.92,
    enable_prefix_caching=True,
)

sampling = SamplingParams(temperature=0.0, max_tokens=256)
prompts = [f"Question {i}: What is {topic}?"
           for i, topic in enumerate(["ML", "DL", "NLP", "CV", "RL"])]

outputs = llm.generate(prompts, sampling)
for out in outputs:
    print(f"[{out.request_id}] {out.outputs[0].text[:80]}...")

Deploying vLLM to Kubernetes in Production

The vLLM project ships an official Production Stack that supports multi-model serving, autoscaling, and load balancing on Kubernetes.

# vllm-production-stack-values.yaml
# vLLM Production Stack Helm Chart configuration
servingEngineSpec:
  runtimeClassName: nvidia
  modelSpec:
    - name: 'llama-70b'
      repository: 'vllm/vllm-openai'
      tag: 'latest'
      modelURL: 'meta-llama/Llama-3.1-70B-Instruct'
      replicaCount: 2
      requestCPU: 8
      requestMemory: '64Gi'
      requestGPU: 4
      gpuType: 'nvidia.com/gpu'
      tensorParallelSize: 4
      maxModelLen: 8192
      extraArgs:
        - '--enable-prefix-caching'
        - '--enable-chunked-prefill'
        - '--gpu-memory-utilization=0.92'
        - '--max-num-seqs=256'

      hpa:
        enabled: true
        minReplicas: 2
        maxReplicas: 8
        targetValue: '70' # Target 70% GPU utilization

routerSpec:
  repository: 'vllm/production-stack-router'
  tag: 'latest'
  replicaCount: 2
  requestCPU: 4
  requestMemory: '8Gi'
  routingStrategy: 'prefix-aware' # Prefix-Cache-friendly routing

# Prometheus metrics collection configuration
monitoring:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
      interval: '15s'
  grafana:
    enabled: true
    dashboards:
      - name: 'vllm-serving'
        url: 'https://grafana.com/grafana/dashboards/vllm'

# Deploy the vLLM Production Stack with Helm
helm repo add vllm https://vllm-project.github.io/production-stack
helm repo update

helm install vllm-serving vllm/vllm-stack \
    -f vllm-production-stack-values.yaml \
    --namespace llm-serving \
    --create-namespace

# Check deployment status
kubectl get pods -n llm-serving
kubectl logs -f deploy/vllm-serving-llama-70b -n llm-serving

SGLang in Depth

RadixAttention: A Breakthrough in KV Cache Reuse

SGLang is an inference engine developed by the LMSYS (UC Berkeley) team, and its key differentiator is an automatic KV Cache reuse mechanism called RadixAttention. It was presented at NeurIPS 2024 and achieved a throughput gain of up to 5x on certain workloads.

The core ideas behind RadixAttention:

1. Radix-Tree-based KV Cache management: the KV Cache of every request is stored in a single Radix Tree. Requests that share a common prefix reuse the KV Cache automatically.
2. LRU cache policy: the KV Cache of frequently used prefixes stays in memory, and older entries are evicted automatically.
3. Automatic prefix detection: even when the user does not name a prefix explicitly, the system detects the common prefix on its own and reuses the KV Cache.

This mechanism is particularly effective on workloads such as:

• Few-shot prompting: a pattern where the same examples (system prompt + few-shot examples) are followed by a variety of questions
• Multi-turn conversation: follow-up requests that share the earlier conversation history
• Tree of Thought: a pattern that explores several branches of reasoning from the same prompt

Compressed Finite State Machine (Structured Generation)

Another core capability of SGLang is how efficiently it produces Structured Output. It compiles constraints such as JSON Schema and regular expressions into a Compressed Finite State Machine, minimizing the overhead of constraint checking.

Existing structured-output engines (Outlines, Guidance and the like) mask the entire vocabulary every time a token is generated, which carries considerable overhead. SGLang compresses the state machine in advance and accelerates the masking operation during decoding by up to 300x or more.

# Running the SGLang server and using the frontend
# 1. Start the server
# python -m sglang.launch_server \
#     --model-path meta-llama/Llama-3.1-70B-Instruct \
#     --tp 4 \
#     --mem-fraction-static 0.88 \
#     --chunked-prefill-size 8192 \
#     --enable-torch-compile \
#     --port 30000

# 2. SGLang frontend (Python DSL)
import sglang as sgl

@sgl.function
def multi_turn_qa(s, system_prompt, questions):
    s += sgl.system(system_prompt)
    answers = []
    for q in questions:
        s += sgl.user(q)
        s += sgl.assistant(sgl.gen("answer", max_tokens=256, temperature=0.7))
        answers.append(s["answer"])
    return answers

# RadixAttention automatically reuses the KV Cache of the system prompt
runtime = sgl.Runtime(
    model_path="meta-llama/Llama-3.1-70B-Instruct",
    tp_size=4,
)
sgl.set_default_backend(runtime)

system = "You are an expert system architect. Provide concise technical answers."
questions_batch = [
    ["What is CQRS?", "How does event sourcing work?"],
    ["What is CQRS?", "When should I avoid CQRS?"],
    ["What is CQRS?", "Compare CQRS with traditional CRUD"],
]

# All 3 requests automatically share the KV Cache for "What is CQRS?"
import concurrent.futures
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
    futures = [
        executor.submit(multi_turn_qa, system, qs)
        for qs in questions_batch
    ]
    results = [f.result() for f in futures]

for i, r in enumerate(results):
    print(f"Batch {i}: {len(r)} answers generated")

runtime.shutdown()

Structured Output Generation in SGLang

# SGLang structured output (JSON Schema based)
import sglang as sgl
from pydantic import BaseModel
from typing import List

class CodeReview(BaseModel):
    file_name: str
    severity: str  # "critical", "warning", "info"
    line_number: int
    issue: str
    suggestion: str

class ReviewResult(BaseModel):
    reviews: List[CodeReview]
    overall_score: int  # 1-10
    summary: str

@sgl.function
def structured_code_review(s, code_snippet):
    s += sgl.system(
        "You are a senior code reviewer. Analyze the given code and provide "
        "structured feedback in JSON format."
    )
    s += sgl.user(f"Review this code:\n```python\n{code_snippet}\n```")
    s += sgl.assistant(
        sgl.gen(
            "review",
            max_tokens=1024,
            temperature=0.0,
            regex=ReviewResult.model_json_schema(),  # JSON Schema constraint
        )
    )

# Compressed FSM compiles the JSON Schema into a state machine
# Guarantees that only valid JSON is produced during decoding
result = structured_code_review(
    code_snippet="""
def process_data(data):
    result = []
    for i in range(len(data)):
        if data[i] > 0:
            result.append(data[i] * 2)
    return result
"""
)

import json
review = json.loads(result["review"])
print(f"Overall Score: {review['overall_score']}/10")
print(f"Issues Found: {len(review['reviews'])}")
for r in review["reviews"]:
    print(f"  [{r['severity']}] Line {r['line_number']}: {r['issue']}")

Benchmark Comparison of the 3 Frameworks

Feature Comparison Table

Throughput Benchmark on H100 (Llama-3.1-70B, TP=4)

Throughput by Model Size (H100 80GB, 64 concurrent requests, FP16)

Performance Across Concurrency Levels (Llama-3.1-70B, TP=4)

What stands out in the benchmark results:

• TensorRT-LLM leads on absolute throughput and latency, and the reason is the depth of its hardware optimization. The caveats are that it is NVIDIA-GPU only and the build process is complex.
• vLLM trails slightly on throughput, but its 42% TTFT reduction on a prefix cache hit is excellent. Its strengths are the broadest hardware support and a mature production stack.
• SGLang is dominant with a 65% TTFT reduction on a prefix cache hit. Thanks to RadixAttention it is the most efficient on repetitive prompt patterns (few-shot, multi-turn).

Production Deployment Architecture

GPU Node Scheduling Strategy

When managing LLM serving nodes in production, allocating GPU resources efficiently is the crux of the job.

# Kubernetes GPU node scheduling - Pod Affinity and Topology settings
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving-70b
  namespace: llm-serving
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-serving
      model: llama-70b
  template:
    metadata:
      labels:
        app: llm-serving
        model: llama-70b
    spec:
      # Schedule onto GPU nodes only
      nodeSelector:
        nvidia.com/gpu.product: 'NVIDIA-H100-80GB-HBM3'
      tolerations:
        - key: 'nvidia.com/gpu'
          operator: 'Exists'
          effect: 'NoSchedule'
      # TP=4, so use 4 GPUs on the same node
      # Topology settings that guarantee NVLink-connected GPUs
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: llm-serving
      containers:
        - name: vllm-engine
          image: vllm/vllm-openai:latest
          args:
            - '--model=meta-llama/Llama-3.1-70B-Instruct'
            - '--tensor-parallel-size=4'
            - '--max-model-len=8192'
            - '--gpu-memory-utilization=0.92'
            - '--enable-prefix-caching'
            - '--port=8000'
          resources:
            limits:
              nvidia.com/gpu: 4
              memory: '128Gi'
              cpu: '16'
            requests:
              nvidia.com/gpu: 4
              memory: '96Gi'
              cpu: '12'
          ports:
            - containerPort: 8000
              name: http
            - containerPort: 8002
              name: metrics
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 120
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 180
            periodSeconds: 30
            failureThreshold: 3

Autoscaling and Monitoring

# KEDA ScaledObject - autoscaling driven by GPU metrics
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-serving-scaler
  namespace: llm-serving
spec:
  scaleTargetRef:
    name: llm-serving-70b
  minReplicaCount: 2
  maxReplicaCount: 8
  cooldownPeriod: 300
  pollingInterval: 15
  triggers:
    # Scale on GPU utilization
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: gpu_utilization
        query: |
          avg(DCGM_FI_DEV_GPU_UTIL{
            namespace="llm-serving",
            pod=~"llm-serving-70b.*"
          })
        threshold: '75'
    # Scale on queue depth
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: pending_requests
        query: |
          sum(vllm:num_requests_waiting{
            namespace="llm-serving"
          })
        threshold: '50'

# Prometheus custom metric collection script
import requests
import time
from prometheus_client import Gauge, start_http_server

# vLLM / SGLang metric collection
TTFT_P50 = Gauge("llm_ttft_p50_ms", "Time to First Token p50 in ms")
TTFT_P99 = Gauge("llm_ttft_p99_ms", "Time to First Token p99 in ms")
THROUGHPUT = Gauge("llm_throughput_rps", "Requests per second")
GPU_KV_CACHE_USAGE = Gauge("llm_gpu_kv_cache_usage", "KV Cache usage ratio")
ACTIVE_REQUESTS = Gauge("llm_active_requests", "Number of active requests")
PENDING_REQUESTS = Gauge("llm_pending_requests", "Number of pending requests")

def collect_vllm_metrics(base_url="http://localhost:8000"):
    """Collect metrics from the vLLM /metrics endpoint"""
    try:
        resp = requests.get(f"{base_url}/metrics", timeout=5)
        lines = resp.text.strip().split("\n")
        for line in lines:
            if line.startswith("#"):
                continue
            if "vllm:time_to_first_token_seconds" in line and "p50" in line:
                TTFT_P50.set(float(line.split()[-1]) * 1000)
            elif "vllm:time_to_first_token_seconds" in line and "p99" in line:
                TTFT_P99.set(float(line.split()[-1]) * 1000)
            elif "vllm:num_requests_running" in line:
                ACTIVE_REQUESTS.set(float(line.split()[-1]))
            elif "vllm:num_requests_waiting" in line:
                PENDING_REQUESTS.set(float(line.split()[-1]))
            elif "vllm:gpu_cache_usage_perc" in line:
                GPU_KV_CACHE_USAGE.set(float(line.split()[-1]))
    except Exception as e:
        print(f"Metric collection failed: {e}")

if __name__ == "__main__":
    start_http_server(9090)
    while True:
        collect_vllm_metrics()
        time.sleep(15)

Failure Cases and Troubleshooting

Case 1: A Serving Outage from OOM, and Memory Management

Situation: while Llama-3.1-70B was being served on 4xH100 with vLLM, concurrent requests went past 200, a CUDA OOM fired, and the entire serving process terminated abnormally.

Root cause analysis:

1. gpu-memory-utilization was set to 0.95, leaving almost no spare memory
2. Some requests produced longer output than expected, so the KV Cache grew explosively
3. Prefix Caching was disabled, so the KV Cache for the same system prompt was allocated more than once

Resolution steps:

1. Lowered gpu-memory-utilization to 0.90 to leave headroom for KV Cache allocation
2. Reduced max-num-seqs from 256 to 128 to cap the number of concurrent requests
3. Enabled enable-prefix-caching to share the KV Cache of the system prompt
4. Capped max-tokens at 2048 to bound the KV Cache of an individual request

# Recovery checklist when an OOM occurs
# 1. Check the current GPU memory state
nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu \
    --format=csv,noheader,nounits

# 2. Check the vLLM process state
curl -s http://localhost:8000/metrics | \
    grep -E "vllm:(num_requests|gpu_cache|cpu_cache)"

# 3. Restart with safe settings
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 128 \
    --max-model-len 4096 \
    --enable-prefix-caching \
    --enable-chunked-prefill

# 4. On Kubernetes, reset the resource requests/limits and roll out an update
kubectl set env deploy/llm-serving-70b \
    VLLM_GPU_MEMORY_UTILIZATION=0.90 \
    -n llm-serving
kubectl rollout restart deploy/llm-serving-70b -n llm-serving
kubectl rollout status deploy/llm-serving-70b -n llm-serving

Case 2: Debugging an Accuracy Drop in a Quantized Model

Situation: Llama-3.1-70B was quantized to INT4 AWQ in TensorRT-LLM and deployed, and in specific domains (medical, legal) the response quality dropped noticeably compared with FP16.

Root cause analysis:

1. The calibration dataset consisted only of general web text, so domain-specific weights were quantized inaccurately
2. The router weights in the MoE (Mixture of Experts) layers were especially sensitive to quantization
3. INT4 loses more information than FP8, so the quality drop stood out on domain-specific patterns

Resolution steps:

1. Included 20% domain-specific text (medical, legal) in the calibration dataset
2. Switched from INT4 AWQ to FP8 quantization (making use of the FP8 Tensor Cores on H100)
3. Applied Mixed Precision, keeping the sensitive layers (the first 2 and the last 2) in FP16
4. Automated the before/after quality comparison with an evaluation pipeline (MMLU, HellaSwag, domain benchmarks)

Recovery Procedure Checklist

Operational Caveats and a Selection Guide

Recommendation Matrix by Use Case

Cost Optimization Strategies

1. Lean on quantization: switching from FP16 to FP8 gives roughly a 1.8x throughput gain on the same GPU. The quality loss is under 1%, which is acceptable in most use cases.
2. Enable prefix caching: if system prompts or few-shot examples repeat, prefix caching saves Prefill computation and can cut cost by 20-60%.
3. Autoscaling + spot instances: run batch inference workloads on spot/preemptible GPU instances and minimize idle-time cost with KEDA-based autoscaling.
4. Right-size the model: a well fine-tuned 8B model often shows better cost-performance than a 70B model within a specific domain. Always confirm with a benchmark before deciding.

Vendor Lock-in Considerations

Choosing TensorRT-LLM buys you top performance, but when you later move to AMD or another GPU you have to replace the entire inference stack. For organizations where a multi-cloud strategy matters, the broad hardware support in vLLM is the advantage. SGLang exposes an OpenAI-compatible API, so swapping the backend is easy, but using the frontend DSL creates a dependency on SGLang.

Conclusion

Choosing an LLM inference serving framework is not a decision to make on benchmark numbers alone. Your team's operational capacity, the hardware environment, workload characteristics, and long-term strategy all have to be weighed together.

The practical recommendations as of today are as follows:

• For most production environments, vLLM is the safest choice. Broad model and hardware support, a mature production stack, and an active community all reduce the operational burden.
• If you need extreme performance in an NVIDIA-only environment, consider TensorRT-LLM. At a scale where the build complexity and the operational burden are worth absorbing, it offers the best performance per dollar.
• If repetitive prompt patterns dominate or structured output is central, SGLang is the best fit. The combination of RadixAttention and Compressed FSM gives it a clear edge over the other frameworks on those workloads.

Whichever framework you choose, keeping an abstraction layer behind an OpenAI-compatible API, so that the cost of switching frameworks later stays as low as possible, is the wisest long-term strategy.

References

• TensorRT-LLM Documentation
• vLLM Documentation
• SGLang Paper (NeurIPS 2024)
• vLLM Production Stack
• Benchmarking vLLM, SGLang, TensorRT-LLM | Cerebrium
• SGLang vs vLLM Benchmark | LMSYS
• PagedAttention Paper
• TensorRT-LLM GitHub