LLM Speculative Decoding Serving Optimization Playbook

• Playbook Overview
• Phase 1: Measuring the Current Serving State
  • Baseline Measurement Script
  • Defining the Metrics
• Phase 2: Choosing the Draft Model
  • Selection Criteria and Candidates
  • Comparison by Draft Model Type
  • Verifying Draft Model Compatibility
• Phase 3: vLLM Serving Configuration
  • Standalone Draft Model Approach
  • EAGLE-3 Approach (Recommended)
  • Tuning the Key Parameters
  • Finding the Optimal num_speculative_tokens
• Phase 4: Accept Ratio Monitoring
  • Collecting Prometheus Metrics
  • Grafana Alert Rules
• Phase 5: Routing by Traffic Class
  • Routing Decision Matrix
  • NGINX-Based Routing Configuration
• Phase 6: Rollback and Fallback
  • Automatic Fallback Decision Logic
• Phase 7: Regular Checkups (Weekly)
  • Weekly Checkup Automation Script
• Troubleshooting
  • 1. Slower after turning speculative decoding on
  • 2. Accept ratio collapses only on certain prompts
  • 3. Tokens arrive in clumps in streaming responses
  • 4. RuntimeError: Draft model and target model have different vocab sizes
• Pre-Deployment Checklist
• Quiz
• References

Playbook Overview

This document is a step-by-step execution guide for introducing speculative decoding into LLM serving. Rather than explaining the concept, it focuses on "in what order, with what settings, and by what criteria do you decide".

The core idea of speculative decoding is simple. A small, fast draft model guesses several tokens at once, and a large, accurate target model verifies them in a single forward pass. When most of the draft tokens are accepted, this is 2-3x faster than having the target model generate one token at a time. The original paper is by Leviathan et al. (arXiv:2211.17192), and it proves that the output distribution is mathematically identical to the target model's.

Phase 1: Measuring the Current Serving State

Before adopting speculative decoding, you have to quantify where you are today. Without a basis for comparison, you cannot prove the effect.

Baseline Measurement Script

import time
import json
import statistics
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

def measure_baseline(
    model: str,
    prompts_file: str,
    num_runs: int = 3,
) -> dict:
    """Measure the latency/throughput baseline of the current serving setup"""
    prompts = json.load(open(prompts_file))

    all_ttft = []    # Time To First Token
    all_tpot = []    # Time Per Output Token
    all_e2e = []     # End-to-End latency
    total_tokens = 0

    for run in range(num_runs):
        for prompt in prompts:
            start = time.perf_counter()
            first_token_time = None
            token_count = 0

            stream = client.chat.completions.create(
                model=model,
                messages=[{"role": "user", "content": prompt["text"]}],
                max_tokens=prompt.get("max_tokens", 256),
                temperature=0.0,
                stream=True,
            )
            for chunk in stream:
                if chunk.choices[0].delta.content:
                    if first_token_time is None:
                        first_token_time = time.perf_counter()
                    token_count += 1

            end = time.perf_counter()

            all_ttft.append(first_token_time - start)
            all_e2e.append(end - start)
            if token_count > 1:
                all_tpot.append((end - first_token_time) / (token_count - 1))
            total_tokens += token_count

    return {
        "ttft_p50_ms": round(statistics.median(all_ttft) * 1000, 1),
        "ttft_p95_ms": round(sorted(all_ttft)[int(len(all_ttft) * 0.95)] * 1000, 1),
        "tpot_p50_ms": round(statistics.median(all_tpot) * 1000, 1),
        "tpot_p95_ms": round(sorted(all_tpot)[int(len(all_tpot) * 0.95)] * 1000, 1),
        "e2e_p50_ms": round(statistics.median(all_e2e) * 1000, 1),
        "e2e_p95_ms": round(sorted(all_e2e)[int(len(all_e2e) * 0.95)] * 1000, 1),
        "total_tokens": total_tokens,
        "avg_tokens_per_sec": round(total_tokens / sum(all_e2e), 1),
    }

# Usage example
baseline = measure_baseline("meta-llama/Llama-3.1-70B-Instruct", "eval_prompts.json")
json.dump(baseline, open("baseline_metrics.json", "w"), indent=2)
print(json.dumps(baseline, indent=2))

Defining the Metrics

Phase 2: Choosing the Draft Model

The draft model you pick decides 70% of speculative decoding performance. Pick badly and you end up slower than the baseline.

Selection Criteria and Candidates

Target model -> Draft model matching guide

Llama 3.1 70B   -> Llama 3.1 8B (same family, identical vocabulary)
                   or an EAGLE-3 draft head (training required, best performance)

Mistral Large   -> Mistral 7B (same tokenizer)

Qwen 2.5 72B    -> Qwen 2.5 1.5B or Qwen 2.5 7B

In-house model  -> a small model with the same tokenizer
                   or train a Medusa head / EAGLE head

Comparison by Draft Model Type

Verifying Draft Model Compatibility

from transformers import AutoTokenizer

def verify_draft_compatibility(target_model: str, draft_model: str) -> dict:
    """Verify tokenizer compatibility between the draft and target models"""
    target_tok = AutoTokenizer.from_pretrained(target_model)
    draft_tok = AutoTokenizer.from_pretrained(draft_model)

    # 1. Check that the vocabulary sizes match
    vocab_match = target_tok.vocab_size == draft_tok.vocab_size

    # 2. Check that the special tokens match
    special_match = (
        target_tok.bos_token_id == draft_tok.bos_token_id and
        target_tok.eos_token_id == draft_tok.eos_token_id and
        target_tok.pad_token_id == draft_tok.pad_token_id
    )

    # 3. Compare the encodings of sample texts
    test_texts = [
        "Hello, how are you?",
        "서울의 날씨는 어떤가요?",
        "def fibonacci(n): return n if n < 2 else fibonacci(n-1) + fibonacci(n-2)",
    ]
    encoding_match = all(
        target_tok.encode(t) == draft_tok.encode(t) for t in test_texts
    )

    return {
        "vocab_size_match": vocab_match,
        "special_tokens_match": special_match,
        "encoding_match": encoding_match,
        "compatible": vocab_match and special_match and encoding_match,
        "target_vocab_size": target_tok.vocab_size,
        "draft_vocab_size": draft_tok.vocab_size,
    }

result = verify_draft_compatibility(
    "meta-llama/Llama-3.1-70B-Instruct",
    "meta-llama/Llama-3.1-8B-Instruct"
)
print(result)
# {'vocab_size_match': True, 'special_tokens_match': True,
#  'encoding_match': True, 'compatible': True, ...}

Phase 3: vLLM Serving Configuration

Standalone Draft Model Approach

# Enable speculative decoding in vLLM (Llama 3.1 70B + 8B)
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --speculative-model meta-llama/Llama-3.1-8B-Instruct \
    --num-speculative-tokens 5 \
    --speculative-disable-mqa-scorer \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 4096 \
    --port 8000

EAGLE-3 Approach (Recommended)

# Use an EAGLE-3 draft head (higher accept ratio)
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --speculative-model eagle3-llama3.1-70b-instruct \
    --speculative-method eagle \
    --num-speculative-tokens 5 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 4096 \
    --use-v2-block-manager \
    --port 8000

Tuning the Key Parameters

# speculative_decoding_config.yaml
# Start from this config, then adjust while watching accept ratio and latency

# Number of draft tokens: too many raises rejects, too few shrinks the gain
num_speculative_tokens: 5 # Starting value. Experiment in the 3-7 range

# Disable automatically when speculative decoding is not paying off
speculative_disable_by_batch_size: 8 # Disable at batch 8 or above

# Handling when temperature > 0
# Use typical acceptance sampling (preserves quality)
speculative_draft_tensor_parallel_size: 1 # TP=1 is enough for the draft

Finding the Optimal num_speculative_tokens

import subprocess
import json

def find_optimal_spec_tokens(
    target_model: str,
    draft_model: str,
    eval_prompts: str,
    candidates: list[int] = [3, 4, 5, 6, 7, 8],
) -> dict:
    """Run the benchmark across a range of num_speculative_tokens values"""
    results = {}
    for n in candidates:
        print(f"Testing num_speculative_tokens={n}")
        # Start the server (managed through subprocess in practice)
        # Only the result-collection logic is shown here
        metrics = run_benchmark(target_model, draft_model, n, eval_prompts)
        results[n] = {
            "accept_ratio": metrics["accept_ratio"],
            "tpot_p50_ms": metrics["tpot_p50_ms"],
            "e2e_speedup": metrics["baseline_e2e"] / metrics["e2e_p50_ms"],
            "gpu_memory_gb": metrics["gpu_memory_gb"],
        }

    # Pick the optimum: the value with the highest speedup
    best_n = max(results, key=lambda n: results[n]["e2e_speedup"])
    results["recommended"] = best_n
    return results

Phase 4: Accept Ratio Monitoring

Accept ratio is the core indicator of how healthy speculative decoding is. vLLM exposes this metric natively.

Collecting Prometheus Metrics

# prometheus.yml - vLLM metric scraping configuration
scrape_configs:
  - job_name: 'vllm-speculative'
    scrape_interval: 15s
    static_configs:
      - targets: ['vllm-server:8000']
    metrics_path: /metrics

The speculative decoding metrics vLLM exposes:

# Draft token acceptance rate
vllm:spec_decode_draft_acceptance_rate

# Acceptance rate by position (highest at position 0, falling off toward the tail)
vllm:spec_decode_per_position_acceptance_rate{position="0"}
vllm:spec_decode_per_position_acceptance_rate{position="1"}

# Mean accepted length
vllm:spec_decode_mean_accepted_length

Grafana Alert Rules

# grafana_alerts.yaml
groups:
  - name: speculative_decoding_alerts
    rules:
      # Warn when accept ratio falls to 0.5 or below
      - alert: LowAcceptRatio
        expr: vllm:spec_decode_draft_acceptance_rate < 0.5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Speculative decoding accept ratio drop'
          description: |
            Accept ratio has dropped to {{ $value | printf "%.2f" }}.
            Below 0.5, speculative decoding turns into pure overhead.
            Consider swapping the draft model or disabling speculative decoding.

      # Recommend disabling immediately when accept ratio is 0.3 or below
      - alert: CriticalAcceptRatio
        expr: vllm:spec_decode_draft_acceptance_rate < 0.3
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Speculative decoding needs to be disabled'
          description: |
            Accept ratio {{ $value | printf "%.2f" }}. Switch to fallback decoding immediately.

Phase 5: Routing by Traffic Class

You should not apply speculative decoding to every request. The effect differs enormously depending on the shape of the request.

Routing Decision Matrix

NGINX-Based Routing Configuration

# /etc/nginx/conf.d/llm-router.conf
upstream vllm_speculative {
    server 10.0.1.10:8000;  # server with speculative decoding enabled
}

upstream vllm_standard {
    server 10.0.1.20:8000;  # standard decoding server
}

# Lua-based dynamic routing
server {
    listen 80;

    location /v1/chat/completions {
        access_by_lua_block {
            local cjson = require "cjson"
            ngx.req.read_body()
            local body = cjson.decode(ngx.req.get_body_data())

            -- Decide the routing condition
            local use_speculative = true

            -- High temperature -> standard decoding
            if body.temperature and body.temperature > 1.0 then
                use_speculative = false
            end

            -- Short max_tokens -> standard decoding
            if body.max_tokens and body.max_tokens < 32 then
                use_speculative = false
            end

            -- Non-streaming and a short response -> standard decoding
            if not body.stream and body.max_tokens and body.max_tokens < 64 then
                use_speculative = false
            end

            if use_speculative then
                ngx.var.upstream = "vllm_speculative"
            else
                ngx.var.upstream = "vllm_standard"
            end
        }

        proxy_pass http://$upstream;
        proxy_set_header Host $host;
    }
}

Phase 6: Rollback and Fallback

Automatic Fallback Decision Logic

import requests
import time
from dataclasses import dataclass

@dataclass
class FallbackConfig:
    accept_ratio_threshold: float = 0.4
    latency_regression_pct: float = 20.0  # 20% or more slower than baseline
    check_interval_sec: int = 60
    consecutive_failures: int = 3

class SpeculativeDecodingGuard:
    """Monitor speculative decoding health and decide on automatic fallback"""

    def __init__(self, config: FallbackConfig, prometheus_url: str):
        self.config = config
        self.prometheus_url = prometheus_url
        self.failure_count = 0

    def query_prometheus(self, query: str) -> float:
        resp = requests.get(
            f"{self.prometheus_url}/api/v1/query",
            params={"query": query},
        )
        result = resp.json()["data"]["result"]
        return float(result[0]["value"][1]) if result else 0.0

    def should_fallback(self) -> tuple[bool, str]:
        # 1. Check the accept ratio
        accept_ratio = self.query_prometheus(
            'vllm:spec_decode_draft_acceptance_rate'
        )
        if accept_ratio < self.config.accept_ratio_threshold:
            self.failure_count += 1
            if self.failure_count >= self.config.consecutive_failures:
                return True, f"accept_ratio={accept_ratio:.2f} < {self.config.accept_ratio_threshold}"
        else:
            self.failure_count = 0

        # 2. Check for latency regression
        current_p95 = self.query_prometheus(
            'histogram_quantile(0.95, rate(vllm:e2e_request_latency_seconds_bucket[5m]))'
        )
        baseline_p95 = self.query_prometheus(
            'vllm:baseline_e2e_p95_seconds'  # the baseline metric must be recorded separately
        )
        if baseline_p95 > 0:
            regression_pct = ((current_p95 - baseline_p95) / baseline_p95) * 100
            if regression_pct > self.config.latency_regression_pct:
                return True, f"latency regression {regression_pct:.1f}% > {self.config.latency_regression_pct}%"

        return False, "healthy"

    def run(self):
        while True:
            should_fb, reason = self.should_fallback()
            if should_fb:
                print(f"[FALLBACK] Disabling speculative decoding: {reason}")
                self.trigger_fallback()
            time.sleep(self.config.check_interval_sec)

    def trigger_fallback(self):
        """Shift traffic to the standard decoding server"""
        # Real implementation: change load balancer weights or toggle a feature flag
        requests.post(
            "http://config-server/api/v1/flags",
            json={"speculative_decoding_enabled": False},
        )

Phase 7: Regular Checkups (Weekly)

Weekly Checkup Automation Script

import json
import datetime
from typing import Any

def weekly_speculative_decoding_report(
    prometheus_url: str,
    baseline_file: str,
) -> dict[str, Any]:
    """Generate the weekly speculative decoding operations report"""
    baseline = json.load(open(baseline_file))

    report = {
        "report_date": datetime.date.today().isoformat(),
        "period": "last_7d",
    }

    # 1. Accept ratio trend
    report["accept_ratio"] = {
        "current_avg": query_prom(prometheus_url,
            'avg_over_time(vllm:spec_decode_draft_acceptance_rate[7d])'),
        "min": query_prom(prometheus_url,
            'min_over_time(vllm:spec_decode_draft_acceptance_rate[7d])'),
        "max": query_prom(prometheus_url,
            'max_over_time(vllm:spec_decode_draft_acceptance_rate[7d])'),
    }

    # 2. Latency improvement
    current_e2e_p50 = query_prom(prometheus_url,
        'histogram_quantile(0.5, rate(vllm:e2e_request_latency_seconds_bucket[7d]))')
    report["speedup"] = {
        "e2e_p50_speedup": round(baseline["e2e_p50_ms"] / (current_e2e_p50 * 1000), 2),
        "baseline_e2e_p50_ms": baseline["e2e_p50_ms"],
        "current_e2e_p50_ms": round(current_e2e_p50 * 1000, 1),
    }

    # 3. Number of fallback occurrences
    report["fallback_count"] = int(query_prom(prometheus_url,
        'count_over_time(ALERTS{alertname="LowAcceptRatio"}[7d])'))

    # 4. Resource usage (the speculative increment)
    report["gpu_memory_overhead_gb"] = query_prom(prometheus_url,
        'avg_over_time(vllm:gpu_cache_usage_perc[7d])') * 80  # based on A100 80GB

    # 5. Recommendations
    recommendations = []
    if report["accept_ratio"]["current_avg"] < 0.55:
        recommendations.append("Recommend swapping the draft model or training an EAGLE-3 head")
    if report["speedup"]["e2e_p50_speedup"] < 1.3:
        recommendations.append("Improvement under 1.3x. Cost vs. benefit needs a second look")
    if report["fallback_count"] > 5:
        recommendations.append(f"{report['fallback_count']} fallbacks this week. Check draft model quality")

    report["recommendations"] = recommendations
    return report

Troubleshooting

1. Slower after turning speculative decoding on

Symptom: E2E latency is 10-30% higher than baseline

Diagnostic order:

# 1. Check the accept ratio
curl -s http://localhost:8000/metrics | grep spec_decode_draft_acceptance_rate
# Below 0.3 points to a draft model problem

# 2. Check draft model inference time
curl -s http://localhost:8000/metrics | grep spec_decode_draft_latency
# Inefficient if draft inference is 50% or more of a single target token inference

# 3. Check for swapping caused by GPU memory pressure
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
# Above 95%, a KV cache shortage cancels out the speculative gain

Fix: If accept ratio < 0.5, swap the draft model. If GPU memory is short, lower num_speculative_tokens to 3.

2. Accept ratio collapses only on certain prompts

Cause: Domains where the draft model is weak, such as code generation and math

Fix: Add a request classifier and route those domains to standard decoding.

3. Tokens arrive in clumps in streaming responses

Symptom: The stream users perceive stalls -> several tokens at once -> stalls again, on repeat

Cause: Speculative decoding verifies draft tokens in one shot, so accepted tokens are delivered in bursts

Fix: Buffer tokens on the client side and render at a fixed interval. Or check the --disable-frontend-multiprocessing option.

4. RuntimeError: Draft model and target model have different vocab sizes

RuntimeError: Draft model vocab size 32000 != target model vocab size 128256

Cause: A draft taken from a model family with a different tokenizer (e.g. Llama 2 draft + Llama 3 target)

Fix: Switch to a draft model from the same model family with the same tokenizer.

Pre-Deployment Checklist

• Baseline measured (TTFT, TPOT, E2E, throughput)
• Draft/target model tokenizer compatibility check passed
• Optimal num_speculative_tokens settled by experiment
• Accept ratio > 0.55 confirmed (production traffic sample)
• E2E latency speedup > 1.3x confirmed
• GPU memory usage confirmed below 95%
• Automatic fallback switching logic implemented and tested
• Routing rules per traffic class configured
• Prometheus metrics + Grafana dashboard set up
• Weekly checkup automation script deployed
• Speculative decoding incident runbook shared with the on-call team

Quiz

References

• Fast Inference from Transformers via Speculative Decoding (arXiv:2211.17192)
• Medusa: Simple LLM Inference Acceleration Framework (arXiv:2401.10774)
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (arXiv:2401.15077)
• vLLM Speculative Decoding Documentation
• vLLM Speculators v0.3.0 Blog Post
• EAGLE-3: Scaling up Inference Acceleration (arXiv:2503.01840)