The Complete LLMOps Guide: Three-Axis Versioning of Model, Prompt, and Eval Set, Canary, Cost Control, and the Platform Team (2025)

Season 4 Ep 11 — If Ep 10 was "the axis that protects," Ep 11 is the axis that keeps things running sustainably. How to avoid "one week to the first release, then a year of hell."

• Prologue — "Is LLMOps MLOps or DevOps?"
• Chapter 1 · Three-Axis Versioning
  • 1.1 The Axes
  • 1.2 Release Metadata
  • 1.3 Linking to Logs
• Chapter 2 · Deployment Strategy
  • 2.1 Shadow
  • 2.2 Canary
  • 2.3 Blue-Green
  • 2.4 A/B Experiments
  • 2.5 Percentage Router
• Chapter 3 · Cost Control
  • 3.1 Token Audit
  • 3.2 Caching
  • 3.3 Model Routing
  • 3.4 Token Diet
  • 3.5 Storage and Network
  • 3.6 Realistic Savings Targets
• Chapter 4 · Building the Platform Team
  • 4.1 Three Layers
  • 4.2 Interfaces
  • 4.3 By Size
  • 4.4 Political Traps
• Chapter 5 · The Eval Harness — The Heart of LLMOps
  • 5.1 Versioning the Eval Set
  • 5.2 CI Integration
  • 5.3 On-demand Experiments
  • 5.4 Production Feedback Loop
• Chapter 6 · Observability and Operations
  • 6.1 Extending Ep 6
  • 6.2 The Core Dashboards
  • 6.3 On-call
  • 6.4 Preparing for Vendor Outages
• Chapter 7 · Data Governance
  • 7.1 Data Classification
  • 7.2 User Consent
  • 7.3 Handling PII
  • 7.4 Licensing
• Chapter 8 · Ten Failure Cases
  • 8.1 Token Blowout
  • 8.2 Prompt Rollback Impossible
  • 8.3 Model Deprecation
  • 8.4 One Region Down, Everything Down
  • 8.5 RAG Index Drift
  • 8.6 Over-retention of User Logs
  • 8.7 Prompt Injection
  • 8.8 False Refusal Surge
  • 8.9 Vector DB Cost Blowout
  • 8.10 Misreading A/B Results
• Chapter 9 · KPIs and OKRs
  • 9.1 Product KPIs
  • 9.2 Engineering KPIs
  • 9.3 AI Quality KPIs
  • 9.4 Operations KPIs
• Chapter 10 · The Korean and Asian Environment
  • 10.1 Cloud
  • 10.2 Vendor Diversification
  • 10.3 Employee Training and Culture
• Chapter 11 · Tooling Roundup (2025)
  • 11.1 Gateway and Routing
  • 11.2 Observability
  • 11.3 Evaluation
  • 11.4 Prompt Registry
  • 11.5 Data and Features
  • 11.6 Serving
  • 11.7 Guardrails and Security
  • 11.8 Cost and Resources
• Chapter 12 · Ten Anti-patterns
  • 12.1 Swapping the Model and Shipping
  • 12.2 Single-vendor Dependence
  • 12.3 Prompts Living Outside Git
  • 12.4 Not Using Shadow or Canary
  • 12.5 No Cost Dashboard
  • 12.6 Eval Set Mixed with Training Data
  • 12.7 No Plan for Vendor Outages
  • 12.8 An Over-powerful AI Platform Team
  • 12.9 No Regard for Regulation and Audit
  • 12.10 No Postmortem Written
• Chapter 13 · Checklist — 12 Things Before an LLMOps Launch
• Chapter 14 · Next Up — Season 4 Ep 12: "AI Product Design"

Prologue — "Is LLMOps MLOps or DevOps?"

Both, and neither.

• Unlike MLOps, model training is rare (what you use is a hosted API or an adapter)
• Unlike DevOps, results are probabilistic, which makes testing hard
• But deployment, observability, rollback, and cost control need every lesson from both

The core questions of LLMOps in 2025:

1. Three-axis versioning (model, prompt, eval set) — how do you keep it consistent?
2. Cost: how do you find and reduce token waste?
3. Organization: where is the boundary between the AI platform team and the product teams?
4. Regulation and audit: how do you leave a trace of every deployment and change?

Chapter 1 · Three-Axis Versioning

1.1 The Axes

• Model: gpt-4o-2025-03-15, claude-3.7-sonnet, qwen2.5-14b-awq, and so on
• Prompt: the Git/registry version of the system prompt
• Eval set: the eval set snapshot tied to each release

1.2 Release Metadata

release: v1.4.2
model: anthropic/claude-3.5-sonnet-2025-02
prompt_version: sales_copilot@v23
eval_set: v12 (scored 91.3/100)
adapters:
  - korean_tone_lora@v3
guardrails:
  - llama_guard@v1
  - custom_policy@v8
rollout:
  canary: 5%

Every release has to pin this metadata as immutable.

1.3 Linking to Logs

• Include release_id in every request log
• Later you can reproduce "which configuration produced this response" with 100% fidelity
• The minimum requirement for a regulatory audit

Chapter 2 · Deployment Strategy

2.1 Shadow

• Call the new version in parallel against production traffic (the response is not used)
• Diff it against the existing response and compare metrics
• Confirm regressions and improvements with no user impact

2.2 Canary

• Expand in sequence: 1–5% → 10% → 50% → 100%
• Automatic gates at each stage (eval set score, p95 latency, error rate)
• Roll back to the previous version immediately on failure

2.3 Blue-Green

• Two environments (Blue live, Green standby) hold the old version and the new version
• The switch is a load balancer flip
• Rollback is immediate (a matter of seconds)

2.4 A/B Experiments

• Secure statistical significance (sample size, duration)
• Metrics: Quality, Cost, Latency, User satisfaction
• End-user segmentation (user-based / organization-based)

2.5 Percentage Router

• A small LLM gateway service splits traffic by user, organization, or feature
• Versions can be changed instantly with a toggle
• Open and SaaS gateways such as Helicone, Portkey, Martian, and LiteLLM

Chapter 3 · Cost Control

3.1 Token Audit

• Record input/output tokens per request
• Aggregate by user, feature, and endpoint
• Weekly review of the highest-cost endpoints and prompts

3.2 Caching

• Prompt cache: native on Anthropic, OpenAI, and Google
• Response cache: reuse a previous response for identical or similar input
• RAG retrieval cache: cache the query → document mapping

3.3 Model Routing

• Simple query → small model → complex query → large model
• The router itself is a classifier (LoRA or rules)
• Open routers such as Martian and RouteLLM rose in 2025

3.4 Token Diet

• Keep trimming the system prompt
• Optimize the number of documents fed into RAG
• Split JSON response fields into required and optional
• Remove waste with Structured Output (Ep 2)

3.5 Storage and Network

• Long-term log retention goes to cold storage
• Compress and partition vector DB indexes
• Minimize cross-region transfer

3.6 Realistic Savings Targets

• First 3 months: 20–40% (low-hanging fruit)
• Month 6: another 15–30% (model routing and cache sophistication)
• After that: 5–10% of maintenance improvements

Chapter 4 · Building the Platform Team

4.1 Three Layers

• Foundation / Infra: GPUs, model serving, observability infrastructure
• AI Platform: shared SDK, prompt registry, eval harness, guardrails
• Product AI: feature implementation for each product (chatbot, RAG, agents, voice, and so on)

4.2 Interfaces

• AI Platform → Product AI: an SDK that is easy to use plus standard observability
• Infra → AI Platform: stable serving and a cost dashboard
• Manage every boundary like an API that has an SLA plus on-call

4.3 By Size

• Startup of 10–50 people: no team split, just "1–2 AI leads plus product engineers"
• Mid-size of 50–500: an AI Platform of 5–15, with an AI owner per product
• Enterprise of 500+: all three layers split out, with separate security and compliance teams

4.4 Political Traps

• "All AI goes through our team" → bottleneck
• "Every product builds its own AI infrastructure" → duplicated waste
• The middle path: a shared platform plus product autonomy was the dominant model in 2025

Chapter 5 · The Eval Harness — The Heart of LLMOps

5.1 Versioning the Eval Set

• Git LFS or a data registry (DVC, Hugging Face Datasets, S3 + metadata)
• A version, schema, labeler, and creation date on every eval set
• Kept strictly separate from training data

5.2 CI Integration

• Run a lightweight eval set automatically on every PR
• Run the large eval set asynchronously after merge
• Slack/Discord alerts when a regression is detected

5.3 On-demand Experiments

• Evaluate new model candidates and prompt variants immediately
• A dashboard comparing cost and results
• Parallel experiments (several variants at once)

5.4 Production Feedback Loop

• User thumbs → label candidates
• Labeler review → added to the eval set
• A monthly "consolidated incident-case release"

Chapter 6 · Observability and Operations

6.1 Extending Ep 6

• Three layers: Trace/Span/Metric
• Built on OpenTelemetry
• LangFuse/LangSmith/Phoenix/Helicone/W&B

6.2 The Core Dashboards

• Quality: eval set score trend, thumbs up/down
• Cost: daily/monthly totals, by feature, by user
• Latency: p50/p95/p99, TTFT, TTL
• Safety: attack attempts, refusal rate, false refusal
• Errors: by status, by vendor, retry rate

6.3 On-call

• Define SLIs/SLOs for the key metrics
• Alert tiers (P1/P2/P3)
• Runbooks: vendor outage, cost blowout, quality regression
• Postmortem within 24–72h of an incident

6.4 Preparing for Vendor Outages

• Multi-provider routing (OpenAI outage → automatic fallback to Anthropic)
• A minimum-functionality mode on a local model
• A clear escalation path

Chapter 7 · Data Governance

7.1 Data Classification

• Public / Internal / Confidential / PII / Regulated
• A model, vendor, and log retention policy for each tier
• Use an automatic classifier (DLP)

7.2 User Consent

• Disclose AI use, be transparent about the scope of log retention
• APIs for deletion and data provision requests
• Comply with GDPR and personal information protection law

7.3 Handling PII

• Detect PII on input, then mask or pseudonymize it
• Re-mask on output
• Keep audit logs as hashes only

7.4 Licensing

• The terms of the model used to generate synthetic data
• Contracts covering third-party data use
• Responding to user copyright concerns (citing RAG retrieval results)

Chapter 8 · Ten Failure Cases

8.1 Token Blowout

Costs spiked after the system prompt was updated from 3000 to 8000 tokens.

8.2 Prompt Rollback Impossible

Inline strings rather than Git, so the previous version could not be found.

8.3 Model Deprecation

The vendor announced end-of-life for a model, and a regression showed up on the replacement.

8.4 One Region Down, Everything Down

Dependence on a single region. No multi-region or multi-vendor fallback.

8.5 RAG Index Drift

Document updates did not trigger an index rebuild, producing stale answers.

8.6 Over-retention of User Logs

Flagged in a regulatory audit, resulting in a fine.

8.7 Prompt Injection

Internal documents leaked through indirect injection.

8.8 False Refusal Surge

The refusal rate on legitimate requests rose after guardrails were tightened.

8.9 Vector DB Cost Blowout

More chunks were generated than expected, doubling the monthly cost.

8.10 Misreading A/B Results

Too few samples and ignored variance, promoting the wrong version.

Chapter 9 · KPIs and OKRs

9.1 Product KPIs

• Task completion rate
• Time saved (per user)
• NPS / Retention
• Adoption / Active usage

9.2 Engineering KPIs

• p95 latency
• Error rate
• Cost per request
• Availability (SLA)

9.3 AI Quality KPIs

• Eval set score (weekly/monthly)
• Hallucination rate
• Safety violations
• False refusal rate

9.4 Operations KPIs

• MTTR (mean time to recovery)
• Incident frequency (weekly/quarterly)
• Release cadence (times per week)
• Test coverage

Chapter 10 · The Korean and Asian Environment

10.1 Cloud

• AWS, GCP, Azure, Oracle + domestic (NHN Cloud, KT Cloud, Naver Cloud)
• Regulatory audits may require a domestic cloud or on-premises
• Prepare for Seoul and Tokyo region latency

10.2 Vendor Diversification

• OpenAI, Anthropic, Google, Cohere, Mistral + domestic (Upstage, LG AI Research)
• Finance and public sector: preferential terms for domestic vendors

10.3 Employee Training and Culture

• AI usage guidelines (preventing internal data leaks)
• Basic prompt engineering training (1–2 hours)
• A security review before any new product adopts AI

Chapter 11 · Tooling Roundup (2025)

11.1 Gateway and Routing

• LiteLLM, Portkey, Helicone, Martian, OpenRouter

11.2 Observability

• LangFuse, LangSmith, Phoenix (Arize), Helicone, W&B Weave, Datadog/NewRelic LLM extensions

11.3 Evaluation

• RAGAS, DeepEval, Promptfoo, Giskard, PyRIT (attack), OpenAI Evals, Confident AI

11.4 Prompt Registry

• LangSmith Prompt Hub, Humanloop, PromptLayer, W&B Weave, Agenta

11.5 Data and Features

• Weights & Biases Artifacts, MLflow, DVC, LakeFS

11.6 Serving

• vLLM, TGI, SGLang, TensorRT-LLM, Ollama, Modal, Anyscale, Together, Fireworks

11.7 Guardrails and Security

• NeMo Guardrails, Guardrails AI, Lakera Guard, Rebuff, Azure Content Safety, Google Cloud Model Armor

11.8 Cost and Resources

• Helicone Cost Insights, OpenMeter, CloudZero, Cast AI, Karpenter (K8s)

Chapter 12 · Ten Anti-patterns

12.1 Swapping the Model and Shipping

Rolling out without rechecking the eval set and the prompt.

12.2 Single-vendor Dependence

The risk of outages, price increases, and changes to the terms.

12.3 Prompts Living Outside Git

Config files, inline code, SaaS-only → versioning disappears.

12.4 Not Using Shadow or Canary

Discovering problems in production.

12.5 No Cost Dashboard

A shock when the month-end invoice arrives.

12.6 Eval Set Mixed with Training Data

Inflated evaluation.

12.7 No Plan for Vendor Outages

Downtime users feel keenly.

12.8 An Over-powerful AI Platform Team

The autonomy of Product AI gets suppressed.

12.9 No Regard for Regulation and Audit

Fines or a forced shutdown after launch.

12.10 No Postmortem Written

The same incident repeats.

Chapter 13 · Checklist — 12 Things Before an LLMOps Launch

• Versioning on all three axes (model, prompt, eval set) with release metadata pinned
• Shadow + Canary + automatic rollback in place
• Cost dashboard (daily/monthly, by feature and by user)
• A multi-vendor fallback path
• Eval harness + CI integration
• Standard observability instrumentation (Trace/Metric)
• Guardrails and red-team CI
• SLI/SLO + an on-call rotation
• Data classification, PII masking, deletion API
• Incident response and postmortem process
• Agreed responsibilities and APIs between the AI platform and the products
• Employee training and a security guide

Chapter 14 · Next Up — Season 4 Ep 12: "AI Product Design"

Once the engineering is stable, what remains is the user experience.

• The boundary between deterministic UI and generative UI
• Design that builds "trust" (citations, uncertainty, editability)
• Feedback UX: patterns beyond 👍/👎
• Handling streaming, latency, and feedback
• Agent UX: progress, approval, interruption, replay
• Voice UX in depth (extending Ep 9)
• The UX of data and learning
• Onboarding and first-impression design
• The UX of failures and outages
• Accessibility and inclusion
• Ethical AI UX
• Specialization for the Korean language and Korean culture

It is not "a good AI product = a good model + good UX" but "a good AI product = UX design that builds trust within constraints."

See you in the next post.

Summary: LLMOps comes down to three-axis versioning, Shadow/Canary, cost control, the platform team, and audit. It absorbs the lessons of MLOps and DevOps, but it needs additional practices that account for the probabilistic nature and vendor dependence unique to LLMs. Building one has become easy, but to build "an LLM product that keeps improving for a year without stopping" the 12-item checklist in this post is the minimum starting point. "AI is not something you deploy, it is something you run" is the lesson of 2025.