RAG Chatbot Evaluation in Practice: From Offline/Online Quality Measurement to Production Guardrails

• An "Average Accuracy" Number Tells You Nothing About RAG Quality
• The Four Dimensions of RAG Evaluation
• Offline Evaluation: Building a Golden Dataset and Measuring Automatically
  • Golden Dataset Structure
  • Measuring Retrieval Quality
  • LLM-as-a-Judge: Measuring Grounding and Answer Quality
• Online Evaluation: Monitoring Quality in Production
  • Collecting Real-time Quality Metrics
  • Setting Quality Alert Thresholds
• Regression Prevention Pipeline: Putting a Quality Gate in CI
  • The Full CI Workflow
• Using the RAGAS Framework
• Handling Each Failure Scenario
  • Scenario 1: Faithfulness Score Collapses
  • Scenario 2: Empty-search-result Rate Spikes
  • Scenario 3: LLM-as-a-Judge Evaluation Cost Spikes
• References

An "Average Accuracy" Number Tells You Nothing About RAG Quality

The most common mistake in evaluating a RAG chatbot is deciding to ship on the basis of a single metric (for example, an overall accuracy of 82%). The following risks hide behind that number.

• Retrieval failure decides answer quality: if the document containing the answer is never retrieved, even the best LLM will hallucinate. That overall accuracy of 82% may be a number produced by a retrieval recall of 60% plus the LLM papering over the gap.
• An answer without grounding is a hallucination: if the LLM ignores the retrieved documents and answers from its own knowledge, the answer is untrustworthy even when it happens to be right. The grounding rate - the proportion of answers grounded in the retrieved documents - has to be measured separately.
• Safety is separate from accuracy: an accurate answer still cannot ship if it contains PII or violates a business rule.

This article measures RAG chatbot quality across four independent dimensions - Retrieval, Grounding, Answer and Safety - and designs the full evaluation pipeline from offline benchmarks to online monitoring.

The Four Dimensions of RAG Evaluation

Offline Evaluation: Building a Golden Dataset and Measuring Automatically

Golden Dataset Structure

"""
Golden Dataset for RAG evaluation.
Each test case contains the question, the expected answer, the documents that
support the answer, and the expected category.
"""
from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class GoldenTestCase:
    """A single evaluation case"""
    question_id: str
    question: str
    expected_answer: str
    relevant_doc_ids: List[str]     # IDs of the documents that contain the answer
    category: str                    # faq, policy, product, troubleshooting
    difficulty: str                  # easy, medium, hard
    expected_citations: List[str]    # document IDs the answer is expected to cite
    negative_assertions: List[str] = field(default_factory=list)  # content that must not appear in the answer

# example evaluation set
GOLDEN_DATASET = [
    GoldenTestCase(
        question_id="faq_001",
        question="How many days does a refund take to process?",
        expected_answer="Refunds are processed within 3-5 business days from the request date.",
        relevant_doc_ids=["doc_refund_policy_v3", "doc_faq_payment"],
        category="faq",
        difficulty="easy",
        expected_citations=["doc_refund_policy_v3"],
        negative_assertions=["instant refund", "same-day processing"],
    ),
    GoldenTestCase(
        question_id="policy_002",
        question="Who pays the customs duty on an international shipment?",
        expected_answer="For international shipments, customs duty and VAT are paid by the recipient.",
        relevant_doc_ids=["doc_shipping_international", "doc_customs_guide"],
        category="policy",
        difficulty="medium",
        expected_citations=["doc_shipping_international"],
        negative_assertions=["duty exempt", "free shipping"],
    ),
    GoldenTestCase(
        question_id="trouble_003",
        question="The app shows payment error P4021. How do I fix it?",
        expected_answer="Error P4021 is caused by a card issuer authentication failure. Enable online payments in the card issuer app and try again.",
        relevant_doc_ids=["doc_error_codes", "doc_payment_troubleshoot"],
        category="troubleshooting",
        difficulty="hard",
        expected_citations=["doc_error_codes"],
        negative_assertions=["contact customer support", "unknown error"],
    ),
]

Measuring Retrieval Quality

"""
Module for measuring the quality of the retrieval step.
Computes Recall@K, MRR (Mean Reciprocal Rank) and nDCG.
"""
from typing import List, Dict
import numpy as np

def recall_at_k(
    retrieved_doc_ids: List[str],
    relevant_doc_ids: List[str],
    k: int = 5,
) -> float:
    """
    The proportion of relevant documents present in the top K search results.
    e.g. relevant_docs = ["A", "B"], retrieved = ["C", "A", "D", "B", "E"]
    recall@5 = 2/2 = 1.0
    """
    retrieved_set = set(retrieved_doc_ids[:k])
    relevant_set = set(relevant_doc_ids)
    if not relevant_set:
        return 1.0  # with no relevant documents, recall is meaningless
    return len(retrieved_set & relevant_set) / len(relevant_set)

def mean_reciprocal_rank(
    retrieved_doc_ids: List[str],
    relevant_doc_ids: List[str],
) -> float:
    """
    The reciprocal rank of the first relevant document.
    e.g. relevant = ["B"], retrieved = ["A", "B", "C"] -> MRR = 1/2 = 0.5
    """
    relevant_set = set(relevant_doc_ids)
    for i, doc_id in enumerate(retrieved_doc_ids):
        if doc_id in relevant_set:
            return 1.0 / (i + 1)
    return 0.0

def evaluate_retrieval(
    test_cases: List[dict],
    retriever,
    k_values: List[int] = [3, 5, 10],
) -> Dict[str, float]:
    """
    Measure retrieval quality across the whole test set.
    """
    results = {f"recall@{k}": [] for k in k_values}
    results["mrr"] = []

    for case in test_cases:
        retrieved = retriever.search(
            query=case["question"],
            top_k=max(k_values),
        )
        retrieved_ids = [doc.id for doc in retrieved]

        for k in k_values:
            r = recall_at_k(retrieved_ids, case["relevant_doc_ids"], k)
            results[f"recall@{k}"].append(r)

        mrr = mean_reciprocal_rank(retrieved_ids, case["relevant_doc_ids"])
        results["mrr"].append(mrr)

    return {
        metric: round(float(np.mean(values)), 4)
        for metric, values in results.items()
    }

LLM-as-a-Judge: Measuring Grounding and Answer Quality

Use an LLM as the judge to measure the answer's faithfulness (fidelity to the retrieved documents) and relevance (relatedness to the question).

"""
Evaluate the Grounding and Relevance of RAG answers with the LLM-as-a-Judge pattern.
Use OpenAI GPT-4o or Claude 3.5 Sonnet as the judge.
"""
from dataclasses import dataclass
from typing import List, Optional
import json

FAITHFULNESS_PROMPT = """You are an expert judge who evaluates the faithfulness of RAG chatbot answers.

Given information:
- User question: {question}
- Retrieved documents: {retrieved_contexts}
- Chatbot answer: {answer}

Evaluation criteria:
1. Is every claim in the answer grounded in the retrieved documents?
2. Does the answer add or invent anything that is not in the documents?
3. Does the answer distort what the documents say?

Respond in the following JSON format only:
{{
  "faithfulness_score": <0.0-1.0>,
  "claims": [
    {{
      "claim": "<a claim extracted from the answer>",
      "supported": <true/false>,
      "evidence": "<the supporting document text, or 'no evidence found'>"
    }}
  ],
  "unsupported_claims_count": <int>,
  "hallucination_detected": <true/false>
}}"""

RELEVANCE_PROMPT = """You are an expert judge who evaluates the relevance of RAG chatbot answers.

Given information:
- User question: {question}
- Chatbot answer: {answer}

Evaluation criteria:
1. Does the answer respond directly to the question?
2. Is it free of excessive unnecessary information?
3. Is any essential information missing?

Respond in the following JSON format only:
{{
  "relevance_score": <0.0-1.0>,
  "addresses_question": <true/false>,
  "missing_information": "<the essential information that is missing, or 'none'>",
  "unnecessary_information": "<the unnecessary information, or 'none'>"
}}"""

@dataclass
class JudgmentResult:
    question_id: str
    faithfulness_score: float
    relevance_score: float
    hallucination_detected: bool
    unsupported_claims: int
    missing_info: str
    raw_judgment: dict

async def evaluate_with_llm_judge(
    question: str,
    answer: str,
    retrieved_contexts: List[str],
    judge_client,  # an OpenAI or Anthropic client
    question_id: str = "",
) -> JudgmentResult:
    """Evaluate faithfulness and relevance with the LLM judge."""

    # Faithfulness evaluation
    faith_prompt = FAITHFULNESS_PROMPT.format(
        question=question,
        retrieved_contexts="\n---\n".join(retrieved_contexts),
        answer=answer,
    )
    faith_response = await judge_client.chat.completions.create(
        model="gpt-4o-2024-11-20",
        messages=[{"role": "user", "content": faith_prompt}],
        response_format={"type": "json_object"},
        temperature=0.0,  # temperature 0 for consistent judgments
    )
    faith_result = json.loads(faith_response.choices[0].message.content)

    # Relevance evaluation
    rel_prompt = RELEVANCE_PROMPT.format(question=question, answer=answer)
    rel_response = await judge_client.chat.completions.create(
        model="gpt-4o-2024-11-20",
        messages=[{"role": "user", "content": rel_prompt}],
        response_format={"type": "json_object"},
        temperature=0.0,
    )
    rel_result = json.loads(rel_response.choices[0].message.content)

    return JudgmentResult(
        question_id=question_id,
        faithfulness_score=faith_result.get("faithfulness_score", 0.0),
        relevance_score=rel_result.get("relevance_score", 0.0),
        hallucination_detected=faith_result.get("hallucination_detected", False),
        unsupported_claims=faith_result.get("unsupported_claims_count", 0),
        missing_info=rel_result.get("missing_information", "unknown"),
        raw_judgment={"faithfulness": faith_result, "relevance": rel_result},
    )

Online Evaluation: Monitoring Quality in Production

Even after a model that passed offline evaluation is deployed to production, the quality against real traffic has to be monitored continuously.

Collecting Real-time Quality Metrics

"""
Middleware that collects real-time quality metrics for a production RAG chatbot.
It collects lightweight quality signals for every request-response pair.
"""
import time
import hashlib
from dataclasses import dataclass, field
from typing import List, Optional, Dict
from prometheus_client import Histogram, Counter, Gauge

# Prometheus metric definitions
RESPONSE_LATENCY = Histogram(
    "rag_response_latency_seconds",
    "Total time taken by a RAG response",
    ["pipeline_stage"],  # retrieval, generation, total
    buckets=[0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0],
)
RETRIEVAL_EMPTY = Counter(
    "rag_retrieval_empty_total",
    "Number of requests that returned zero search results",
)
CITATION_RATE = Gauge(
    "rag_citation_rate",
    "Proportion of the last 100 responses that included a citation",
)
THUMBS_DOWN = Counter(
    "rag_thumbs_down_total",
    "Number of negative user feedback events",
    ["category"],
)

@dataclass
class QualitySignals:
    """Lightweight quality signals collected from one request-response pair"""
    request_id: str
    retrieval_count: int              # number of documents retrieved
    retrieval_latency_ms: float
    top_similarity_score: float       # similarity score of the top search result
    generation_latency_ms: float
    response_length_chars: int
    has_citation: bool                # whether the response includes a source citation
    language_detected: str            # the language of the response
    contains_hedging: bool            # whether it contains hedging such as "probably" or "may be"

def collect_quality_signals(
    request_id: str,
    query: str,
    retrieved_docs: list,
    response: str,
    retrieval_time: float,
    generation_time: float,
) -> QualitySignals:
    """Extract the quality signals from a request-response pair."""
    top_score = max((doc.score for doc in retrieved_docs), default=0.0)
    has_citation = any(
        marker in response
        for marker in ["[Source:", "[Ref:", "[Doc", "Source:"]
    )
    hedging_phrases = ["probably", "may be", "might not be accurate", "needs to be confirmed"]
    contains_hedging = any(phrase in response for phrase in hedging_phrases)

    signals = QualitySignals(
        request_id=request_id,
        retrieval_count=len(retrieved_docs),
        retrieval_latency_ms=retrieval_time * 1000,
        top_similarity_score=top_score,
        generation_latency_ms=generation_time * 1000,
        response_length_chars=len(response),
        has_citation=has_citation,
        language_detected="ko" if any('\uac00' <= c <= '\ud7a3' for c in response) else "en",
        contains_hedging=contains_hedging,
    )

    # record the Prometheus metrics
    RESPONSE_LATENCY.labels(pipeline_stage="retrieval").observe(retrieval_time)
    RESPONSE_LATENCY.labels(pipeline_stage="generation").observe(generation_time)
    RESPONSE_LATENCY.labels(pipeline_stage="total").observe(retrieval_time + generation_time)

    if signals.retrieval_count == 0:
        RETRIEVAL_EMPTY.inc()

    return signals

Setting Quality Alert Thresholds

# prometheus-rules.yaml
groups:
  - name: rag_quality_alerts
    interval: 30s
    rules:
      # warn when the empty-search-result rate exceeds 10%
      - alert: RAGRetrievalEmptyRateHigh
        expr: |
          rate(rag_retrieval_empty_total[10m]) /
          rate(rag_response_latency_seconds_count{pipeline_stage="total"}[10m]) > 0.10
        for: 5m
        labels:
          severity: warning
          team: chatbot
        annotations:
          summary: 'The RAG empty-search-result rate exceeds 10%'
          description: |
            Current empty-result rate: {{ $value | humanizePercentage }}
            Check the index state, the embedding model and the query preprocessing.

      # warn when the citation rate drops below 70%
      - alert: RAGCitationRateLow
        expr: rag_citation_rate < 0.70
        for: 10m
        labels:
          severity: warning
          team: chatbot
        annotations:
          summary: 'The RAG citation rate is below 70%'
          description: |
            Current citation rate: {{ $value | humanizePercentage }}
            Check the citation instruction in the LLM prompt, or the retrieval quality.

      # warn when response latency spikes
      - alert: RAGResponseLatencyHigh
        expr: |
          histogram_quantile(0.95,
            rate(rag_response_latency_seconds_bucket{pipeline_stage="total"}[5m])
          ) > 5.0
        for: 3m
        labels:
          severity: critical
          team: chatbot
        annotations:
          summary: 'RAG p95 response latency exceeds 5 seconds'

      # spike in negative user feedback
      - alert: RAGNegativeFeedbackSpike
        expr: |
          rate(rag_thumbs_down_total[30m]) > 2 * rate(rag_thumbs_down_total[24h] offset 1d)
        for: 15m
        labels:
          severity: warning
          team: chatbot
        annotations:
          summary: 'Negative user feedback has more than doubled versus the previous day'

Regression Prevention Pipeline: Putting a Quality Gate in CI

Every time you change a component of the RAG pipeline - the prompt, the retrieval configuration, the LLM model - a quality regression has to be detected automatically.

"""
RAG quality regression test.
Measures quality across the 4 dimensions on the golden dataset and
decides whether there is a regression against the previous version.
"""
import json
import sys
from pathlib import Path
from typing import Dict

# quality baseline (the scores of the previously deployed version)
BASELINE_SCORES = {
    "recall@5": 0.88,
    "mrr": 0.75,
    "faithfulness": 0.91,
    "relevance": 0.87,
    "safety_pass_rate": 1.00,
    "citation_rate": 0.82,
}

# acceptable drop (absolute value)
REGRESSION_TOLERANCE = {
    "recall@5": 0.03,       # a drop of up to 3% is allowed
    "mrr": 0.03,
    "faithfulness": 0.02,   # faithfulness is held strictly
    "relevance": 0.03,
    "safety_pass_rate": 0.0,  # no regression allowed on safety
    "citation_rate": 0.05,
}

def check_regression(current_scores: Dict[str, float]) -> dict:
    """
    Compare the current scores against the baseline and decide whether there is a regression.
    Returns: {"passed": bool, "regressions": [...], "improvements": [...]}
    """
    regressions = []
    improvements = []

    for metric, baseline in BASELINE_SCORES.items():
        current = current_scores.get(metric, 0.0)
        tolerance = REGRESSION_TOLERANCE.get(metric, 0.0)
        delta = current - baseline

        if delta < -tolerance:
            regressions.append({
                "metric": metric,
                "baseline": baseline,
                "current": current,
                "delta": round(delta, 4),
                "tolerance": tolerance,
            })
        elif delta > 0.01:  # an improvement of 1% or more
            improvements.append({
                "metric": metric,
                "baseline": baseline,
                "current": current,
                "delta": round(delta, 4),
            })

    passed = len(regressions) == 0
    return {
        "passed": passed,
        "regressions": regressions,
        "improvements": improvements,
        "summary": (
            f"PASSED: {len(improvements)} improvements, 0 regressions"
            if passed
            else f"FAILED: {len(regressions)} regressions detected"
        ),
    }

if __name__ == "__main__":
    # run in CI: python eval/regression_check.py results.json
    results_path = sys.argv[1] if len(sys.argv) > 1 else "eval_results.json"
    with open(results_path) as f:
        current_scores = json.load(f)

    result = check_regression(current_scores)
    print(json.dumps(result, indent=2, ensure_ascii=False))

    if not result["passed"]:
        print("\n=== REGRESSION DETAILS ===")
        for reg in result["regressions"]:
            print(f"  {reg['metric']}: {reg['baseline']:.4f} -> {reg['current']:.4f} "
                  f"(delta: {reg['delta']:.4f}, tolerance: {reg['tolerance']:.4f})")
        sys.exit(1)

The Full CI Workflow

# .github/workflows/rag-eval.yml
name: RAG Quality Evaluation
on:
  push:
    paths:
      - 'src/rag/**'
      - 'prompts/**'
      - 'config/retrieval/**'
  pull_request:
    paths:
      - 'src/rag/**'
      - 'prompts/**'

jobs:
  offline-eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install -r requirements-eval.txt

      - name: Run retrieval evaluation
        run: |
          python eval/retrieval_eval.py \
            --dataset eval/golden_dataset.json \
            --k-values 3,5,10 \
            --output eval_results_retrieval.json

      - name: Run LLM judge evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python eval/llm_judge_eval.py \
            --dataset eval/golden_dataset.json \
            --judge-model gpt-4o-2024-11-20 \
            --output eval_results_judge.json

      - name: Run safety evaluation
        run: |
          python eval/safety_eval.py \
            --dataset eval/golden_dataset.json \
            --output eval_results_safety.json

      - name: Merge results and check regression
        run: |
          python eval/merge_results.py \
            eval_results_retrieval.json \
            eval_results_judge.json \
            eval_results_safety.json \
            --output eval_results.json

          python eval/regression_check.py eval_results.json

      - name: Upload evaluation report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-report
          path: eval_results*.json

Using the RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment) is an open-source framework for evaluating RAG. You can replace the metrics implemented by hand above with RAGAS, or complement them with it.

"""
An example of RAG evaluation using the RAGAS framework.
pip install ragas==0.2.6
"""
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

# prepare the evaluation data (RAGAS format)
eval_data = {
    "question": [
        "How many days does a refund take to process?",
        "Who pays the customs duty on an international shipment?",
    ],
    "answer": [
        "Refunds are processed within 3-5 business days.",
        "For international shipments, the customs duty and VAT are paid by the recipient.",
    ],
    "contexts": [
        ["A refund is processed within 3-5 business days from the refund request date. Weekends and public holidays do not count as business days."],
        ["The customs duty and VAT incurred on an international shipment are paid by the recipient. The duty amount varies with the tax rate of the importing country."],
    ],
    "ground_truth": [
        "Refunds are processed within 3-5 business days from the request date.",
        "For international shipments, customs duty and VAT are paid by the recipient.",
    ],
}

dataset = Dataset.from_dict(eval_data)

# run the RAGAS evaluation
results = evaluate(
    dataset=dataset,
    metrics=[
        faithfulness,        # is the answer faithful to the context
        answer_relevancy,    # is the answer relevant to the question
        context_precision,   # is the retrieved context precise
        context_recall,      # was all the relevant context retrieved
    ],
)

print(results)
# {'faithfulness': 0.95, 'answer_relevancy': 0.92,
#  'context_precision': 0.88, 'context_recall': 0.90}

Handling Each Failure Scenario

Scenario 1: Faithfulness Score Collapses

Symptom: faithfulness dropped from 0.91 to 0.72 in the offline evaluation
     User reports: "there are more answers with no basis"

Investigation order:
  1. Check whether the LLM model version changed
     -> it was updated from gpt-4o-2024-08-06 to gpt-4o-2024-11-20
  2. Check the grounding instruction in the system prompt
     -> the prompt was never tuned after the model update

Fix:
  1. Strengthen the system prompt: "Answer strictly on the basis of the retrieved
     documents. For anything not in the documents, answer 'I cannot find that information.'"
  2. Establish the rule that offline evaluation must be run before a model update
  3. Version the prompt per model (a model_version -> prompt_version mapping table)

Scenario 2: Empty-search-result Rate Spikes

Symptom: the rag_retrieval_empty_total counter is 5x its usual value
     Error logs: none (the search itself succeeds but returns 0 results)

Cause: after the embedding model update, the vector space of the new query
     embeddings differs from that of the existing document embeddings
     (cosine similarity dropped across the board)

Fix:
  1. Roll back to the previous embedding model immediately
  2. Re-embed every document whenever a new embedding model is adopted
  3. Add a "confirm reindexing complete" step to the embedding model change procedure

Scenario 3: LLM-as-a-Judge Evaluation Cost Spikes

Symptom: the monthly LLM Judge bill went from $3,000 to $12,000
Cause: the test set grew from 50 cases to 500, and it runs on every PR

Fix:
  1. Split the golden dataset into core (50 cases) + extended (450 cases)
  2. Run only core on a PR; include extended on a merge to main
  3. Cache the LLM Judge: cache the judgment for identical input for 30 days
  4. Filter first with a cheaper model (gpt-4o-mini) and re-judge only the failures with gpt-4o

References

• RAGAS official documentation - Metrics Reference
• DeepEval - LLM Evaluation Framework
• TruLens - Feedback Functions for LLM Apps
• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401)
• Es et al., "RAGAS: Automated Evaluation of Retrieval Augmented Generation" (arXiv:2309.15217)
• OpenAI Evals Framework
• OWASP Top 10 for LLM Applications