- Introduction
- Original RAG (Lewis et al., 2020)
- REALM and RETRO: Integrating Large-Scale Retrieval
- Atlas: Few-Shot Learning and Retrieval
- Self-RAG: Adaptive Retrieval Through Self-Reflection
- Corrective RAG (CRAG)
- From Naive RAG to Advanced RAG and Modular RAG
- Benchmark Comparison
- Practical Considerations
- Future Research Directions
- Conclusion
- References
Introduction
Large language models (LLMs) show remarkable language understanding and generation ability, but they carry two fundamental limits. First, hallucination: they produce content that is not factual but sounds plausible. Second, the knowledge cutoff of their training data keeps them from reflecting recent information. Storing knowledge in parameters does not scale indefinitely, and the cost of retraining a model is astronomical.
Retrieval-Augmented Generation (RAG) has emerged as the most practical answer to this problem. The core idea is simple. Given a question, retrieve relevant documents from an external knowledge store, then use them as context to generate the answer. That reflects up-to-date knowledge and reduces hallucination without modifying the model parameters.
This article traces the evolution of RAG research through its key papers. Starting from the Original RAG of Lewis et al. in 2020, it compares architectures and benchmarks across the large-scale retrieval integration of REALM and RETRO, the few-shot learning of Atlas, the self-reflection mechanism of Self-RAG, and the retrieval-quality evaluation of Corrective-RAG.
Original RAG (Lewis et al., 2020)
Architecture Overview
Lewis et al. presented "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" at NeurIPS 2020, the origin of the RAG paradigm. Its core structure combines a DPR (Dense Passage Retrieval) retriever with a BART seq2seq generator.
The model draws on two kinds of memory.
- Parametric Memory: knowledge stored in BART's pretrained parameters
- Non-parametric Memory: an external knowledge store built by indexing a Wikipedia dump with FAISS
RAG-Sequence vs RAG-Token
The paper proposes two model variants.
- RAG-Sequence: uses the same document for the entire sequence. Given a document z, the whole output y is generated in one pass
- RAG-Token: can reference a different document for each token. The document distribution is recomputed at every token generation step
Implementing DPR-Based Document Retrieval
import torch
import numpy as np
from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer
from transformers import DPRContextEncoder, DPRContextEncoderTokenizer
class DPRRetriever:
"""Dense Passage Retrieval implementation based on DPR"""
def __init__(self, model_name="facebook/dpr-question_encoder-single-nq-base"):
self.q_encoder = DPRQuestionEncoder.from_pretrained(model_name)
self.q_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained(model_name)
ctx_model = "facebook/dpr-ctx_encoder-single-nq-base"
self.ctx_encoder = DPRContextEncoder.from_pretrained(ctx_model)
self.ctx_tokenizer = DPRContextEncoderTokenizer.from_pretrained(ctx_model)
self.document_embeddings = None
self.documents = []
def encode_documents(self, documents: list[str]) -> np.ndarray:
"""Convert a document corpus into embeddings"""
self.documents = documents
embeddings = []
for doc in documents:
inputs = self.ctx_tokenizer(
doc, return_tensors="pt",
max_length=256, truncation=True, padding=True
)
with torch.no_grad():
output = self.ctx_encoder(**inputs)
embeddings.append(output.pooler_output.numpy())
self.document_embeddings = np.vstack(embeddings)
# Apply L2 normalization
norms = np.linalg.norm(self.document_embeddings, axis=1, keepdims=True)
self.document_embeddings = self.document_embeddings / norms
return self.document_embeddings
def retrieve(self, query: str, top_k: int = 5) -> list[dict]:
"""Retrieve the top k relevant documents for a query"""
inputs = self.q_tokenizer(
query, return_tensors="pt",
max_length=64, truncation=True, padding=True
)
with torch.no_grad():
q_embedding = self.q_encoder(**inputs).pooler_output.numpy()
q_embedding = q_embedding / np.linalg.norm(q_embedding)
scores = np.dot(self.document_embeddings, q_embedding.T).squeeze()
top_indices = np.argsort(scores)[::-1][:top_k]
results = []
for idx in top_indices:
results.append({
"document": self.documents[idx],
"score": float(scores[idx]),
"index": int(idx)
})
return results
# Usage example
retriever = DPRRetriever()
corpus = [
"RAG is a model that combines retrieval and generation.",
"Transformer uses the Self-Attention mechanism.",
"BERT is a bidirectional pretrained language model.",
"DPR uses dense vectors to retrieve passages.",
]
retriever.encode_documents(corpus)
results = retriever.retrieve("How does document retrieval work in RAG?")
for r in results:
print(f"[Score: {r['score']:.4f}] {r['document']}")
Original RAG reached 44.5 EM on Natural Questions and 56.8 EM on TriviaQA, proving the promise of a generative approach against the extractive QA methods of the time.
REALM and RETRO: Integrating Large-Scale Retrieval
REALM: Retrieval at the Pretraining Stage
REALM (Retrieval-Enhanced Language Model) by Guu et al. (2020) came one step ahead of RAG and was the first work to integrate retrieval from the pretraining stage onward. During Masked Language Modeling it retrieves external documents in order to predict the masked tokens, and that retrieval process is trained along with the model through backpropagation.
Its key contribution was showing that a retriever and a generator can be jointly trained end-to-end.
RETRO: A 2-Trillion-Token Database
RETRO (Retrieval-Enhanced Transformer) by Borgeaud et al. (2022) scaled retrieval up dramatically. It builds a database on the order of 2 trillion tokens and introduces the Chunked Cross-Attention (CCA) mechanism to use the retrieved chunks efficiently.
RETRO's core design principles are as follows.
| Property | RETRO | GPT-3 |
|---|---|---|
| Parameter count | 7.5B | 175B |
| Retrieval database | 2T tokens | None |
| Pile test perplexity | Comparable | Baseline |
| Training cost | Relatively low | High |
It matched the performance of GPT-3 with roughly 25x fewer parameters. That is empirical evidence that not all knowledge has to be stored in the parameters.
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
class ChunkedCrossAttention(nn.Module):
"""RETRO-style Chunked Cross-Attention implementation"""
def __init__(self, d_model: int = 512, n_heads: int = 8, chunk_size: int = 64):
super().__init__()
self.d_model = d_model
self.n_heads = n_heads
self.d_k = d_model // n_heads
self.chunk_size = chunk_size
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
self.layer_norm = nn.LayerNorm(d_model)
def forward(
self,
hidden_states: torch.Tensor,
retrieved_chunks: torch.Tensor
) -> torch.Tensor:
"""
Args:
hidden_states: (B, seq_len, d_model) - decoder hidden states
retrieved_chunks: (B, n_chunks, chunk_len, d_model) - retrieved neighbor chunks
"""
B, seq_len, D = hidden_states.shape
n_chunks = seq_len // self.chunk_size
# Split the sequence into chunks
h_chunks = hidden_states[:, :n_chunks * self.chunk_size].reshape(
B, n_chunks, self.chunk_size, D
)
# Cross-attend each chunk to its retrieved neighbors
Q = self.W_q(h_chunks) # (B, n_chunks, chunk_size, D)
K = self.W_k(retrieved_chunks) # (B, n_chunks, chunk_len, D)
V = self.W_v(retrieved_chunks)
# Split into multiple heads
Q = Q.reshape(B, n_chunks, self.chunk_size, self.n_heads, self.d_k).permute(0, 1, 3, 2, 4)
K = K.reshape(B, n_chunks, -1, self.n_heads, self.d_k).permute(0, 1, 3, 2, 4)
V = V.reshape(B, n_chunks, -1, self.n_heads, self.d_k).permute(0, 1, 3, 2, 4)
# Scaled Dot-Product Attention
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
attn_weights = F.softmax(scores, dim=-1)
attn_output = torch.matmul(attn_weights, V)
# Merge the heads and project the output
attn_output = attn_output.permute(0, 1, 3, 2, 4).reshape(
B, n_chunks, self.chunk_size, D
)
attn_output = self.W_o(attn_output)
# Residual connection and layer normalization
output = self.layer_norm(h_chunks + attn_output)
output = output.reshape(B, n_chunks * self.chunk_size, D)
# Restore the leftover tokens (when the length is not divisible by chunk_size)
if seq_len > n_chunks * self.chunk_size:
remainder = hidden_states[:, n_chunks * self.chunk_size:]
output = torch.cat([output, remainder], dim=1)
return output
# RETRO-style retrieval pipeline example
cca = ChunkedCrossAttention(d_model=512, n_heads=8, chunk_size=64)
hidden = torch.randn(2, 256, 512) # batch 2, sequence 256
retrieved = torch.randn(2, 4, 32, 512) # 4 chunks, 32 tokens each
output = cca(hidden, retrieved)
print(f"Input shape: {hidden.shape} -> Output shape: {output.shape}")
Atlas: Few-Shot Learning and Retrieval
Atlas by Izacard et al. (2023) combined a Contriever retriever with a Fusion-in-Decoder (FiD) generator. Its key finding is that when retrieval quality is high enough, a model can compete with large models even after cutting the parameter count drastically.
The Atlas 11B model beat PaLM 540B on Natural Questions with only 64 examples (64-shot). A model 50x smaller in parameter count can beat a large one when its retrieval mechanism is good enough.
| Model | Parameters | NQ (64-shot) | TriviaQA (64-shot) |
|---|---|---|---|
| PaLM | 540B | 39.6 | 81.4 |
| Atlas | 11B | 42.4 | 84.7 |
| Chinchilla | 70B | 35.5 | 72.3 |
The notable part of the Atlas training strategy is Attention Distillation. Fine-tuning the retriever with the generator's cross-attention distribution creates a virtuous cycle in which retriever and generator reinforce each other.
Self-RAG: Adaptive Retrieval Through Self-Reflection
The ICLR 2024 Oral Paper
Self-RAG (Self-Reflective Retrieval-Augmented Generation) by Asai et al. (2023) was selected for an Oral presentation at ICLR 2024, roughly the top 1%. It takes on a fundamental limitation of conventional RAG head-on. Conventional pipelines always retrieve, whatever the question type, and for simple commonsense questions or creative tasks the unnecessary retrieval can actually degrade performance.
The Reflection Token Mechanism
The core innovation of Self-RAG is a set of 4 reflection tokens.
| Reflection token | Role | Output values |
|---|---|---|
| Retrieve | Decide whether retrieval is needed | Yes, No, Continue |
| ISREL | Judge the relevance of a retrieved document | Relevant, Irrelevant |
| ISSUP | Whether the generated content is sufficiently grounded | Fully Supported, Partially Supported, No Support |
| ISUSE | Overall usefulness of the response | 1-5 |
The model emits these tokens itself during generation, judging on its own whether to retrieve, how relevant the documents are, and how good the response is.
Performance Comparison
Self-RAG shows an overwhelming improvement over the existing methods.
| Model | PopQA | Bio | ASQA (EM) |
|---|---|---|---|
| Llama2-7B | 14.7 | 31.6 | 21.9 |
| Llama2 + RAG | 38.2 | 36.7 | 25.3 |
| Self-RAG (7B) | 55.8 | 51.5 | 30.1 |
| ChatGPT | 29.3 | 41.2 | 27.8 |
On PopQA it achieved more than a 270% improvement over Llama2 and more than a 90% improvement over ChatGPT.
from dataclasses import dataclass
from enum import Enum
from typing import Optional
class RetrieveDecision(Enum):
YES = "yes"
NO = "no"
CONTINUE = "continue"
class RelevanceScore(Enum):
RELEVANT = "relevant"
IRRELEVANT = "irrelevant"
class SupportScore(Enum):
FULLY_SUPPORTED = "fully_supported"
PARTIALLY_SUPPORTED = "partially_supported"
NO_SUPPORT = "no_support"
@dataclass
class ReflectionResult:
retrieve: RetrieveDecision
relevance: Optional[RelevanceScore] = None
support: Optional[SupportScore] = None
utility: Optional[int] = None # 1-5 score
class SelfRAGPipeline:
"""Self-RAG-style adaptive retrieve-and-generate pipeline"""
def __init__(self, generator, retriever, reflection_model):
self.generator = generator
self.retriever = retriever
self.reflection_model = reflection_model
def should_retrieve(self, query: str, partial_output: str = "") -> RetrieveDecision:
"""Reflection step that decides for itself whether retrieval is needed"""
prompt = (
f"Query: {query}\n"
f"Partial output: {partial_output}\n"
"Does this query require external knowledge retrieval? "
"Answer: yes, no, or continue"
)
decision = self.reflection_model.predict(prompt)
return RetrieveDecision(decision.strip().lower())
def evaluate_relevance(self, query: str, document: str) -> RelevanceScore:
"""Evaluate the relevance of a retrieved document (simulates the ISREL token)"""
prompt = (
f"Query: {query}\n"
f"Document: {document}\n"
"Is this document relevant to answering the query? "
"Answer: relevant or irrelevant"
)
score = self.reflection_model.predict(prompt)
return RelevanceScore(score.strip().lower())
def evaluate_support(
self, query: str, document: str, response: str
) -> SupportScore:
"""Evaluate how well the output is grounded (simulates the ISSUP token)"""
prompt = (
f"Query: {query}\n"
f"Document: {document}\n"
f"Response: {response}\n"
"Is the response supported by the document? "
"Answer: fully_supported, partially_supported, or no_support"
)
score = self.reflection_model.predict(prompt)
return SupportScore(score.strip().lower())
def generate_with_reflection(self, query: str) -> dict:
"""Run the full Self-RAG pipeline"""
# Step 1: decide whether retrieval is needed
retrieve_decision = self.should_retrieve(query)
if retrieve_decision == RetrieveDecision.NO:
# Retrieval unnecessary - generate directly
response = self.generator.generate(query)
return {
"response": response,
"retrieved": False,
"reflection": ReflectionResult(retrieve=RetrieveDecision.NO)
}
# Step 2: retrieve documents
documents = self.retriever.retrieve(query, top_k=5)
# Step 3: filter documents by relevance
relevant_docs = []
for doc in documents:
relevance = self.evaluate_relevance(query, doc["text"])
if relevance == RelevanceScore.RELEVANT:
relevant_docs.append(doc)
if not relevant_docs:
# No relevant documents - generate without retrieval
response = self.generator.generate(query)
return {
"response": response,
"retrieved": True,
"relevant_docs": 0,
"reflection": ReflectionResult(
retrieve=RetrieveDecision.YES,
relevance=RelevanceScore.IRRELEVANT
)
}
# Step 4: generate and evaluate candidate responses
best_response = None
best_score = -1
for doc in relevant_docs:
context = f"Context: {doc['text']}\nQuery: {query}"
candidate = self.generator.generate(context)
support = self.evaluate_support(query, doc["text"], candidate)
# Compute the support score
support_score = {
SupportScore.FULLY_SUPPORTED: 3,
SupportScore.PARTIALLY_SUPPORTED: 1,
SupportScore.NO_SUPPORT: 0
}.get(support, 0)
if support_score > best_score:
best_score = support_score
best_response = candidate
best_support = support
return {
"response": best_response,
"retrieved": True,
"relevant_docs": len(relevant_docs),
"reflection": ReflectionResult(
retrieve=RetrieveDecision.YES,
relevance=RelevanceScore.RELEVANT,
support=best_support,
utility=min(best_score + 2, 5)
)
}
Corrective RAG (CRAG)
Introducing a Retrieval Quality Evaluator
Corrective RAG (CRAG) by Yan et al. (2024) attacks another weak point of conventional RAG. Conventional pipelines use the retrieved documents as they are, without verifying whether they are actually useful. When retrieval quality is low, the inaccurate context can make hallucination worse instead of better.
CRAG introduces a lightweight retrieval evaluator that scores the confidence of the retrieval result quantitatively and triggers one of three actions depending on that score.
| Verdict | Confidence condition | Action |
|---|---|---|
| Correct | High confidence | Refine the key knowledge from the retrieved documents and use it |
| Incorrect | Low confidence | Switch to an alternative knowledge source such as web search |
| Ambiguous | Medium confidence | Combine the refined retrieval results with web search results |
The Decompose-then-Recompose Algorithm
CRAG's other key contribution is the Decompose-then-Recompose algorithm. It removes irrelevant information from the retrieved documents, extracts only the essential knowledge, and reassembles it.
- Decompose the retrieved document into fine-grained knowledge strips
- Evaluate the relevance of each strip individually
- Select only the relevant strips and recombine them
- Generate the final response from the recombined context
from dataclasses import dataclass
from enum import Enum
import numpy as np
class ConfidenceLevel(Enum):
CORRECT = "correct"
INCORRECT = "incorrect"
AMBIGUOUS = "ambiguous"
@dataclass
class EvaluationResult:
confidence: ConfidenceLevel
score: float
action: str
class CRAGPipeline:
"""Corrective RAG-style pipeline implementation"""
def __init__(
self,
retriever,
evaluator,
generator,
web_searcher,
upper_threshold: float = 0.7,
lower_threshold: float = 0.3
):
self.retriever = retriever
self.evaluator = evaluator
self.generator = generator
self.web_searcher = web_searcher
self.upper_threshold = upper_threshold
self.lower_threshold = lower_threshold
def evaluate_retrieval(self, query: str, documents: list[dict]) -> EvaluationResult:
"""Score the confidence of the retrieval result"""
scores = []
for doc in documents:
score = self.evaluator.score(query, doc["text"])
scores.append(score)
max_score = max(scores) if scores else 0.0
if max_score >= self.upper_threshold:
return EvaluationResult(
confidence=ConfidenceLevel.CORRECT,
score=max_score,
action="refine_and_use"
)
elif max_score <= self.lower_threshold:
return EvaluationResult(
confidence=ConfidenceLevel.INCORRECT,
score=max_score,
action="web_search_fallback"
)
else:
return EvaluationResult(
confidence=ConfidenceLevel.AMBIGUOUS,
score=max_score,
action="combine_sources"
)
def decompose_then_recompose(
self, query: str, document: str
) -> str:
"""Decompose-then-Recompose: extract only the relevant knowledge from a document"""
# Step 1: decompose the document into fine-grained knowledge strips
sentences = document.split(". ")
knowledge_strips = [s.strip() + "." for s in sentences if s.strip()]
# Step 2: evaluate the relevance of each knowledge strip
relevant_strips = []
for strip in knowledge_strips:
relevance = self.evaluator.score(query, strip)
if relevance > 0.5:
relevant_strips.append((strip, relevance))
# Step 3: sort by relevance and recombine
relevant_strips.sort(key=lambda x: x[1], reverse=True)
refined_context = " ".join([s[0] for s in relevant_strips])
return refined_context if refined_context else document
def process_query(self, query: str) -> dict:
"""Run the full CRAG pipeline"""
# Step 1: initial document retrieval
documents = self.retriever.retrieve(query, top_k=10)
# Step 2: evaluate retrieval quality
evaluation = self.evaluate_retrieval(query, documents)
context = ""
sources = []
if evaluation.confidence == ConfidenceLevel.CORRECT:
# Retrieval trusted - refine the key knowledge and use it
for doc in documents[:3]:
refined = self.decompose_then_recompose(query, doc["text"])
context += refined + "\n"
sources = ["internal_retrieval"]
elif evaluation.confidence == ConfidenceLevel.INCORRECT:
# Retrieval not trusted - fall back to web search
web_results = self.web_searcher.search(query)
for result in web_results[:3]:
context += result["snippet"] + "\n"
sources = ["web_search"]
else: # AMBIGUOUS
# Combine both sources
for doc in documents[:2]:
refined = self.decompose_then_recompose(query, doc["text"])
context += refined + "\n"
web_results = self.web_searcher.search(query)
for result in web_results[:2]:
context += result["snippet"] + "\n"
sources = ["internal_retrieval", "web_search"]
# Step 3: generate the final response
prompt = f"Context: {context}\nQuery: {query}\nAnswer:"
response = self.generator.generate(prompt)
return {
"response": response,
"confidence": evaluation.confidence.value,
"score": evaluation.score,
"sources": sources
}
From Naive RAG to Advanced RAG and Modular RAG
The survey paper "Retrieval-Augmented Generation for Large Language Models: A Survey" by Gao et al. (2024) classifies the development of RAG into three stages.
Architecture Evolution Comparison
| Dimension | Naive RAG | Advanced RAG | Modular RAG |
|---|---|---|---|
| Period | 2020~2022 | 2022~2023 | 2023~ |
| Retrieval strategy | Plain similarity search | Query rewriting, HyDE | Adaptive retrieval, routing |
| Chunking | Fixed size | Semantic chunking | Hierarchical, recursive chunking |
| Post-retrieval | None | Re-ranking, compression | Self-reflection, correction |
| Limits | Low retrieval precision, hallucination | Pipeline complexity | Design-space explosion |
| Representative models | RAG (Lewis) | RETRO, Atlas | Self-RAG, CRAG |
Pre-retrieval, Retrieval, and Post-retrieval Optimization
Since Advanced RAG, a variety of optimization techniques have appeared for each stage.
Pre-retrieval optimization:
- Query Rewriting: turn the original question into a form optimized for retrieval
- HyDE (Hypothetical Document Embeddings): generate a hypothetical document first, then use it as the search query
- Step-back Prompting: convert the question into an abstract one to search more broadly
Retrieval optimization:
- Hybrid Search: combine BM25 (sparse) with vector search (dense)
- Multi-vector retrieval: ColBERT-style token-level interaction
- Recursive retrieval: search repeatedly on the basis of the initial results
Post-retrieval optimization:
- Re-ranking: reorder the retrieval results with a Cross-Encoder
- Context compression: remove unnecessary information
- Self-RAG / CRAG: self-reflection and correction
Benchmark Comparison
Combined Performance Comparison of the Major Models
| Model | Type | NQ (EM) | TriviaQA (EM) | PopQA (F1) | FEVER (Acc) |
|---|---|---|---|---|---|
| RAG (Lewis, 2020) | Naive | 44.5 | 56.8 | - | - |
| REALM (Guu, 2020) | Pre-train | 40.4 | - | - | - |
| RETRO (Borgeaud, 2022) | Pre-train | - | - | - | - |
| Atlas-11B (Izacard, 2023) | Few-shot | 42.4 | 84.7 | - | - |
| Self-RAG-7B (Asai, 2023) | Adaptive | - | - | 55.8 | - |
| CRAG (Yan, 2024) | Corrective | - | - | - | - |
A direct comparison on identical benchmarks is difficult because each paper uses a different evaluation setup (number of shots, retrieval corpus size, model size). The overall trend is clear all the same. Performance improves as adaptive retrieval and self-reflection mechanisms are introduced.
CRAG Benchmark (Meta, NeurIPS 2024)
The CRAG Benchmark, presented by Meta at NeurIPS 2024, systematically evaluates RAG systems across 8 domains and a range of question types.
| Approach | Overall accuracy | Hallucination rate |
|---|---|---|
| Pure LLM (no retrieval) | 34% | High |
| Naive RAG | 44% | Medium |
| Advanced RAG | 55% | Low |
| SOTA RAG system | 63% | Very low |
The result suggests two things. First, RAG delivers a clear improvement over a pure LLM (34% vs 44%). Second, moving from naive RAG to advanced RAG can add another 20%p or more.
Practical Considerations
Choosing a Retriever: Dense vs Sparse vs Hybrid
In practice the choice of retriever depends on the characteristics of the data and the requirements.
| Retrieval method | Strengths | Weaknesses | Good fit for |
|---|---|---|---|
| Sparse (BM25) | Accurate keyword matching, fast | Does not capture semantic similarity | Technical terms, code search |
| Dense (vector) | Captures semantic similarity | Can miss on keywords | General QA, conversational search |
| Hybrid | Combines the strengths of both | Complex to implement, needs weight tuning | Production systems |
Implementing a Hybrid Retrieval Pipeline
import numpy as np
from dataclasses import dataclass, field
from typing import Optional
import re
from collections import Counter
import math
@dataclass
class Document:
text: str
doc_id: str
metadata: dict = field(default_factory=dict)
@dataclass
class SearchResult:
document: Document
score: float
source: str # "sparse", "dense", or "hybrid"
class BM25Retriever:
"""Simplified BM25 sparse retriever implementation"""
def __init__(self, k1: float = 1.5, b: float = 0.75):
self.k1 = k1
self.b = b
self.documents: list[Document] = []
self.doc_freqs: dict[str, int] = {}
self.doc_lengths: list[int] = []
self.avg_doc_length: float = 0
self.doc_term_freqs: list[dict[str, int]] = []
def _tokenize(self, text: str) -> list[str]:
return re.findall(r'\w+', text.lower())
def index(self, documents: list[Document]):
self.documents = documents
for doc in documents:
tokens = self._tokenize(doc.text)
self.doc_lengths.append(len(tokens))
term_freq = Counter(tokens)
self.doc_term_freqs.append(term_freq)
for term in set(tokens):
self.doc_freqs[term] = self.doc_freqs.get(term, 0) + 1
self.avg_doc_length = (
sum(self.doc_lengths) / len(self.doc_lengths) if self.doc_lengths else 0
)
def search(self, query: str, top_k: int = 10) -> list[SearchResult]:
query_tokens = self._tokenize(query)
n_docs = len(self.documents)
scores = []
for i, doc in enumerate(self.documents):
score = 0.0
for term in query_tokens:
if term not in self.doc_term_freqs[i]:
continue
tf = self.doc_term_freqs[i][term]
df = self.doc_freqs.get(term, 0)
idf = math.log((n_docs - df + 0.5) / (df + 0.5) + 1)
dl = self.doc_lengths[i]
numerator = tf * (self.k1 + 1)
denominator = tf + self.k1 * (
1 - self.b + self.b * dl / self.avg_doc_length
)
score += idf * numerator / denominator
scores.append(score)
top_indices = np.argsort(scores)[::-1][:top_k]
return [
SearchResult(
document=self.documents[i],
score=float(scores[i]),
source="sparse"
)
for i in top_indices if scores[i] > 0
]
class DenseRetriever:
"""Dense Vector Retriever (embedding-based)"""
def __init__(self, embedding_fn):
self.embedding_fn = embedding_fn
self.documents: list[Document] = []
self.embeddings: Optional[np.ndarray] = None
def index(self, documents: list[Document]):
self.documents = documents
texts = [doc.text for doc in documents]
self.embeddings = self.embedding_fn(texts)
# L2 normalization
norms = np.linalg.norm(self.embeddings, axis=1, keepdims=True)
self.embeddings = self.embeddings / (norms + 1e-10)
def search(self, query: str, top_k: int = 10) -> list[SearchResult]:
q_emb = self.embedding_fn([query])
q_emb = q_emb / (np.linalg.norm(q_emb) + 1e-10)
scores = np.dot(self.embeddings, q_emb.T).squeeze()
top_indices = np.argsort(scores)[::-1][:top_k]
return [
SearchResult(
document=self.documents[i],
score=float(scores[i]),
source="dense"
)
for i in top_indices
]
class HybridRetriever:
"""Hybrid Retrieval: combine BM25 and dense search"""
def __init__(
self,
sparse: BM25Retriever,
dense: DenseRetriever,
alpha: float = 0.5
):
self.sparse = sparse
self.dense = dense
self.alpha = alpha # Dense weight (1-alpha = sparse weight)
def _normalize_scores(self, results: list[SearchResult]) -> dict[str, float]:
"""Min-Max normalization"""
if not results:
return {}
scores = [r.score for r in results]
min_s, max_s = min(scores), max(scores)
range_s = max_s - min_s if max_s != min_s else 1.0
return {
r.document.doc_id: (r.score - min_s) / range_s
for r in results
}
def search(self, query: str, top_k: int = 10) -> list[SearchResult]:
"""Hybrid search based on Reciprocal Rank Fusion"""
sparse_results = self.sparse.search(query, top_k=top_k * 2)
dense_results = self.dense.search(query, top_k=top_k * 2)
sparse_scores = self._normalize_scores(sparse_results)
dense_scores = self._normalize_scores(dense_results)
# Collect every unique document
all_doc_ids = set(sparse_scores.keys()) | set(dense_scores.keys())
doc_map = {}
for r in sparse_results + dense_results:
doc_map[r.document.doc_id] = r.document
# Weighted combination
hybrid_scores = {}
for doc_id in all_doc_ids:
s_score = sparse_scores.get(doc_id, 0.0)
d_score = dense_scores.get(doc_id, 0.0)
hybrid_scores[doc_id] = (
(1 - self.alpha) * s_score + self.alpha * d_score
)
# Sort and return the top k
sorted_docs = sorted(
hybrid_scores.items(), key=lambda x: x[1], reverse=True
)[:top_k]
return [
SearchResult(
document=doc_map[doc_id],
score=score,
source="hybrid"
)
for doc_id, score in sorted_docs
]
# Usage example
bm25 = BM25Retriever()
docs = [
Document("RAG combines retrieval and generation.", "doc1"),
Document("Self-RAG uses reflection tokens.", "doc2"),
Document("CRAG evaluates retrieval quality.", "doc3"),
Document("RETRO uses a 2 trillion token database.", "doc4"),
]
bm25.index(docs)
sparse_results = bm25.search("How is retrieval quality evaluated in RAG?")
for r in sparse_results:
print(f"[BM25 Score: {r.score:.4f}] {r.document.text}")
Chunking Strategy and the Cost-Performance Trade-off
Chunking has a decisive effect on RAG performance.
| Chunking strategy | Chunk size | Strengths | Weaknesses |
|---|---|---|---|
| Fixed size | 256~512 tokens | Simple to implement | Breaks context |
| Sentence-based | 3~5 sentences | Natural boundaries | Uneven sizes |
| Semantic | Variable | Keeps topical coherence | Embedding cost |
| Recursive | Hierarchical | Multi-level retrieval | Complex to implement |
class SemanticChunker:
"""Semantic chunking: detect natural boundaries via embedding similarity"""
def __init__(self, embedding_fn, similarity_threshold: float = 0.75):
self.embedding_fn = embedding_fn
self.threshold = similarity_threshold
def chunk(self, text: str, min_chunk_size: int = 100) -> list[str]:
"""Split where the semantic similarity between sentences changes"""
sentences = [s.strip() for s in text.split(". ") if s.strip()]
if len(sentences) <= 1:
return [text]
# Compute the embedding of each sentence
embeddings = self.embedding_fn(sentences)
# Compute cosine similarity between adjacent sentences
chunks = []
current_chunk = [sentences[0]]
for i in range(1, len(sentences)):
sim = np.dot(embeddings[i], embeddings[i - 1]) / (
np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i - 1])
+ 1e-10
)
if sim < self.threshold and len(". ".join(current_chunk)) >= min_chunk_size:
# Similarity below the threshold starts a new chunk
chunks.append(". ".join(current_chunk) + ".")
current_chunk = [sentences[i]]
else:
current_chunk.append(sentences[i])
if current_chunk:
chunks.append(". ".join(current_chunk) + ".")
return chunks
Future Research Directions
Agentic RAG: Combining Tool Use with Retrieval
The direction drawing the most attention lately is Agentic RAG. Beyond simply retrieving documents, an LLM agent actively gathers the information it needs using a range of tools: API calls, database queries, code execution. Retrieval itself becomes one tool among many, and the agent picks the best action for the situation among retrieval, computation, and API calls.
Multi-modal RAG: Retrieving Images and Tables
Multi-modal RAG, which retrieves and uses images, tables, graphs and other modalities rather than text alone, is also under active research. Typical scenarios are retrieving an architecture diagram from technical documentation, or parsing a table in a financial report to answer a numeric question. Vision-language model based retrievers such as ColPali are the representative work in this direction.
Real-Time Knowledge Updates
In production RAG systems, real-time updates to the knowledge store remain an unsolved problem. How to refresh the embedding index efficiently when documents are added, modified or deleted, along with version management and consistency, are the core research topics. Streaming indexing and incremental update techniques are drawing attention.
Conclusion
The evolution of RAG shows a shift from plain "retrieve then generate" to intelligent, adaptive use of knowledge. The key trajectory can be summarized as follows.
- Original RAG (2020): proved that retrieval and generation can be combined
- RETRO (2022): maximized parameter efficiency through large-scale retrieval
- Atlas (2023): demonstrated that retrieval quality can substitute for model size
- Self-RAG (2023): made retrieval itself selective and secured quality through self-reflection
- CRAG (2024): evaluated the confidence of retrieval results and corrected with alternative sources
In practice the key is to combine the ideas from these papers selectively. A simple internal QA system may do fine with Naive RAG plus BM25, but in the medical and legal domains, where high accuracy is required, the reflection mechanism of Self-RAG or the quality-evaluation technique of CRAG is essential.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - Lewis et al., 2020
- Improving Language Models by Retrieving from Trillions of Tokens (RETRO) - Borgeaud et al., 2022
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection - Asai et al., 2023
- Corrective Retrieval Augmented Generation - Yan et al., 2024
- Retrieval-Augmented Generation for Large Language Models: A Survey - Gao et al., 2024
- CRAG Benchmark - Meta, 2024
- REALM: Retrieval-Enhanced Language Model Pre-Training - Guu et al., 2020
- Atlas: Few-shot Learning with Retrieval Augmented Language Models - Izacard et al., 2023