LabHub

Blog

Elasticsearch, OpenSearch, and Lucene Internals — Inverted Index, BM25, Sharding, Vector Search, Hybrid RAG (2025)

2026-04-15 · 11 min read · #elasticsearch#opensearch#lucene#search#vector-search#rag#culture

한국어English日本語

"Search is not a feature. It's a philosophy of how humans interact with information." — Doug Cutting (creator of Lucene, 1999)

When Doug Cutting built Lucene in Java in 1999, he believed "anyone should be able to add Google-class search to their own data." 26 years later, Elasticsearch, OpenSearch, and Solr all stand on Lucene. Log analytics, product search, autocomplete — and since 2024, the core infrastructure of RAG (Retrieval-Augmented Generation).

But "using Elasticsearch" and "understanding Lucene" are worlds apart. This article is a map from the fundamentals of search to the hybrid search of 2025.

1. Why LIKE '%keyword%' in an RDB Doesn't Work

The Linear Scan Wall

SELECT * FROM articles WHERE content LIKE '%postgresql%';

Postgres GIN helps partially, but:

This is why dedicated search engines exist.

2. Inverted Index — The Mathematical Heart of Search

The Basic Idea

Document 1: "The quick brown fox"
Document 2: "The lazy brown dog"
Document 3: "Foxes and dogs"

Flipped to word to document list:

brown  -> [1, 2]
dog    -> [2]
dogs   -> [3]
fox    -> [1]
foxes  -> [3]
lazy   -> [2]
quick  -> [1]
the    -> [1, 2]

Query "brown dog":

This is the essence of the Inverted Index. Close to O(log⁡N)O(\log N) even across billions of documents.

Lucene's On-Disk Units

  1. Term Dictionary — sorted dictionary of all terms (FST, Finite State Transducer)
  2. Postings List — document list per term + frequency/position
  3. Stored Fields — original documents (compressed)
  4. Doc Values — columnar storage for aggregations/sorting
  5. Norms — field length normalization values

FST — Prefix Sharing for Memory Savings

Common prefixes of terms like "fox, foxes, foxy" are compressed with a finite state transducer. Tens of millions of terms fit in a few MB. Also the core structure behind autocomplete.

3. Segment — Lucene's Immutability Principle

Immutable Segments

In Lucene, once a file is written it never changes. That's what makes Lucene fast and safe.

Segment Files

_0.cfs     — composite file
_0.cfe     — entry point
_0.si      — segment info
_0.fdt/fdx — field data
_0.tim/tip — term dictionary
_0.doc/pos — postings
_0.dvm/dvd — doc values
_0.liv     — live docs (delete bitmap)

Merge Policy

Refresh vs Flush vs Commit

TermMeaningTiming
RefreshTurn in-memory buffer into a searchable Segmentevery 1s by default
Flushfsync the segment to diskautomatic (memory threshold)
CommitFull durability including translogless frequent

The secret of "Near Real-Time Search": Refresh every 1s, Flush later. The "1-second lag" is Elasticsearch's trademark.

Translog

4. BM25 — Replacing TF-IDF

Limits of TF-IDF

tf-idf(t,d)=tf(t,d)×log⁡Ndf(t)\text{tf-idf}(t, d) = \text{tf}(t, d) \times \log\frac{N}{\text{df}(t)}

BM25 Formula

score(d,q)=∑t∈qIDF(t)⋅f(t,d)⋅(k1+1)f(t,d)+k1⋅(1−b+b⋅∣d∣avgdl)\text{score}(d, q) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t, d) \cdot (k_1 + 1)}{f(t, d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\text{avgdl}})}

Key changes:

Why BM25 Is the Default

Lower b (around 0.3) is common for product names or query-log fields.

5. Analyzer — The Art of Tokenization

Three-Stage Pipeline

  1. Character Filter — strip HTML, char replacement
  2. Tokenizer — split into words
  3. Token Filter — lowercase, stemming, synonyms, stopwords

The Korean Hell

English: whitespace tokenization is easy.

Korean: agglutinative, conjugated endings, particles. "검색했다/검색한다/검색은/검색을" must all match "검색". Use nori (Korean), kuromoji (Japanese), ik (Chinese).

Example: nori analysis

POST _analyze
{
  "analyzer": "nori",
  "text": "Elasticsearch는 검색엔진입니다"
}

-> "elasticsearch", "는", "검색", "엔진", "입니다"

Remove particles/endings with "filter": ["nori_part_of_speech"].

Synonym Expansion

"shoe, sneaker, runner"
-> query "shoe" also matches sneaker/runner docs

Half of search quality lives in the synonym dictionary. Tedious but highest ROI.

6. Shard & Replica — The Basics of Distribution

Primary Shard

Replica Shard

Limits and Rules

Cluster State and Split Brain

7. Query DSL — The JSON Maze

Main Query Types

QueryUse
matchAnalyzed full-text matching
termExact match without analysis (keyword fields)
match_phraseOrder-preserving phrase
multi_matchSearch across multiple fields
boolAND/OR/NOT composition
rangeRange
function_scoreCustom scoring
rank_featureBoosting field

Four Clauses of bool

{
  "bool": {
    "must": [],
    "should": [],
    "filter": [],
    "must_not": []
  }
}

Use filter aggressively — it's cached and fast. Score with match; make fixed conditions filter.

Term vs Match — The Most Common Mistake

{"term": {"name": "User Name"}}
{"match": {"name": "User Name"}}

match for text fields, term for keyword fields.

Aggregations

{
  "aggs": {
    "by_category": {
      "terms": {"field": "category"},
      "aggs": {
        "avg_price": {"avg": {"field": "price"}}
      }
    }
  }
}

The SQL GROUP BY equivalent — essential for log analysis and dashboards.

8. Vector Search — The Revolution After 2022

Why Vectors

ES/OpenSearch kNN

Native kNN since Elasticsearch 8.0 (2022), built on Lucene 9.0 HNSW.

{
  "mappings": {
    "properties": {
      "title_vector": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

{
  "knn": {
    "field": "title_vector",
    "query_vector": [0.1, 0.2],
    "k": 10,
    "num_candidates": 100
  }
}

HNSW Params

Quantization

9. Hybrid Search — The Answer for the RAG Era

BM25 vs Vector

AspectBM25Vector
Exact match (SKUs)StrongWeak
Semantic similarityWeakStrong
Rare termsStrongWeak
TyposWeakMedium
MultilingualWeakStrong

You need both.

RRF — Reciprocal Rank Fusion

RRF(d)=∑i1k+ranki(d)\text{RRF}(d) = \sum_i \frac{1}{k + \text{rank}_i(d)}

ES rrf retriever

{
  "retriever": {
    "rrf": {
      "retrievers": [
        {"standard": {"query": {"match": {"content": "query"}}}},
        {"knn": {"field": "vec", "query_vector": [], "k": 50}}
      ],
      "rank_window_size": 50,
      "rank_constant": 60
    }
  }
}

Cross-Encoder Reranker

10. The 2021 License War — OpenSearch Is Born

Background

Aftermath

2024-2025 Status

ProductLicenseLead
Elasticsearch 8Elastic License 2 / SSPLElastic
Elasticsearch 8.14+AGPL added (2024.8)Elastic (recovery attempt)
OpenSearch 2.xApache 2.0AWS -> Linux Foundation

Choosing

11. Operational Hell — Common Failure Patterns

JVM Heap

Circuit Breaker

circuit_breaking_exception: Data too large

Hot Shard

Shard Explosion

Snapshot & Restore

12. Ingest Pipelines

Logstash

Beats

Elastic Agent + Fleet

OpenTelemetry

Ingest Node Pipeline

{
  "processors": [
    {"set": {"field": "indexed_at", "value": "{{_ingest.timestamp}}"}},
    {"grok": {"field": "message", "patterns": ["%{COMBINEDAPACHELOG}"]}}
  ]
}

13. ES|QL — The Return of SQL (2024)

Elastic's long-awaited SQL-like query language.

FROM logs-*
| WHERE status >= 500
| STATS count = COUNT(*) BY host
| SORT count DESC
| LIMIT 10

OpenSearch added similar functionality with PPL (Piped Processing Language) in 2024.

14. Search Quality Evaluation

15. Top 10 Anti-Patterns

  1. Default 1-shard/1-replica for tens of TB indexes
  2. Manual _id causing routing skew
  3. Too many indexes/shards
  4. JVM heap 64 GB (exceeds Compressed OOPs)
  5. Deep pagination (from=10000) — use scroll/search_after
  6. term on analyzed text fields
  7. Cluster health check per query
  8. Frequent bulk deletes without _forcemerge
  9. No snapshots
  10. Vector-only, abandoning BM25 — hybrid almost always wins

16. Checklist for Running ES/OpenSearch Wisely

Closing — Giants Standing on Lucene

Elasticsearch, OpenSearch, Solr — different names, all on the shoulders of the giant named Lucene. And Lucene itself began as Doug Cutting's single Java library to democratize search in 1999.

In 2025, search finds logs, products, content recommendations, LLM context, autocomplete, and typo correction. It is everywhere, but not everyone understands it. When Inverted Index, Segment, BM25, vector search, Shard, and Routing click into place, you move from being a search user to a search designer.

"A good search engine doesn't just find what you typed. It finds what you meant." — Peter Norvig

Read the original: https://www.youngju.dev/blog/culture/2026-04-15-elasticsearch-opensearch-lucene-internals-inverted-index-bm25-sharding-vector-hybrid-search-deep-dive-guide-2025.en

Comments

No comments yet.

Sign in to leave a comment