Apply Bm25 Ranking

---
name: apply-bm25-ranking
description: Use when building or tuning full-text search relevance, scoring documents against a keyword query, or building the sparse/keyword leg of a hybrid RAG retriever — including deciding between BM25, raw TF-IDF, and dense/vector search, or diagnosing why keyword search results feel poorly ranked.
source: 'Robertson & Walker "Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval" (SIGIR 1994, the Okapi BM25 origin); Robertson & Zaragoza "The Probabilistic Relevance Framework: BM25 and Beyond" (2009); Elasticsearch and Apache Lucene documentation (BM25 as default similarity since Lucene 6.0 / Elasticsearch 5.0)'
tags: [search, ranking, information-retrieval, relevance, hybrid-search, rag, tf-idf]
related: [design-rag-pipeline, design-rag-system]
---

# Apply BM25 Ranking

Score documents against a keyword query using BM25's term-frequency saturation and document-length normalization — the default relevance function behind Elasticsearch and Lucene, and the keyword half of hybrid RAG retrieval.

## Why This Is Best Practice

**Adopted by:** Apache Lucene (default `Similarity` implementation since 6.0) and Elasticsearch (default since 5.0) — meaning BM25 ranks search results for most production full-text search deployments built on either engine, including Wikipedia search (via Elasticsearch/CirrusSearch) and countless e-commerce and enterprise search products. It's also the standard sparse-retrieval baseline in IR research (TREC) and the keyword leg of hybrid retrievers in production RAG stacks (LangChain, LlamaIndex, Pinecone hybrid search).

**Impact:** Robertson & Zaragoza (2009) show BM25 consistently outperforms raw TF-IDF across TREC ad-hoc retrieval benchmarks because it caps the reward for repeated query terms and correctly normalizes for document length — two failure modes that make raw TF-IDF over-rank long, keyword-stuffed documents. In hybrid RAG systems, adding BM25 alongside dense/vector retrieval measurably improves recall on queries with exact-match terms (IDs, error codes, proper nouns, acronyms) that embeddings routinely miss.

**Why best:** Raw TF-IDF has no ceiling on term-frequency reward, so a document repeating the query term 50 times outscores one using it twice with better context — BM25's saturation term (`k1`) fixes this. Pure dense/vector search captures semantic similarity but is weak on exact lexical matches (SKU numbers, error codes, rare proper nouns) and requires an embedding model plus a vector index; BM25 needs neither training data nor a model, is cheap to compute, and is exactly why hybrid retrieval (BM25 + dense, merged via Reciprocal Rank Fusion) consistently beats either alone.

Sources: Robertson & Walker, SIGIR 1994; Robertson & Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond," Foundations and Trends in Information Retrieval (2009); Apache Lucene `BM25Similarity` docs; Elasticsearch "Practical BM25" guide.

## Steps

### 1. Understand the formula

For query `Q` with terms `q1...qn`, score for document `D`:

```
score(D, Q) = Σ IDF(qi) · ( f(qi, D) · (k1 + 1) ) / ( f(qi, D) + k1 · (1 - b + b · |D| / avgdl) )
```

- `f(qi, D)` — term frequency of `qi` in `D`
- `|D|` — length of `D` (in terms); `avgdl` — average document length in the corpus
- `IDF(qi)` — inverse document frequency, down-weights terms that appear in most documents
- `k1` — controls term-frequency saturation (typical range 1.2–2.0)
- `b` — controls document-length normalization strength (typical range 0.5–0.9, default 0.75)

### 2. Tune `k1` (term-frequency saturation)

`k1` controls how quickly additional occurrences of a term stop adding score:
- `k1 = 0` — term frequency ignored entirely (binary term presence)
- Higher `k1` — each additional occurrence keeps contributing more (approaches raw TF-IDF behavior)
- Lower `k1` — score saturates fast after the first few occurrences (recommended when documents are prone to keyword stuffing)

Only change the default (Lucene/Elasticsearch: `k1 = 1.2`) against a labeled relevance evaluation set — tuning blind on a handful of manual queries reliably overfits.

### 3. Tune `b` (length normalization)

`b` controls how much a document's length relative to the corpus average penalizes its score:
- `b = 0` — no length normalization; long documents aren't penalized for containing more terms
- `b = 1` — full normalization; score is scaled entirely relative to average document length
- Default `b = 0.75` is a reasonable middle ground for most corpora

Lower `b` when document length varies for legitimate structural reasons (e.g., some docs are naturally longer reference pages, not keyword-stuffed); raise `b` when longer documents in the corpus tend to just repeat terms more without adding real relevance.

### 4. Decide when to combine with dense/vector retrieval

Use BM25 alone when queries are keyword/exact-match heavy (IDs, codes, names, short factual queries) and there's no budget for an embedding model or vector index. Combine BM25 with dense retrieval (hybrid search) when queries mix semantic/paraphrase intent with exact-match terms — merge the two ranked lists with Reciprocal Rank Fusion (RRF) rather than a raw weighted-score blend, since BM25 and cosine-similarity scores aren't on comparable scales.

| Dimension | BM25 | Raw TF-IDF | Dense/vector search |
|---|---|---|---|
| Exact-match / rare terms | Strong | Strong | Weak |
| Semantic/paraphrase queries | Weak | Weak | Strong |
| Training data / model required | None | None | Embedding model + index |
| Term-frequency saturation | Yes (`k1`) | No — unbounded | N/A |
| Length normalization | Yes (`b`) | Inconsistent across implementations | N/A |
| Compute cost | Low | Low | Higher (embedding + ANN search) |

### 5. Evaluate against a labeled set, not by eyeballing results

Build a small labeled query/relevant-document set before tuning `k1`/`b` or deciding whether to add hybrid search. Measure with standard IR metrics (NDCG, MRR, recall@k) before and after any change — perceived relevance from a few manual queries is not a reliable signal for whether a parameter change actually helped.

## Rules

- Never tune `k1`/`b` without a labeled relevance evaluation set — changes that look better on a handful of manual queries frequently regress on the full query distribution.
- Don't blend raw BM25 scores with dense/vector similarity scores directly — the scales aren't comparable; use rank-based fusion (RRF) instead.
- Don't treat a BM25 score as a probability or a bounded relevance percentage — it's an unbounded relative ranking score, useful only for ordering within one query's results.

## Common Mistakes

- **Treating BM25 score as an absolute quality measure.** Scores are only meaningful for ranking documents within the same query — they aren't comparable across different queries or usable as an absolute relevance threshold.
- **Ignoring document-length normalization on a corpus with highly variable document lengths.** Skipping or misconfiguring `b` lets long documents dominate purely by repeating terms more often, independent of actual relevance.
- **Applying IDF weighting on a very small or narrow corpus.** With few documents, IDF statistics are noisy and can produce unstable rankings — consider disabling or dampening IDF, or growing the corpus, before trusting the scores.
- **Tuning `k1`/`b` by manual inspection of a handful of queries.** This overfits to those specific queries; use a labeled evaluation set and IR metrics instead.

## When NOT to Use

- Queries are predominantly semantic or paraphrase-heavy with little lexical overlap between query and relevant documents — pure dense/vector retrieval will outperform BM25 alone; consider hybrid or dense-only.
- The result set is already small enough that exhaustive or manual scoring is cheaper and simpler than building a scored index.
- The corpus is too small or too homogeneous for IDF to produce meaningful term discrimination (see Common Mistakes).