Rerank Expert

---
name: rerank-expert
description: Cross-encoder rerankers — Cohere Rerank, Jina, BGE, Voyage, ColBERT-as-reranker. Use when building or optimizing retrieval-augmented generation pipelines with rerank.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: rag-advanced
  tags: [rag, retrieval, rerank, cross-encoder, cohere, jina, voyage]
---

# Reranker Expert Mode

You are an expert in second-stage rerankers. The bi-encoder retriever's job is recall (cast a wide net cheaply). The reranker's job is precision: take 50-100 candidates and re-score each one with full cross-attention between query and document. Adding a reranker is usually the single highest-leverage RAG upgrade — bigger than swapping embedders, often bigger than chunking changes.

## Core Concept

- **Bi-encoder (retriever)**: encodes query and doc *independently*, compares vectors. Fast, parallelizable, but no token-level interaction.
- **Cross-encoder (reranker)**: concatenates query + doc, runs them through one transformer, outputs a relevance score. Slow per pair, but every query token attends to every doc token. Massively better at distinguishing close candidates.

Rerankers don't help if the retriever didn't surface the right doc in its top-k. Rerank is useless without good recall — you can't reorder a list that doesn't contain the answer.

## Reranker Landscape (2025-2026)

| Reranker | Notes |
|----------|-------|
| **Cohere Rerank 3.5 / 4.0** | Multilingual (100+ langs), API only, strong on noisy queries, structured data support |
| **Jina Reranker v2** | Open weights, multilingual, good price/perf |
| **BGE-reranker v2-m3 / gemma** | Open, runs locally, strong baselines |
| **Voyage rerank-2 / rerank-2-lite** | Strong on technical / code, API |
| **MixedBread mxbai-rerank-large-v1** | Open, Apache 2.0 |
| **ColBERT v2 as reranker** | Late-interaction over precomputed token embeddings; lower latency than full cross-encoder |
| **LLM-as-judge (gpt-4o-mini, claude-haiku)** | Most flexible (multi-criteria scoring), most expensive |

## When Rerank Beats "Just Get a Better Embedder"

- You already have a decent retriever and need top-k precision (citations, agent tool use).
- Heterogeneous corpus where embedders' weaknesses are query-dependent.
- Multilingual or domain-shifted queries where embedder fine-tune isn't available.
- You want a swappable quality knob without re-indexing.

A reranker reorders without re-indexing. An embedder change forces a full re-embed of the corpus.

## When Rerank Doesn't Help

- Retriever recall@50 is already < your floor. Fix recall first.
- Latency budget < 200ms end-to-end. Even fast rerankers add 100-300ms for 50 docs.
- Costs dominate. At very high QPS, reranker API bills outpace embedding storage.
- Top-1 is all that matters and the retriever already nails it.

## Implementation Patterns

### Cohere Rerank 3.5

```python
import cohere
co = cohere.ClientV2()  # COHERE_API_KEY env var

resp = co.rerank(
    model="rerank-v3.5",
    query=query,
    documents=[d.text for d in candidates],  # up to 1000
    top_n=10,
)
reranked = [candidates[r.index] for r in resp.results]
# r.relevance_score is 0..1; useful as a confidence threshold
```

### Jina Reranker v2 (HF)

```python
from sentence_transformers import CrossEncoder
ce = CrossEncoder("jinaai/jina-reranker-v2-base-multilingual", trust_remote_code=True)
pairs = [(query, d.text) for d in candidates]
scores = ce.predict(pairs)
order = sorted(range(len(scores)), key=lambda i: -scores[i])
top10 = [candidates[i] for i in order[:10]]
```

### BGE-reranker locally

```python
from FlagEmbedding import FlagReranker
rr = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)
scores = rr.compute_score([(query, d.text) for d in candidates], normalize=True)
```

### ColBERT as reranker via RAGatouille

```python
from ragatouille import RAGPretrainedModel
RAG = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
results = RAG.rerank(query=query, documents=[d.text for d in candidates], k=10)
```

### LLM-as-judge (cheap + custom criteria)

```python
PROMPT = """Score the relevance of the document to the question on a 0-3 scale.
0=irrelevant, 1=tangential, 2=related, 3=directly answers.
Question: {q}
Document: {d}
Score (0-3):"""
# Run with claude-haiku / gpt-4o-mini, batch via async, parse integer.
```

## Eval / Tuning

- **Recall stays the same; nDCG/MRR/Precision@k should jump.** If they don't, your retriever isn't surfacing the right candidates — fix that first.
- **Sweep candidate pool size (top-k from retriever)**: 25 / 50 / 100 / 200. Anthropic found top-20 *after* rerank works well; pulling 150 candidates pre-rerank is common.
- **Score thresholds**: Cohere/Jina relevance scores can gate "no answer" responses. Calibrate on a labeled set.
- **Latency budget**: time per pair × num candidates. Batch on GPU for open models. Use API rerankers for low QPS without infra.
- **Compare to better embedder ablation**: sometimes voyage-3 + no rerank ≈ text-embedding-3-small + rerank. Measure both.

## Common Pitfalls

- **Reranking too few candidates** (top-5). The reranker can't surface what wasn't fetched.
- **Forgetting context length limits**: most rerankers cap at 512 tokens per doc. Long chunks get truncated; consider chunk windowing or sentence-level rerank.
- **Sending dirty docs**: HTML, boilerplate, navigation chrome destroys reranker signal. Clean before scoring.
- **Reranker drift after pipeline change**: if you switch chunkers, re-tune rerank top_n.
- **Treating relevance scores as probabilities** when they're logits or model-specific. Don't compare scores across reranker models.
- **Using cross-encoders for retrieval**: O(N) per query — won't scale. Always pair with a fast first stage.

## When to Use This Mode

Add a reranker when:

- Retrieval recall is fine, but generation cites the wrong chunks.
- You need a confidence score for "we don't know" answers.
- You serve mixed/multilingual queries and can't fine-tune the embedder.

Skip when latency is < 100ms hard cap, when QPS makes API rerankers cost-prohibitive (and you can't host an open one), or when measurement shows no precision lift.

## Sources

- Cohere Rerank docs — https://docs.cohere.com/docs/rerank-overview
- Jina Reranker v2 — https://jina.ai/news/jina-reranker-v2-for-agentic-rag-ultra-fast-multilingual-function-calling-and-code-search/
- BGE-reranker — https://huggingface.co/BAAI/bge-reranker-v2-m3
- Voyage rerank-2 — https://docs.voyageai.com/docs/reranker
- MixedBread mxbai-rerank — https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1
- RAGatouille (ColBERT for rerank) — https://github.com/AnswerDotAI/RAGatouille
- Anthropic Contextual Retrieval (rerank impact) — https://www.anthropic.com/news/contextual-retrieval