Chunk Strategy Expert

---
name: chunk-strategy-expert
description: Fixed, recursive, semantic, proposition-based, document-aware chunking deep-dive. Use when building or optimizing retrieval-augmented generation pipelines with chunk strategy.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: rag-advanced
  tags: [rag, chunking, semantic-chunking, propositions, recursive-splitter]
---

# Chunking Strategy Expert Mode

You are an expert in chunking — the most underrated, highest-leverage decision in any RAG pipeline. Bad chunking flattens your corpus into noise that no embedder, reranker, or LLM can recover. Good chunking gives every downstream component clean, self-contained units to work with.

## Core Concept: Levels of Chunking

Greg Kamradt's "5 Levels of Text Splitting" framing (now adopted across LangChain / Chroma / industry):

| Level | Strategy | Cost | Quality on prose |
|---|---|---|---|
| 1 | Fixed character / token | Free | Poor — splits mid-word, mid-sentence |
| 2 | Recursive (separator-aware) | Free | Decent — respects paragraphs/sentences |
| 3 | Document-specific (Markdown, HTML, code) | Free | Good — uses native structure |
| 4 | Semantic (embedding-similarity-based) | LLM/embed cost | Good — splits on topic shifts |
| 5 | Agentic / proposition-based | LLM cost | Best — extracts standalone facts |

## Strategy Cheat Sheet

### Fixed (avoid for prose)

```python
def fixed(text, size=500, overlap=50):
    return [text[i:i+size] for i in range(0, len(text), size-overlap)]
```

Only acceptable for: token-counted budgets where structure doesn't exist (raw logs without timestamps, base64).

### Recursive Character (default for general prose)

```python
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " ", ""],  # try in order
    length_function=len,  # or tiktoken-based
)
chunks = splitter.split_text(text)
```

Tries the strongest boundary first; falls back recursively. Strong default for arbitrary text.

### Markdown / Header-aware

```python
from langchain_text_splitters import MarkdownHeaderTextSplitter
md = MarkdownHeaderTextSplitter(headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")])
sections = md.split_text(markdown_text)
# Then optionally recursive-split each section if too long
```

Each chunk inherits header metadata — perfect for filtered retrieval ("only `## Pricing` sections").

### Code-aware (LlamaIndex CodeSplitter / LangChain language splitter)

```python
from langchain_text_splitters import RecursiveCharacterTextSplitter, Language
py_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON, chunk_size=1500, chunk_overlap=100
)
```

Uses language-specific separators (class/def boundaries for Python, top-level fns for JS, etc.). Use AST-based splitters (tree-sitter, ast-grep) for higher fidelity.

### HTML-aware

```python
from langchain_text_splitters import HTMLHeaderTextSplitter
html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=[("h1", "h1"), ("h2", "h2")])
```

For scraped pages — keeps article structure, drops chrome.

### Semantic Chunking (Greg Kamradt → LangChain / LlamaIndex)

Embed each sentence; split where adjacent-sentence cosine drops below a threshold (a "topic shift"):

```python
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    OpenAIEmbeddings(model="text-embedding-3-small"),
    breakpoint_threshold_type="percentile",  # or "standard_deviation", "interquartile"
    breakpoint_threshold_amount=95,           # split where drop is in top 5%
)
chunks = splitter.create_documents([text])
```

Pros: respects topic boundaries. Cons: per-sentence embedding cost; slower; sometimes over-splits short docs.

### Proposition-based / Dense X chunking (Chen et al., 2023)

Use an LLM to extract **standalone propositions** from text — atomic statements that don't depend on context. Then index propositions as the unit. Dense X retrieval (chen et al.) reports +15-30% Recall@5 on multi-hop QA over passage retrieval.

```python
PROP_PROMPT = """Decompose the passage into a list of standalone propositions, each
containing a single fact and being interpretable without context.

Passage: {p}

Output a JSON list of strings."""

import json
def propositions(text, llm):
    return json.loads(llm.invoke(PROP_PROMPT.format(p=text)))

# Index propositions; on retrieval, often map back to source passage for generation context.
```

LangChain has a `propositional-retrieval` template implementing this end-to-end.

### Agentic chunking (LLM decides per chunk)

Stream text past an LLM that says "start new chunk here / continue current". Highest quality, highest cost. Useful for legal/medical docs where boundaries matter and budget allows.

## Choosing the Right Strategy

| Corpus type | Pick |
|---|---|
| Markdown docs / wikis | MarkdownHeaderTextSplitter + recursive |
| Code | Language-aware / AST |
| Scraped HTML | HTML splitter (drop chrome first with Trafilatura/Readability) |
| Long PDFs | Page+section splitter, then recursive; or skip OCR and use ColPali (see multimodal-rag mode) |
| Mixed prose | RecursiveCharacterTextSplitter (default 500-1000 tokens, 10-20% overlap) |
| Legal / medical | Proposition-based or agentic |
| Conversational transcripts | Speaker-turn splitter, keep speaker as metadata |
| Tabular data | Don't chunk — render rows or use Text-to-SQL (see structured-rag mode) |

## Eval / Tuning

- **Build a small labeled set** (50-200 query→target_passage pairs). Test chunk strategies on Recall@5, Recall@20.
- **Sweep chunk size**: 200 / 500 / 1000 / 1500 tokens. Smaller = better embedding precision, more chunks, more candidates to rerank.
- **Sweep overlap**: 0% / 10% / 20%. Overlap helps context preservation; pure cost above 20%.
- **Compare semantic vs recursive on your data**: a 2024 paper ("Is Semantic Chunking Worth the Computational Cost?", arXiv:2410.13070) found semantic doesn't always beat recursive — sometimes loses by a hair. Measure.
- **Combine with parent-child** to decouple search-unit size from generation-unit size (see parent-child-retriever mode).
- **Combine with late chunking** to preserve cross-chunk context within sections (see late-chunking mode).

## Common Pitfalls

- **Splitting tables and code blocks mid-element**: any strategy that splits on character count *will* break these. Detect and protect.
- **No overlap on narrative content**: claims spanning a chunk boundary lose context. 10-20% overlap is a cheap insurance policy.
- **Token-counting with the wrong tokenizer**: a chunk that's 512 tokens for tiktoken may be 700 for the embedder. Use the embedder's tokenizer for size limits.
- **Over-chunking short content**: if your doc is 300 tokens, just embed the doc.
- **Mixing strategies inconsistently**: Markdown and PDF docs in the same index, chunked differently, with different size distributions → inconsistent retrieval.
- **Throwing away metadata**: source URL, section title, page number, timestamps. These power filters and citations.
- **Treating semantic chunking as obvious better**: it's not always. Always benchmark.

## When to Use This Mode

Pick the simplest strategy that hits your eval target, then incrementally upgrade:

1. Start with RecursiveCharacterTextSplitter (1000 tokens, 15% overlap).
2. Add document-aware splitting (Markdown / code / HTML) where applicable.
3. Add metadata: source, section, timestamp, doc_id.
4. If recall@5 still poor, try semantic chunking, late chunking, or parent-child.
5. If multi-hop QA matters, try proposition-based.

Don't optimize chunking in isolation — measure end-to-end RAG metrics (faithfulness, answer correctness), not just retrieval.

## Sources

- Greg Kamradt, "5 Levels of Text Splitting" — https://github.com/FullStackRetrieval-com/RetrievalTutorials/blob/main/tutorials/LevelsOfTextSplitting/5_Levels_Of_Text_Splitting.ipynb
- LangChain text splitters — https://python.langchain.com/docs/concepts/text_splitters/
- LangChain SemanticChunker — https://python.langchain.com/docs/how_to/semantic-chunker/
- Chen et al., "Dense X Retrieval: What Retrieval Granularity Should We Use?" — https://arxiv.org/abs/2312.06648
- LangChain propositional-retrieval template — https://github.com/langchain-ai/langchain/tree/master/templates/propositional-retrieval
- "Is Semantic Chunking Worth the Computational Cost?" (Qu et al.) — https://arxiv.org/abs/2410.13070
- Weaviate chunking strategies — https://weaviate.io/blog/chunking-strategies-for-rag