Crumb Eval

---
name: crumb-eval
description: Evaluates information retrieval models on complex, multi-aspect, and logically structured queries across eight diverse domains. It probes the model's ability to handle nuanced document alignments, set-based operations, and context-rich instructions beyond simple keyword matching. Use when the user wants to benchmark on CRUMB, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.07253
  bibtex_key: [REDACTED]
  confidence: high
---

# crumb-eval

> Benchmarking Information Retrieval Models on Complex Retrieval Tasks — Killingback et al. (2025) (arXiv:2509.07253, 2025)

## What this evaluates

Evaluates information retrieval models on complex, multi-aspect, and logically structured queries across eight diverse domains. It probes the model's ability to handle nuanced document alignments, set-based operations, and context-rich instructions beyond simple keyword matching.

## Datasets

- **CRUMB** — total ?; splits: test (-1), val (-1); repo https://github.com/jfkback/crumb

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of graded relevance scores of retrieved documents up to position 10, normalized by the ideal DCG@10.
- `R@100` — range: [0, 1]
  - Recall at 100. The proportion of relevant documents retrieved within the top 100 results out of the total number of relevant documents.

## Input / output format

**Input**: Query text (optionally prefixed with an instruction like 'Instruct: <instruction>\nQuery: <query>'). Document corpus provided as either full markdown-formatted documents or contextualized chunks (≤512 BERT tokens, prepended with hierarchical header paths).

**Output**: Ranked list of retrieved documents or chunks, typically returned as a list of document IDs or text snippets ordered by relevance score.

## Scoring recipe

```python
def compute_metrics(retrieved_ids, relevant_ids, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, rid in enumerate(retrieved_ids[:k]) if rid in relevant_ids)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_ids), k)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    recall = min(len(set(retrieved_ids[:100]) & set(relevant_ids)), len(relevant_ids)) / max(len(relevant_ids), 1)
    return {'nDCG@10': ndcg, 'R@100': recall}
```

## Common pitfalls

- Using newline-based chunking instead of the provided contextualized chunking strategy, which strips hierarchical headers and degrades performance.
- Ignoring the MaxP recommendation for passage-level evaluation, which can unfairly penalize models that retrieve relevant full documents but miss specific chunks.
- Applying LLM query rewriting uniformly, as it improves weaker models but degrades performance on strong baselines.

## Evidence (verbatim from paper)

> The benchmark reveals that even top retrieval models achieve only average nDCG@10 of 0.346 and R@100 of 0.587, with poor performance in low-overlap scenarios and set-based logical queries.

## Citation

```bibtex
@misc{killingback2025crumb,
  title={Benchmarking Information Retrieval Models on Complex Retrieval Tasks},
  author={Killingback et al. (2025)},
  year={2025},
  note={arXiv:2509.07253}
}
```

- arXiv: 2509.07253