Loco1 Eval

---
name: loco1-eval
description: Evaluates long-context retrieval capabilities on real-world documents where relevant information spans entire texts, such as legal contracts and medical notes. It specifically probes a model's ability to locate and rank relevant passages without relying on truncation or chunking strategies that often bias standard retrievers. Use when the user wants to benchmark on LoCoV1, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.07440
  bibtex_key: saadfalcon2024loco
  confidence: high
---

# loco1-eval

> Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT — Jon Saad-Falcon et al. (arXiv:2402.07440, 2024)

## What this evaluates

Evaluates long-context retrieval capabilities on real-world documents where relevant information spans entire texts, such as legal contracts and medical notes. It specifically probes a model's ability to locate and rank relevant passages without relying on truncation or chunking strategies that often bias standard retrievers.

## Datasets

- **LoCoV1** — total ?; splits: test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Measures ranking quality by comparing the discounted cumulative gain of the predicted ranking to the ideal ranking, normalized to [0,1]. Accounts for both position and relevance quality of retrieved items.

## Input / output format

**Input**: A query and a long document (or concatenated passages) containing the relevant information.

**Output**: A ranked list of retrieved documents/passages.

## Scoring recipe

```python
def compute_ndcg_at_10(retrieved, relevant):
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(retrieved[:10]))
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(sorted(relevant, reverse=True)[:10]))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Assuming truncation or chunking with embedding averaging is a fair baseline for long-context retrieval; the paper shows these often fail or worsen performance.
- Ignoring position bias in long documents; models with shorter max sequence lengths drop significantly when relevant info is at the end.
- Evaluating efficiency on average chunk time instead of full-document embedding throughput.

## Evidence (verbatim from paper)

> We use nDCG@10 as the quality metric for LoCoV1. nDCG@10 measures the ranking quality of information retrieval systems, accounting for both the position and quality of the items in the retrieved sequence.

## Citation

```bibtex
@misc{saadfalcon2024loco,
  title={Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT},
  author={Jon Saad-Falcon et al.},
  year={2024},
  note={arXiv:2402.07440}
}
```

- arXiv: 2402.07440