Longbench Eval

---
name: longbench-eval
description: Evaluates large language models' ability to understand and process long contexts across bilingual (English and Chinese) multitask scenarios, including single/multi-document QA, summarization, few-shot learning, code completion, and synthetic tasks. Use when the user wants to benchmark on LongBench, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.14508
  bibtex_key: [REDACTED]
  confidence: high
---

# longbench-eval

> LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding — Bai et al. (2023) (arXiv:2308.14508, 2023)

## What this evaluates

Evaluates large language models' ability to understand and process long contexts across bilingual (English and Chinese) multitask scenarios, including single/multi-document QA, summarization, few-shot learning, code completion, and synthetic tasks.

## Datasets

- **LongBench** — total ?; splits: test (-1); repo https://github.com/THUDM/LongBench

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall over token overlap between prediction and gold answer. Used primarily for QA and few-shot tasks.
- `ROUGE-L` — range: [0, 1]
  - Measures the longest common subsequence (LCS) between the generated summary and the reference summary, capturing both precision and recall.
- `Edit Sim` — range: [0, 1]
  - Normalized Levenshtein distance measuring the minimum number of single-character edits (insertions, deletions, substitutions) required to change one code snippet into another.

## Input / output format

**Input**: Long context document(s) concatenated with an instruction or question. If input length L exceeds the model's maximum context length M, the sequence is truncated from the middle: [S_1:floor(M/2); S_L-floor(M/2)-1:L]. Few-shot tasks include examples within the context. Chat-style prompts are explicitly omitted for few-shot and code completion tasks.

**Output**: Free-form text generation using greedy decoding. For few-shot learning and code completion tasks, only the first line of the model's response is extracted for evaluation.

## Scoring recipe

```python
def evaluate(predictions, golds, task_types):
    scores = []
    for pred, gold, task in zip(predictions, golds, task_types):
        if task in ['fewshot', 'code']:
            pred = pred.split('\n')[0]  # Extract first line
        if task == 'qa':
            scores.append(f1_score(pred, gold))
        elif task == 'summarization':
            scores.append(rouge_l_score(pred, gold))
        elif task == 'code':
            scores.append(edit_similarity(pred, gold))
        else:
            scores.append(exact_match(pred, gold))
    return sum(scores) / len(scores)
```

## Common pitfalls

- Middle truncation is used when context exceeds model limits, which differs from standard prefix truncation and can discard crucial middle information.
- Chat-style prompts are deliberately avoided for few-shot and code tasks to force completion-style generation; applying standard chat templates will invalidate scores.
- Memorization confounds results on Wikipedia-derived datasets; readers must compare against the 'w/o context' baseline to isolate true long-context understanding.

## Evidence (verbatim from paper)

> F1 and ROUGE-L (Lin, 2004) are two popular N-gram based metrics widely adopted in QA and summarization tasks. Edit Sim (Levenshtein distance) is popularly used in code generation evaluation (Svyatkovskiy et al., 2020). For the few-shot learning tasks, we extract the first line of the response. For the two code completion tasks, we extract the first line of model generation that is not comment.

## Citation

```bibtex
@misc{bai2023longbench,
  title={LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding},
  author={Bai et al. (2023)},
  year={2023},
  note={arXiv:2308.14508}
}
```

- arXiv: 2308.14508