Longbench V2 Eval

---
name: longbench-v2-eval
description: Evaluates large language models' ability to comprehend and reason over realistic, extremely long contexts (up to 2M words) across six multitask domains. It probes deep understanding rather than shallow extraction by using challenging multiple-choice questions that require extended reasoning and careful reading. Use when the user wants to benchmark on LongBench v2, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.15204
  bibtex_key: [REDACTED]
  confidence: high
---

# longbench-v2-eval

> LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai et al. (2024) (arXiv:2412.15204, 2024)

## What this evaluates

Evaluates large language models' ability to comprehend and reason over realistic, extremely long contexts (up to 2M words) across six multitask domains. It probes deep understanding rather than shallow extraction by using challenging multiple-choice questions that require extended reasoning and careful reading.

## Datasets

- **LongBench v2** — total 503; splits: test (503); repo https://github.com/THUDM/LongBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Invalid or non-compliant model responses are compensated by assigning a 25% accuracy score (random guessing baseline) instead of being discarded.

## Input / output format

**Input**: Long context document(s) (up to 2M words, truncated to model's max context window if needed) concatenated with a multiple-choice question and its options. For RAG evaluation, the context is split into 512-token chunks and top-N most similar chunks are retrieved.

**Output**: A single letter/option corresponding to the correct answer. In the CoT setting, the model first generates a chain of thought followed by the final answer choice.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    total = len(predictions)
    for pred, gold in zip(predictions, golds):
        if pred is a valid choice:
            if pred == gold:
                correct += 1
        else:
            correct += 0.25  # Compensate invalid responses with random guess baseline
    return (correct / total) * 100
```

## Common pitfalls

- Invalid model outputs must be compensated with a 25% score (random baseline) rather than dropped or counted as zero.
- Contexts exceeding the model's context window require middle truncation to ensure fair comparison.
- CoT and zero-shot settings yield different results; mixing evaluation settings without explicit labeling invalidates comparisons.

## Evidence (verbatim from paper)

> To account for model responses and human responses that do not yield a valid choice, we report the compensated results in Table 5, where these cases are counted towards the accuracy with a random probability of 25%.

## Citation

```bibtex
@misc{bai2024longbenchv2,
  title={LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks},
  author={Yushi Bai et al. (2024)},
  year={2024},
  note={arXiv:2412.15204}
}
```

- arXiv: 2412.15204