Quality Eval

---
name: quality-eval
description: This benchmark evaluates a model's ability to comprehend and reason over long documents (2k–8k tokens) to answer multiple-choice questions. It specifically probes whether models can integrate global context rather than relying on local keyword matching or summaries, with a subset (HARD) filtering for questions that require full reading rather than skimming. Use when the user wants to benchmark on QuALITY, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.08608
  bibtex_key: pang2021quality
  confidence: high
---

# quality-eval

> QuALITY: Question Answering with Long Input Texts, Yes! — Pang et al. (2021) (arXiv:2112.08608, 2021)

## What this evaluates

This benchmark evaluates a model's ability to comprehend and reason over long documents (2k–8k tokens) to answer multiple-choice questions. It specifically probes whether models can integrate global context rather than relying on local keyword matching or summaries, with a subset (HARD) filtering for questions that require full reading rather than skimming.

## Datasets

- **QuALITY** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard multiple-choice accuracy: the proportion of questions where the model's predicted option matches the gold answer. Evaluated separately on the full test set and the QuALITY-HARD subset.

## Input / output format

**Input**: A long English passage (2k–8k tokens) followed by a multiple-choice question with four answer options. In some baselines, the passage is truncated or replaced by extracted sentences based on retrieval methods.

**Output**: A single selected answer option from the four provided choices.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Models often rely on local keyword matching or summaries rather than integrating global context, leading to poor performance on long passages.
- The QuALITY-HARD subset specifically contains questions that human annotators could not answer within a 45-second skimming window, so evaluating on it requires true long-document comprehension.
- Question-only baselines (passage omitted) still achieve ~43% accuracy, indicating potential dataset artifacts or answer bias that can inflate scores if not controlled.

## Evidence (verbatim from paper)

> Table 6 shows model performance on the test set. ... Table 6: Accuracy on the full QuALITY test set and the QuALITY-HARD subset (formatted as full / HARD).

## Citation

```bibtex
@misc{pang2021quality,
  title={QuALITY: Question Answering with Long Input Texts, Yes!},
  author={Pang et al. (2021)},
  year={2021},
  note={arXiv:2112.08608}
}
```

- arXiv: 2112.08608