Human Evaluation Framework

---
name: human-evaluation-framework
description: Evaluates text generation models across multiple NLP tasks using standardized human annotation, focusing on reproducibility, annotator quality detection, and scalar scoring of qualities like fluency and correctness. Use when the user has predictions and gold and needs to compute human scores.
metadata:
  skill_kind: metric
  source_arxiv: 2101.06561
  bibtex_key: khashabi2021genie
  confidence: medium
---

# human-evaluation-framework

> GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation — Khashabi et al. (2021) (arXiv:2101.06561, 2021)

## What this evaluates

Evaluates text generation models across multiple NLP tasks using standardized human annotation, focusing on reproducibility, annotator quality detection, and scalar scoring of qualities like fluency and correctness.

## Datasets

- **Machine Translation** — total ?; splits: test (-1)
- **Summarization** — total ?; splits: test (-1)
- **Commonsense Reasoning** — total ?; splits: test (-1)
- **Machine Comprehension** — total ?; splits: test (-1)

## Metrics

- `human scores` **(primary)** — range: other
  - Absolute scalar rating assigned by human annotators to model-generated text, aggregated across a diverse pool of raters after filtering noisy annotators via a probabilistic quality model.

## Input / output format

**Input**: Task-specific prompts or source texts (e.g., for machine translation, summarization, commonsense reasoning, or machine comprehension) and the corresponding model-generated text.

**Output**: A scalar score per instance reflecting the quality of the generated text, as rated by human annotators.

## Scoring recipe

```python
def compute_metric(dataset):
    all_ratings = []
    for instance in dataset:
        annotators = get_assigned_annotators(instance)
        for annotator in annotators:
            rating = annotator.rate(instance.generation)
            all_ratings.append((annotator.id, rating))
    valid_ratings = filter_noisy_annotators(all_ratings)
    return mean([r for _, r in valid_ratings])
```

## Common pitfalls

- Annotator quality varies widely; failing to use the probabilistic noise-filtering model leads to unreliable rankings.
- Evaluations must be time-spread to ensure temporal consistency; batch evaluations introduce temporal bias.
- Scores must be absolute scalars, not pairwise comparisons, to enable longitudinal model ranking.

## Evidence (verbatim from paper)

> Evaluated across four core tasks—machine translation, summarization, commonsense reasoning, and machine comprehension—GENIE produces reliable, comparable human scores that outperform automatic metrics in capturing nuanced qualities like fluency and correctness, enabling longitudinal model ranking and fostering community-driven, standardized benchmarking.

## Citation

```bibtex
@misc{khashabi2021genie,
  title={GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation},
  author={Khashabi et al. (2021)},
  year={2021},
  note={arXiv:2101.06561}
}
```

- arXiv: 2101.06561