Unieval Eval

---
name: unieval-eval
description: Evaluates natural language generation models across multiple quality dimensions (e.g., coherence, fluency, consistency, relevance) by reframing assessment as a Boolean QA task. Measures how well automated scores align with human judgments using correlation metrics. Use when the user wants to benchmark on SummEval, Topical-Chat, SFRES, SFHOT, QAGS, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.07197
  bibtex_key: zhong2022unieval
  confidence: high
---

# unieval-eval

> Towards a Unified Multi-Dimensional Evaluator for Text Generation — Zhong et al. (2022) (arXiv:2210.07197, 2022)

## What this evaluates

Evaluates natural language generation models across multiple quality dimensions (e.g., coherence, fluency, consistency, relevance) by reframing assessment as a Boolean QA task. Measures how well automated scores align with human judgments using correlation metrics.

## Datasets

- **SummEval** — total ?; splits: test (-1)
- **Topical-Chat** — total ?; splits: test (-1)
- **SFRES** — total ?; splits: test (-1)
- **SFHOT** — total ?; splits: test (-1)
- **QAGS** — total ?; splits: test (-1)

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Ranks the predicted scores and human scores, then computes the Pearson correlation between the rank vectors. Measures monotonic relationship.
- `Kendall-Tau` — range: [-1, 1]
  - Computes the difference between concordant and discordant pairs of predictions and human scores, normalized by total pairs. Measures ordinal association.
- `Pearson correlation` — range: [-1, 1]
  - Computes the covariance of predicted and human scores divided by the product of their standard deviations. Measures linear relationship.

## Input / output format

**Input**: A targeted Boolean question for a specific evaluation dimension (e.g., 'Is this a coherent summary?'), the source/context text, and the generated text.

**Output**: Boolean answer (Yes/No) or probability of 'Yes'.

## Scoring recipe

```python
def compute_score(predictions, human_scores, dimension):
    if dimension in ['fluency', 'consistency']:
        score = mean(sentence_level_scores)
    elif dimension == 'engagingness':
        score = sum(sentence_level_scores)
    else:
        score = compute_equation_1(prediction, human_label)
    return spearmanr(predictions, human_scores)
```

## Common pitfalls

- Confusing summary-level aggregation (for SummEval) with turn-level aggregation (for Topical-Chat), which changes the correlation baseline.
- Using similarity-based metrics like ROUGE or BERTScore for dimensions they were not designed for (e.g., coherence), leading to artificially low correlations.
- Ignoring the continual learning schedule (coherence -> fluency -> consistency -> relevance), which significantly impacts zero-shot transfer performance.

## Evidence (verbatim from paper)

> Following Liu et al. (2021a), we use summary-level Spearman and Kendall-Tau correlation to assess the performance of different evaluators for summarization.

## Citation

```bibtex
@misc{zhong2022unieval,
  title={Towards a Unified Multi-Dimensional Evaluator for Text Generation},
  author={Zhong et al. (2022)},
  year={2022},
  note={arXiv:2210.07197}
}
```

- arXiv: 2210.07197