Spearman Correlation

---
name: spearman-correlation
description: Measures the rank-order agreement between an NLG evaluator's predicted scores and human reference judgments. It probes the evaluator's ability to capture task-specific quality dimensions such as fluency, coherence, consistency, and groundedness across summarization and dialogue generation. Use when the user has predictions and gold and needs to compute Spearman correlation.
metadata:
  skill_kind: metric
  source_arxiv: 2303.16634
  bibtex_key: liu2023geval
  confidence: high
---

# spearman-correlation

> G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment — Liu et al. (2023) (arXiv:2303.16634, 2023)

## What this evaluates

Measures the rank-order agreement between an NLG evaluator's predicted scores and human reference judgments. It probes the evaluator's ability to capture task-specific quality dimensions such as fluency, coherence, consistency, and groundedness across summarization and dialogue generation.

## Datasets

- **SummEval** — total ?; splits: test (-1)
- **Topical-Chat** — total ?; splits: test (-1)
- **QAGS** — total ?; splits: test (-1)

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Non-parametric measure of rank correlation between the predicted scores from the NLG evaluator and the human reference scores. Calculated on the ranks of the two score vectors. Ranges from -1 to 1, where 1 indicates perfect agreement in ranking.

## Input / output format

**Input**: Source document/context, generated summary or dialogue response, and a set of evaluation criteria (e.g., fluency, coherence, consistency, relevance, naturalness, engagingness, groundedness).

**Output**: A numerical score (typically on a 1–5 scale) for each specified criterion, generated via a chain-of-thought form-filling prompt.

## Scoring recipe

```python
def compute_spearman(predictions, gold):
    # predictions: list of evaluator scores per instance
    # gold: list of human scores per instance
    from scipy.stats import spearmanr
    corr, _ = spearmanr(predictions, gold)
    return corr
```

## Common pitfalls

- GPT-4 lacks native token probability output; the paper estimates probabilities by sampling n=20 times with temperature=1, introducing sampling variance.
- LLM-based evaluators exhibit a strong bias toward LLM-generated text over human-written content, which can artificially inflate scores in self-reinforcement training loops.
- Correlation is computed separately for each evaluation aspect (e.g., coherence, groundedness) rather than as a single aggregate metric.

## Evidence (verbatim from paper)

> We adopt the same approach as Zhong et al. (2022) to evaluate different summarization metrics using summary-level Spearman and Kendall-Tau correlation.

## Citation

```bibtex
@misc{liu2023geval,
  title={G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2303.16634}
}
```

- arXiv: 2303.16634