Summarization Metrics Eval

---
name: summarization-metrics-eval
description: Evaluates the correlation between automatic summarization metrics and LLM-as-a-Judge models against human judgments across five quality criteria (coherence, consistency, fluency, relevance, 5W1H) in Spanish and Basque. Use when the user wants to benchmark on BASSE, or asks about evaluating this task. Reports Spearman's $
ho$.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.17039
  bibtex_key: [REDACTED]
  confidence: high
---

# summarization-metrics-eval

> Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans? — Barnes et al. (2025) (arXiv:2503.17039, 2025)

## What this evaluates

Evaluates the correlation between automatic summarization metrics and LLM-as-a-Judge models against human judgments across five quality criteria (coherence, consistency, fluency, relevance, 5W1H) in Spanish and Basque.

## Datasets

- **BASSE** — total 2040; splits: test (-1); repo https://github.com/hitz-zentroa/summarization

## Metrics

- `Spearman's $
ho$` **(primary)** — range: [-1, 1]
  - System-level rank correlation coefficient between human average scores and automatic/LLM scores. Calculated following established NLG evaluation protocols (Louis and Nenkova, 2013; Fabbri et al., 2021).
- `ROUGE` — range: [0, 1]
  - Token subsequence overlap between candidate and reference summaries. Includes ROUGE-1 to ROUGE-4, ROUGE-L, and ROUGE-su* variants.
- `BertScore` — range: [0, 1]
  - Average token-level cosine similarity between greedily aligned tokens using multilingual BERT base uncased. Reports precision, recall, and F1.
- `BLEU` — range: [0, 1]
  - Corpus-level average n-gram precision with a brevity penalty, using SacreBLEU implementation.
- `LLM-Judge Score` — range: unknown
  - Numeric score assigned by an LLM judge (e.g., GPT-4o, Prometheus 2, Qwen2.5) for a single criterion, prompted with instructions, rubric, reference, and summary in English.

## Input / output format

**Input**: For automatic metrics: candidate summary and reference summary. For LLM judges: original instruction, annotation rubric, human reference answer, and candidate summary.

**Output**: Numeric score per summary/criterion for each metric or judge model.

## Scoring recipe

```python
def compute_spearman_rho(human_scores, metric_scores):
    n = len(human_scores)
    if n < 2:
        return 0.0
    h_ranks = assign_ranks(human_scores)
    m_ranks = assign_ranks(metric_scores)
    d_sq_sum = sum((h - m) ** 2 for h, m in zip(h_ranks, m_ranks))
    rho = 1 - (6 * d_sq_sum) / (n * (n ** 2 - 1))
    return rho
```

## Common pitfalls

- Metrics require language-specific preprocessing: ROUGE-we needs FastText embeddings, BertScore uses mBERT, CIDEr uses Snowball stemmer, and METEOR requires lowercasing.
- LLM judges must be prompted in English despite evaluating non-English summaries, as models struggle to follow non-English instructions.
- Correlation is computed at the system-level (averaging scores per system), not instance-level, which changes the statistical interpretation and requires careful aggregation.

## Evidence (verbatim from paper)

> We calculate both Spearman’s ($
ho$) and Kendall’s ($	au$) rank correlation coefficients at system-level following Louis and Nenkova (2013) and Fabbri et al. (2021), but report only Spearman in the subsequent section, as the results do not vary depending on the coefficient.

## Citation

```bibtex
@misc{barnes2025summarizationmetrics,
  title={Summarization Metrics for Spanish and Basque: Do Automatic Scores and LLM-Judges Correlate with Humans?},
  author={Barnes et al. (2025)},
  year={2025},
  note={arXiv:2503.17039}
}
```

- arXiv: 2503.17039