Ssa Mte Eval

---
name: ssa-mte-eval
description: Evaluates machine translation quality estimation metrics on under-resourced African languages by comparing their predicted scores against human-annotated Direct Assessment (DA) judgments. It probes a model's ability to correlate with human perception of translation adequacy across diverse language pairs, including both reference-based and reference-free settings. Use when the user wants to benchmark on SSA-MTE, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.04557
  bibtex_key: li2025ssacomet
  confidence: high
---

# ssa-mte-eval

> SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? — Senyu Li et al. (2025) (arXiv:2506.04557, 2025)

## What this evaluates

Evaluates machine translation quality estimation metrics on under-resourced African languages by comparing their predicted scores against human-annotated Direct Assessment (DA) judgments. It probes a model's ability to correlate with human perception of translation adequacy across diverse language pairs, including both reference-based and reference-free settings.

## Datasets

- **SSA-MTE** — total 73000; splits: train (-1), dev (-1), test (-1)

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Spearman rank correlation coefficient computed between the model's predicted quality scores and the human-annotated Direct Assessment (DA) scores across all test instances.

## Input / output format

**Input**: Source sentence, reference translation (for MTE), and hypothesis translation. For QE, only source and hypothesis are provided.

**Output**: A continuous quality score normalized to the [0, 1] range.

## Scoring recipe

```python
def compute_spearman(predictions, gold_scores):
    from scipy.stats import spearmanr
    corr, _ = spearmanr(predictions, gold_scores)
    return corr
```

## Common pitfalls

- Human DA scores must be pre-processed with z-score normalization per evaluator followed by min-max scaling to [0, 1] before training or evaluation to ensure comparability.
- LLM-based baselines require a 5-shot in-context prompt sampled from the training split; performance drops significantly without these demonstrations.

## Evidence (verbatim from paper)

> As shown in Table[3], SSA-COMET-MTL achieves the second highest average Spearman correlation with human judgments in the MTE setting, outperforming all prior AfriCOMET variants as well as the strong prompting-based baselines such as Gemini-2.5 Pro.

## Citation

```bibtex
@misc{li2025ssacomet,
  title={SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?},
  author={Senyu Li et al. (2025)},
  year={2025},
  note={arXiv:2506.04557}
}
```

- arXiv: 2506.04557