Span Mt Metaeval

---
name: span-mt-metaeval
description: Evaluates the reliability and fairness of span-level error detection metrics for machine translation auto-evaluators. It probes whether standard micro-averaged precision/recall/F1 scores produce consistent rankings compared to a proposed partial overlap matching strategy. Use when the user wants to benchmark on MQM 2022-2024, or asks about evaluating this task. Reports micro-averaged precision/recall/F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.19921
  bibtex_key: [REDACTED]
  confidence: medium
---

# span-mt-metaeval

> Span-Level Machine Translation Meta-Evaluation — Perrella et al. (2026) (arXiv:2603.19921, 2026)

## What this evaluates

Evaluates the reliability and fairness of span-level error detection metrics for machine translation auto-evaluators. It probes whether standard micro-averaged precision/recall/F1 scores produce consistent rankings compared to a proposed partial overlap matching strategy.

## Datasets

- **MQM 2022-2024** — total ?; splits: test (-1)

## Metrics

- `micro-averaged precision/recall/F1` **(primary)** — range: [0, 1]
  - Computed as the harmonic mean of micro-averaged precision and recall across all predicted and gold span-level errors. Precision is the fraction of predicted spans that match gold spans, and recall is the fraction of gold spans matched by predictions.
- `mpp` — range: [0, 1]
  - Match with partial overlap and partial credit, using micro-averaging. Assigns fractional credit based on the degree of overlap between predicted and gold spans rather than requiring exact matches.

## Input / output format

**Input**: Source sentence, machine-translated hypothesis, reference sentence, and gold-standard span-level error annotations.

**Output**: Predicted span-level error annotations (start/end offsets and error types) for the hypothesis.

## Scoring recipe

```python
def compute_micro_f1(pred_spans, gold_spans):
    tp = sum(1 for p in pred_spans if p in gold_spans)
    fp = len(pred_spans) - tp
    fn = len(gold_spans) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Exact span matching is often too strict for MT error detection, leading to artificially low scores.
- Micro-averaging across diverse error types or language pairs can mask model weaknesses in specific categories.
- Arbitrary methodological choices in span alignment cause inconsistent auto-evaluator rankings.

## Evidence (verbatim from paper)

> common metrics like micro-averaged precision/recall/F1 produce inconsistent rankings due to arbitrary methodological choices. We demonstrate that some meta-evaluation strategies are unsuitable for MT error detection by measuring the performance of sentinel auto-evaluators alongside normal auto-evaluators.

## Citation

```bibtex
@misc{perrella2026spanmtmetaeval,
  title={Span-Level Machine Translation Meta-Evaluation},
  author={Perrella et al. (2026)},
  year={2026},
  note={arXiv:2603.19921}
}
```

- arXiv: 2603.19921