Mt Incremental Eval

---
name: mt-incremental-eval
description: This benchmark evaluates how well automatic machine translation metrics track quality improvements in commercial systems over time. It probes whether metrics consistently rank newer systems higher than older ones, and how their reliability changes as system quality improves or when synthetic references are used. Use when the user wants to benchmark on Commercial MT Systems Corpus, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.03277
  bibtex_key: wu2024evaluatingautomaticmetrics
  confidence: high
---

# mt-incremental-eval

> Evaluating Automatic Metrics with Incremental Machine Translation Systems — Wu et al. (2024) (arXiv:2407.03277, 2024)

## What this evaluates

This benchmark evaluates how well automatic machine translation metrics track quality improvements in commercial systems over time. It probes whether metrics consistently rank newer systems higher than older ones, and how their reliability changes as system quality improves or when synthetic references are used.

## Datasets

- **Commercial MT Systems Corpus** — total ?; splits: full (-1); repo https://github.com/gjwubyron/Evo

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Ratio of rank agreements between metric score differences and time differences to the total number of system pairs. Formula: |sign(metricΔ) == sign(timeΔ)| / |all system pairs|. Only pairs with a time gap of less than one year are considered.
- `Spearman correlation` — range: [-1, 1]
  - Rank correlation coefficient measuring the consistency of upward trends between metric scores and chronological system order across language pairs.

## Input / output format

**Input**: Pairs of machine translation system outputs (source text, reference, hypothesis) with chronological timestamps, filtered to have a time gap of less than one year.

**Output**: A scalar accuracy score or Spearman correlation coefficient.

## Scoring recipe

```python
def compute_accuracy(metric_scores, timestamps):
    valid_pairs = []
    for i in range(len(metric_scores)):
        for j in range(i + 1, len(metric_scores)):
            if abs(timestamps[j] - timestamps[i]) < 1.0:
                valid_pairs.append((i, j))
    matches = 0
    for i, j in valid_pairs:
        m_delta = metric_scores[j] - metric_scores[i]
        t_delta = timestamps[j] - timestamps[i]
        if (m_delta > 0 and t_delta > 0) or (m_delta < 0 and t_delta < 0):
            matches += 1
    return matches / len(valid_pairs) if valid_pairs else 0.0
```

## Common pitfalls

- Must strictly filter system pairs to a time gap of less than one year; otherwise, large quality gaps overestimate metric reliability.
- Rolling window size (N) significantly impacts trend direction; smaller windows may show upward trends for surface metrics while larger windows show declines.
- Synthetic references do not universally improve accuracy; performance drops in some language pairs compared to human references.

## Evidence (verbatim from paper)

> For each system pair, we calculate the difference of the metric scores (metricΔ) and the difference in time (timeΔ). Accuracy for a specific metric is calculated as the ratio of rank agreements between metric and time deltas to the total number of comparisons: Accuracy = |sign(metricΔ) == sign(timeΔ)| / |all system pairs|. Since the systems span from 2018 to 2024, those separated by a substantial time interval might exhibit considerable quality gaps, potentially resulting in an overestimate of metric reliability. Consequently, we only pair systems with a gap of less than a year.

## Citation

```bibtex
@misc{wu2024evaluatingautomaticmetrics,
  title={Evaluating Automatic Metrics with Incremental Machine Translation Systems},
  author={Wu et al. (2024)},
  year={2024},
  note={arXiv:2407.03277}
}
```

- arXiv: 2407.03277