Mt Reasoning Scaling Eval

---
name: mt-reasoning-scaling-eval
description: This evaluation probes the test-time scaling properties of reasoning models across diverse machine translation tasks. It measures how varying reasoning budgets and iterative self-correction workflows impact translation quality across literary, biomedical, cultural, and commonsense domains. Use when the user wants to benchmark on WMT24-Literary, MetaphorTrans, LitEval-Corpus, WMT24-Biomedical, WMT23-Biomedical, CAMT, Commonsense-MT, RTT, RAGTrans, or asks about evaluating this task. Reports COMET-22.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06471
  bibtex_key: li2025testtimescaling
  confidence: high
---

# mt-reasoning-scaling-eval

> Test-Time Scaling of Reasoning Models for Machine Translation — Li et al. (2025) (arXiv:2510.06471, 2025)

## What this evaluates

This evaluation probes the test-time scaling properties of reasoning models across diverse machine translation tasks. It measures how varying reasoning budgets and iterative self-correction workflows impact translation quality across literary, biomedical, cultural, and commonsense domains.

## Datasets

- **WMT24-Literary** — total 43; splits: test (43)
- **MetaphorTrans** — total 2000; splits: test (2000)
- **LitEval-Corpus** — total 187; splits: test (187)
- **WMT24-Biomedical** — total 600; splits: test (600)
- **WMT23-Biomedical** — total 585; splits: test (585)
- **CAMT** — total 6948; splits: test (6948)
- **Commonsense-MT** — total 1200; splits: test (1200)
- **RTT** — total 100; splits: test (100)
- **RAGTrans** — total 1999; splits: test (1999)

## Metrics

- `COMET-22` **(primary)** — range: [0, 1]
  - Reference-based neural metric that predicts translation quality by comparing the hypothesis to the reference and source text. Scores are normalized to [0, 1].
- `COMETKiwi-22` — range: [0, 1]
  - Reference-free neural metric that predicts translation quality using only the source and hypothesis. Scores are normalized to [0, 1].
- `GRB` — range: [0, 100]
  - LLM-as-judge metric (Gemini-2.0-Flash) that scores translation quality on a 0-100 scale using the reference text.
- `GRF` — range: [0, 100]
  - LLM-as-judge metric (Gemini-2.0-Flash) that scores translation quality on a 0-100 scale without using the reference text.
- `GEA100` — range: [0, 100]
  - Specialized LLM-as-judge metric for literary translation that assesses style and expressiveness on a 0-100 scale.
- `GEA5` — range: [1, 5]
  - Coarse-grained version of GEA that assesses literary style and expressiveness on a 1-5 scale.

## Input / output format

**Input**: Source text for direct translation; source text plus draft translation and optionally a numerical quality score for post-editing.

**Output**: Target language translation (or post-edited translation).

## Scoring recipe

```python
def compute_metrics(predictions, references):
    comet22_scores = [comet22_model.predict(src=r, hyp=p) for p, r in zip(predictions, references)]
    grb_scores = [gemini_judge.score(src=r, hyp=p) for p, r in zip(predictions, references)]
    return {
        'COMET-22': mean(comet22_scores),
        'GRB': mean(grb_scores)
    }
```

## Common pitfalls

- Applying maximum reasoning budgets to direct translation tasks degrades performance due to forced extrapolation.
- Standard reference-based metrics (COMET) fail to capture literary style nuances; specialized LLM-as-judge prompts (GEA) are required for literary benchmarks.
- Post-editing without an explicit quality score signal (QS prompt) yields minimal self-correction improvements.

## Evidence (verbatim from paper)

> We assess translation quality using a suite of automatic metrics, encompassing both reference-based and reference-free approaches, alongside a specialized LLM-based judge for literary texts. For a standardized assessment, we employ two variants from the COMET framework: the reference-based COMET-22 and the reference-free COMETKiwi-22. For LLM-based evaluation, we employ Gemini-2.0-Flash. We first define two general-purpose metrics, Gemini Reference-Based (GRB) and Gemini Reference-Free (GRF), which provide a quality score on a 0-100 scale.

## Citation

```bibtex
@misc{li2025testtimescaling,
  title={Test-Time Scaling of Reasoning Models for Machine Translation},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2510.06471}
}
```

- arXiv: 2510.06471