Ment Eval

---
name: ment-eval
description: Evaluates the reliability of machine translation evaluation metrics across reference-based, quality estimation, and LLM-as-a-judge paradigms when applied to non-literal content such as internet slang, idioms, and literary expressions. Use when the user wants to benchmark on MENT, or asks about evaluating this task. Reports Composite Meta Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.07338
  bibtex_key: tian2026beyond
  confidence: high
---

# ment-eval

> Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation — Yanzhi Tian et al. (2026) (arXiv:2601.07338, 2026)

## What this evaluates

Evaluates the reliability of machine translation evaluation metrics across reference-based, quality estimation, and LLM-as-a-judge paradigms when applied to non-literal content such as internet slang, idioms, and literary expressions.

## Datasets

- **MENT** — total ?; splits: test (-1); repo https://github.com/BITHLP/RATE

## Metrics

- `Accuracy (Acc)` — range: [0, 1]
  - Proportion of correctly ranked or scored translation pairs compared to human judgments.
- `Accuracy-t (Acc-t)` — range: [0, 1]
  - Thresholded accuracy metric assessing metric alignment with human scores within a specified tolerance margin.
- `Pearson Correlation (r)` — range: [-1, 1]
  - Linear correlation coefficient measuring the strength of the linear relationship between metric scores and human reference scores.
- `Spearman correlation (ρ)` — range: [-1, 1]
  - Rank-based correlation coefficient measuring the monotonic relationship between metric and human scores.
- `Composite Meta Score` **(primary)** — range: [0, 1]
  - Average of Accuracy, Accuracy-t, Pearson Correlation, and Spearman Correlation computed across both system and segment levels.

## Input / output format

**Input**: Source sentence, reference translation, and machine-translated hypothesis (or hypothesis and reference for reference-free metrics).

**Output**: A scalar quality score representing the estimated translation quality.

## Scoring recipe

```python
def compute_composite_meta_score(predictions, gold):
    acc = compute_accuracy(predictions, gold)
    acc_t = compute_accuracy_t(predictions, gold)
    pearson_r = compute_pearson_correlation(predictions, gold)
    spearman_rho = compute_spearman_correlation(predictions, gold)
    return (acc + acc_t + pearson_r + spearman_rho) / 4.0
```

## Common pitfalls

- LLM-as-a-Judge metrics suffer from knowledge cutoffs when evaluating emerging slang or idioms absent from pre-training corpora.
- Reference-free (QE) models often fail on non-literal content because they rely on literal/formal pre-training data and cannot verify semantic nuances.
- Domain-specific challenges require different mitigation strategies; e.g., search agents are needed for historical allusions in poetry, while knowledge retrieval helps with cross-cultural slang.

## Evidence (verbatim from paper)

> Following WMT23 Metrics Shared Task Freitag et al. ([2023]), we adopt the composite meta score to assess metric performance across both system level and segment level. Our evaluation includes Accuracy (Acc, Acc-t), Pearson Correlation ($r$), and Spearman correlation ($
ho$). The final meta score is calculated as the average of individual statistics (Feng et al., [2025b]).

## Citation

```bibtex
@misc{tian2026beyond,
  title={Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation},
  author={Yanzhi Tian et al. (2026)},
  year={2026},
  note={arXiv:2601.07338}
}
```

- arXiv: 2601.07338