Canmt Eval

---
name: canmt-eval
description: Evaluates large language models and specialized MT systems on culture-aware machine translation across 12 language pairs. It probes the models' ability to preserve cultural nuances and adapt to explicit semantic versus communicative translation constraints. Use when the user wants to benchmark on CanMT, or asks about evaluating this task. Reports translation performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.24361
  bibtex_key: yuan2026culturemt
  confidence: medium
---

# canmt-eval

> Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation — Yuan et al. (2026) (arXiv:2604.24361, 2026)

## What this evaluates

Evaluates large language models and specialized MT systems on culture-aware machine translation across 12 language pairs. It probes the models' ability to preserve cultural nuances and adapt to explicit semantic versus communicative translation constraints.

## Datasets

- **CanMT** — total ?; splits: test (-1); repo https://github.com/zkyuan-scir/CanMT-a-Culture-Aware-Novel-Driven-Parallel-Dataset-for-Machine-Translation

## Metrics

- `translation performance` **(primary)** — range: [1, 5]
  - A 1-to-5 scale rating assessing translation quality, fluency, and cultural appropriateness. Scores are averaged across 12 language pairs to compute an overall average.

## Input / output format

**Input**: Source sentence or passage from literary novels in English, Spanish, Japanese, or Russian.

**Output**: Translated sentence or passage in the target language (English, Spanish, Japanese, Chinese, or Russian).

## Scoring recipe

```python
def compute_metric(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        # Rated on a 1-5 scale for fluency, adequacy, and cultural appropriateness
        score = llm_judge_score(pred, ref)
        scores.append(score)
    return sum(scores) / len(scores) if scores else 0.0
```

## Common pitfalls

- Models often demonstrate a 'knowledge-application gap', recognizing cultural knowledge but failing to operationalize it in the final translation.
- LLM-as-a-judge evaluations can be unreliable without reference translations to ground the scoring criteria.
- Performance heavily depends on the translation paradigm constraint (semantic vs. communicative) applied in the prompt.

## Evidence (verbatim from paper)

> Table 4: Overall translation performance across language directions. For the Qwen3 series and Seed-X models, only the non-reasoning variants are included.

## Citation

```bibtex
@misc{yuan2026culturemt,
  title={Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation},
  author={Yuan et al. (2026)},
  year={2026},
  note={arXiv:2604.24361}
}
```

- arXiv: 2604.24361