Turkish Mt Eval

---
name: turkish-mt-eval
description: Measures the quality of bidirectional translation between Turkish and English. It evaluates how well models capture cross-lingual semantic alignment and syntactic restructuring across different corpus types. Use when the user wants to benchmark on Wmt-16 (Turkish-English subset), MuST-C (Turkish-English subset), or asks about evaluating this task. Reports BLEU Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.01215
  bibtex_key: [REDACTED]
  confidence: high
---

# turkish-mt-eval

> Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)

## What this evaluates

Measures the quality of bidirectional translation between Turkish and English. It evaluates how well models capture cross-lingual semantic alignment and syntactic restructuring across different corpus types.

## Datasets

- **Wmt-16 (Turkish-English subset)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese
- **MuST-C (Turkish-English subset)** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese

## Metrics

- `BLEU Score` **(primary)** — range: [0, 100]
  - Standard n-gram overlap metric (Papineni et al. 2002) computed over the test set. Evaluated in both tr-en and en-tr directions.

## Input / output format

**Input**: Source language sentence (Turkish or English).

**Output**: Translated target language sentence.

## Scoring recipe

```python
def score_mt(predictions, gold):
    # Standard BLEU calculation
    bleu = compute_bleu(reference=[gold], hypothesis=predictions)
    return bleu * 100  # Paper reports scores on 0-100 scale
```

## Common pitfalls

- BLEU scores are direction-dependent; tr-en and en-tr must be evaluated separately and reported distinctly.
- Pre-trained models like mBART50 are fine-tuned separately per dataset, so cross-dataset generalization is not measured.
- Results from scratch vs pre-trained models are reported in the same table but trained under different protocols.

## Evidence (verbatim from paper)

> We evaluate our models on the relevant test sets for translation in both directions. We utilize Bleu Score Papineni et al. (2002) for the assessment of translation quality.

## Citation

```bibtex
@misc{safaya2022mukayese,
  title={Mukayese: Turkish NLP Strikes Back},
  author={Safaya et al. (2022)},
  year={2022},
  note={arXiv:2203.01215}
}
```

- arXiv: 2203.01215