Codet Eval

---
name: codet-eval
description: This benchmark probes the robustness of machine translation systems to dialectal variations by measuring how consistently they translate semantically similar sentences in standard vs. dialectal forms. It evaluates whether models maintain translation quality and coherence when exposed to lexical and morphosyntactic variations across multiple languages. Use when the user wants to benchmark on CODET, or asks about evaluating this task. Reports COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.17267
  bibtex_key: alam2023codet
  confidence: high
---

# codet-eval

> CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation — Alam et al. (2023) (arXiv:2305.17267, 2023)

## What this evaluates

This benchmark probes the robustness of machine translation systems to dialectal variations by measuring how consistently they translate semantically similar sentences in standard vs. dialectal forms. It evaluates whether models maintain translation quality and coherence when exposed to lexical and morphosyntactic variations across multiple languages.

## Datasets

- **CODET** — total ?; splits: test (-1); repo https://github.com/mahfuzibnalam/dialect_mt

## Metrics

- `COMET` **(primary)** — range: [0, 1]
  - A neural framework that leverages both the source input and a target-language reference translation to predict MT quality, correlating with human judgments. In this protocol, the standard-variety output serves as the pseudo-reference.
- `BLEU` — range: [0, 100]
  - Compares n-gram matches between the candidate translation and the reference translation to determine similarity. Calculated using SacreBLEU.

## Input / output format

**Input**: A pair of sentences: a dialectal input sentence ($\mathbf{x}$) and its standard-variety counterpart ($\tilde{\mathbf{x}}$).

**Output**: Two translations: $\mathbf{y}$ (model output for $\mathbf{x}$) and $\tilde{\mathbf{y}}$ (model output for $\tilde{\mathbf{x}}$).

## Scoring recipe

```python
scores = []
for x, x_tilde in dataset:
    y = model.translate(x)
    y_tilde = model.translate(x_tilde)
    # Use standard output as pseudo-reference
    score = comet(candidate=y, reference=y_tilde, source=x)
    scores.append(score)
return mean(scores)
```

## Common pitfalls

- Varying numbers of contrastive sentences across dialects prevent direct cross-dialect score comparison without subsampling.
- For languages with empty dialect intersections (e.g., Italian), scores are derived from random 100-sentence samples averaged over 100 runs, introducing sampling variance.
- Using the standard-variety output as a pseudo-reference assumes the model translates the standard input perfectly, which may not hold for weak models.

## Evidence (verbatim from paper)

> The core of the idea is that we can treat $\tilde{\mathbf{y}}$, the output of the MT system on the "standard" input, as a pseudo-reference for the translation. Intuitively, a robust system should produce the same output for inputs with similar meanings regardless of the small dialectal variations. Hence, we can calculate any MT metric such as BLEU or COMET by comparing $\mathbf{y}$ to $\tilde{\mathbf{y}}$.

## Citation

```bibtex
@misc{alam2023codet,
  title={CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation},
  author={Alam et al. (2023)},
  year={2023},
  note={arXiv:2305.17267}
}
```

- arXiv: 2305.17267