Terminology Aware Translation Eval

---
name: terminology-aware-translation-eval
description: Evaluates a machine translation system's ability to balance overall translation quality with strict adherence to specified terminology constraints across different languages. It measures how well the model enforces lexical rules without degrading fluency or adequacy, particularly in morphologically complex languages. Use when the user wants to benchmark on EN-{DE,ES,RU} translation test sets, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.07461
  bibtex_key: jaswal2025dual
  confidence: high
---

# terminology-aware-translation-eval

> It Takes Two: A Dual Stage Approach for Terminology-Aware Translation — Jaswal (2025) (arXiv:2511.07461, 2025)

## What this evaluates

Evaluates a machine translation system's ability to balance overall translation quality with strict adherence to specified terminology constraints across different languages. It measures how well the model enforces lexical rules without degrading fluency or adequacy, particularly in morphologically complex languages.

## Datasets

- **EN-{DE,ES,RU} translation test sets** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram overlap metric between reference and hypothesis translations, computed with sentence averaging and smoothing. Reported as a percentage (0-100).
- `chrF2++` — range: [0, 100]
  - Character-level n-gram F-score combining unigrams and bigrams with character n-grams, designed to be robust to morphological variations and spelling differences.
- `terminology success rate (proper)` — range: [0, 1]
  - Proportion of required terminology terms correctly inserted into the generated translation.
- `terminology success rate (random)` — range: [0, 1]
  - Proportion of randomly assigned terminology terms successfully inserted into the translation, used as a control to measure the model's constraint enforcement capability independent of contextual appropriateness.

## Input / output format

**Input**: Source sentence in English

**Output**: Translated sentence in the target language (German, Spanish, or Russian)

## Scoring recipe

```python
def compute_metrics(hypotheses, references, required_terms):
    bleu = sacrebleu.corpus_bleu(hypotheses, [references]).score
    chrf = sacrebleu.corpus_chrf(hypotheses, [references]).score
    
    proper_sr = 0.0
    random_sr = 0.0
    for hyp, terms in zip(hypotheses, required_terms):
        if len(terms) == 0: continue
        proper_sr += sum(1 for t in terms if t in hyp) / len(terms)
        random_sr += sum(1 for t in terms if t in hyp) / len(terms)
    proper_sr /= len(hypotheses)
    random_sr /= len(hypotheses)
    return {'BLEU': bleu, 'chrF2++': chrf, 'proper_SR': proper_sr, 'random_SR': random_sr}
```

## Common pitfalls

- High terminology success rates do not guarantee high BLEU or chrF2++ scores if constraints are forced inappropriately, compromising contextual fluency.
- Morphologically complex languages (e.g., Russian) exhibit larger performance gaps between constrained and unconstrained translation, making evaluation more sensitive to constraint handling.
- Random SR measures raw insertion capability rather than contextual appropriateness, potentially masking translation quality issues when constraints are misapplied.

## Evidence (verbatim from paper)

> We evaluate the system using three complementary metrics used by the WMT organizers: BLEU for overall translation adequacy, chrF2++ for character-level fluency and robustness, and terminology success rates (proper and random) to directly measure constraint satisfaction (papineni-etal-2002-bleu; popovic-2015-chrf).

## Citation

```bibtex
@misc{jaswal2025dual,
  title={It Takes Two: A Dual Stage Approach for Terminology-Aware Translation},
  author={Jaswal (2025)},
  year={2025},
  note={arXiv:2511.07461}
}
```

- arXiv: 2511.07461