Bouquet Mt Eval

---
name: bouquet-mt-eval
description: Evaluates machine translation systems on a contamination-free, multilingual dataset covering diverse domains and registers. It measures translation quality at both sentence and paragraph levels to assess how well models handle linguistic diversity and cultural authenticity across 8 major languages. Use when the user wants to benchmark on BOUQuET, or asks about evaluating this task. Reports CometKiwi.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.04314
  bibtex_key: andrews2025bouquet
  confidence: high
---

# bouquet-mt-eval

> BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation — Andrews et al. (2025) (arXiv:2502.04314, 2025)

## What this evaluates

Evaluates machine translation systems on a contamination-free, multilingual dataset covering diverse domains and registers. It measures translation quality at both sentence and paragraph levels to assess how well models handle linguistic diversity and cultural authenticity across 8 major languages.

## Datasets

- **BOUQuET** — total ?; splits: test (-1)

## Metrics

- `CometKiwi` **(primary)** — range: [0, 1]
  - Neural MT quality metric predicting human-like scores on a 0-1 scale, where higher values indicate better translation quality. Uses the CometKiwi-da-xl model.
- `MetricX` — range: [0, 25]
  - Hybrid MT quality metric predicting Direct Assessment scores on a 0-25 scale, where lower values indicate better translation quality. Uses the MetricX-24-hybrid-xl-v2p6 model.

## Input / output format

**Input**: Source sentence or paragraph in one of the 9 source languages (8 BOUQuET languages plus English) paired with a reference translation.

**Output**: Model-generated translation in the target language.

## Scoring recipe

```python
def evaluate(system, dataset, level='sentence'):
    com_scores, metx_scores = [], []
    for src, ref, pred in dataset[level]:
        com_scores.append(cometkiwi_da_xl.predict(src, pred))
        metx_scores.append(metricx_24.predict(src, pred))
    avg_com = mean(com_scores)
    avg_metx = mean(metx_scores)
    ranking = rank_systems_by(com_scores)
    return {'COM': avg_com, 'MetX': avg_metx, 'ranking': ranking}
# Compare rankings across datasets using swap count and pearson correlation
```

## Common pitfalls

- Evaluating at sentence-level versus paragraph-level yields significantly different system rankings and lower Pearson correlation, so the evaluation level must be explicitly reported.
- MetricX uses an inverse scale (lower is better) compared to CometKiwi (higher is better), which can cause confusion when comparing or ranking systems across datasets.
- The dataset covers 8 languages plus English as a pivot, so evaluation directions must be specified and averaged correctly across all 9 source languages.

## Evidence (verbatim from paper)

> Following the official evaluation metrics of WMT 2024 (Kocmi et al., [2024]), we use two automatic metrics: CometKiwi (CometKiwi-da-xl, range 0-1 and ↑ better, COM) (Chimoto and Bassett, [2022]) and MetricX (MetricX-24-hybrid-xl-v2p6, range 0-25 and ↓ better, MetX) (Juraska et al., [2024]). We include in the benchmarking datasets that cover Source-BOUQuET languages (FLORES+ and NTREX-128).

## Citation

```bibtex
@misc{andrews2025bouquet,
  title={BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation},
  author={Andrews et al. (2025)},
  year={2025},
  note={arXiv:2502.04314}
}
```

- arXiv: 2502.04314