Bayling2 Multilingual Eval

---
name: bayling2-multilingual-eval
description: Evaluates multilingual translation quality and cross-lingual reasoning across high-resource and low-resource languages. It probes the model's ability to align languages and transfer capabilities from high-resource to low-resource settings without extensive low-resource instruction data. Use when the user wants to benchmark on Flores-101, WMT22, Belebele, XNLI, GSM8K, or asks about evaluating this task. Reports BLEU (sacrebleu), COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.16300
  bibtex_key: zhang2024bayling2
  confidence: high
---

# bayling2-multilingual-eval

> BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment — Zhang et al. (2024) (arXiv:2411.16300, 2024)

## What this evaluates

Evaluates multilingual translation quality and cross-lingual reasoning across high-resource and low-resource languages. It probes the model's ability to align languages and transfer capabilities from high-resource to low-resource settings without extensive low-resource instruction data.

## Datasets

- **Flores-101** — total ?; splits: test (-1)
- **WMT22** — total ?; splits: test (-1)
- **Belebele** — total ?; splits: test (-1)
- **XNLI** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)

## Metrics

- `BLEU (sacrebleu)` **(primary)** — range: percent
  - Measures statistical similarity based on n-gram precision between the generated translation and reference translations, typically reported as a percentage.
- `COMET` **(primary)** — range: percent
  - Measures semantic similarity using cross-lingual pre-trained models, regarded as the most human-aligned evaluation metric for translation tasks.
- `Accuracy` — range: [0, 1]
  - Calculates the proportion of correctly selected options in multiple-choice benchmarks.

## Input / output format

**Input**: Source language sentence (for translation) or multiple-choice question with options (for multi-task/general benchmarks).

**Output**: Target language sentence (for translation) or selected option letter/text (for multi-task/general benchmarks).

## Scoring recipe

```python
def score(predictions, golds, metric):
    if metric == 'bleu':
        return sacrebleu.corpus_bleu(predictions, [golds]).score
    elif metric == 'comet':
        return comet_model.predict(predictions, golds).score
    else:  # accuracy
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
```

## Common pitfalls

- Flores-101 low-resource languages require a 1-shot in-context example to prevent off-target generation, unlike the 0-shot setting used for other benchmarks.
- COMET evaluates semantic similarity via cross-lingual models rather than surface n-gram overlap, so it can diverge significantly from BLEU scores.

## Evidence (verbatim from paper)

> For metrics, BLEU (sacrebleu) (Post, [2018]) and COMET (Rei et al., [2022]) are used to assess the quality of LLMs’ translation. BLEU score measures the statistical similarity based on n-gram accuracy, COMET score measures the semantic similarity using cross-lingual pre-trained models, which is currently regarded as the most human-aligned evaluation metric for translation tasks.

## Citation

```bibtex
@misc{zhang2024bayling2,
  title={BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.16300}
}
```

- arXiv: 2411.16300