Bianet Mt Eval

---
name: bianet-mt-eval
description: Evaluates the impact of the Bianet parallel corpus on Neural Machine Translation performance for English-Turkish and English-Kurdish language pairs in the news domain. It compares baseline models trained on existing corpora against models augmented with Bianet data, and assesses multilingual transfer learning benefits. Use when the user wants to benchmark on WMT2016, Bianet, SETIMES, Ubuntu & GNUME, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1805.05095
  bibtex_key: ataman2018bianet
  confidence: high
---

# bianet-mt-eval

> Bianet: A Parallel News Corpus in Turkish, Kurdish and English — Ataman et al. (2018) (arXiv:1805.05095, 2018)

## What this evaluates

Evaluates the impact of the Bianet parallel corpus on Neural Machine Translation performance for English-Turkish and English-Kurdish language pairs in the news domain. It compares baseline models trained on existing corpora against models augmented with Bianet data, and assesses multilingual transfer learning benefits.

## Datasets

- **WMT2016** — total 4001; splits: dev (1001), test (3000)
- **Bianet** — total 48956; splits: train (48956)
- **SETIMES** — total 205706; splits: train (205706)
- **Ubuntu & GNUME** — total 65357; splits: train (65357)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram precision with brevity penalty, computed corpus-wide. Reported on a 0-100 scale.
- `chrF3` — range: [0, 1]
  - Character n-gram F-score with n=3, computed corpus-wide.

## Input / output format

**Input**: Source sentence in English, Turkish, or Kurdish.

**Output**: Target sentence in the corresponding target language (Turkish, English, or Kurdish).

## Scoring recipe

```python
def evaluate(predictions, references):
    # predictions and references are lists of strings
    bleu = corpus_bleu(predictions, [references]).score
    chrf = corpus_chrf(predictions, [references], order=3).score
    return {'BLEU': bleu, 'chrF3': chrf}
```

## Common pitfalls

- The English-Kurdish evaluation uses a custom sampled split from Bianet (500 dev, 500 test) because no official WMT test set exists for this pair, unlike English-Turkish which uses WMT2016.
- BLEU scores are reported on a 0-100 scale in the tables, not the 0-1 scale used in some implementations.
- Models are trained with subword segmentation (BPE) and evaluated on detokenized output; mismatched tokenization can skew chrF3/bleu.

## Evidence (verbatim from paper)

> We use the BLEU (Papineni et al., 2002) and chrF3 (Popovic, 2015) automatic evaluation metrics and the Multeval (Clark et al., 2011) significance test for evaluating the accuracy of the models. The translation accuracy obtained on the WMT Turkish testing sets are given in Table 4.

## Citation

```bibtex
@misc{ataman2018bianet,
  title={Bianet: A Parallel News Corpus in Turkish, Kurdish and English},
  author={Ataman et al. (2018)},
  year={2018},
  note={arXiv:1805.05095}
}
```

- arXiv: 1805.05095