Bhashaverse Translation Eval

---
name: bhashaverse-translation-eval
description: Evaluates multilingual machine translation and related sequence-to-sequence tasks across 36 Indian subcontinent languages. It probes the model's ability to handle morphological complexity, script diversity, code-mixing, and domain-specific adaptation through reference-based and reference-free metrics. Use when the user wants to benchmark on FLORES + IN22, Reserved Development Corpora, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.04351
  bibtex_key: mujadia2024bhashaverse
  confidence: high
---

# bhashaverse-translation-eval

> BhashaVerse : Translation Ecosystem for Indian Subcontinent Languages — Mujadia et al. (2024) (arXiv:2412.04351, 2024)

## What this evaluates

Evaluates multilingual machine translation and related sequence-to-sequence tasks across 36 Indian subcontinent languages. It probes the model's ability to handle morphological complexity, script diversity, code-mixing, and domain-specific adaptation through reference-based and reference-free metrics.

## Datasets

- **FLORES + IN22** — total 902198; splits: test (902198)
- **Reserved Development Corpora** — total 22000; splits: dev (22000)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Geometric mean of precisions up to n-grams (typically 4-gram) with a brevity penalty to penalize overly short translations.
- `CHRF3` — range: [0, 100]
  - Character n-gram F-score using trigrams, measuring overlap between reference and hypothesis at the character level.
- `COMET-22` — range: [0, 1]
  - Reference-based neural metric trained on human judgments, outputting a score between 0 and 1 representing translation quality.
- `Spearman Correlation` — range: [-1, 1]
  - Rank-based correlation coefficient measuring the monotonic relationship between predicted scores and human judgments.
- `F1 Score` — range: [0, 1]
  - Harmonic mean of precision and recall for token-level or span-level error identification tasks.

## Input / output format

**Input**: Source sentence (English or Indian subcontinent language) or grammatically incorrect sentence for correction tasks.

**Output**: Target sentence (translated or corrected output).

## Scoring recipe

```python
def compute_metrics(preds, refs):
    bleu = sacrebleu.corpus_bleu(preds, [refs]).score
    chrf = sacrebleu.corpus_chrf(preds, [refs]).score
    f1 = f1_score(refs, preds, average='macro')
    spearman = spearmanr(refs, preds).correlation
    comet = comet_model.predict(preds, refs)
    return {'bleu': bleu, 'chrf3': chrf, 'f1': f1, 'spearman': spearman, 'comet22': comet}
```

## Common pitfalls

- The development corpora for non-MT tasks are explicitly curated from the training data, risking data leakage if treated as independent test sets.
- Average scores are reported across 36 languages with vastly different resource levels, masking performance on low-resource pairs.
- CHRF3 is used instead of the more common CHRF++ or standard CHRF, requiring careful implementation to match the reported scores.

## Evidence (verbatim from paper)

> The model is evaluated on machine translation using benchmark corpora such as FLORES and IN22, covering available languages, including English and other Indian subcontinent languages. With a combined dataset size of 902,198 sentences, the model achieved an average BLEU score (Papineni et al., [2002]) of 25.45, CHRF3 score (Popović, [2015]) of 53.81, and COMET-22 score (Rei et al., [2022a]) of 0.8138.

## Citation

```bibtex
@misc{mujadia2024bhashaverse,
  title={BhashaVerse : Translation Ecosystem for Indian Subcontinent Languages},
  author={Mujadia et al. (2024)},
  year={2024},
  note={arXiv:2412.04351}
}
```

- arXiv: 2412.04351