Milpac Eval

---
name: milpac-eval
description: Evaluates the quality of machine translation systems translating English legal text into nine Indian languages. It benchmarks commercial and open-source models against human-translated references to assess domain-specific translation accuracy and metric correlation. Use when the user wants to benchmark on MILPaC, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.09765
  bibtex_key: mahapatra2023milpac
  confidence: high
---

# milpac-eval

> MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages — Mahapatra et al. (2023) (arXiv:2310.09765, 2023)

## What this evaluates

Evaluates the quality of machine translation systems translating English legal text into nine Indian languages. It benchmarks commercial and open-source models against human-translated references to assess domain-specific translation accuracy and metric correlation.

## Datasets

- **MILPaC** — total ?; splits: MILPaC-IP (-1), MILPaC-Acts (-1), MILPaC-CCI-FAQ (1460)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - BP * exp(sum_{n=1}^4 w_n log p_n), where p_n is clipped n-gram precision (n=1..4) and BP is brevity penalty. Scaled to [0,100].
- `GLEU` — range: [0, 100]
  - Minimum of n-gram precision and recall for n=1..4. Scaled to [0,100].
- `chrF++` — range: [0, 100]
  - Arithmetic mean of character n-gram (order 6) and word n-gram (order 2) F-scores. Scaled to [0,100].

## Input / output format

**Input**: English legal text unit ($x_i$) to be translated to a target Indian language. For LLMs, a one-shot prompt containing an example English-to-target translation pair is prepended.

**Output**: Translated text in the target Indian language.

## Scoring recipe

```python
def compute_metrics(hypothesis, reference, lang):
    hyp_tok = indicnlp_tokenize(hypothesis, lang)
    ref_tok = indicnlp_tokenize(reference, lang)
    p_n = [clipped_precision(hyp_tok, ref_tok, n) for n in range(1, 5)]
    bp = 1.0 if len(hyp_tok) > len(ref_tok) else math.exp(1 - len(ref_tok)/len(hyp_tok))
    bleu = bp * math.exp(sum(0.25 * math.log(p) for p in p_n))
    prec = ngram_precision(hyp_tok, ref_tok, max_n=4)
    rec = ngram_recall(hyp_tok, ref_tok, max_n=4)
    gleu = min(prec, rec)
    chrf = char_word_fscore(hyp_tok, ref_tok, nc=6, nw=2)
    return bleu * 100, gleu * 100, chrf * 100
```

## Common pitfalls

- LLMs (Davinci-003, GPT-3.5T-Inst) have a 4096 token limit, causing the 77 longest text units in MILPaC-CCI-FAQ to be excluded from their evaluation.
- mBART-50 lacks support for Oriya and Punjabi, so it is not evaluated for those languages.
- All metric scores are scaled to [0, 100] for reporting, not the standard [0, 1] range.
- Tokenization must use IndicNLP for Indian languages before computing BLEU/GLEU/chrF++ to ensure reproducibility.

## Evidence (verbatim from paper)

> We use the following standard metrics to evaluate the performance of MT systems: BLEU: BLEU (Bi-Lingual Evaluation Understudy) is an automatic MT evaluation metric (Papineni et al., 2002). It measures the overlap between the translation generated by an MT system (t_h) and the reference translation (t_r), by considering n-grams based precision (n=1,2,3,4).

## Citation

```bibtex
@misc{mahapatra2023milpac,
  title={MILPaC: A Novel Benchmark for Evaluating Translation of Legal Text to Indian Languages},
  author={Mahapatra et al. (2023)},
  year={2023},
  note={arXiv:2310.09765}
}
```

- arXiv: 2310.09765