Nmt Low Resource Indonesian Eval

---
name: nmt-low-resource-indonesian-eval
description: Evaluates neural machine translation performance across eight translation directions involving Indonesian and four low-resource Indonesian local languages (Javanese, Sundanese, Minangkabau, Balinese). It probes how different training paradigms (unsupervised, semi-supervised) and data augmentation strategies impact translation quality when parallel data is scarce. Use when the user wants to benchmark on Indonesian Local Language NMT Corpus, or asks about evaluating this task. Reports spm200BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.00998
  bibtex_key: susanto2023replicable
  confidence: high
---

# nmt-low-resource-indonesian-eval

> Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia — Susanto et al. (2023) (arXiv:2311.00998, 2023)

## What this evaluates

Evaluates neural machine translation performance across eight translation directions involving Indonesian and four low-resource Indonesian local languages (Javanese, Sundanese, Minangkabau, Balinese). It probes how different training paradigms (unsupervised, semi-supervised) and data augmentation strategies impact translation quality when parallel data is scarce.

## Datasets

- **Indonesian Local Language NMT Corpus** — total ?; splits: test (-1)

## Metrics

- `spm200BLEU` **(primary)** — range: other
  - Sentence-level BLEU score computed on text tokenized using a SentencePiece vocabulary of 200,000 subwords. It measures n-gram overlap between the model's translation and the reference, typically with a geometric mean of unigrams to 4-grams.

## Input / output format

**Input**: Source sentence in Indonesian or a local Indonesian language (Javanese, Sundanese, Minangkabau, or Balinese).

**Output**: Translated target sentence in the corresponding local language or Indonesian.

## Scoring recipe

```python
def compute_spm200bleu(predictions, references):
    pred_tok = [spm200.tokenize(p) for p in predictions]
    ref_tok = [[spm200.tokenize(r)] for r in references]
    scores = [sentence_bleu(p, r) for p, r in zip(pred_tok, ref_tok)]
    return mean(scores) * 100
```

## Common pitfalls

- The metric is specifically spm200BLEU (SentencePiece 200k vocabulary), not standard BLEU, making direct comparison with other NMT benchmarks difficult.
- Performance is highly asymmetric across translation directions (e.g., id->jv vs jv->id) and heavily dependent on the availability of parallel training data, so reporting a single aggregate score without specifying direction and language pair is misleading.
- Synthetic data augmentation (AUG) is only applied to semi-supervised/supervised training paradigms and does not affect unsupervised training results.

## Evidence (verbatim from paper)

> The results of these experiments (all metrics are inspm200BLEU, shown in Table 3), reveal a consistent trend: CodeXL approach results in a significantly better performing NMT systems compared to Scratch and PreXL.

## Citation

```bibtex
@misc{susanto2023replicable,
  title={Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia},
  author={Susanto et al. (2023)},
  year={2023},
  note={arXiv:2311.00998}
}
```

- arXiv: 2311.00998