Nusax Eval

---
name: nusax-eval
description: Evaluates sentiment classification and machine translation capabilities across 10 low-resource Indonesian local languages, Indonesian, and English. It probes cross-lingual transferability, multilingual training benefits, and data efficiency for underrepresented Austronesian languages. Use when the user wants to benchmark on NusaX, or asks about evaluating this task. Reports macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.15960
  bibtex_key: winata2022nusax
  confidence: high
---

# nusax-eval

> NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages — Winata et al. (2022) (arXiv:2205.15960, 2022)

## What this evaluates

Evaluates sentiment classification and machine translation capabilities across 10 low-resource Indonesian local languages, Indonesian, and English. It probes cross-lingual transferability, multilingual training benefits, and data efficiency for underrepresented Austronesian languages.

## Datasets

- **NusaX** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/IndoNLP/nusax

## Metrics

- `macro-F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score across all sentiment classes, computed per language and averaged for overall results.
- `SacreBLEU` — range: [0, 100]
  - Standard BLEU score with sentence-level n-gram matching and brevity penalty, computed using the SacreBLEU toolkit.

## Input / output format

**Input**: For sentiment analysis: a single sentence in one of the 12 languages (10 local Indonesian languages, Indonesian, or English). For machine translation: a source sentence in a source language (e.g., Indonesian, English, or a local language).

**Output**: For sentiment analysis: a predicted sentiment class label. For machine translation: a translated sentence in the target language.

## Scoring recipe

```python
def compute_macro_f1(preds, golds):
    classes = sorted(set(preds + golds))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

def compute_sacrebleu(preds, golds):
    return sacrebleu.corpus_bleu(preds, [golds]).score
```

## Common pitfalls

- Copy baseline scores can be artificially high for languages with high lexical or grammatical overlap with Indonesian, misleadingly suggesting good translation performance.
- SacreBLEU may not capture semantic fidelity or handle rare words well, as models often copy or mistranslate them due to limited training data.
- Cross-lingual transfer results can be inflated by shared vocabulary or similar syntax rather than true generalization, especially between closely related Austronesian languages.

## Evidence (verbatim from paper)

> Table 4: Results of the machine translation task from other languages to Indonesian (x → ind) based on SacreBLEU. Table 7: Sentiment analysis results for macro-F1 (%) of XLM-R_LARGE in the multilingual setting.

## Citation

```bibtex
@misc{winata2022nusax,
  title={NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages},
  author={Winata et al. (2022)},
  year={2022},
  note={arXiv:2205.15960}
}
```

- arXiv: 2205.15960