Indonlu Eval

---
name: indonlu-eval
description: This benchmark evaluates Indonesian natural language understanding across 12 diverse tasks, including single-sentence classification, sentence-pair classification, and sequence labeling/tagging. It probes a model's ability to handle sentiment analysis, aspect-based sentiment, textual entailment, part-of-speech tagging, named entity recognition, keyphrase extraction, and question answering in Indonesian. Use when the user wants to benchmark on IndoNLU, or asks about evaluating this task. Reports macro-averaged F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.05387
  bibtex_key: wilie2020indonlu
  confidence: high
---

# indonlu-eval

> IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding — Wilie et al. (2020) (arXiv:2009.05387, 2020)

## What this evaluates

This benchmark evaluates Indonesian natural language understanding across 12 diverse tasks, including single-sentence classification, sentence-pair classification, and sequence labeling/tagging. It probes a model's ability to handle sentiment analysis, aspect-based sentiment, textual entailment, part-of-speech tagging, named entity recognition, keyphrase extraction, and question answering in Indonesian.

## Datasets

- **IndoNLU** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `macro-averaged F1` **(primary)** — range: [0, 1]
  - Macro-averaged F1 score computed separately for classification and sequence labeling tasks. For classification, it uses top-1 predictions. For sequence labeling, it uses word-level IOB matching following the CoNLL evaluation script. Two separate mean F1 scores are reported.

## Input / output format

**Input**: Tokenized Indonesian text (single sentence or sentence pair) with corresponding gold labels (class labels for classification, IOB tags for sequence labeling).

**Output**: Predicted class labels (top-1) for classification tasks, or predicted IOB tag sequences for sequence labeling tasks.

## Scoring recipe

```python
def compute_macro_f1(preds, golds, task_type):
    if task_type == 'classification':
        preds = [p.argmax() for p in preds]
    f1_scores = []
    for label in unique_labels:
        tp = sum(1 for p, g in zip(preds, golds) if p == label and g == label)
        fp = sum(1 for p, g in zip(preds, golds) if p == label and g != label)
        fn = sum(1 for p, g in zip(preds, golds) if p != label and g == label)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- Using micro-averaged F1 instead of the specified macro-averaged F1.
- Reporting a single overall F1 score instead of two separate mean F1 scores for classification and sequence labeling tasks.
- Evaluating sequence labeling at the character or span level rather than using the specified word-level IOB matching protocol.

## Evidence (verbatim from paper)

> We use the F1 score to measure the evaluation performance of all tasks. For the binary and multi-label classification tasks, we measure the macro-averaged F1 score by taking the top-1 prediction from the model. For the sequence labeling task, we calculate word-level sequence labeling macro-averaged F1-score for all models by following the sequence labeling evaluation method described in the CoNLL evaluation script. We calculate two mean F1-scores separately for classification and sequence labeling tasks to evaluate models on our IndoNLU benchmark.

## Citation

```bibtex
@misc{wilie2020indonlu,
  title={IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding},
  author={Wilie et al. (2020)},
  year={2020},
  note={arXiv:2009.05387}
}
```

- arXiv: 2009.05387