Indolem Eval

---
name: indolem-eval
description: Evaluates Indonesian NLP capabilities across morpho-syntax, semantics, and discourse. It probes token-level labeling (POS, NER), syntactic structure (dependency parsing), text classification (sentiment), generation (summarization), and discourse coherence (next tweet prediction, tweet ordering). Use when the user wants to benchmark on INDOLEM, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.00677
  bibtex_key: koto2020indolem
  confidence: high
---

# indolem-eval

> IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP — Koto et al. (2020) (arXiv:2011.00677, 2020)

## What this evaluates

Evaluates Indonesian NLP capabilities across morpho-syntax, semantics, and discourse. It probes token-level labeling (POS, NER), syntactic structure (dependency parsing), text classification (sentiment), generation (summarization), and discourse coherence (next tweet prediction, tweet ordering).

## Datasets

- **INDOLEM** — total ?; splits: train (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels or classes out of the total number of instances.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall. For NER, computed at the entity level; for classification tasks, typically macro or micro averaged.
- `UAS/LAS` — range: [0, 1]
  - Unlabeled/Labelled Attachment Score: percentage of words whose head word and dependency label are predicted correctly.
- `ROUGE-1/2/L` — range: [0, 1]
  - Recall-oriented overlap metrics measuring the similarity between reference and generated summaries using unigrams, bigrams, and longest common subsequence.
- `Spearman's ρ` — range: [-1, 1]
  - Rank correlation coefficient measuring the monotonic relationship between predicted and true tweet orderings.

## Input / output format

**Input**: Indonesian text inputs including sentences for POS/NER/parsing, tweets for discourse tasks, and source documents for summarization.

**Output**: Task-specific outputs: token-level labels, entity spans, dependency head/label pairs, sentiment class labels, generated summary text, or ranked/predicted next tweet indices.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # Accuracy
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    # F1 (entity-level for NER)
    tp = sum(1 for p, g in zip(predictions, gold) if p == g and g != 'O')
    prec = tp / max(1, sum(1 for p in predictions if p != 'O'))
    rec = tp / max(1, sum(1 for g in gold if g != 'O'))
    f1 = 2 * prec * rec / max(1e-9, prec + rec)
    return {'accuracy': acc, 'f1': f1}
```

## Common pitfalls

- Comparing results across studies without verifying test partition consistency (e.g., UD-Indo-PUD versions differ between papers, making direct comparison invalid).
- Human baseline performance was only evaluated on a random 100-instance subset, not the full test set, which may not represent overall human capability on the benchmark.
- POS tagging evaluation uses 5-fold cross-validation, while NER and other tasks report single test-set scores, requiring careful aggregation when comparing model performance.

## Evidence (verbatim from paper)

> Table 3: Results on POS and NER tasks using accuracy averaged over five folds for POS tagging task, and entity-level F1 over the test set for the NER tasks.

## Citation

```bibtex
@misc{koto2020indolem,
  title={IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP},
  author={Koto et al. (2020)},
  year={2020},
  note={arXiv:2011.00677}
}
```

- arXiv: 2011.00677