Indonesian Pos Tagging Eval

---
name: indonesian-pos-tagging-eval
description: Evaluates sequence labeling performance on Indonesian text by assigning part-of-speech tags to tokens. It probes morphological feature extraction, contextual understanding, and robustness to annotation inconsistencies and rare lexical categories. Use when the user wants to benchmark on IDN Tagged Corpus, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1809.03391
  bibtex_key: kurniawan2018indonesian
  confidence: high
---

# indonesian-pos-tagging-eval

> Toward a Standardized and More Accurate Indonesian Part-of-Speech Tagging — Kurniawan et al. (2018) (arXiv:1809.03391, 2018)

## What this evaluates

Evaluates sequence labeling performance on Indonesian text by assigning part-of-speech tags to tokens. It probes morphological feature extraction, contextual understanding, and robustness to annotation inconsistencies and rare lexical categories.

## Datasets

- **IDN Tagged Corpus** — total ?; splits: 5-fold CV (-1); repo https://github.com/kmkurn/id-pos-tagging

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Token-level F1 score computed over all POS tags. Calculated as 2 * (precision * recall) / (precision + recall), where precision and recall are derived from exact match counts between predicted and gold tags across all tokens.

## Input / output format

**Input**: A sequence of Indonesian words/tokens.

**Output**: A sequence of POS tags corresponding to each input token.

## Scoring recipe

```python
def compute_f1(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    total = len(gold)
    precision = correct / total
    recall = correct / total
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1
```

## Common pitfalls

- Annotation inconsistency for WH and SC tags causes models to struggle with these specific categories.
- The rare 'X' tag contains typos, slang, and foreign terms, often misclassified as nouns due to ambiguous morphological cues.
- Evaluating without 5-fold cross-validation may yield unreliable results due to dataset distribution variance.

## Evidence (verbatim from paper)

> The F1 scores are averaged over the 5 cross-validation folds. We see that Major baseline performs very poorly compared with the Memo baseline, which surprisingly achieves over 90 F1 points. This result suggests that Memo is a more suitable baseline for this dataset in contrast with Major. The result also provides evidence to the usefulness of our evaluation metric which heavily penalizes a simple majority vote model. Furthermore, we notice that the rule-based tagger by Rashel et al. performs worse than Memo, indicating that Memo is not just suitable but also quite a strong baseline. Moving on, we observe how CRF has 6 points advantage over Memo, signaling that incorporating contextual features and modeling tag-to-tag transitions are useful. Lastly, the biLSTM with CRF tagger performs the best with 97.47 F1 score.

## Citation

```bibtex
@misc{kurniawan2018indonesian,
  title={Toward a Standardized and More Accurate Indonesian Part-of-Speech Tagging},
  author={Kurniawan et al. (2018)},
  year={2018},
  note={arXiv:1809.03391}
}
```

- arXiv: 1809.03391