Liputan6 Eval

---
name: liputan6-eval
description: Evaluates the ability of models to generate concise, accurate summaries of Indonesian news articles. It probes both extractive and abstractive summarization capabilities, highlighting performance gaps on more abstract summaries and the limitations of n-gram overlap metrics. Use when the user wants to benchmark on Liputan6, or asks about evaluating this task. Reports ROUGE F-1 (R1, R2, RL).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.00679
  bibtex_key: koto2020liputan6
  confidence: high
---

# liputan6-eval

> Liputan6: A Large-scale Indonesian Dataset for Text Summarization — Koto et al. (2020) (arXiv:2011.00679, 2020)

## What this evaluates

Evaluates the ability of models to generate concise, accurate summaries of Indonesian news articles. It probes both extractive and abstractive summarization capabilities, highlighting performance gaps on more abstract summaries and the limitations of n-gram overlap metrics.

## Datasets

- **Liputan6** — total ?; splits: train (-1), dev (-1), canonical test (-1), Xtreme test (-1); repo https://github.com/fajri91/sum_liputan6

## Metrics

- `ROUGE F-1 (R1, R2, RL)` **(primary)** — range: [0, 1]
  - Computes F-1 scores based on unigram (R1), bigram (R2), and longest common subsequence (RL) overlap between the generated summary and the reference summary.
- `BERTSCORE (F-1)` — range: [0, 1]
  - Computes F-1 scores based on contextual embeddings from bert-base-multilingual-cased (layer 9), matching the protocol used for machine translation evaluation.

## Input / output format

**Input**: Indonesian news article (document text)

**Output**: Generated summary text

## Scoring recipe

```python
def score(predictions, references):
    scores = {'R1': 0, 'R2': 0, 'RL': 0, 'BERTSCORE': 0}
    for pred, ref in zip(predictions, references):
        scores['R1'] += rouge_f1(pred, ref, ngram=1)
        scores['R2'] += rouge_f1(pred, ref, ngram=2)
        scores['RL'] += rouge_f1(pred, ref, ngram='LCS')
        scores['BERTSCORE'] += bertscore_f1(pred, ref, model='bert-base-multilingual-cased', layer=9)
    return {k: v / len(predictions) for k, v in scores.items()}
```

## Common pitfalls

- ROUGE scores heavily penalize abstractive summaries due to n-gram mismatch, which may not reflect factual accuracy or fluency.
- BERTSCORE must be computed using bert-base-multilingual-cased at layer 9 to match the paper's reported values.
- The 'Xtreme' test set contains more abstract summaries, causing a significant performance drop compared to the 'canonical' test set.

## Evidence (verbatim from paper)

> We use three ROUGE (Lin, 2004) F-1 scores as evaluation metrics: R1 (unigram overlap), R2 (bigram overlap), and RL (longest common subsequence overlap). In addition, we also provide BERTSCORE (F-1), as has recently been used for machine translation evaluation (Zhang et al., 2020b). We use the development set to select the best checkpoint during training, and report the evaluation scores for the canonical and Xtreme test sets in Table 4.

## Citation

```bibtex
@misc{koto2020liputan6,
  title={Liputan6: A Large-scale Indonesian Dataset for Text Summarization},
  author={Koto et al. (2020)},
  year={2020},
  note={arXiv:2011.00679}
}
```

- arXiv: 2011.00679