Nepali Summarization Eval

---
name: nepali-summarization-eval
description: Evaluates decoder-based language models on abstractive text summarization for Nepali news articles, testing generation quality and context handling. Use when the user wants to benchmark on Nepali Summarization Dataset (Bhandari 2024), or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.15734
  bibtex_key: thapa2024development
  confidence: high
---

# nepali-summarization-eval

> Development of Pre-Trained Transformer-based Models for the Nepali Language — Thapa et al. (2024) (arXiv:2411.15734, 2024)

## What this evaluates

Evaluates decoder-based language models on abstractive text summarization for Nepali news articles, testing generation quality and context handling.

## Datasets

- **Nepali Summarization Dataset (Bhandari 2024)** — total 7258; splits: train (5806), test (1452)

## Metrics

- `ROUGE-1` — range: percent
  - Unigram overlap between the generated summary and the reference summary.
- `ROUGE-2` — range: percent
  - Bigram overlap between the generated summary and the reference summary.
- `ROUGE-L` **(primary)** — range: percent
  - Longest common subsequence overlap between the generated summary and the reference summary.

## Input / output format

**Input**: Source Nepali text sequences (typically ≤400 tokens for optimal performance due to model context limits).

**Output**: Abstractive summary text.

## Scoring recipe

```python
def rouge_eval(predictions, golds):
    scores = {'ROUGE-1': [], 'ROUGE-2': [], 'ROUGE-L': []}
    for pred, gold in zip(predictions, golds):
        for metric in scores:
            scores[metric].append(calculate_rouge(pred, gold, metric))
    return {k: sum(v)/len(v) for k, v in scores.items()}
```

## Common pitfalls

- Model performance degrades significantly on inputs exceeding 400 tokens due to the original 512-token context window limit.
- Evaluation relies on a single publicly available dataset; results may not generalize to other Nepali domains or genres.

## Evidence (verbatim from paper)

> There were no existing benchmarks for NLG tasks, so we used abstractive summarization for the evaluation of GPT-2... The dataset consists of 7,258 data points, where we used 5,806 (80%) data points for training and the remaining 1,452 (20%) data points for evaluation... We used the ROUGE score (Lin, 2004) Lin and Och ([2004]) as our evaluation metric.

## Citation

```bibtex
@misc{thapa2024development,
  title={Development of Pre-Trained Transformer-based Models for the Nepali Language},
  author={Thapa et al. (2024)},
  year={2024},
  note={arXiv:2411.15734}
}
```

- arXiv: 2411.15734