L3cube Mahasum Eval

---
name: l3cube-mahasum-eval
description: Evaluates abstractive text summarization models on their ability to generate concise, fluent, and coherent Marathi news summaries from longer source articles. It benchmarks performance against both a newly curated large-scale dataset (MahaSum) and an existing multilingual benchmark (XL-Sum Marathi subset). Use when the user wants to benchmark on XLsum, MahaSum, or asks about evaluating this task. Reports ROUGE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.09184
  bibtex_key: deshmukh2024l3cubemahasum
  confidence: high
---

# l3cube-mahasum-eval

> L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi — Deshmukh et al. (2024) (arXiv:2410.09184, 2024)

## What this evaluates

Evaluates abstractive text summarization models on their ability to generate concise, fluent, and coherent Marathi news summaries from longer source articles. It benchmarks performance against both a newly curated large-scale dataset (MahaSum) and an existing multilingual benchmark (XL-Sum Marathi subset).

## Datasets

- **XLsum** — total 10903; splits: train (8722), test (1090), val (1090); repo https://github.com/csebuetnlp/xl-sum
- **MahaSum** — total 25374; splits: train (20299), test (2537), val (2537); repo https://github.com/l3cube-pune/MarathiNLP

## Metrics

- `ROUGE` **(primary)** — range: [0, 1]
  - Recall and precision of overlapping unigrams, bigrams, and skip-bigrams between the generated summary and reference summaries. Typically reported as ROUGE-1, ROUGE-2, and ROUGE-L (longest common subsequence).

## Input / output format

**Input**: Marathi news article text (full document, average ~315-653 words)

**Output**: Abstractive summary text (average ~18-25 words)

## Scoring recipe

```python
def compute_rouge(predictions, references):
    scores = {}
    for rouge_type in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        r = rouge_score(references, predictions, rouge_types=[rouge_type])
        scores[rouge_type] = r[rouge_type].fmeasure
    return scores
```

## Common pitfalls

- Summaries are highly abstractive and concise, so exact string matching or BLEU will yield artificially low scores; ROUGE is the standard metric used.
- The dataset uses a fixed 80/10/10 train/val/test split; using different splits or cross-validation will change reported performance.
- MahaSum is scraped from specific Marathi news sources (Lokmat, Loksatta), so models may overfit to domain-specific vocabulary and struggle on out-of-domain text.

## Evidence (verbatim from paper)

> By using MahaSum alongside XL-Sum, we evaluate how well the IndicBART model performs in summarizing Marathi text. ... performance validated via ROUGE metrics.

## Citation

```bibtex
@misc{deshmukh2024l3cubemahasum,
  title={L3Cube-MahaSum: A Comprehensive Dataset and BART Models for Abstractive Text Summarization in Marathi},
  author={Deshmukh et al. (2024)},
  year={2024},
  note={arXiv:2410.09184}
}
```

- arXiv: 2410.09184