Varta Headline Eval

---
name: varta-headline-eval
description: Evaluates abstractive headline generation across 15 Indic languages and English. It probes cross-lingual transfer, script normalization effects, and the impact of language-family-specific pretraining on low-resource generation. Use when the user wants to benchmark on Varta, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.05858
  bibtex_key: aralikatte2023varta
  confidence: high
---

# varta-headline-eval

> V\=arta: A Large-Scale Headline-Generation Dataset for Indic Languages — Aralikatte et al. (2023) (arXiv:2305.05858, 2023)

## What this evaluates

Evaluates abstractive headline generation across 15 Indic languages and English. It probes cross-lingual transfer, script normalization effects, and the impact of language-family-specific pretraining on low-resource generation.

## Datasets

- **Varta** — total ?; splits: train (-1), test (-1); repo https://github.com/rahular/varta

## Metrics

- `ROUGE-L` **(primary)** — range: percent
  - Longest Common Subsequence (LCS) recall and precision between generated and reference headlines, combined into a single F-measure score.

## Input / output format

**Input**: Article text (document) in one of 15 Indic languages or English.

**Output**: Single-line generated headline string.

## Scoring recipe

```python
def rouge_l(predictions, references):
    lcs_scores = []
    for pred, ref in zip(predictions, references):
        lcs = longest_common_subsequence(pred.split(), ref.split())
        lcs_scores.append(2 * len(lcs) / (len(pred.split()) + len(ref.split())))
    return sum(lcs_scores) / len(lcs_scores) * 100
```

## Common pitfalls

- Evaluating on transliterated scripts (Latin/Devanagari) vs original scripts changes tokenization and can inflate/deflate scores.
- Cross-lingual zero-shot evaluation requires strict language-code matching to avoid contamination.
- ROUGE-L is a surface-form metric that may penalize valid paraphrases or stylistic variations in headlines.

## Evidence (verbatim from paper)

> We finetune each model described in §4.2 on Värta in five settings: (i) en: finetune only on English data from the SMALL training set, and evaluate on all language test sets in original scripts. ... Table 3: Headline generation results for the three baseline models trained in all five data settings: English only (en), Hindi only (hi), Latin transliterated data (latin), Devanagari transliterated data (dvn.), and original script data (all). Only ROUGE-L scores are shown here.

## Citation

```bibtex
@misc{aralikatte2023varta,
  title={V\=arta: A Large-Scale Headline-Generation Dataset for Indic Languages},
  author={Aralikatte et al. (2023)},
  year={2023},
  note={arXiv:2305.05858}
}
```

- arXiv: 2305.05858