Bart Eval

---
name: bart-eval
description: Evaluates a denoising sequence-to-sequence pre-trained model across discriminative comprehension, abstractive text generation, dialogue response, and machine translation tasks to measure cross-task generalization and generation quality. Use when the user wants to benchmark on SQuAD 1.1, SQuAD 2.0, GLUE, CNN/DailyMail, XSum, ConvAI2, ELI5, WMT'16 RO-EN, or asks about evaluating this task. Reports ROUGE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.13461
  bibtex_key: lewis2019bart
  confidence: high
---

# bart-eval

> BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension — Mike Lewis et al. (arXiv:1910.13461, 2019)

## What this evaluates

Evaluates a denoising sequence-to-sequence pre-trained model across discriminative comprehension, abstractive text generation, dialogue response, and machine translation tasks to measure cross-task generalization and generation quality.

## Datasets

- **SQuAD 1.1** — total ?; splits: test (-1)
- **SQuAD 2.0** — total ?; splits: test (-1)
- **GLUE** — total ?; splits: dev (-1)
- **CNN/DailyMail** — total ?; splits: test (-1)
- **XSum** — total ?; splits: test (-1)
- **ConvAI2** — total ?; splits: validation (-1)
- **ELI5** — total ?; splits: test (-1)
- **WMT'16 RO-EN** — total ?; splits: test (-1)

## Metrics

- `ROUGE` **(primary)** — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation. Measures overlap of unigrams, bigrams, and longest common subsequence between generated and reference text.
- `BLEU` — range: [0, 1]
  - Bilingual Evaluation Understudy. Geometric mean of modified n-gram precisions with a brevity penalty to penalize overly short translations.
- `Exact Match (EM)` — range: [0, 1]
  - Percentage of predictions that exactly match the ground truth answer span character-for-character.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for token overlap in open-ended QA and dialogue.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances across discriminative tasks.
- `Matthews Correlation Coefficient (MCC)` — range: [-1, 1]
  - Correlation coefficient between predicted and actual binary classifications, robust to class imbalance.
- `Perplexity (PPL)` — range: [0, inf)
  - Exponential of the average negative log-likelihood of the reference tokens under the model.

## Input / output format

**Input**: Task-specific text input: document/question/dialogue context for comprehension, source sentence for generation/translation.

**Output**: Task-specific target: span/label for comprehension, generated summary/response/translation for generation tasks.

## Scoring recipe

```python
def score(predictions, gold, task):
    if task in ['summarization', 'qa']:
        return rouge_l(predictions, gold)
    elif task == 'translation':
        return bleu(predictions, gold)
    elif task == 'squad':
        return exact_match(predictions, gold)
    elif task == 'glue':
        return accuracy(predictions, gold)
    elif task == 'convai2':
        return f1(predictions, gold), perplexity(predictions, gold)
    return 0
```

## Common pitfalls

- Generation decoding requires specific beam search settings (size 5, trigram blocking, length penalty tuning) rather than greedy decoding.
- ROUGE and BLEU scores are highly sensitive to tokenization; official tokenizer normalization must be applied for fair comparison.
- SQuAD 2.0 includes unanswerable questions, requiring a threshold-based prediction strategy not needed for SQuAD 1.1.

## Evidence (verbatim from paper)

> BART outperforms the best previous work, which leverages BERT, by roughly 6.0 points on all ROUGE metrics—representing a significant advance in performance on this problem.

## Citation

```bibtex
@misc{lewis2019bart,
  title={BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension},
  author={Mike Lewis et al.},
  year={2019},
  note={arXiv:1910.13461}
}
```

- arXiv: 1910.13461