Trec 2020 Podcast Summarisation Eval

---
name: trec-2020-podcast-summarisation-eval
description: Evaluates the ability of summarization systems to generate concise, accurate, and informative summaries of long-form spoken podcast episodes. It probes handling of speech-specific challenges like redundancy, speaker turns, and informal language, as well as factual recall of key entities and events. Use when the user wants to benchmark on TREC 2020 Podcast Summarisation Track, or asks about evaluating this task. Reports Avg.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.02535
  bibtex_key: manakul2020cued_speech
  confidence: high
---

# trec-2020-podcast-summarisation-eval

> CUED_speech at TREC 2020 Podcast Summarisation Track — Manakul et al. (2020) (arXiv:2012.02535, 2020)

## What this evaluates

Evaluates the ability of summarization systems to generate concise, accurate, and informative summaries of long-form spoken podcast episodes. It probes handling of speech-specific challenges like redundancy, speaker turns, and informal language, as well as factual recall of key entities and events.

## Datasets

- **TREC 2020 Podcast Summarisation Track** — total 179; splits: test (179)

## Metrics

- `Avg` **(primary)** — range: [0, 3]
  - Average score computed by mapping ordinal human ratings to Bad=0, Fair=1, Good=2, Excellent=3, then averaging across all annotated episodes.
- `ROUGE-L (F1)` — range: percent
  - Longest common subsequence F1 score between generated summaries and reference summaries. Computed using both official NIST references and a processed version of episode descriptions.

## Input / output format

**Input**: Raw or filtered transcriptions of podcast episodes.

**Output**: Abstractive text summaries of the podcast episodes.

## Scoring recipe

```python
def compute_avg_human(annotations):
    score_map = {'Bad': 0, 'Fair': 1, 'Good': 2, 'Excellent': 3}
    return sum(score_map[a] for a in annotations) / len(annotations)

def compute_rouge_l_f1(generated, reference):
    # Standard ROUGE-L F1 calculation on longest common subsequence
    precision = len(lcs(generated, reference)) / len(generated)
    recall = len(lcs(generated, reference)) / len(reference)
    if precision + recall == 0: return 0.0
    return 2 * (precision * recall) / (precision + recall) * 100
```

## Common pitfalls

- Human evaluation uses an ordinal scale (0-3) but is averaged as if interval; results should not be interpreted as percentages or probabilities.
- ROUGE-L scores vary significantly depending on whether official NIST references or processed/cleaned references are used, as spoken transcripts contain noise like URLs and filler words.
- Ensemble size (3 vs 9 models) impacts performance, so single-model results may not match reported best scores.

## Evidence (verbatim from paper)

> 179 episodes were selected randomly, and NIST annotators evaluated them on the Bad/Fair/Good/Excellent scale... Table 6 shows that our CUED-ensemble3 submission receives the highest human rating at 1.777 in average... When using an automatic evaluation, CUED-ensemble3 also achieves the highest score shown in Table 7 (note that our ROUGE scores, computed using processed episode descriptions, suggest the performance of CUED-ensemble3 and CUED-ensemble9 are similar).

## Citation

```bibtex
@misc{manakul2020cued_speech,
  title={CUED_speech at TREC 2020 Podcast Summarisation Track},
  author={Manakul et al. (2020)},
  year={2020},
  note={arXiv:2012.02535}
}
```

- arXiv: 2012.02535