Meetingbank Eval

---
name: meetingbank-eval
description: Evaluates the quality of abstractive and extractive summarization systems on city council meeting transcripts and videos. It probes a model's ability to capture key decisions, maintain factual accuracy, and produce fluent, coherent, and non-redundant summaries. Use when the user wants to benchmark on MeetingBank, or asks about evaluating this task. Reports Average Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.17529
  bibtex_key: hu2023meetingbank
  confidence: high
---

# meetingbank-eval

> MeetingBank: A Benchmark Dataset for Meeting Summarization — Hu et al. (2023) (arXiv:2305.17529, 2023)

## What this evaluates

Evaluates the quality of abstractive and extractive summarization systems on city council meeting transcripts and videos. It probes a model's ability to capture key decisions, maintain factual accuracy, and produce fluent, coherent, and non-redundant summaries.

## Datasets

- **MeetingBank** — total 1366; splits: train (-1), test (200)

## Metrics

- `Average Score` **(primary)** — range: [1, 5]
  - Mean of five 5-point Likert scale scores: informativeness, factuality, fluency, coherence, and redundancy. Higher values indicate better summary quality.

## Input / output format

**Input**: Video segment (typically ≤30 minutes), corresponding transcript, and a system-generated summary.

**Output**: Five scores (1–5) for each criterion, plus an averaged total score.

## Scoring recipe

```python
def compute_average_score(predictions):
    criteria = ['informativeness', 'factuality', 'fluency', 'coherence', 'redundancy']
    scores = [predictions[c] for c in criteria]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Summaries must be presented in random order to evaluators to prevent order-based bias.
- Human evaluation is only performed on a subset of 200 randomly selected instances, not the full dataset.
- Evaluators assess both the video and transcript alongside the summary, which differs from text-only benchmarks.

## Evidence (verbatim from paper)

> The workers are asked to watch a video segment, typically 30 minutes or less, read the transcript, and then evaluate the quality of each system summary based on five criteria: informativeness, factuality, fluency, coherence, and redundancy. A 5-point Likert scale is used to evaluate each criterion. The scores are then averaged, and standard deviation is also reported. ... AVERAGE SCORE

## Citation

```bibtex
@misc{hu2023meetingbank,
  title={MeetingBank: A Benchmark Dataset for Meeting Summarization},
  author={Hu et al. (2023)},
  year={2023},
  note={arXiv:2305.17529}
}
```

- arXiv: 2305.17529