Qmsum Eval

---
name: qmsum-eval
description: Evaluates an agent's ability to distill and synthesize key information from high-noise, multi-turn meeting transcripts based on specific user queries. It probes the model's capacity to maintain global context while filtering irrelevant dialogue turns to produce a query-relevant summary. Use when the user wants to benchmark on QMSUM, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14930
  bibtex_key: lv2026ieascache
  confidence: high
---

# qmsum-eval

> IE as Cache: Information Extraction Enhanced Agentic Reasoning — Hang Lv et al. (2026) (arXiv:2604.14930, 2026)

## What this evaluates

Evaluates an agent's ability to distill and synthesize key information from high-noise, multi-turn meeting transcripts based on specific user queries. It probes the model's capacity to maintain global context while filtering irrelevant dialogue turns to produce a query-relevant summary.

## Datasets

- **QMSUM** — total ?; splits: test (-1)

## Metrics

- `ROUGE-1` **(primary)** — range: [0, 1]
  - Computes the unigram overlap between the generated summary and the reference summary, typically averaged across all test instances. Reflects query-relevant content coverage.

## Input / output format

**Input**: Extensive multi-turn meeting transcripts paired with user queries specifying the summarization focus.

**Output**: A generated summary text addressing the user query.

## Scoring recipe

```python
def score_rouge1(pred, gold):
    # Standard ROUGE-1 calculation (unigram overlap)
    pred_tokens = pred.lower().split()
    gold_tokens = gold.lower().split()
    if not gold_tokens: return 0.0
    overlap = sum(1 for t in pred_tokens if t in gold_tokens)
    return overlap / len(gold_tokens)

# Average over test set
rouge_scores = [score_rouge1(p, g) for p, g in zip(predictions, gold_summaries)]
final_metric = sum(rouge_scores) / len(rouge_scores)
```

## Common pitfalls

- ReAct agents often underperform compared to simple baselines because frequent context switching fragments the narrative flow of the meeting.
- Summarization quality heavily depends on retaining global context while filtering noise; static extraction fails to capture evolving dialogue states.
- ROUGE metrics penalize paraphrasing, so semantically correct but lexically different summaries may score lower than expected.

## Evidence (verbatim from paper)

> For QMSUM, we report ROUGE-1, reflecting query-relevant content coverage in the generated summary.

## Citation

```bibtex
@misc{lv2026ieascache,
  title={IE as Cache: Information Extraction Enhanced Agentic Reasoning},
  author={Hang Lv et al. (2026)},
  year={2026},
  note={arXiv:2604.14930}
}
```

- arXiv: 2604.14930