Vietmed Sum Eval

---
name: vietmed-sum-eval
description: Evaluates abstractive summarization capabilities on real-world and simulated medical conversations in Vietnamese, testing both human-transcribed and ASR-generated noisy transcripts. Use when the user wants to benchmark on VietMed-Sum, or asks about evaluating this task. Reports ROUGE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.15888
  bibtex_key: leduc2024realtimespeechsummarization
  confidence: medium
---

# vietmed-sum-eval

> Real-time Speech Summarization for Medical Conversations — Le-Duc et al. (2024) (arXiv:2406.15888, 2024)

## What this evaluates

Evaluates abstractive summarization capabilities on real-world and simulated medical conversations in Vietnamese, testing both human-transcribed and ASR-generated noisy transcripts.

## Datasets

- **VietMed-Sum** — total ?; splits: train (-1), test (-1); repo https://github.com/leduckhai/MultiMed

## Metrics

- `ROUGE` **(primary)** — range: [0, 1]
  - Measures n-gram overlap between generated summaries and reference summaries. The paper evaluates both local and global summaries using standard ROUGE variants.

## Input / output format

**Input**: Transcribed speech text (either human-transcribed or ASR-generated) representing medical conversation utterances.

**Output**: Abstractive summary text (generated locally after N utterances and globally at conversation end).

## Scoring recipe

```python
def compute_rouge(predictions, references):
    scores = {}
    for metric in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        metric_scores = [rouge_score(ref, pred, use_stemmer=True) for ref, pred in zip(references, predictions)]
        scores[metric] = sum(metric_scores) / len(metric_scores)
    return scores
```

## Common pitfalls

- The paper evaluates on both human transcripts and ASR transcripts, but does not specify if ROUGE scores are reported separately for each or averaged.
- ROUGE variants (ROUGE-1, ROUGE-2, ROUGE-L) are not explicitly detailed in the main text; readers must check the Appendix for exact configuration.
- Local vs. global summary evaluation is mentioned, but the exact split or aggregation method for these two types of summaries is not fully specified in the provided section.

## Evidence (verbatim from paper)

> We use ROUGE *[[14]]*, a metric commonly used for summarization, to evaluate our models. To evaluate their performance, we calculate their ROUGE scores on the local and global summaries in the test set.

## Citation

```bibtex
@misc{leduc2024realtimespeechsummarization,
  title={Real-time Speech Summarization for Medical Conversations},
  author={Le-Duc et al. (2024)},
  year={2024},
  note={arXiv:2406.15888}
}
```

- arXiv: 2406.15888