Medical Summarization Eval

---
name: medical-summarization-eval
description: This evaluation probes the ability of large language models to generate accurate and faithful summaries of medical texts under high out-of-vocabulary (OOV) conditions. It specifically measures how tokenization fragmentation and domain-specific terminology affect summarization quality and concept preservation across multiple medical benchmarks. Use when the user wants to benchmark on PubMedQA, EBM, BioASQ-M, BioASQ-S, or asks about evaluating this task. Reports Rouge-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21242
  bibtex_key: [REDACTED]
  confidence: high
---

# medical-summarization-eval

> Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings — Balde et al. (2025) (arXiv:2505.21242, 2025)

## What this evaluates

This evaluation probes the ability of large language models to generate accurate and faithful summaries of medical texts under high out-of-vocabulary (OOV) conditions. It specifically measures how tokenization fragmentation and domain-specific terminology affect summarization quality and concept preservation across multiple medical benchmarks.

## Datasets

- **PubMedQA** — total 500; splits: test (500)
- **EBM** — total 424; splits: test (424)
- **BioASQ-M** — total 963; splits: test (963)
- **BioASQ-S** — total 496; splits: test (496)

## Metrics

- `Rouge-L` **(primary)** — range: [0, 1]
  - Measures the longest common subsequence between the generated summary and the reference summary to assess informativeness and coherence.
- `Concept-Score` — range: [0, 1]
  - Measures faithfulness by computing the overlap of UMLS medical concepts extracted from both the generated and reference summaries using the QuickUMLS tool.

## Input / output format

**Input**: Query appended to a PubMed abstract (or relevant snippets) as the source document, formatted into an in-context learning prompt with similarity-sampled examples.

**Output**: A concise text summary addressing the query based on the provided context.

## Scoring recipe

```python
def compute_rouge_l(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        scores.append(rouge_score(ref, pred, rouge_types=['rougeL']))
    return sum(scores) / len(scores)

def compute_concept_score(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        gen_concepts = quickumls.extract(pred)
        ref_concepts = quickumls.extract(ref)
        overlap = len(gen_concepts & ref_concepts)
        union = len(gen_concepts | ref_concepts)
        scores.append(overlap / union if union > 0 else 0.0)
    return sum(scores) / len(scores)
```

## Common pitfalls

- High OOV concentration in medical texts causes severe tokenization fragmentation, which can artificially depress ROUGE scores independent of actual summarization quality.
- Concept-Score depends on the UMLS ontology via QuickUMLS, meaning it may fail to capture novel or non-standard medical terms not present in the reference ontology.
- In-context learning examples are selected using PubMedBERT similarity, which may not perfectly align with semantic relevance for medical summarization tasks.

## Evidence (verbatim from paper)

> We evaluate the model-generated summaries using Rouge-L (R-L) to measure informativeness and coherence, and Concept-Score (CSr) to measure faithfulness*Zhang et al. ([2023])*. Concept-Score measures the overlap of UMLS medical concepts (computed using QuickUMLS tool*Soldaini and Goharian ([2016])*) between the generated and reference summaries. We use Rouge-L as the primary comparison metric, in line with prior studies*Fabbri et al. ([2021]); Yuan et al. ([2022]); Balde et al. ([2024a], [b])*.

## Citation

```bibtex
@misc{balde2025evaluation,
  title={Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings},
  author={Balde et al. (2025)},
  year={2025},
  note={arXiv:2505.21242}
}
```

- arXiv: 2505.21242