Medical Radiology Similarity Eval

---
name: medical-radiology-similarity-eval
description: This evaluation probes the ability of automated metrics to capture deep clinical semantics in radiology reports. It compares LLM-generated similarity scores against traditional lexical overlap metrics, measuring how well each aligns with ground truth annotations derived from clinical NLP tools. Use when the user wants to benchmark on Radiology Report Pairs (CheXpert/NegBio-derived), or asks about evaluating this task. Reports GPT_sim.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.11398
  bibtex_key: xu2024reasoning
  confidence: medium
---

# medical-radiology-similarity-eval

> Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis — Shaochen Xu et al. (2024) (arXiv:2402.11398, 2024)

## What this evaluates

This evaluation probes the ability of automated metrics to capture deep clinical semantics in radiology reports. It compares LLM-generated similarity scores against traditional lexical overlap metrics, measuring how well each aligns with ground truth annotations derived from clinical NLP tools.

## Datasets

- **Radiology Report Pairs (CheXpert/NegBio-derived)** — total 62500; splits: test (62500)

## Metrics

- `GPT_sim` **(primary)** — range: [0, 1]
  - Semantic similarity score generated by GPT-4 reasoning over radiology report pairs.
- `ROUGE-1 F1` — range: [0, 1]
  - Unigram lexical overlap F1 score between two text pairs.
- `ROUGE-2 F1` — range: [0, 1]
  - Bigram lexical overlap F1 score between two text pairs.
- `ROUGE-L F1` — range: [0, 1]
  - Longest common subsequence lexical overlap F1 score between two text pairs.
- `BLEU` — range: [0, 1]
  - Bilingual evaluation understudy score measuring n-gram precision with brevity penalty.

## Input / output format

**Input**: Pairs of radiology report texts.

**Output**: A single similarity score per text pair, representing the predicted semantic alignment.

## Scoring recipe

```python
# For each of the 62,500 report pairs:
pred_score = compute_similarity(pair, method="GPT_sim|ROUGE-1|ROUGE-2|ROUGE-L|BLEU")
gt_score = get_gt_similarity(pair, source="CheXpert|NegBio")
# Aggregate across all pairs
mean_diff = mean(abs(pred_score - gt_score))
# Report mean_diff per method per GT source
```

## Common pitfalls

- Traditional lexical metrics (ROUGE/BLEU) yield higher mean scores but indicate greater deviation from clinical ground truth, contrary to intuition that higher overlap equals better alignment.
- Ground truth is derived from automated clinical annotation tools (CheXpert/NegBio) rather than direct physician ratings, which may not capture all clinical nuances.
- The evaluation emphasizes mean differences, potentially obscuring distributional misalignments; the authors supplement this with hexbin plots covering the 5th–95th percentile range.

## Evidence (verbatim from paper)

> The comparison leveraged several metrics, including GPT_sim (GPT-4 generated similarities), ROUGE-1 F1, ROUGE-2 F1, ROUGE-L F1, and BLEU scores, to evaluate their performance in aligning with the GT derived from CheXpert and NegBio annotations.

## Citation

```bibtex
@misc{xu2024reasoning,
  title={Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis},
  author={Shaochen Xu et al. (2024)},
  year={2024},
  note={arXiv:2402.11398}
}
```

- arXiv: 2402.11398