Dmid Mammography Report Eval

---
name: dmid-mammography-report-eval
description: Evaluates a vision-language model's ability to generate clinically accurate and linguistically fluent mammography reports from multi-view breast images. It probes both natural language generation quality and domain-specific diagnostic reasoning, specifically BI-RADS categorization and breast density assessment. Use when the user wants to benchmark on DMID, or asks about evaluating this task. Reports BI-RADS Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.09225
  bibtex_key: sung2025amrg
  confidence: high
---

# dmid-mammography-report-eval

> AMRG: Extend Vision Language Models for Automatic Mammography Report Generation — Sung et al. (2025) (arXiv:2508.09225, 2025)

## What this evaluates

Evaluates a vision-language model's ability to generate clinically accurate and linguistically fluent mammography reports from multi-view breast images. It probes both natural language generation quality and domain-specific diagnostic reasoning, specifically BI-RADS categorization and breast density assessment.

## Datasets

- **DMID** — total ?; splits: train (-1), test (-1)

## Metrics

- `ROUGE-L` — range: [0, 1]
  - Measures the overlap of longest common subsequences between generated and reference reports, weighted by recall.
- `METEOR` — range: [0, 1]
  - Computes a weighted harmonic mean of precision and recall, incorporating synonymy and stemming to capture semantic similarity.
- `CIDEr` — range: [0, 1]
  - Ranks n-grams by inverse document frequency to penalize common words and reward distinctive, clinically informative terms.
- `BI-RADS Accuracy` **(primary)** — range: [0, 1]
  - Proportion of exact matches between predicted and ground-truth BI-RADS diagnostic categories.
- `Density Accuracy` — range: [0, 1]
  - Proportion of exact matches between predicted and ground-truth breast density labels.

## Input / output format

**Input**: Multi-view mammography images.

**Output**: Textual mammography report containing descriptive findings, BI-RADS category, and breast density assessment.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    nlp_scores = {}
    for metric in ['bleu-1','rouge-1','rouge-2','rouge-l','meteor','cider','f1']:
        nlp_scores[metric] = evaluate_nlp(predictions, golds, metric)
    birads_pred = extract_label(predictions, 'birads')
    birads_gold = extract_label(golds, 'birads')
    density_pred = extract_label(predictions, 'density')
    density_gold = extract_label(golds, 'density')
    nlp_scores['BI-RADS Accuracy'] = sum(p==g for p,g in zip(birads_pred, birads_gold)) / len(golds)
    nlp_scores['Density Accuracy'] = sum(p==g for p,g in zip(density_pred, density_gold)) / len(golds)
    return nlp_scores
```

## Common pitfalls

- Using high LoRA ranks (e.g., r=64) causes overfitting on the relatively small DMID dataset, degrading both NLP and clinical metrics.
- Surface-level n-gram metrics (BLEU-1, ROUGE-2) may favor syntactically fluent but clinically inaccurate outputs, whereas semantic metrics and exact label matching better reflect diagnostic utility.
- General-purpose VLMs often hallucinate benign structures or omit critical findings despite high fluency scores.

## Evidence (verbatim from paper)

> In particular, the configuration $(r\=32,\alpha\=16)$ achieves the highest scores across all NLP metrics (e.g., ROUGE-L 0.52, METEOR 0.5194, CIDEr 0.5336) and clinical metrics (BI-RADS accuracy 0.55, density accuracy 0.35), outperforming both the base model and other LoRA variants.

## Citation

```bibtex
@misc{sung2025amrg,
  title={AMRG: Extend Vision Language Models for Automatic Mammography Report Generation},
  author={Sung et al. (2025)},
  year={2025},
  note={arXiv:2508.09225}
}
```

- arXiv: 2508.09225