Mammography Report Eval

---
name: mammography-report-eval
description: Evaluates the ability of local vision-language models to generate clinically styled mammography reports and perform multi-task classification (e.g., BI-RADS, breast density, calcifications) from medical images. It probes the models' robustness under zero-shot, few-shot, Chain-of-Thought prompting, and Retrieval-Augmented Generation (RAG), as well as the impact of parameter-efficient fine-tuning (QLoRA). Use when the user wants to benchmark on VinDr-Mammo, DMID, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.22462
  bibtex_key: [REDACTED]
  confidence: high
---

# mammography-report-eval

> MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation — Raiyan Jahangir et al. (2026) (arXiv:2602.22462, 2026)

## What this evaluates

Evaluates the ability of local vision-language models to generate clinically styled mammography reports and perform multi-task classification (e.g., BI-RADS, breast density, calcifications) from medical images. It probes the models' robustness under zero-shot, few-shot, Chain-of-Thought prompting, and Retrieval-Augmented Generation (RAG), as well as the impact of parameter-efficient fine-tuning (QLoRA).

## Datasets

- **VinDr-Mammo** — total ?; splits: test (-1)
- **DMID** — total ?; splits: test (-1)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, computed as macro-average across classes to handle class imbalance.
- `BERTScore` — range: [0, 1]
  - Similarity metric using contextual embeddings from BERT to compute precision, recall, and F1 between generated and reference text.
- `ROUGE-L` — range: [0, 1]
  - Recall-oriented overlap of the longest common subsequence between generated and reference text.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly predicted labels out of total instances.

## Input / output format

**Input**: Mammography images paired with prompts (zero-shot, few-shot, Chain-of-Thought, or RAG-augmented with retrieved clinical examples).

**Output**: Structured JSON containing fields for BI-RADS, Breast Density, Calcification, Mass, Asymmetry, Suspicion, and Findings text; or single-task generation of one field at a time.

## Scoring recipe

```python
def evaluate(predictions, gold):
    # Classification metrics
    acc = accuracy_score(gold['labels'], predictions['labels'])
    f1 = f1_score(gold['labels'], predictions['labels'], average='macro')
    # Text similarity metrics
    bert = bertscore.compute(predictions=predictions['text'], references=gold['text'])['f1']
    rouge = rouge_score(gold['text'], predictions['text'], rouge_types=['rougeL'])['rougeL']
    return {'accuracy': acc, 'f1': f1, 'bertscore': bert, 'rougeL': rouge}
```

## Common pitfalls

- RAG can degrade performance on certain fields (e.g., findings text) if retrieved context mismatches the target distribution or introduces lexical noise.
- Fine-tuning output format (multi-task full JSON vs. single-task one-field) significantly impacts results; single-task may outperform multi-task for some labels but fail for others.
- Optimal training epochs are task-dependent and non-monotonic; early stopping or checkpoint selection is required rather than assuming more epochs always improve performance.

## Evidence (verbatim from paper)

> We first evaluate report-generation similarity using only text-similarity metrics (BERTScore and ROUGE-L) on narrative fields, BI-RADS, density, and findings directly read by clinicians.

## Citation

```bibtex
@misc{jahangir2026mammowise,
  title={MammoWise: Multi-Model Local RAG Pipeline for Mammography Report Generation},
  author={Raiyan Jahangir et al. (2026)},
  year={2026},
  note={arXiv:2602.22462}
}
```

- arXiv: 2602.22462