Regradient 160k Eval

---
name: regradient-160k-eval
description: Evaluates the ability of vision-language models to generate accurate and semantically rich chest X-ray radiology reports from medical images. It tests both lexical overlap and clinical semantic alignment of the generated findings and impressions against ground-truth reports. Use when the user wants to benchmark on ReXGradient-160K, or asks about evaluating this task. Reports COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.09887
  bibtex_key: gulluk2026sem enrich
  confidence: high
---

# regradient-160k-eval

> SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning — Gulluk et al. (2026) (arXiv:2604.09887, 2026)

## What this evaluates

Evaluates the ability of vision-language models to generate accurate and semantically rich chest X-ray radiology reports from medical images. It tests both lexical overlap and clinical semantic alignment of the generated findings and impressions against ground-truth reports.

## Datasets

- **ReXGradient-160K** — total 160000; splits: train (-1), test (-1)

## Metrics

- `COMET` **(primary)** — range: [0, 100]
  - Semantic similarity metric that penalizes unnecessary, irrelevant, and repetitive sentences. Scores are averaged across the test set.
- `BERTScore-F1` — range: [0, 1]
  - Contextual embedding-based similarity metric evaluating semantic alignment between generated and reference texts by balancing precision and recall.
- `Sentence BLEU-1` — range: [0, 100]
  - Measures unigram n-gram overlap between generated and reference sentences.
- `CheXbert-14 F1` — range: [0, 100]
  - F1 score for 14 clinical findings extracted from radiology reports using the CheXbert pipeline.
- `RadGraph-F1` — range: [0, 100]
  - F1 score for finding-modality-attribute triples extracted from reports using the RadGraph pipeline.

## Input / output format

**Input**: Chest X-ray image (224x224 pixels) paired with a ground-truth radiology report (findings and impression sections).

**Output**: Generated radiology report text (findings and/or impression sections).

## Scoring recipe

```python
def compute_metrics(predictions, references):
    comet_scores = [comet_model.predict(pred, ref) for pred, ref in zip(predictions, references)]
    bert_scores = [bertscore_f1(pred, ref) for pred, ref in zip(predictions, references)]
    bleu_scores = [sentence_bleu_1(pred, ref) for pred, ref in zip(predictions, references)]
    chexbert_f1 = compute_chexbert_f1(predictions, references)
    radgraph_f1 = compute_radgraph_f1(predictions, references)
    return {
        'COMET': np.mean(comet_scores),
        'BERTScore-F1': np.mean(bert_scores),
        'Sentence BLEU-1': np.mean(bleu_scores),
        'CheXbert-14 F1': chexbert_f1,
        'RadGraph-F1': radgraph_f1
    }
```

## Common pitfalls

- Corpus BLEU scores are explicitly noted as biased in favor of longer outputs and may be artificially inflated by the enrichment method adding sentences from the dataset itself.
- The test dataset is never enriched during evaluation; enrichment is only applied during training/inference batch loading, so metrics must be computed on raw test references.
- CheXbert and RadGraph scores can vary significantly across different base models, requiring careful per-model reporting rather than just averaging.

## Evidence (verbatim from paper)

> One of the most important metrics is COMET score in our case, it is because our method involves adding new sentences to the existings sentences for each sample. As COMETis a metric that punishes unnecessary, irrelevant and repetitive sentences, we might expect that our model would perform worse than the baseline training if the added sentences were noisy.

## Citation

```bibtex
@misc{gulluk2026sem enrich,
  title={SemEnrich: Self-Supervised Semantic Enrichment of Radiology Reports for Vision-Language Learning},
  author={Gulluk et al. (2026)},
  year={2026},
  note={arXiv:2604.09887}
}
```

- arXiv: 2604.09887