Rexxkg Eval

---
name: rexxkg-eval
description: Evaluates the medical knowledge understanding and entity/relation coverage of AI-generated chest X-ray radiology reports by comparing structured knowledge graphs extracted from generated text against ground-truth clinical reports. It specifically probes whether models capture nuanced anatomical relationships, medical devices, and quantified measurements beyond surface-level lexical overlap. Use when the user wants to benchmark on CheXpert Plus, MIMIC-CXR, or asks about evaluating this task. Reports ReXKG-NSC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.14397
  bibtex_key: zhang2024uncovering
  confidence: high
---

# rexxkg-eval

> Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs — Zhang et al. (2024) (arXiv:2408.14397, 2024)

## What this evaluates

Evaluates the medical knowledge understanding and entity/relation coverage of AI-generated chest X-ray radiology reports by comparing structured knowledge graphs extracted from generated text against ground-truth clinical reports. It specifically probes whether models capture nuanced anatomical relationships, medical devices, and quantified measurements beyond surface-level lexical overlap.

## Datasets

- **CheXpert Plus** — total 223228; splits: benchmark (24086), intra-dataset-baseline (24085)
- **MIMIC-CXR** — total 377110; splits: intra-dataset-baseline (24085)

## Metrics

- `ReXKG-NSC` **(primary)** — range: [0, 1]
  - Node Similarity Coverage: Measures the overlap and similarity of extracted entity nodes (anatomy, disorder, concept, device, procedure, size) between the generated report's knowledge graph and the ground-truth clinical report's graph.
- `ReXKG-AMS` — range: [0, 1]
  - Edge Distribution Similarity: Evaluates the distributional similarity of relation edges (suggestive of, located at, modify) between the generated and ground-truth knowledge graphs.
- `ReXKG-SCS` — range: [0, 1]
  - Subgraph Coverage Score: Assesses how well the generated report covers important subgraphs (k=2 nodes, top 10% by importance) from the ground-truth knowledge graph.

## Input / output format

**Input**: Chest X-ray image (and optionally clinical prompt) for report generation models; ground-truth radiology reports for knowledge graph construction.

**Output**: Free-text radiology report (specifically the findings section).

## Scoring recipe

```python
# Extract entities and relations from ground truth (GT) and generated reports using PURE/BERT pipeline
gt_entities, gt_relations = extract_kg(gt_report)
gen_entities, gen_relations = extract_kg(gen_report)

# Compute ReXKG-NSC (Node Similarity Coverage)
nsc = len(set(gt_entities) & set(gen_entities)) / len(set(gt_entities))

# Compute ReXKG-AMS (Edge Distribution Similarity)
ams = distribution_similarity(gt_relations, gen_relations)

# Compute ReXKG-SCS (Subgraph Coverage Score)
gt_subgraphs = get_top_k_subgraphs(gt_entities, gt_relations, k=2, top_pct=0.1)
scs = coverage_score(gt_subgraphs, gen_entities, gen_relations)

return {"ReXKG-NSC": nsc, "ReXKG-AMS": ams, "ReXKG-SCS": scs}
```

## Common pitfalls

- Relies on a fixed IE pipeline (PURE + BERT) that may fail to extract rare or complex medical entities, artificially deflating scores.
- Evaluates only the 'findings' section, ignoring impressions, indications, or other structured report components.
- Ground-truth knowledge graphs are constructed from a single dataset (CheXpert Plus I), limiting assessment of cross-institutional generalization.

## Evidence (verbatim from paper)

> It proposes three novel metrics—ReXKG-NSC (node similarity), ReXKG-AMS (edge distribution), and ReXKG-SCS (subgraph coverage)—to assess model understanding beyond surface-level report similarity.

## Citation

```bibtex
@misc{zhang2024uncovering,
  title={Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphs},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2408.14397}
}
```

- arXiv: 2408.14397