Gaps Eval

---
name: gaps-eval
description: Evaluates AI clinicians on clinical reasoning depth, answer completeness, robustness to input perturbations, and safety risk mitigation. It probes how well models retrieve, synthesize, and apply evidence-based guidelines under varying cognitive loads and adversarial conditions. Use when the user wants to benchmark on GAPS-NCCN-NSCLC-preview, or asks about evaluating this task. Reports GAPS score (normalized rubric-based).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13734
  bibtex_key: chen2025gaps
  confidence: high
---

# gaps-eval

> GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians — Chen et al. (2025) (arXiv:2510.13734, 2025)

## What this evaluates

Evaluates AI clinicians on clinical reasoning depth, answer completeness, robustness to input perturbations, and safety risk mitigation. It probes how well models retrieve, synthesize, and apply evidence-based guidelines under varying cognitive loads and adversarial conditions.

## Datasets

- **GAPS-NCCN-NSCLC-preview** — total 92; splits: test (92); repo https://github.com/AQ-MedAI/MedicalAiBenchEval

## Metrics

- `GAPS score (normalized rubric-based)` **(primary)** — range: [0, 1]
  - Scores are normalized to [0, 1] using rule-based and rubric-based scoring evaluated by an LLM-as-a-judge ensemble. The score reflects the proportion of correctly satisfied rubric elements against a guideline-anchored standard.
- `hit rate` — range: [0, 1]
  - Proportion of satisfied positive rubric elements (A1 Must-have, A2 Should-have, A3 Nice-to-have) or avoided negative rubric elements (S2 Near miss, S3 Suboptimal, S4 Critical) out of total rubrics per item.

## Input / output format

**Input**: Clinical question prompts (clean P0 or perturbed P1–P3) derived from NCCN NSCLC guidelines, often accompanied by evidence corpora or knowledge graph structures.

**Output**: Free-text clinical recommendations or answers addressing the prompt.

## Scoring recipe

```python
def score_response(response, gold_rubrics):
    satisfied = 0
    total = len(gold_rubrics)
    for rubric in gold_rubrics:
        if llm_judge_check(response, rubric):  # LLM-as-a-judge verifies alignment
            satisfied += 1
    return satisfied / total if total > 0 else 0.0

# Adequacy hit rate = score_response(response, adequacy_rubrics)
# Safety hit rate = score_response(response, safety_rubrics)
# Overall GAPS score = normalized aggregate across Grounding, Adequacy, Safety, and Perturbation axes
```

## Common pitfalls

- High factual recall (G1/G2) does not translate to complex decision-making (G3/G4), leading to overestimation of clinical reasoning capabilities.
- Models are highly sensitive to adversarial premise perturbations (P3), often aligning with misleading cues rather than critically evaluating them.
- Safety violations (S4) are not uniformly distributed; they spike dramatically under high cognitive load and vary significantly across different LLM architectures.

## Evidence (verbatim from paper)

> We next evaluate representative state-of-the-art LLMs on the automatically constructed GAPS benchmark GAPS-NCCN-NSCLC-preview using rule-based and rubric-based scoring (normalized to [0,1]), details of which are covered in Section[3.6]. Scores were obtained by an ensemble of LLM-as-a-judge models, with clinician audits on stratified subsets for validation. Performance was analyzed across the four GAPS axes: Grounding levels (G1–G4), Perturbations (P1–P3), and Adequacy and Safety outcomes (hit rates).

## Citation

```bibtex
@misc{chen2025gaps,
  title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2510.13734}
}
```

- arXiv: 2510.13734