Rextthewild Eval

---
name: rextthewild-eval
description: Evaluates multimodal models' ability to understand real-world medical photographs by answering clinician-verified multiple-choice questions across seven clinical domains. It probes capabilities in geometric perception, anatomical localization, clinical characterization, and causal reasoning. Use when the user wants to benchmark on ReXInTheWild, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.19517
  bibtex_key: [REDACTED]
  confidence: high
---

# rextthewild-eval

> ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding — Banerjee et al. (2026) (arXiv:2603.19517, 2026)

## What this evaluates

Evaluates multimodal models' ability to understand real-world medical photographs by answering clinician-verified multiple-choice questions across seven clinical domains. It probes capabilities in geometric perception, anatomical localization, clinical characterization, and causal reasoning.

## Datasets

- **ReXInTheWild** — total 955; splits: test (955)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total number of questions. Reported with 95% confidence intervals.

## Input / output format

**Input**: A real-world medical photograph paired with a multiple-choice question.

**Output**: Selection of the correct option from the provided multiple-choice answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Models frequently confuse left/right sides of the body or flexed/extended joints (geometric errors).
- Causality errors often co-occur with lower-level geometric or localization mistakes, making it hard to isolate reasoning failures.
- Performance varies significantly across clinical domains (e.g., 'Eyes' vs 'Head & Neck'), so aggregate accuracy may mask domain-specific weaknesses.

## Evidence (verbatim from paper)

> Gemini-3 achieved the strongest performance with 78% accuracy (95% CI: 0.75, 0.81) across all questions, followed by Claude Opus 4.5 with 72% accuracy (95% CI: 0.69, 0.75) and GPT-5 with 68% accuracy (95% CI: 0.65, 0.71).

## Citation

```bibtex
@misc{banerjee2026rextthewild,
  title={ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding},
  author={Banerjee et al. (2026)},
  year={2026},
  note={arXiv:2603.19517}
}
```

- arXiv: 2603.19517