Healthbench Eval

---
name: healthbench-eval
description: Evaluates LLM responses to realistic clinical queries using a fine-grained, rubric-based scoring system. It measures medical accuracy, instruction following, completeness, context awareness, and safety by assigning positive or negative points to specific behavioral criteria, then normalizing the total to a [0, 1] scale. Use when the user wants to benchmark on HealthBench, or asks about evaluating this task. Reports HealthBench Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.02594
  bibtex_key: ravichandran2025healthbench
  confidence: high
---

# healthbench-eval

> OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries — Ravichandran et al. (2025) (arXiv:2509.02594, 2025)

## What this evaluates

Evaluates LLM responses to realistic clinical queries using a fine-grained, rubric-based scoring system. It measures medical accuracy, instruction following, completeness, context awareness, and safety by assigning positive or negative points to specific behavioral criteria, then normalizing the total to a [0, 1] scale.

## Datasets

- **HealthBench** — total ?; splits: test (-1)

## Metrics

- `HealthBench Score` **(primary)** — range: [0, 1]
  - Sum of point values for all met criteria (positive for correct/helpful, negative for harmful/misinformation). Normalized by dividing by the sum of absolute point values in the rubric. Finally clipped to [0, 1].

## Input / output format

**Input**: Clinical query/prompt from the user and a predefined rubric containing criteria with associated point values.

**Output**: Model-generated response to the clinical query, which is evaluated against the rubric to produce a normalized HealthBench Score between 0 and 1.

## Scoring recipe

```python
total_points = 0
max_possible_points = sum([abs(p) for (_, p) in rubric])
for (criterion, point_value) in rubric:
    if criterion_is_met(response, criterion):
        total_points += point_value
normalized_score = total_points / max_possible_points
final_score = max(0, min(1, normalized_score))
```

## Common pitfalls

- Point values can be negative, so raw sums may be negative before normalization, but the final score is strictly clipped to [0, 1].
- Normalization divides by the sum of absolute values of all rubric points, not just the positive ones or the maximum possible positive score.
- The criterion_is_met function is not algorithmically defined in the paper, requiring human expert annotation or a separate judge model to evaluate.

## Evidence (verbatim from paper)

> The scoring algorithm follows these steps: total_points=0; max_possible_points=sum([abs(p)for(_,p)inrubric]); for(criterion,point_value)inrubric: ifcriterion_is_met(response,criterion): total_points+=point_value; normalized_score=total_points/max_possible_points; final_score=max(0,min(1,normalized_score)). Normalized HB (HealthBench) Score: HB Score = total_points / max_possible_points.

## Citation

```bibtex
@misc{ravichandran2025healthbench,
  title={OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries},
  author={Ravichandran et al. (2025)},
  year={2025},
  note={arXiv:2509.02594}
}
```

- arXiv: 2509.02594