Head Qa V2 Eval

---
name: head-qa-v2-eval
description: This benchmark evaluates large language models on complex medical reasoning using real Spanish medical licensing exam questions. It probes domain-specific knowledge retention, cross-lingual generalization, and the effectiveness of various inference strategies like prompting, retrieval-augmented generation, and log-probability selection. Use when the user wants to benchmark on HEAD-QA v2, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.15355
  bibtex_key: [REDACTED]
  confidence: high
---

# head-qa-v2-eval

> HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning — Correa-Guillén et al. (2025) (arXiv:2511.15355, 2025)

## What this evaluates

This benchmark evaluates large language models on complex medical reasoning using real Spanish medical licensing exam questions. It probes domain-specific knowledge retention, cross-lingual generalization, and the effectiveness of various inference strategies like prompting, retrieval-augmented generation, and log-probability selection.

## Datasets

- **HEAD-QA v2** — total 12751; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions out of the total number of questions.
- `normalized exam score` — range: [0, 1]
  - Calculated as (correct_answers - wrong_answers / 3) / total_items, following the official Spanish medical exam penalty scheme where three wrong answers cancel one correct answer.
- `unanswered ratio` — range: [0, 1]
  - Fraction of questions for which the model produces no valid response.

## Input / output format

**Input**: Multiple-choice medical question with options, provided in either English or Spanish. Inputs may optionally include few-shot examples, retrieved context passages (RAG), or be presented as isolated options for log-probability scoring.

**Output**: A single selected option (e.g., A, B, C, D) or a generated text response containing the answer. For log-probability evaluation, the model outputs log-probabilities for each option.

## Scoring recipe

```python
def compute_metrics(predictions, golds, total_items):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    wrong = sum(1 for p, g in zip(predictions, golds) if p != g and p is not None)
    unanswered = sum(1 for p in predictions if p is None)
    accuracy = correct / len(golds)
    exam_score = (correct - wrong / 3) / total_items
    unanswered_ratio = unanswered / len(golds)
    return {'accuracy': accuracy, 'normalized_exam_score': exam_score, 'unanswered_ratio': unanswered_ratio}
```

## Common pitfalls

- Chain-of-thought (CoT) prompting consistently reduces accuracy and increases non-response rates in this domain, contrary to typical LLM behavior.
- Retrieval-augmented generation (RAG) does not reliably improve performance; retrieved context is often noisy or weakly relevant, and models may ignore it in favor of internal knowledge.
- Log-probability selection yields lower scores than prompting because it evaluates options independently, preventing joint elimination reasoning.

## Evidence (verbatim from paper)

> Performance is evaluated using three metrics:(1) accuracy, the proportion of correct answers; (2) the normalized exam score, based on the official Spanish medical exam scheme (three wrong answers cancel one correct) and normalized by total items; and (3) the unanswered ratio, the fraction of questions with no valid response.

## Citation

```bibtex
@misc{correaguillen2025headqav2,
  title={HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning},
  author={Correa-Guillén et al. (2025)},
  year={2025},
  note={arXiv:2511.15355}
}
```

- arXiv: 2511.15355