Head Qa Eval

---
name: head-qa-eval
description: Evaluates complex reasoning and domain-specific knowledge integration in healthcare by testing models on multi-choice questions derived from real Spanish medical specialization exams. It probes the ability to handle long, context-rich questions requiring cross-domain inference and precise medical knowledge. Use when the user wants to benchmark on HEAD-QA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.04701
  bibtex_key: vilares2019headqa
  confidence: high
---

# head-qa-eval

> HEAD-QA: A Healthcare Dataset for Complex Reasoning — Vilares et al. (2019) (arXiv:1906.04701, 2019)

## What this evaluates

Evaluates complex reasoning and domain-specific knowledge integration in healthcare by testing models on multi-choice questions derived from real Spanish medical specialization exams. It probes the ability to handle long, context-rich questions requiring cross-domain inference and precise medical knowledge.

## Datasets

- **HEAD-QA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions out of the total number of questions.
- `POINTS` — range: other
  - A scoring system from the official exams where a correct answer adds 3 points and an incorrect answer subtracts 1 point.

## Input / output format

**Input**: A multi-choice question with multiple answer options, derived from real healthcare specialization exams.

**Output**: A single selected answer option from the provided choices.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    wrong = sum(1 for p, g in zip(predictions, golds) if p != g)
    total = len(golds)
    accuracy = correct / total
    points = (3 * correct) - (1 * wrong)
    return {'accuracy': accuracy, 'POINTS': points}
```

## Common pitfalls

- The POINTS metric penalizes incorrect answers (-1 point), so models that guess randomly will score negatively, unlike standard accuracy.
- Performance heavily depends on question length and domain; medicine questions (MIR) are consistently the hardest due to longer text, while pharmacology (FIR) and biology (BIR) are easier.
- Cross-lingual information retrieval models sometimes outperform in-domain Spanish models, indicating translation quality or retrieval robustness plays a major role.

## Evidence (verbatim from paper)

> Metrics We use accuracy and a POINTS metric (used in the official exams): a right answer counts 3 points and a wrong one subtracts 1 point.

## Citation

```bibtex
@misc{vilares2019headqa,
  title={HEAD-QA: A Healthcare Dataset for Complex Reasoning},
  author={Vilares et al. (2019)},
  year={2019},
  note={arXiv:1906.04701}
}
```

- arXiv: 1906.04701