Dermabench Eval

---
name: dermabench-eval
description: Evaluates vision-language models on dermatological visual question answering and clinical reasoning. It probes the model's ability to understand skin lesions across diverse Fitzpatrick skin types, answer structured diagnostic questions, and reason about morphology and distribution. Use when the user wants to benchmark on DermaBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.14084
  bibtex_key: yilmaz2026dermabench
  confidence: medium
---

# dermabench-eval

> DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning — Yilmaz et al. (2026) (arXiv:2601.14084, 2026)

## What this evaluates

Evaluates vision-language models on dermatological visual question answering and clinical reasoning. It probes the model's ability to understand skin lesions across diverse Fitzpatrick skin types, answer structured diagnostic questions, and reason about morphology and distribution.

## Datasets

- **DermaBench** — total 14474; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard VQA exact-match accuracy: percentage of questions where the model's predicted answer exactly matches the ground truth answer (case-insensitive).

## Input / output format

**Input**: A clinical dermatology image paired with a structured question from a hierarchical schema (Q0–Q21) covering diagnosis, morphology, distribution, and clinical reasoning.

**Output**: A text answer corresponding to the question, formatted as a single choice, multiple choice, or open-ended response depending on the question type.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(gold_answers)
```

## Common pitfalls

- The dataset spans Fitzpatrick skin types I–VI, so performance may vary significantly by skin tone; models must be evaluated with fairness-aware stratification.
- Questions follow a hierarchical branching logic; models must handle conditional sub-questions correctly rather than answering all 22 questions uniformly.
- The benchmark emphasizes clinical reasoning and morphology grounding, so superficial visual matching without diagnostic justification may yield high accuracy but fail clinical validity checks.

## Evidence (verbatim from paper)

> All data follow a VQA-style JSON schema that includes image path, modality tag, Fitzpatrick phototype, diagnostic and morphological labels, question category, answer text, annotator identifier, and additional metadata fields supporting fairness and reasoning analyses. The dataset is released for research use under a permissive license and is accompanied by documentation detailing the annotation protocol, consensus process, and recommended evaluation workflows.

## Citation

```bibtex
@misc{yilmaz2026dermabench,
  title={DermaBench: A Clinician-Annotated Benchmark Dataset for Dermatology Visual Question Answering and Reasoning},
  author={Yilmaz et al. (2026)},
  year={2026},
  note={arXiv:2601.14084}
}
```

- arXiv: 2601.14084