Brazilian Medical Exam Eval

---
name: brazilian-medical-exam-eval
description: Evaluates zero-shot medical knowledge and clinical reasoning of LLMs and MLLMs on a Brazilian Portuguese medical residency exam. Probes text-only comprehension versus multimodal image interpretation across five clinical domains. Use when the user wants to benchmark on HCFMUSP Brazilian Portuguese Medical Residency Exam, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.19885
  bibtex_key: [REDACTED]
  confidence: high
---

# brazilian-medical-exam-eval

> Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam — Truyts et al. (2025) (arXiv:2507.19885, 2025)

## What this evaluates

Evaluates zero-shot medical knowledge and clinical reasoning of LLMs and MLLMs on a Brazilian Portuguese medical residency exam. Probes text-only comprehension versus multimodal image interpretation across five clinical domains.

## Datasets

- **HCFMUSP Brazilian Portuguese Medical Residency Exam** — total 117; splits: test (117)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total 117 items.
- `processing_time` — range: seconds
  - Mean time in seconds to generate a response per question.
- `safety_harm_rating` — range: binary
  - Binary classification (safe/unsafe) of model explanations based on potential patient harm, assessed by three clinical experts using Gwet's AC1 for inter-rater agreement.

## Input / output format

**Input**: Multiple-choice medical exam questions in Brazilian Portuguese, provided either as text-only or with accompanying medical images (radiological or non-radiological).

**Output**: Model selects the correct answer option from the provided choices.

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers, processing_times):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    accuracy = (correct / len(gold_answers)) * 100
    mean_time = sum(processing_times) / len(processing_times)
    return {'accuracy': accuracy, 'processing_time': mean_time}
```

## Common pitfalls

- Models are evaluated in a strict zero-shot setting with no prompt tuning or few-shot examples, which may penalize models not optimized for direct exam-style prompting.
- Multimodal performance drops significantly for radiological images, but the evaluation does not separate vision-language alignment errors from pure medical knowledge gaps.
- Expert safety/harm assessment uses Gwet's AC1 for inter-rater reliability, but agreement is notably lower for incorrectly answered questions, introducing subjectivity in safety labeling.

## Evidence (verbatim from paper)

> The accuracy and processing time for Claude-3-Sonnet, Claude-3-Opus, Claude-3.5-Sonnet and Claude-3-Haiku were analyzed on all 117 questions (Table[2]). Overall, we have observed a tendency to decrease accuracy (except for Claude 3.5 sonnet) mean processing time per question increases were observed with the addiction of questions containing images (Fig.[3], filled circles).

## Citation

```bibtex
@misc{truyts2025zeroshot,
  title={Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam},
  author={Truyts et al. (2025)},
  year={2025},
  note={arXiv:2507.19885}
}
```

- arXiv: 2507.19885