Smmile Eval

---
name: smmile-eval
description: Evaluates the ability of multimodal large language models (MLLMs) to perform in-context learning (ICL) in medical domains. It probes how effectively models leverage provided image-question-answer demonstrations to answer new clinical queries, while also measuring robustness to irrelevant examples, recency bias, and the gap between automated and expert clinical judgment. Use when the user wants to benchmark on SMMILE, SMMILE++, or asks about evaluating this task. Reports LLM-as-a-Judge.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.21355
  bibtex_key: rieff2025smmile
  confidence: high
---

# smmile-eval

> SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning — Rieff et al. (2025) (arXiv:2506.21355, 2025)

## What this evaluates

Evaluates the ability of multimodal large language models (MLLMs) to perform in-context learning (ICL) in medical domains. It probes how effectively models leverage provided image-question-answer demonstrations to answer new clinical queries, while also measuring robustness to irrelevant examples, recency bias, and the gap between automated and expert clinical judgment.

## Datasets

- **SMMILE** — total 111; splits: test (111)
- **SMMILE++** — total 1038; splits: test (1038)

## Metrics

- `Exact Match (EM)` — range: percent
  - Binary score (0 or 100) indicating whether the model's generated answer exactly matches the ground-truth answer after normalization for formatting, punctuation, and capitalization.
- `LLM-as-a-Judge` **(primary)** — range: percent
  - A text-only LLM (Llama3.3 70B) evaluates the model's generation against the ground-truth answer, outputting a binary judgment (0 for incorrect, 1 for correct). The final score is the percentage of outputs judged correct.
- `MCQA Accuracy` — range: percent
  - Percentage of multiple-choice questions where the model correctly selects the ground-truth option.
- `Expert Rating` — range: percent
  - Binary rating (correct/incorrect) assigned by clinical experts to model responses. Score is the percentage of responses rated correct.

## Input / output format

**Input**: A system message, a set of multimodal in-context examples (image-question-answer triplets), and a query consisting of a medical image and a corresponding question.

**Output**: Free-text response (up to 512 tokens) for open-ended tasks, or a selected option letter/text for multiple-choice questions.

## Scoring recipe

```python
def compute_em(pred, gold):
    pred_norm = normalize_text(pred) # handles formatting, punctuation, capitalization
    return 100 if pred_norm == normalize_text(gold) else 0

def compute_llm_judge(pred, gold):
    prompt = f'Ground truth: {gold}\nModel output: {pred}\nJudge accuracy (0/1):'
    judge = llama3_3_70b.generate(prompt)
    return 100 if judge.strip() == '1' else 0

def compute_mcqa(pred, options):
    return 100 if pred in options else 0
```

## Common pitfalls

- LLM-as-a-Judge can be overly lenient in ICL settings, accepting responses that merely match the format/phrasing of in-context examples without ensuring clinical adequacy.
- In-context learning can degrade performance if irrelevant examples are included or due to recency bias, where placing the most relevant example last significantly boosts scores.
- Exact Match (EM) is extremely strict and often yields much lower scores than LLM-as-a-Judge or MCQA, masking actual clinical reasoning capability.

## Evidence (verbatim from paper)

> For open-ended evaluations, we evaluate MLLM-generated outputs using two metrics. First, the Exact Match (EM) metric counts a model generation as correct (score of 100) if it exactly matches the ground-truth answer, and incorrect (score of 0) otherwise. During evaluation, answers are normalized to account for minor variations in formatting, punctuation, and capitalization before comparison. Second, the LLM-as-a-Judge approach provides a text-only LLM (Llama3.3 70B) with both the model generation and the ground-truth answer; the model is then prompted to evaluate accuracy. The LLM provides a binary judgment (0 for incorrect, 1 for correct) for each generated output, and the final score represents the percentage of outputs judged as correct.

## Citation

```bibtex
@misc{rieff2025smmile,
  title={SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning},
  author={Rieff et al. (2025)},
  year={2025},
  note={arXiv:2506.21355}
}
```

- arXiv: 2506.21355