Fewmmbench Eval

---
name: fewmmbench-eval
description: Evaluates multimodal large language models on few-shot learning capabilities across nine diverse tasks. It probes the models' ability to leverage in-context demonstrations (0, 4, or 8 shots) and chain-of-thought reasoning under controlled retrieval settings, measuring performance relative to zero-shot baselines. Use when the user wants to benchmark on FewMMBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.21854
  bibtex_key: dogan2026fewmmbench
  confidence: high
---

# fewmmbench-eval

> FewMMBench: A Benchmark for Multimodal Few-Shot Learning — Dogan et al. (2026) (arXiv:2602.21854, 2026)

## What this evaluates

Evaluates multimodal large language models on few-shot learning capabilities across nine diverse tasks. It probes the models' ability to leverage in-context demonstrations (0, 4, or 8 shots) and chain-of-thought reasoning under controlled retrieval settings, measuring performance relative to zero-shot baselines.

## Datasets

- **FewMMBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions based on ground-truth labels.

## Input / output format

**Input**: Image-text query with 2-4 candidate answers in a multiple-choice format, optionally augmented with few-shot demonstrations (0, 4, or 8 examples) retrieved via random or graph-cut-based methods, and optionally chain-of-thought reasoning prompts.

**Output**: Model generates a text response selecting one candidate answer. Predictions are normalized via regex-based post-processing or evaluated via perplexity-based pairwise ranking over candidates.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        normalized_pred = extract_option_regex(pred)
        if normalized_pred == gold:
            correct += 1
    return correct / len(gold_labels)
```

## Common pitfalls

- Regex-based post-processing may misinterpret open-ended or unexpected model outputs.
- Answer order bias can affect accuracy if models are sensitive to candidate placement.
- Perplexity-based ranking is an alternative that may yield different results than exact-match accuracy.

## Evidence (verbatim from paper)

> The primary metric is accuracy, computed as the proportion of correctly answered questions based on ground-truth labels. To ensure consistency in answer extraction, especially across models with varying output styles, we apply regex-based post-processing to normalize model predictions.

## Citation

```bibtex
@misc{dogan2026fewmmbench,
  title={FewMMBench: A Benchmark for Multimodal Few-Shot Learning},
  author={Dogan et al. (2026)},
  year={2026},
  note={arXiv:2602.21854}
}
```

- arXiv: 2602.21854