Provision Eval

---
name: provision-eval
description: Evaluates multimodal language models' vision-centric instruction following, multi-image reasoning, and general visual understanding. It probes the model's ability to process single and multiple images, answer factual questions, and perform complex reasoning across a suite of standard benchmarks. Use when the user wants to benchmark on CV-Bench (CVB-2D, CVB-3D), SEED-Bench, MMBench (MMB), MME, QBench2, MMMU, RealWorldQA, MMStar, MMVet, Mantis-Eval, MMT-Bench (MMT), TextVQA, or asks about evaluating this task. Reports Avg..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.07012
  bibtex_key: zhang2024provision
  confidence: high
---

# provision-eval

> ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models — Zhang et al. (2024) (arXiv:2412.07012, 2024)

## What this evaluates

Evaluates multimodal language models' vision-centric instruction following, multi-image reasoning, and general visual understanding. It probes the model's ability to process single and multiple images, answer factual questions, and perform complex reasoning across a suite of standard benchmarks.

## Datasets

- **CV-Bench (CVB-2D, CVB-3D)** — total ?; splits: test (-1)
- **SEED-Bench** — total ?; splits: test (-1)
- **MMBench (MMB)** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **QBench2** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **RealWorldQA** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **MMVet** — total ?; splits: test (-1)
- **Mantis-Eval** — total ?; splits: test (-1)
- **MMT-Bench (MMT)** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)

## Metrics

- `Avg.` **(primary)** — range: percent
  - The arithmetic mean of the accuracy scores reported on each individual benchmark listed in the datasets array.

## Input / output format

**Input**: One or more images paired with a text instruction or question.

**Output**: A short textual answer or a selected option from a multiple-choice list.

## Scoring recipe

```python
def compute_avg_score(predictions, golds, benchmarks):
    scores = {}
    for bench in benchmarks:
        correct = sum(1 for p, g in zip(predictions[bench], golds[bench]) if p == g)
        scores[bench] = correct / len(golds[bench]) * 100
    return sum(scores.values()) / len(scores)
```

## Common pitfalls

- The choice between short-answer and multiple-choice formats significantly impacts performance; mixing them ('half-half') often yields the best results.
- Replacing base training data vs. augmenting it changes the optimization landscape, with augmentation generally providing more stable gains.
- Manual vs. model-generated scene graphs in the training data lead to different scaling behaviors, with manual graphs generally outperforming at smaller scales.

## Evidence (verbatim from paper)

> | CVB-2D | CVB-3D | SEED | MMB | MME | QBench2 | MMMU | RealWorldQA | Avg. |

## Citation

```bibtex
@misc{zhang2024provision,
  title={ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.07012}
}
```

- arXiv: 2412.07012