Vhelm Eval

---
name: vhelm-eval
description: Holistic evaluation of vision-language models across multiple dimensions including visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Use when the user wants to benchmark on VHELM Scenarios, or asks about evaluating this task. Reports scenario_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.07112
  bibtex_key: lee2024vhelm
  confidence: medium
---

# vhelm-eval

> VHELM: A Holistic Evaluation of Vision Language Models — Lee et al. (2024) (arXiv:2410.07112, 2024)

## What this evaluates

Holistic evaluation of vision-language models across multiple dimensions including visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety.

## Datasets

- **VHELM Scenarios** — total 915000; splits: test (-1); repo https://github.com/stanford-crfm/helm

## Metrics

- `scenario_score` **(primary)** — range: other
  - Not specified in the provided section.

## Input / output format

**Input**: Text and image inputs per scenario, processed with standard inference parameters.

**Output**: Model-generated text responses.

## Scoring recipe

```python
# Not specified in the provided section.
# Evaluation runs sample up to 1,000 instances per scenario.
# Total instances per run: 915K across 22 models.
```

## Common pitfalls

- Sampling is capped at a maximum of 1,000 instances per scenario to save cost/time, which may not reflect full dataset performance.
- Standard inference parameters are maintained per model, but specific hyperparameters or prompting details are not provided in this section.

## Evidence (verbatim from paper)

> Our evaluation run randomly samples a maximum of 1,000 instances for each of the scenarios in order to alleviate monetary and time constraints; a single evaluation run on the 22 VLMs uses a total of 915K instances and consumes 51.6M input text tokens, 9.4M output text tokens, and 915K images.

## Citation

```bibtex
@misc{lee2024vhelm,
  title={VHELM: A Holistic Evaluation of Vision Language Models},
  author={Lee et al. (2024)},
  year={2024},
  note={arXiv:2410.07112}
}
```

- arXiv: 2410.07112