Vibe Eval Eval

---
name: vibe-eval-eval
description: Probes multimodal reasoning and visual understanding on real-world images. Specifically designed with a 'hard' subset of prompts that are unsolvable by current frontier models to measure genuine performance gaps and contamination-free generalization. Use when the user wants to benchmark on Vibe-Eval, or asks about evaluating this task. Reports Vibe-Eval Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.02287
  bibtex_key: padlewski2024vibeeval
  confidence: high
---

# vibe-eval-eval

> Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models — Padlewski et al. (2024) (arXiv:2405.02287, 2024)

## What this evaluates

Probes multimodal reasoning and visual understanding on real-world images. Specifically designed with a 'hard' subset of prompts that are unsolvable by current frontier models to measure genuine performance gaps and contamination-free generalization.

## Datasets

- **Vibe-Eval** — total 269; splits: hard (100), normal (169); repo https://github.com/reka-ai/reka-vibe-eval

## Metrics

- `Vibe-Eval Score` **(primary)** — range: percent
  - Automated score generated by the Reka Core evaluator on a 1-5 scale. The final metric is the mean score across all prompts, reported as a percentage. For robustness, the evaluator is run three times per prompt with temperature 0.4, and the mean of these runs is taken.

## Input / output format

**Input**: Multimodal prompts consisting of real-world images paired with text instructions/questions. Evaluated in a zero-shot chat style without exemplars.

**Output**: Open-ended text responses generated by the multimodal language model.

## Scoring recipe

```python
scores = []
for prompt in dataset:
    run_scores = []
    for _ in range(3):
        response = model.generate(prompt, temperature=0.0)
        score = reka_core.evaluate(prompt, response, temperature=0.4) # Returns 1-5
        run_scores.append(score)
    scores.append(mean(run_scores))
final_score = mean(scores) * 100
```

## Common pitfalls

- The hard subset is explicitly seeded using prompts where Reka Core fails, introducing a negative bias against Core on that split.
- Assigning partial credit for hard prompts requires domain expertise and leads to high variance among human raters.
- The automatic evaluator (Reka Core) tends to be generous and compresses the score range, reducing sensitivity to minor performance differences on small subsets.

## Evidence (verbatim from paper)

> For automated evaluation, we run the Core evaluator three times with a temperature of 0.4 and take the mean of the scores. Table 2: Vibe-Eval score and ranking for existing multimodal language models sorted by overall score.

## Citation

```bibtex
@misc{padlewski2024vibeeval,
  title={Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models},
  author={Padlewski et al. (2024)},
  year={2024},
  note={arXiv:2405.02287}
}
```

- arXiv: 2405.02287