Gym V Eval

---
name: gym-v-eval
description: Evaluates agentic vision models on zero-shot generalization across 179 procedurally generated environments spanning 10 domains. It measures task completion via answer correctness for single-turn interactions and cumulative performance via normalized episodic return for multi-turn interactions. Use when the user wants to benchmark on Gym-V, or asks about evaluating this task. Reports normalized episodic return.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15432
  bibtex_key: meng2026gymv
  confidence: high
---

# gym-v-eval

> Gym-V: A Unified Vision Environment System for Agentic Vision Research — Meng et al. (2026) (arXiv:2603.15432, 2026)

## What this evaluates

Evaluates agentic vision models on zero-shot generalization across 179 procedurally generated environments spanning 10 domains. It measures task completion via answer correctness for single-turn interactions and cumulative performance via normalized episodic return for multi-turn interactions.

## Datasets

- **Gym-V** — total 179; splits: test (-1); repo https://github.com/ModalMinds/gym-v

## Metrics

- `answer correctness` — range: [0, 100] percent
  - Binary score (1 or 0) indicating whether the model's generated answer exactly matches the ground truth for single-turn environments.
- `normalized episodic return` **(primary)** — range: [0, 100] percent
  - Sum of step-wise rewards over an episode, normalized, with any negative values explicitly clipped to zero. Used for multi-turn environments.

## Input / output format

**Input**: Visual observations from procedurally generated environments across 10 domains. Models interact via single-turn (one observation to answer) or multi-turn (sequential observations to actions) protocols.

**Output**: Single-turn: a final answer string. Multi-turn: a sequence of discrete actions over an episode.

## Scoring recipe

```python
def score(prediction, gold, env_type, rewards=None):
    if env_type == 'single-turn':
        return 1.0 if prediction == gold else 0.0
    else:  # multi-turn
        raw_return = sum(rewards) if rewards else 0.0
        return max(0.0, raw_return)  # negative values clipped to zero
```

## Common pitfalls

- Failing to distinguish between single-turn (answer correctness) and multi-turn (episodic return) scoring protocols, which use fundamentally different metrics.
- Overlooking that negative episodic returns are explicitly clipped to zero before being converted to percentages, which can mask poor performance in multi-turn tasks.
- Assuming 'mean@3' refers to top-3 generation; in this context it denotes averaging scores over three independent runs or seeds per environment.

## Evidence (verbatim from paper)

> Single-turn environments are scored by answer correctness, while multi-turn environments report normalized episodic return (negative values clipped to zero). All scores are shown as percentages ($\times 100$).

## Citation

```bibtex
@misc{meng2026gymv,
  title={Gym-V: A Unified Vision Environment System for Agentic Vision Research},
  author={Meng et al. (2026)},
  year={2026},
  note={arXiv:2603.15432}
}
```

- arXiv: 2603.15432