Pixel Reasoner Eval

---
name: pixel-reasoner-eval
description: Evaluates multimodal models' ability to perform fine-grained visual reasoning, object counting, temporal video understanding, and complex infographic parsing. It specifically probes whether models can effectively leverage pixel-space operations (e.g., zooming, frame selection) rather than defaulting to text-only reasoning pathways. Use when the user wants to benchmark on V* (V-Star), TallyQA, MVBench, InfographicVQA, or asks about evaluating this task. Reports Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.15966
  bibtex_key: [REDACTED]
  confidence: high
---

# pixel-reasoner-eval

> Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning — Wang et al. (2025) (arXiv:2505.15966, 2025)

## What this evaluates

Evaluates multimodal models' ability to perform fine-grained visual reasoning, object counting, temporal video understanding, and complex infographic parsing. It specifically probes whether models can effectively leverage pixel-space operations (e.g., zooming, frame selection) rather than defaulting to text-only reasoning pathways.

## Datasets

- **V* (V-Star)** — total ?; splits: V* test (-1)
- **TallyQA** — total ?; splits: TallyQA-complex (-1)
- **MVBench** — total ?; splits: MVBench-test (-1)
- **InfographicVQA** — total ?; splits: InfoVQA-test (-1)

## Metrics

- `Acc` **(primary)** — range: percent
  - Accuracy: the percentage of questions where the model's generated answer exactly matches the ground truth label.
- `ANLS` — range: [0, 1]
  - Average Normalized Levenshtein Similarity: the mean of (1 - normalized Levenshtein distance) across all predictions and their corresponding gold answers.

## Input / output format

**Input**: Multimodal inputs (images, webpages, or videos) paired with natural language questions/prompts.

**Output**: Textual answers generated via greedy decoding.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_name):
    if metric_name == 'Acc':
        correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
        return (correct / len(golds)) * 100
    elif metric_name == 'ANLS':
        total_sim = 0.0
        for p, g in zip(predictions, golds):
            dist = levenshtein_distance(p, g)
            max_len = max(len(p), len(g))
            total_sim += 1.0 - (dist / max_len) if max_len > 0 else 1.0
        return total_sim / len(golds)
    return 0.0
```

## Common pitfalls

- Using beam search or sampling instead of greedy decoding, which the paper explicitly specifies for all evaluations.
- Evaluating on the full TallyQA dataset instead of the 'complex' split, which significantly changes difficulty and reported scores.
- Ignoring that several baseline models only report results on V* Bench, making cross-benchmark comparisons for those models incomplete.

## Evidence (verbatim from paper)

> | Metric |  | Acc | Acc | Acc | ANLS |

## Citation

```bibtex
@misc{wang2025pixelreasoner,
  title={Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.15966}
}
```

- arXiv: 2505.15966