Roboarena Eval

---
name: roboarena-eval
description: This benchmark evaluates the ranking accuracy and sample efficiency of generalist robot policies in real-world environments. It probes whether a distributed, pairwise comparison framework can reliably approximate an exhaustive oracle ranking across diverse scenes and tasks. Use when the user wants to benchmark on RoboArena Real-World Policy Evaluation, or asks about evaluating this task. Reports Pearson correlation r.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.18123
  bibtex_key: atreya2025roboarena
  confidence: high
---

# roboarena-eval

> RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies — Atreya et al. (2025) (arXiv:2506.18123, 2025)

## What this evaluates

This benchmark evaluates the ranking accuracy and sample efficiency of generalist robot policies in real-world environments. It probes whether a distributed, pairwise comparison framework can reliably approximate an exhaustive oracle ranking across diverse scenes and tasks.

## Datasets

- **RoboArena Real-World Policy Evaluation** — total 4284; splits: test (4284)

## Metrics

- `Pearson correlation r` **(primary)** — range: [-1, 1]
  - Measures the linear correlation between the predicted policy ranking and the oracle ranking derived from exhaustive evaluation. Ranges from -1 to 1, where 1 indicates perfect rank agreement.
- `Mean Maximum Rank Violation (MMRV)` — range: other
  - A ranking metric that quantifies the maximum discrepancy in rank positions between the predicted and oracle rankings, explicitly accounting for performance differences between policies rather than just ordinal swaps.

## Input / output format

**Input**: Pairwise video observations of two robot policies executing the same task instruction in a real-world scene, accompanied by the task prompt.

**Output**: Per comparison: progress scores for each policy, a preference/winner label, and a task category. Aggregated output: a global ranked list of policies.

## Scoring recipe

```python
def compute_metrics(predicted_ranking, oracle_ranking):
    policies = list(predicted_ranking.keys())
    pred_vals = [predicted_ranking[p] for p in policies]
    oracle_vals = [oracle_ranking[p] for p in policies]
    r = pearsonr(pred_vals, oracle_vals)
    mmrv = max(abs(predicted_ranking[p] - oracle_ranking[p]) for p in policies)
    return r, mmrv
```

## Common pitfalls

- Progress-based rankings alone can miss nuanced policy behaviors (e.g., speed or confidence) when evaluators assign identical progress scores to both policies in a pair.
- Conventional fixed-task evaluations are insufficient for generalist policies because they lack the environmental and task diversity needed to capture robust performance.
- MMRV accounts for performance differences between policies, not just ordinal rank swaps, so interpreting it as a simple rank-distance metric is incorrect.

## Evidence (verbatim from paper)

> We follow Li et al. [29] and report Pearson correlation $r$ as well as Mean Maximum Rank Violation (MMRV), a ranking metric that takes the performance difference between policies into account.

## Citation

```bibtex
@misc{atreya2025roboarena,
  title={RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies},
  author={Atreya et al. (2025)},
  year={2025},
  note={arXiv:2506.18123}
}
```

- arXiv: 2506.18123