Rmbench Eval

---
name: rmbench-eval
description: This benchmark evaluates robotic manipulation policies on memory-dependent, non-Markovian tasks. It probes a model's ability to retain and utilize historical visual and state information over long horizons to complete multi-step dual-arm manipulation sequences. Use when the user wants to benchmark on RMBench, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01229
  bibtex_key: chen2026rmbench
  confidence: high
---

# rmbench-eval

> RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design — Chen et al. (2026) (arXiv:2603.01229, 2026)

## What this evaluates

This benchmark evaluates robotic manipulation policies on memory-dependent, non-Markovian tasks. It probes a model's ability to retain and utilize historical visual and state information over long horizons to complete multi-step dual-arm manipulation sequences.

## Datasets

- **RMBench** — total ?; splits: test (-1); repo https://github.com/robotwin-Platform/rmbench

## Metrics

- `success rate` **(primary)** — range: percent
  - Calculated as the number of successfully completed rollout episodes divided by the total number of evaluated episodes, multiplied by 100 to yield a percentage.

## Input / output format

**Input**: Visual observations (image tokens) and task instructions, optionally augmented with memory tokens (anchor, sliding window, or key memory) from previous timesteps.

**Output**: Next robotic action (e.g., joint commands or end-effector pose).

## Scoring recipe

```python
def compute_success_rate(predictions, gold, total_episodes):
    successes = 0
    for i in range(total_episodes):
        if predictions[i] == gold[i]:  # Binary flag: 1=success, 0=failure
            successes += 1
    return (successes / total_episodes) * 100
```

## Common pitfalls

- Baseline models trained under Markovian assumptions fail on non-Markovian tasks because they only use current observations, ignoring necessary historical context.
- Inaccurate subtask termination detection by the classifier can cause repeated actions or missed contacts, artificially driving success rates to zero on specific tasks (e.g., Press Button).
- Fixed observation windows or sliding memory without anchor retention lead to progressive loss of task-critical information over long horizons.

## Evidence (verbatim from paper)

> On average, Mem-0 improves success rates by 38.4% on $M(1)$ tasks and 21.2% on $M(n)$ tasks relative to the baselines, underscoring the critical role of memory modules in addressing memory-dependent manipulation in RMBench.

## Citation

```bibtex
@misc{chen2026rmbench,
  title={RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.01229}
}
```

- arXiv: 2603.01229