Memoryvla Eval

---
name: memoryvla-eval
description: Evaluates long-horizon robotic manipulation capabilities of vision-language-action models under non-Markovian dynamics. It probes the model's ability to maintain and retrieve perceptual and semantic memory over extended task horizons using only third-person visual observations and language instructions. Use when the user wants to benchmark on SimplerEnv-Bridge, SimplerEnv-Fractal, LIBERO, Real-world Manipulation, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.19236
  bibtex_key: shi2025memoryvla
  confidence: high
---

# memoryvla-eval

> MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation — Shi et al. (2025) (arXiv:2508.19236, 2025)

## What this evaluates

Evaluates long-horizon robotic manipulation capabilities of vision-language-action models under non-Markovian dynamics. It probes the model's ability to maintain and retrieve perceptual and semantic memory over extended task horizons using only third-person visual observations and language instructions.

## Datasets

- **SimplerEnv-Bridge** — total ?; splits: test (-1)
- **SimplerEnv-Fractal** — total ?; splits: test (-1)
- **LIBERO** — total ?; splits: test (-1)
- **Real-world Manipulation** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of successful task executions out of total trials per task. Calculated as (number of successful trials / total trials) × 100.

## Input / output format

**Input**: Single third-person RGB frame at 224×224 together with the language instruction.

**Output**: 7-DoF actions.

## Scoring recipe

```python
def compute_success_rate(trials, success_condition):
    successes = sum(1 for trial in trials if success_condition(trial))
    return (successes / len(trials)) * 100
```

## Common pitfalls

- Baselines like π₀ and CogACT often use additional proprioceptive states or wrist-camera views; MemoryVLA strictly uses only third-person RGB and language, making direct comparison sensitive to input modality assumptions.
- Real-world long-horizon tasks use step-wise scoring to track progress over sub-goals rather than binary end-state success, which can inflate or deflate scores depending on the evaluation script.
- SimplerEnv-Fractal's Visual Aggregation (VA) setting intentionally alters backgrounds, lighting, and distractors to stress-test robustness, differing from standard visual matching protocols.

## Evidence (verbatim from paper)

> Results are reported at the best validation step, and each task is evaluated with 24 trials to compute success rates.

## Citation

```bibtex
@misc{shi2025memoryvla,
  title={MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulation},
  author={Shi et al. (2025)},
  year={2025},
  note={arXiv:2508.19236}
}
```

- arXiv: 2508.19236