Helm Eval

---
name: helm-eval
description: Evaluates long-horizon vision-language-action (VLA) manipulation capabilities, specifically testing a model's ability to maintain cross-phase context, predict action failures before execution, and recover from perturbations via rollback or replanning. Use when the user wants to benchmark on LIBERO-LONG, CALVIN ABC→D, LIBERO-Recovery, or asks about evaluating this task. Reports TSR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18791
  bibtex_key: zeng2026helm
  confidence: high
---

# helm-eval

> HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation — Zijian Zeng et al. (arXiv:2604.18791, 2026)

## What this evaluates

Evaluates long-horizon vision-language-action (VLA) manipulation capabilities, specifically testing a model's ability to maintain cross-phase context, predict action failures before execution, and recover from perturbations via rollback or replanning.

## Datasets

- **LIBERO-LONG** — total 500; splits: test (500)
- **CALVIN ABC→D** — total ?; splits: test (-1)
- **LIBERO-Recovery** — total ?; splits: test (-1)

## Metrics

- `TSR` **(primary)** — range: percent
  - Task Success Rate: percentage of episodes where the entire task chain is completed successfully.
- `SCR` — range: percent
  - Subgoal Completion Rate: percentage of individual subgoals completed successfully out of all attempted subgoals.
- `RSR` — range: percent
  - Recovery Success Rate: percentage of perturbed episodes where the agent successfully recovers and completes the task after a failure.
- `Chains` — range: other
  - Average number of completed task chains per episode (maximum 5).

## Input / output format

**Input**: Language instructions, current and past visual observations, and retrieved episodic memory context.

**Output**: Sequential robot actions (e.g., joint velocities), binary failure predictions, and recovery/rollback decisions.

## Scoring recipe

```python
def compute_metrics(episodes):
    tsr = sum(1 for e in episodes if e.success) / len(episodes) * 100
    scr = sum(e.completed_subgoals for e in episodes) / sum(e.total_subgoals for e in episodes) * 100
    rsr = sum(1 for e in episodes if e.perturbed and e.recovered) / sum(1 for e in episodes if e.perturbed) * 100
    chains = sum(e.completed_chains for e in episodes) / len(episodes)
    return {'TSR': tsr, 'SCR': scr, 'RSR': rsr, 'Chains': chains}
```

## Common pitfalls

- RSR is only computed on perturbed episodes in LIBERO-Recovery, not on the standard LIBERO-LONG test set.
- Context length alone (H=32) does not substitute for explicit memory retrieval; increasing H yields diminishing returns compared to the EMM+SV framework.

## Evidence (verbatim from paper)

> LIBERO-Recovery (ours): LIBERO-LONG with controlled perturbations (object displacement ±5 cm or gripper state flip) injected at a random subgoal boundary; reports Recovery Success Rate (RSR). Table 2: Main results. TSR = Task Success Rate (%), SCR = Subgoal Completion Rate (%), RSR = Recovery Success Rate (%).

## Citation

```bibtex
@misc{zeng2026helm,
  title={HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation},
  author={Zijian Zeng et al.},
  year={2026},
  note={arXiv:2604.18791}
}
```

- arXiv: 2604.18791