Larybench Eval

---
name: larybench-eval
description: Evaluates how well vision models and latent action representations capture semantic action categories and map visual features to low-level robotic control trajectories. It probes both high-level action understanding and physical grounding for generalizable vision-to-action alignment across diverse robotic and human motion datasets. Use when the user wants to benchmark on VLABench, CALVIN, RoboCOIN, AgiBotWorld-Beta, or asks about evaluating this task. Reports Top-1 Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11689
  bibtex_key: nie2026lary
  confidence: high
---

# larybench-eval

> LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment — Nie et al. (2026) (arXiv:2604.11689, 2026)

## What this evaluates

Evaluates how well vision models and latent action representations capture semantic action categories and map visual features to low-level robotic control trajectories. It probes both high-level action understanding and physical grounding for generalizable vision-to-action alignment across diverse robotic and human motion datasets.

## Datasets

- **VLABench** — total ?; splits: train (-1), val (-1)
- **CALVIN** — total ?; splits: train (-1), val (-1)
- **RoboCOIN** — total ?; splits: train (-1), val (-1)
- **AgiBotWorld-Beta** — total ?; splits: train (-1), val (-1)

## Metrics

- `Top-1 Accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted action categories averaged across all atomic and composite classes. Computed as the number of correct predictions divided by the total number of samples.
- `Mean Squared Error (MSE)` — range: [0, inf)
  - Average squared difference between predicted and ground-truth end-effector trajectory values. Computed as the mean of (predicted - true)^2 across all dimensions and time steps.

## Input / output format

**Input**: Classification: 9 uniformly sampled frames per video clip, resized to 224×224. Regression: Pairs of images separated by a 5-frame interval. Both tasks use continuous latent embeddings extracted by the vision encoder or latent action model.

**Output**: Classification: Discrete action category label (atomic or composite). Regression: Continuous end-effector trajectory chunk (7/12/16-DoF per chunk).

## Scoring recipe

```python
# Classification
preds = probe(latent_features)
acc = (preds == gold_labels).float().mean()

# Regression
traj_pred = mlp(latent_features)
mse = ((traj_pred - gold_traj) ** 2).mean()
```

## Common pitfalls

- Using discretized codebook indices instead of continuous latent embeddings causes significant information loss for regression tasks.
- Uniform frame sampling fails on composite tasks with varying FPS and motion speeds; requires motion-guided sampling to capture temporal dynamics.
- Latent dimensions vary across encoders; a projection layer is required to align dimensions before classification for fair comparison.

## Evidence (verbatim from paper)

> For semantic action classification, we report the Top-1 Accuracy averaged across all action categories. For low-level control regression, we utilize the Mean Squared Error (MSE) to measure physical fidelity.

## Citation

```bibtex
@misc{nie2026lary,
  title={LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment},
  author={Nie et al. (2026)},
  year={2026},
  note={arXiv:2604.11689}
}
```

- arXiv: 2604.11689