Wiser Eval

---
name: wiser-eval
description: Evaluates a robot's ability to generalize visuomotor planning to unseen visual signals, referring expressions, and spatial relationships during cube-picking and placing tasks. It probes semantic generalization and robustness to distribution shifts in embodied instruction following. Use when the user wants to benchmark on WISER Benchmark, or asks about evaluating this task. Reports Success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11751
  bibtex_key: [REDACTED]
  confidence: high
---

# wiser-eval

> Grounded World Model for Semantically Generalizable Planning — Quanyi Li et al. (2026) (arXiv:2604.11751, 2026)

## What this evaluates

Evaluates a robot's ability to generalize visuomotor planning to unseen visual signals, referring expressions, and spatial relationships during cube-picking and placing tasks. It probes semantic generalization and robustness to distribution shifts in embodied instruction following.

## Datasets

- **WISER Benchmark** — total ?; splits: train (-1), test (288); repo https://github.com/QuanyiLi/gwm-wiser

## Metrics

- `Success` **(primary)** — range: [0, 1]
  - Fraction of tasks where the robot successfully executes the instructed manipulation (grasping the correct object and placing it at the target location). Computed as successful tasks divided by total tasks.

## Input / output format

**Input**: Natural language task instructions (system prompt + task prompt), current robot observations (images from wrist and other cameras), and robot state.

**Output**: Action sequences or trajectories for robot execution, typically proposed via MPC and scored via cosine similarity to instruction embeddings.

## Scoring recipe

```python
def compute_success(predictions, gold):
    correct = 0
    for pred, gold_inst in zip(predictions, gold):
        if check_grasp_correctness(pred, gold_inst) and check_place_correctness(pred, gold_inst):
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Models often overfit to training sentence structures and lose the ability to understand decomposed instructions.
- Performance is bottlenecked by the scoring model (Qwen3-VL-Embedding); even ground-truth future embeddings do not yield 100% success.
- Delta vs. absolute action spaces do not inherently improve generalizability, contrary to some assumptions.

## Evidence (verbatim from paper)

> The GWM-MPC achieves the best test-scene performance, yielding an 87% success rate across 288 test tasks that feature unseen referring expressions, spatial relationship descriptions, and visual signals.

## Citation

```bibtex
@misc{li2026groundedworldmodel,
  title={Grounded World Model for Semantically Generalizable Planning},
  author={Quanyi Li et al. (2026)},
  year={2026},
  note={arXiv:2604.11751}
}
```

- arXiv: 2604.11751