Domino Eval

---
name: domino-eval
description: Evaluates a robot policy's ability to perform manipulation tasks in environments with moving objects and dynamic spatiotemporal changes. It probes the model's capacity for historical context integration and future state anticipation to maintain control stability and task success under motion. Use when the user wants to benchmark on DOMINO@0.1, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.15620
  bibtex_key: fang2026domino
  confidence: high
---

# domino-eval

> Towards Generalizable Robotic Manipulation in Dynamic Environments — Fang et al. (2026) (arXiv:2603.15620, 2026)

## What this evaluates

Evaluates a robot policy's ability to perform manipulation tasks in environments with moving objects and dynamic spatiotemporal changes. It probes the model's capacity for historical context integration and future state anticipation to maintain control stability and task success under motion.

## Datasets

- **DOMINO@0.1** — total 110000; splits: train (-1), test (-1); repo https://github.com/H-EmbodVis/DOMINO

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - Percentage of evaluation episodes where the robot successfully completes the specified manipulation task, calculated as (successful_episodes / total_episodes) * 100.
- `Manipulation Score (MS)` — range: other
  - Continuous metric quantifying the quality of interaction with moving targets, reflecting control stability, trajectory tracking accuracy, and temporal consistency during manipulation.

## Input / output format

**Input**: Multi-view RGB images, historical optical flow frames, object-centric predictive queries, robot proprioceptive state, and natural language task instructions.

**Output**: Robot action commands (e.g., joint velocities or end-effector poses) for each control step.

## Scoring recipe

```python
def compute_sr(predictions, gold):
    success_count = 0
    for pred, gold in zip(predictions, gold):
        if check_task_completion(pred, gold):
            success_count += 1
    return (success_count / len(gold)) * 100

def compute_ms(predictions, gold):
    scores = [calculate_interaction_quality(pred, gold) for pred, gold in zip(predictions, gold)]
    return sum(scores) / len(scores)
```

## Common pitfalls

- Assuming fine-tuning on dynamic data alone bridges the static-to-dynamic performance gap (improvements are <3%).
- Using single-frame observations instead of historical context causes severe degradation in dynamic settings.
- Confusing zero-shot static-to-dynamic transfer (S→D) with fine-tuned dynamic-to-dynamic (D→D) evaluation protocols.

## Evidence (verbatim from paper)

> We primarily evaluate on our proposed DOMINO@0.1 benchmark, reporting the Success Rate (SR) and Manipulation Score (MS). ... Specifically, PUMA achieves the highest average success rate of 17.20%, substantially outperforming recent strong baselines ... Furthermore, our method attains a peak Manipulation Score of 34.97, indicating a higher quality of interaction with moving targets.

## Citation

```bibtex
@misc{fang2026domino,
  title={Towards Generalizable Robotic Manipulation in Dynamic Environments},
  author={Fang et al. (2026)},
  year={2026},
  note={arXiv:2603.15620}
}
```

- arXiv: 2603.15620