Manipulationnet Eval

---
name: manipulationnet-eval
description: Evaluates real-world robot manipulation capabilities across two complementary tracks: physical skills (sensorimotor execution under contact, clearance, and perceptual constraints) and embodied reasoning (multimodal grounding of natural language and visual instructions into grounded actions). Use when the user wants to benchmark on ManipulationNet Benchmark, or asks about evaluating this task. Reports task success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.04363
  bibtex_key: chen2026manipulationnet
  confidence: high
---

# manipulationnet-eval

> ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning — Chen et al. (2026) (arXiv:2603.04363, 2026)

## What this evaluates

Evaluates real-world robot manipulation capabilities across two complementary tracks: physical skills (sensorimotor execution under contact, clearance, and perceptual constraints) and embodied reasoning (multimodal grounding of natural language and visual instructions into grounded actions).

## Datasets

- **ManipulationNet Benchmark** — total ?; splits: test (-1)

## Metrics

- `declutter rate` — range: [0, 1]
  - Ratio of objects successfully grasped and removed to the total number of spawned objects in a scene.
- `grasp success rate` — range: [0, 1]
  - Ratio of successful grasps to the total number of grasp attempts made during a scene.
- `time efficiency` — range: seconds
  - Total wall-clock time required to complete the task or scene layout.
- `task success rate` **(primary)** — range: [0, 1]
  - Binary or graded success determined by centralized judges applying task-specific metrics after integrity verification.

## Input / output format

**Input**: Standardized physical object set configured on a workstation; task instructions (language prompts, visual prompts, or projected scene layouts) delivered via mnet-client; external camera recording setup.

**Output**: Robot execution logs, real-time status messages, cryptographic hashes of video frames, and a complete recorded video of the trial.

## Scoring recipe

```python
def score_submission(video, logs, task_goal):
    if not verify_one_time_code(video): return None
    if not verify_hashes(video, logs): return None
    if task_goal == 'grasping_in_clutter':
        declutter_rate = count_success(logs) / count_spawned(logs)
        grasp_success_rate = count_success(logs) / count_attempts(logs)
        time_eff = logs['end_time'] - logs['start_time']
        return declutter_rate, grasp_success_rate, time_eff
    else:
        return judge_task_success(video, logs, task_goal)
```

## Common pitfalls

- Submissions must display a server-generated one-time code in the camera view; missing it invalidates the trial.
- Integrity relies on real-time cryptographic hashes of frames and a final video hash; pre-recording or post-processing is explicitly blocked.
- Metrics are not auto-computed by the client; final scoring requires centralized committee verification against task-specific criteria.

## Evidence (verbatim from paper)

> The grasping in clutter benchmark evaluates the grasping performance based on the declutter rate (the ratio of objects successfully grasped and removed to all spawned objects), the grasp success rate (the ratio of successful grasps to the total number of grasp attempts), and time efficiency. Task-specific performance metrics are applied only after these integrity checks have been passed, ensuring objective and comparable evaluations across systems.

## Citation

```bibtex
@misc{chen2026manipulationnet,
  title={ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2603.04363}
}
```

- arXiv: 2603.04363