Partinstruct Eval

---
name: partinstruct-eval
description: This benchmark probes a robot policy's ability to follow fine-grained, part-level natural language instructions for long-horizon manipulation. It specifically tests zero-shot task decomposition, 3D part grounding, and multi-step planning under varying object, part, and task generalization conditions. Use when the user wants to benchmark on PartInstruct, or asks about evaluating this task. Reports success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21652
  bibtex_key: yin2025partinstruct
  confidence: high
---

# partinstruct-eval

> PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation — Yin et al. (2025) (arXiv:2505.21652, 2025)

## What this evaluates

This benchmark probes a robot policy's ability to follow fine-grained, part-level natural language instructions for long-horizon manipulation. It specifically tests zero-shot task decomposition, 3D part grounding, and multi-step planning under varying object, part, and task generalization conditions.

## Datasets

- **PartInstruct** — total ?; splits: train (10000), test (1800)

## Metrics

- `success` **(primary)** — range: [0, 1]
  - Binary metric per episode. Returns 1 if the agent successfully completes every skill in the predefined chain, and 0 otherwise. The final score is the average success across all evaluation episodes.

## Input / output format

**Input**: Multimodal observations (RGB images, depth maps, point clouds, object/part annotations, proprioceptive joint states and end-effector poses) paired with a single natural language task instruction.

**Output**: A 7-dimensional action vector per timestep: 3D Cartesian coordinates, roll/pitch/yaw angles, and a single gripper position control.

## Scoring recipe

```python
def compute_success(predictions, gold):
    # predictions: sequence of 7D action vectors
    # gold: list of goal predicates for each skill in the chain
    success = True
    for skill_goal in gold:
        if not check_predicate_satisfaction(predictions, skill_goal):
            success = False
            break
    return 1.0 if success else 0.0
```

## Common pitfalls

- Models are trained with explicit skill instructions but evaluated only with high-level task instructions, requiring zero-shot decomposition.
- Success is strictly defined as completing the entire chain of base skills; partial task completion does not count.
- The benchmark tests five distinct generalization conditions (OS, OI, TP, TC, OC); reporting a single aggregate score hides specific failure modes.

## Evidence (verbatim from paper)

> As defined in Section [III-C], each part-level skill has a binary success criterion. A completion of the entire task means the agent manages to complete every single skill defined in the skill chain.

## Citation

```bibtex
@misc{yin2025partinstruct,
  title={PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation},
  author={Yin et al. (2025)},
  year={2025},
  note={arXiv:2505.21652}
}
```

- arXiv: 2505.21652