Isaacsim Kitchen Eval

---
name: isaacsim-kitchen-eval
description: Evaluates a robot's ability to decompose high-level language instructions into executable task plans and execute them in a simulated kitchen environment. It jointly measures planning accuracy and low-level control success under strict time and spatial constraints. Use when the user wants to benchmark on IsaacSim Kitchen Benchmark, or asks about evaluating this task. Reports EM.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.15663
  bibtex_key: kachaev2025mindandmotion
  confidence: high
---

# isaacsim-kitchen-eval

> Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation — Kachaev et al. (2025) (arXiv:2508.15663, 2025)

## What this evaluates

Evaluates a robot's ability to decompose high-level language instructions into executable task plans and execute them in a simulated kitchen environment. It jointly measures planning accuracy and low-level control success under strict time and spatial constraints.

## Datasets

- **IsaacSim Kitchen Benchmark** — total ?; splits: test (-1)

## Metrics

- `EM` **(primary)** — range: [0, 1]
  - Exact Match accuracy for task planning steps. Calculated as the average per-plan accuracy where each step must exactly match the ground-truth string character-by-character.
- `SR` — range: [0, 1]
  - Success Rate for mobile manipulation tasks. Equals 1 if a task completes within 120 seconds and meets spatial criteria (navigation: base within 10 cm of target; manipulation: object within 5 cm of target), else 0.
- `MSE` — range: other
  - Mean Squared Error between the predicted trajectory and the expert trajectory, averaged across all predicted trajectories.

## Input / output format

**Input**: High-level natural language instruction describing a kitchen task.

**Output**: For planning: a sequence of text strings representing task steps. For control: a predicted trajectory or action sequence for each task step.

## Scoring recipe

```python
def compute_joint_metric(predictions, ground_truth, instructions):
    total_score = 0.0
    N = len(instructions)
    for i, instr in enumerate(instructions):
        pred_plan = predictions['plan'][i]
        gt_plan = ground_truth['plan'][i]
        n_i = len(gt_plan)
        em_i = sum(1 for p, g in zip(pred_plan, gt_plan) if p == g) / n_i
        sr_i = 0.0
        for task in gt_plan:
            if execute_task_with_policy(task, predictions['policy']) and time < 120s:
                if check_spatial_criteria(task):
                    sr_i += 1.0
        sr_i /= n_i
        total_score += (em_i + sr_i)
    return total_score / N
```

## Common pitfalls

- Evaluating the controller using the model's predicted plan instead of the ground-truth plan, which conflates planning failures with execution failures.
- Ignoring the strict 120-second time limit per task, which automatically marks a task as failed (SR=0) regardless of spatial proximity.
- Using fuzzy string matching for plan steps; the benchmark requires exact character-by-character matches.

## Evidence (verbatim from paper)

> For the online joint validation of the task planning and mobile manipulation subproblems in the simulator, we use the EM and Success Rate (SR) metrics. The SR is equal to 1 for a decomposed task if the task is completed successfully within the 120-second time limit for model inference. Success is defined as follows. For a navigation task: the robot’s base geometric center is within 10 cm of the target position. For a manipulation task: the geometric center of the manipulated object is within 5 cm of its target position.

## Citation

```bibtex
@misc{kachaev2025mindandmotion,
  title={Mind and Motion Aligned: A Joint Evaluation IsaacSim Benchmark for Task Planning and Low-Level Policies in Mobile Manipulation},
  author={Kachaev et al. (2025)},
  year={2025},
  note={arXiv:2508.15663}
}
```

- arXiv: 2508.15663