Emmo Eval

---
name: emmo-eval
description: This benchmark evaluates embodied mobile manipulation agents in open environments, testing their ability to execute long-horizon, language-conditioned tasks that require interleaved high-level planning and low-level continuous navigation/manipulation. It specifically probes reasoning fidelity, execution success, adaptability to failures, and path efficiency compared to expert trajectories. Use when the user wants to benchmark on EMMOE-100, or asks about evaluating this task. Reports PLWSR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.08604
  bibtex_key: li2025emmoe
  confidence: high
---

# emmo-eval

> EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments — Dongping Li et al. (2025) (arXiv:2503.08604, 2025)

## What this evaluates

This benchmark evaluates embodied mobile manipulation agents in open environments, testing their ability to execute long-horizon, language-conditioned tasks that require interleaved high-level planning and low-level continuous navigation/manipulation. It specifically probes reasoning fidelity, execution success, adaptability to failures, and path efficiency compared to expert trajectories.

## Datasets

- **EMMOE-100** — total 100; splits: train (90), test (10)

## Metrics

- `PLWSR` **(primary)** — range: [0, 1]
  - Path Length Weighted Success Rate. Calculated as SR × (length of successful trajectory) / max(length of expert trajectory, length of successful trajectory). It measures the ability gap between the agent and the expert specifically on successful trajectories.
- `SR` — range: [0, 1] or percent
  - Success Rate. The proportion of tasks completed successfully out of total execution attempts.
- `TP` — range: [0, 1] or percent
  - Task Progress metric (defined in Section 2.3 of the full paper). Tracks overall task advancement or step efficiency.
- `SRR` — range: [0, 1] or percent
  - Success Re-planning Rate. Reflects the model's ability to adapt to environments and adjust from failure using feedback information.
- `SER` — range: [0, 1] or percent
  - Success End Rate. Reflects the model's ability to correctly determine when a task is completed and should be terminated.

## Input / output format

**Input**: Language-conditioned task instructions, object names, background/environmental state information, and execution history/feedback from previous steps.

**Output**: Sequential action commands (e.g., 'Go to', 'Pick', 'Place', 'Open', 'Close', 'End'), low-level model selection choices, and re-planning outputs upon failure.

## Scoring recipe

```python
def compute_emmoe_metrics(predictions, gold, max_steps=20, runs=3):
    success_flags = []
    traj_lengths = []
    for run_preds in predictions:
        completed = check_task_completion(run_preds, gold)
        success_flags.append(completed)
        traj_lengths.append(len(run_preds))
    sr = sum(success_flags) / len(success_flags)
    successful_trajs = [l for l, s in zip(traj_lengths, success_flags) if s]
    expert_len = gold['expert_trajectory_length']
    avg_succ_len = sum(successful_trajs) / len(successful_trajs) if successful_trajs else 0
    plwsr = sr * (avg_succ_len / max(expert_len, avg_succ_len))
    # TP, SRR, SER computed per Sec 2.3 definitions
    return {'SR': sr, 'PLWSR': plwsr, 'TP': ..., 'SRR': ..., 'SER': ...}
```

## Common pitfalls

- DPO alignment significantly improves training split performance but degrades generalization on the test split, particularly causing a sharp drop in SRR.
- High subtask success rates do not guarantee high overall task success; models often fail long-horizon tasks due to context forgetting or hallucination despite completing individual steps.
- Evaluation averages results over 3 runs per task with a strict 20-step limit, which heavily penalizes inefficient planners and can mask sporadic successes.

## Evidence (verbatim from paper)

> In addition to SR, TP, SER and SRR introduced in Section[2.3], we also choose Path Length Weighted SR (PLWSR)*[[27]]* as one of our evaluation metrics. PLWSR is defined as SR×(length of successful trajectory) / $max$(length of expert trajectory, length of successful trajectory) and measures the ability gap between the agent and the expert in successful trajectories. All tasks in EMMOE-100 will be used for evaluation, and the remaining ten untrained tasks will serve as our test set. Each task is executed three times with a maximum step limit of 20 each time, the average execution results will be used for the final calculation.

## Citation

```bibtex
@misc{li2025emmoe,
  title={EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environments},
  author={Dongping Li et al. (2025)},
  year={2025},
  note={arXiv:2503.08604}
}
```

- arXiv: 2503.08604