Embodied Reasoner Eval

---
name: embodied-reasoner-eval
description: Evaluates an agent's ability to perform long-horizon embodied interactive tasks by synergizing visual search, reasoning, and action. It probes spatial reasoning, self-reflection, and planning capabilities in both simulated and real-world environments. Use when the user wants to benchmark on Unspecified (Simulated & Real-world tasks), or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.21696
  bibtex_key: zhang2025embodiedreasoner
  confidence: high
---

# embodied-reasoner-eval

> Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks — Wenqi Zhang et al. (2025) (arXiv:2503.21696, 2025)

## What this evaluates

Evaluates an agent's ability to perform long-horizon embodied interactive tasks by synergizing visual search, reasoning, and action. It probes spatial reasoning, self-reflection, and planning capabilities in both simulated and real-world environments.

## Datasets

- **Unspecified (Simulated & Real-world tasks)** — total ?; splits: simulated (-1), real-world (30)

## Metrics

- `success_rate` **(primary)** — range: percent
  - The percentage of tasks successfully completed out of the total number of evaluated tasks.
- `search_efficiency` — range: percent
  - A relative measure of how quickly or effectively the model locates target objects compared to baselines, reported as percentage improvement.
- `task_completeness` — range: percent
  - An assessment of whether all required sub-goals and final objectives within a multi-step task are fully fulfilled.
- `repetitive_exploration_rate` — range: percent
  - Measures how often the model navigates to the same area within its trajectory, quantifying redundant search behavior.

## Input / output format

**Input**: Real-time visual input (images or camera feed) paired with natural language task instructions.

**Output**: Action command to be executed by the environment or human operator.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, trajectories):
    success_rate = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(predictions) * 100
    rer_scores = []
    for traj in trajectories:
        repeated = count_repeated_area_visits(traj)
        rer_scores.append(repeated / len(traj) * 100)
    return {'success_rate': success_rate, 'repetitive_exploration_rate': sum(rer_scores)/len(rer_scores)}
```

## Common pitfalls

- Models may over-explore on simpler tasks, leading to missed detections of nearby objects.
- Baseline models frequently exhibit repetitive searching behaviors due to weak temporal reasoning and context awareness.
- Performance drops significantly on long-horizon tasks exceeding five key actions for baseline models.

## Evidence (verbatim from paper)

> To quantify this, we define a repetitive exploration rate (RER), which measures how often the model navigates to the same area within its trajectory. As shown in [Fig. 6], our models (*Embodied-Reasoner / Explorer*) consistently exhibit a significantly lower RER (-50%) compared to baseline models across all four tasks.

## Citation

```bibtex
@misc{zhang2025embodiedreasoner,
  title={Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks},
  author={Wenqi Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.21696}
}
```

- arXiv: 2503.21696