Omniear Eval

---
name: omniear-eval
description: Evaluates embodied agent reasoning by testing how well models infer capability gaps, dynamic tool acquisition, and coordination needs from environmental constraints. Probes the ability to ground abstract reasoning in physical reality under partial observability. Use when the user wants to benchmark on EAR-Bench, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.05614
  bibtex_key: wang2025omniear
  confidence: high
---

# omniear-eval

> OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks — Wang et al. (2025) (arXiv:2508.05614, 2025)

## What this evaluates

Evaluates embodied agent reasoning by testing how well models infer capability gaps, dynamic tool acquisition, and coordination needs from environmental constraints. Probes the ability to ground abstract reasoning in physical reality under partial observability.

## Datasets

- **EAR-Bench** — total 2800; splits: test (2800); repo https://github.com/ZJU-REAL/OmniEmbodied

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - Percentage of tasks completed successfully out of the total number of test scenarios.
- `Step Count` — range: other
  - Average number of actions executed for tasks that were completed successfully.
- `Relative Step Ratio (RSR)` — range: other
  - Ratio of expert trajectory length to model trajectory length (L_expert / L_model), measuring execution efficiency.

## Input / output format

**Input**: Standardized prompts, environment descriptions, and task contexts under partial observability. Agents must explore to discover object locations and properties, with tool-dependent actions dynamically enabled based on context.

**Output**: A sequence of actions or tool-use commands selected from a standardized action vocabulary.

## Scoring recipe

```python
def compute_metrics(predictions, gold_trajectories):
    success_count = 0
    total_steps_successful = 0
    successful_count = 0
    for pred, gold in zip(predictions, gold_trajectories):
        if is_task_successful(pred, gold):
            success_count += 1
            total_steps_successful += len(pred)
            successful_count += 1
    sr = (success_count / len(predictions)) * 100
    avg_steps = total_steps_successful / successful_count if successful_count > 0 else 0
    return sr, avg_steps
```

## Common pitfalls

- Assuming full environmental observability: The protocol explicitly implements partial observability, requiring agents to explore to discover object locations and properties.
- Equating model scale with reasoning capability: Performance on physical constraint tasks plateaus early regardless of parameter count, showing that scale improves execution/planning but not physical grounding.
- Ignoring the distinction between explicit and implicit collaboration: The benchmark isolates pure multi-agent reasoning by using centralized coordination, meaning models must infer coordination needs rather than rely on communication protocols.

## Evidence (verbatim from paper)

> Success Rate (SR) measures task completion percentage, Step Count indicates average actions for successful completion. Each model completes 2,800 test scenarios across seven task categories with three independent runs for statistical reliability.

## Citation

```bibtex
@misc{wang2025omniear,
  title={OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2508.05614}
}
```

- arXiv: 2508.05614