Mimo Embodied Eval

---
name: mimo-embodied-eval
description: Evaluates a cross-embodied foundation model's capabilities in affordance prediction, task planning, spatial understanding, and autonomous driving perception/prediction/planning across diverse visual and language inputs. Use when the user wants to benchmark on RoboRefIt, Where2Place, VABench-Point, Part-Afford, RoboAfford-Eval, EgoPlan2, RoboVQA, Cosmos-Reason1, CV-Bench, ERQA, EmbSpatial, SAT, RoboSpatial, RefSpatial-Bench, CRPE-relation, MetaVQA, VSI-Bench, CODA-LM, DRAMA, MME-RealWorld, IDKB, OmniDrive, NuInstruct, DriveLM, MAPLM, nuScenes-QA, LingoQA, BDD-X, DriveAction, or asks about evaluating this task. Reports precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.16518
  bibtex_key: hao2025mimoembodied
  confidence: high
---

# mimo-embodied-eval

> MiMo-Embodied: X-Embodied Foundation Model Technical Report — Hao et al. (2025) (arXiv:2511.16518, 2025)

## What this evaluates

Evaluates a cross-embodied foundation model's capabilities in affordance prediction, task planning, spatial understanding, and autonomous driving perception/prediction/planning across diverse visual and language inputs.

## Datasets

- **RoboRefIt** — total ?; splits: test (-1)
- **Where2Place** — total ?; splits: test (-1)
- **VABench-Point** — total ?; splits: test (-1)
- **Part-Afford** — total ?; splits: test (-1)
- **RoboAfford-Eval** — total ?; splits: test (-1)
- **EgoPlan2** — total ?; splits: test (-1)
- **RoboVQA** — total ?; splits: test (-1)
- **Cosmos-Reason1** — total ?; splits: test (-1)
- **CV-Bench** — total ?; splits: test (-1)
- **ERQA** — total ?; splits: test (-1)
- **EmbSpatial** — total ?; splits: test (-1)
- **SAT** — total ?; splits: test (-1)
- **RoboSpatial** — total ?; splits: test (-1)
- **RefSpatial-Bench** — total ?; splits: test (-1)
- **CRPE-relation** — total ?; splits: test (-1)
- **MetaVQA** — total ?; splits: test (-1)
- **VSI-Bench** — total ?; splits: test (-1)
- **CODA-LM** — total ?; splits: test (-1)
- **DRAMA** — total ?; splits: test (-1)
- **MME-RealWorld** — total ?; splits: test (-1)
- **IDKB** — total ?; splits: test (-1)
- **OmniDrive** — total ?; splits: test (-1)
- **NuInstruct** — total ?; splits: test (-1)
- **DriveLM** — total ?; splits: test (-1)
- **MAPLM** — total ?; splits: test (-1)
- **nuScenes-QA** — total ?; splits: test (-1)
- **LingoQA** — total ?; splits: test (-1)
- **BDD-X** — total ?; splits: test (-1)
- **DriveAction** — total ?; splits: test (-1)

## Metrics

- `precision` **(primary)** — range: percent
  - Percentage of correct predictions or spatially valid outputs (e.g., predicted points falling within human-annotated regions) relative to the total number of test instances.

## Input / output format

**Input**: Multimodal inputs consisting of single-view images, multi-view video frames, or panoramic scenes paired with natural language instructions, queries, or action goals.

**Output**: Textual answers, predicted coordinate points, or generated action sequences/plans corresponding to the input instructions or queries.

## Scoring recipe

```python
def compute_precision(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if isinstance(gold_item, dict) and 'region' in gold_item:
            if point_in_region(pred, gold_item['region']):
                correct += 1
        else:
            if pred == gold_item or is_semantically_equivalent(pred, gold_item):
                correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Many benchmarks require custom evaluation scripts (marked with * in tables) rather than official leaderboard scores, so reproducing results requires accessing the authors' evaluation framework.
- Spatial grounding tasks (e.g., VABench-Point) require specific coordinate tolerance thresholds that are not explicitly detailed in the text, making exact replication difficult.
- Some benchmarks evaluate multi-step planning or causal reasoning where outputs are open-ended, requiring LLM-based or heuristic scoring not fully specified.

## Evidence (verbatim from paper)

> The pointing subset of VABench *[[63]]* (VABench-Point) evaluates a model’s precision in grounding natural language commands to specific coordinate locations for robotic manipulation, requiring predicted points to fall within human-annotated regions of target objects or free space. Results marked with * are obtained using our evaluation framework.

## Citation

```bibtex
@misc{hao2025mimoembodied,
  title={MiMo-Embodied: X-Embodied Foundation Model Technical Report},
  author={Hao et al. (2025)},
  year={2025},
  note={arXiv:2511.16518}
}
```

- arXiv: 2511.16518