Reasonplan3d Eval

---
name: reasonplan3d-eval
description: Probes a model's ability to infer implicit human intentions from natural language and generate multi-step, route-aware activity plans grounded in 3D scene segmentation. It evaluates both textual planning coherence and spatial reasoning over 3D environments. Use when the user wants to benchmark on ReasonPlan3D, or asks about evaluating this task. Reports BLEU-4.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.12974
  bibtex_key: jiang2025exploring3d
  confidence: high
---

# reasonplan3d-eval

> Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planning — Xueying Jiang et al. (arXiv:2503.12974, 2025)

## What this evaluates

Probes a model's ability to infer implicit human intentions from natural language and generate multi-step, route-aware activity plans grounded in 3D scene segmentation. It evaluates both textual planning coherence and spatial reasoning over 3D environments.

## Datasets

- **ReasonPlan3D** — total ?; splits: val (-1)

## Metrics

- `BLEU-4` **(primary)** — range: [0, 1]
  - Computes the geometric mean of 1- to 4-gram precisions between predicted and reference text, penalizing brevity via a brevity penalty. Standard convention uses weights (0.25, 0.25, 0.25, 0.25) and sentence-level averaging.
- `CIDEr` — range: [0, 1]
  - Measures consensus between predicted and reference texts using TF-IDF weighted n-gram similarities. Higher scores indicate better alignment with human reference descriptions.
- `METEOR` — range: [0, 1]
  - Aligns predicted and reference texts using exact, stem, synonym, and paraphrase matches, then applies a penalty for fragmentation. Designed to correlate better with human judgment than BLEU.
- `ROUGE` — range: [0, 1]
  - Recall-oriented metric that measures overlap of n-grams, longest common subsequences, or word sequences between predictions and references. Typically reported as ROUGE-L or ROUGE-SU4 in NLG benchmarks.

## Input / output format

**Input**: Point clouds (P), multi-view images (I_mv), and implicit human instructions (X_inst). Internally processed into scene-level embeddings, 3D segmentation masks, and a scene graph (G) before being fed to the MLLM.

**Output**: A sequence of textual one-step plans terminated by a custom [END] stop token. Each plan includes step-by-step activity instructions and inter-step route details.

## Scoring recipe

```python
def evaluate(predictions, references):
    bleu4 = nltk.bleu([references], predictions, weights=(0.25, 0.25, 0.25, 0.25))
    cider = compute_cider(predictions, references)
    meteor = compute_meteor(predictions, references)
    rouge = compute_rouge(predictions, references)
    return {'BLEU-4': bleu4, 'CIDEr': cider, 'METEOR': meteor, 'ROUGE': rouge}
```

## Common pitfalls

- The model generates a variable number of steps ending with a custom [END] token; evaluating with fixed-length truncation or ignoring the stop token will skew metrics.
- Standard n-gram metrics (BLEU, ROUGE) measure lexical overlap but do not verify spatial consistency or 3D route feasibility, potentially rewarding syntactically correct but physically impossible plans.
- CIDEr relies on TF-IDF weighting from a reference corpus; if the ReasonPlan3D reference set is small or domain-specific, scores may not generalize.

## Evidence (verbatim from paper)

> Benchmarking on the ReasonPlan3D validation set for the 3D Reasoning-Driven Planning task with evaluation metrics BLEU, CIDEr, METEOR, and ROUGE. Best in bold, second underlined.

## Citation

```bibtex
@misc{jiang2025exploring3d,
  title={Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planning},
  author={Xueying Jiang et al.},
  year={2025},
  note={arXiv:2503.12974}
}
```

- arXiv: 2503.12974