Mini Behavior Eval

---
name: mini-behavior-eval
description: Probes long-horizon decision-making and multi-state object interaction in a procedurally generated 3D gridworld. Evaluates an agent's ability to plan and execute complex household tasks under sparse and dense reward signals. Use when the user wants to benchmark on Mini-BEHAVIOR, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.01824
  bibtex_key: jin2023minibehavior
  confidence: high
---

# mini-behavior-eval

> Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI — Jin et al. (2023) (arXiv:2310.01824, 2023)

## What this evaluates

Probes long-horizon decision-making and multi-state object interaction in a procedurally generated 3D gridworld. Evaluates an agent's ability to plan and execute complex household tasks under sparse and dense reward signals.

## Datasets

- **Mini-BEHAVIOR** — total ?; splits: test (-1); repo https://github.com/StanfordVL/mini_behavior

## Metrics

- `success_rate` **(primary)** — range: [0, 1]
  - Binary indicator: 1 if the agent completes the specified task within the 1000-step time limit, 0 otherwise.
- `normalized_reward` — range: [0, 1]
  - Cumulative reward scaled such that 1.0 corresponds to successfully completing the task, used to track progress under dense reward shaping.

## Input / output format

**Input**: 3D gridworld state/observation and natural language task instruction.

**Output**: Discrete action (e.g., move, pick, place, open, close) executed sequentially within a 1000-step episode limit.

## Scoring recipe

```python
def compute_success_rate(predictions, gold):
    successes = 0
    for pred in predictions:
        if pred.completed_task:
            successes += 1
    return successes / len(predictions)
```

## Common pitfalls

- Procedural generation yields infinite task variations, so fixed test splits must be explicitly sampled or regenerated for reproducibility.
- Sparse reward (+1 only on completion) causes learning failure for medium/hard tasks within 1M steps, necessitating dense reward shaping for meaningful evaluation.
- Fixed 1000-step time limit may truncate valid but longer successful trajectories.

## Evidence (verbatim from paper)

> We first examine these tasks under the sparse reward setting, where the agent will only receive a reward of +1 if it successfully completes the task. ... Notice that we normalize the reward in the plots such that a reward of 1 would correspond to successfully completing the task.

## Citation

```bibtex
@misc{jin2023minibehavior,
  title={Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI},
  author={Jin et al. (2023)},
  year={2023},
  note={arXiv:2310.01824}
}
```

- arXiv: 2310.01824