Marketgen Eval

---
name: marketgen-eval
description: Evaluates embodied agents and multimodal LLMs on long-horizon manipulation tasks in procedurally generated supermarket environments. Specifically, it probes spatial reasoning, occlusion handling, and collision avoidance during checkout unloading and in-aisle item collection. Use when the user wants to benchmark on MarketGen Benchmark, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.21161
  bibtex_key: hu2025marketgen
  confidence: high
---

# marketgen-eval

> MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments — Hu et al. (2025) (arXiv:2511.21161, 2025)

## What this evaluates

Evaluates embodied agents and multimodal LLMs on long-horizon manipulation tasks in procedurally generated supermarket environments. Specifically, it probes spatial reasoning, occlusion handling, and collision avoidance during checkout unloading and in-aisle item collection.

## Datasets

- **MarketGen Benchmark** — total ?; splits: test (-1)

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - Percentage of trials where the agent successfully completes the full task objective (e.g., unloading all basket items or collecting the target shelf item) without violating constraints.
- `Success Path Length (SPL)` — range: percent
  - Composite efficiency metric combining task success with path optimality, reported as a percentage.
- `Path Length (PL)` — range: other
  - Total distance traveled by the agent during the trial, measured in meters.

## Input / output format

**Input**: Visual observations (rendered images/point clouds) of the supermarket scene and natural language task instructions.

**Output**: Sequential robot control commands (navigation waypoints, grasp poses, placement actions) executed in the simulation or real robot.

## Scoring recipe

```python
def compute_sr(predictions, gold_tasks):
    successful = 0
    for pred, task in zip(predictions, gold_tasks):
        if pred.task_completed and pred.no_collision and pred.all_items_collected:
            successful += 1
    return (successful / len(predictions)) * 100
```

## Common pitfalls

- Success Rate (SR) values are consistently low (<15%) for MLLMs, indicating that decoupled planning and affordance models struggle with high-clutter, long-horizon tasks.
- SPL and Path Length (PL) metrics are only reported for the Checkout Unloading track, preventing direct efficiency comparisons with the In-Aisle track.
- Sim-to-real validation uses a fixed set of 4 object shapes with only 20 trials each, which does not represent the full 1,000+ product asset library.

## Evidence (verbatim from paper)

> We evaluate the performance of several representative Multi-modal Large Language Models (MLLMs) on our two benchmark tracks: Checkout Unloading and In-Aisle Item Collection. As shown in Tab.[2], the results highlight the significant challenge posed by long-horizon, high-clutter tasks in realistic supermarket scenarios. While Gemini-2.5-Pro achieves the highest overall Success Rate (SR), its performance remains below 15%, indicating that the modular methods struggle with the complexity of these tasks.

## Citation

```bibtex
@misc{hu2025marketgen,
  title={MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments},
  author={Hu et al. (2025)},
  year={2025},
  note={arXiv:2511.21161}
}
```

- arXiv: 2511.21161