Swarmbench Eval

---
name: swarmbench-eval
description: This benchmark evaluates emergent decentralized coordination in LLM-driven multi-agent systems under strict local perception and communication constraints. It simulates five canonical swarm tasks—Pursuit, Synchronization, Foraging, Flocking, and Transport—in a 2D grid environment to probe whether LLMs can form adaptive group strategies and execute robust long-range planning without global information. Use when the user wants to benchmark on SwarmBench, or asks about evaluating this task. Reports Performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.04364
  bibtex_key: ruan2025swarmbench
  confidence: high
---

# swarmbench-eval

> Benchmarking LLMs' Swarm intelligence — Ruan et al. (2025) (arXiv:2505.04364, 2025)

## What this evaluates

This benchmark evaluates emergent decentralized coordination in LLM-driven multi-agent systems under strict local perception and communication constraints. It simulates five canonical swarm tasks—Pursuit, Synchronization, Foraging, Flocking, and Transport—in a 2D grid environment to probe whether LLMs can form adaptive group strategies and execute robust long-range planning without global information.

## Datasets

- **SwarmBench** — total ?; splits: test (-1); repo https://github.com/x66ccff/swarmbench

## Metrics

- `Performance` **(primary)** — range: other
  - Average task performance score across five independent simulation runs per model and task. Task-specific success conditions vary by environment (e.g., target capture rate, synchronization accuracy, food collection, flock cohesion, object transport).

## Input / output format

**Input**: A 5x5 local grid view (k×k observation window) and local communication messages from neighboring agents in the preceding round, provided within a 2D grid environment.

**Output**: Discrete action selection per agent per time step (e.g., movement direction, interaction/push, or communication broadcast) based on local observations and received messages.

## Scoring recipe

```python
def compute_performance(actions, task_type, n_runs=5):
    scores = []
    for _ in range(n_runs):
        env = SwarmBenchEnvironment(task_type, view_size=5)
        for step in range(max_steps):
            for agent in agents:
                obs = env.get_local_view(agent)
                msgs = env.get_received_messages(agent)
                action = model.predict(obs, msgs)
                env.step(action)
            scores.append(env.get_task_score(task_type))
    return mean(scores)
```

## Common pitfalls

- Assuming explicit message content directly drives task success; the paper shows physical dynamics and implicit coordination are stronger predictors.
- Interpreting high permutation importance of messages as strategic group-level success; messages influence local tactical decisions but do not guarantee optimal global outcomes.
- Assuming larger perception ranges (e.g., k=7) always improve performance; the paper notes diminishing returns or degradation due to increased prompt complexity and noise.

## Evidence (verbatim from paper)

> Performance, averaged over five runs per model/task, reveals significant variation across both LLMs and tasks, highlighting the inherent difficulty of achieving decentralized coordination under strict local constraints.

## Citation

```bibtex
@misc{ruan2025swarmbench,
  title={Benchmarking LLMs' Swarm intelligence},
  author={Ruan et al. (2025)},
  year={2025},
  note={arXiv:2505.04364}
}
```

- arXiv: 2505.04364