Crew Wildfire Eval

---
name: crew-wildfire-eval
description: Probes LLM-based multi-agent coordination in dynamic, partially observable wildfire disaster response scenarios. It evaluates capabilities such as spatial reasoning, task designation, plan adaptation, and heterogeneous team collaboration under stochastic dynamics and long-horizon objectives. Use when the user wants to benchmark on CREW-Wildfire, or asks about evaluating this task. Reports task success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.05178
  bibtex_key: hyun2025crewwildfire
  confidence: high
---

# crew-wildfire-eval

> CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale — Hyun et al. (2025) (arXiv:2507.05178, 2025)

## What this evaluates

Probes LLM-based multi-agent coordination in dynamic, partially observable wildfire disaster response scenarios. It evaluates capabilities such as spatial reasoning, task designation, plan adaptation, and heterogeneous team collaboration under stochastic dynamics and long-horizon objectives.

## Datasets

- **CREW-Wildfire** — total ?; splits: test (-1)

## Metrics

- `task success` **(primary)** — range: [0, 1]
  - Binary or proportional measure of whether all level-specific objectives are completed (e.g., all trees cut, all civilians rescued, fire extinguished).
- `Composite Score (Trees Destroyed + 20×Agents Lost + 100×Civilians Lost)` — range: other
  - Level-specific scoring functions aggregate task completion, damage minimization, and safety penalties. For the Full Environment level, it is explicitly Trees Destroyed + 20×Agents Lost + 100×Civilians Lost.

## Input / output format

**Input**: Procedurally generated wildfire environment state including map dimensions, agent types (Firefighters, Bulldozers, Drones, Helicopters), civilian and fire locations, partial observability constraints, and level-specific objectives.

**Output**: Agent action sequences or coordination plans executed over a long-horizon simulation, evaluated against level-specific scoring functions and behavioral competency markers.

## Scoring recipe

```python
def compute_score(level, final_state):
    if level == 'Full Environment':
        return final_state.trees_destroyed + 20 * final_state.agents_lost + 100 * final_state.civilians_lost
    elif level == 'Cut Trees':
        return final_state.trees_cut_in_labeled_cells
    elif level == 'Scout Fire':
        return min(2, final_state.drones_over_fire)
    elif level in ['Transport', 'Rescue']:
        return final_state.units_at_target
    else:
        return final_state.trees_destroyed + 20 * final_state.agents_lost
```

## Common pitfalls

- Assuming a single fixed scoring formula applies across all 12 levels; each level uses a distinct scoring function (e.g., trees cut vs. drones over fire vs. composite penalties).
- Overlooking partial observability and stochastic fire spread, which require agents to adapt plans and share observations rather than relying on global state.
- Treating behavioral competencies as optional; they are explicitly marked per level to diagnose specific coordination failures.

## Evidence (verbatim from paper)

> We measure the performance via task success, damage minimization, and agent/civilian safety. We also assign high-level behavioral goals to each level to evaluate a variety of high-level behaviors. The behavioral goals include: 1) Task Designation (TD): the ability to divide tasks among agents; 2) Agent Capitalization (AC): the ability to recognize and capitalize on the strengths and weaknesses in heterogeneous teams; 3) Spatial Reasoning (SR): The ability to reason and plan accordingly with spatial information; 4) Observation Sharing (OS): the ability to communicate useful observations when necessary; 5) Realtime Coordination (RC): the ability to communicate and rely on other agents to perform synchronized tasks; 6) Plan Adaptation (PA): the ability to adapt and revise plans; 7) Objective Prioritization (OP): the ability to rank competing goals contextually. ... Full Environment Scoring Function: Trees Destroyed + 20×Agents Lost + 100×Civilians Lost

## Citation

```bibtex
@misc{hyun2025crewwildfire,
  title={CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale},
  author={Hyun et al. (2025)},
  year={2025},
  note={arXiv:2507.05178}
}
```

- arXiv: 2507.05178