Flatland 2020 Eval

---
name: flatland-2020-eval
description: Evaluates multi-agent coordination and path planning for train rescheduling in a dynamic grid-world railway simulation. It probes the ability of agents to adapt to partial observability, handle congestion, and coordinate under switching constraints and dynamic disruptions. Use when the user wants to benchmark on Flatland Competition 2020, or asks about evaluating this task. Reports overall score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.16511
  bibtex_key: laurent2021flatland
  confidence: high
---

# flatland-2020-eval

> Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World — Laurent et al. (2021) (arXiv:2103.16511, 2021)

## What this evaluates

Evaluates multi-agent coordination and path planning for train rescheduling in a dynamic grid-world railway simulation. It probes the ability of agents to adapt to partial observability, handle congestion, and coordinate under switching constraints and dynamic disruptions.

## Datasets

- **Flatland Competition 2020** — total ?; splits: test (-1)

## Metrics

- `overall score` **(primary)** — range: other
  - A composite metric that balances two competing goals: maximizing the percentage of trains that reach their targets (completion rate) and minimizing the total time required to bring them there. Exact weighting is not specified in the text.

## Input / output format

**Input**: Grid-based railway simulation environment with dynamic disruptions; agents receive observations such as local tree structures, global conflict graphs, or hand-crafted features depending on the solution.

**Output**: Action commands to coordinate train movements and reschedule paths in real-time to avoid collisions and reach targets.

## Scoring recipe

```python
def compute_score(completed_trains, total_trains, total_time):
    completion_rate = completed_trains / total_trains
    # Score balances completion rate and time minimization
    # Higher completion and lower time yield a higher overall score
    return composite_metric(completion_rate, total_time)
```

## Common pitfalls

- The final ranking depends on a composite score, not just completion rate; a team with fewer completed environments can rank higher if their average score per environment is better.
- Both centralized and decentralized execution paradigms achieved comparable performance, so algorithm choice (centralized vs decentralized) is not a strict performance predictor.
- The environment features dynamic disruptions and switching constraints, requiring real-time adaptation rather than static path planning.

## Evidence (verbatim from paper)

> Table 1 shows their overall scores, completion percentages and number of encountered environments. The OR solution (An_Old_Driver) tackled the most environments with a total of 363, followed by the winning RL solution (JBR_HSE) with 336. Interestingly, the second RL solution (Netcetera) tackled 1 environment less (229) than the fourth RL solution (MARMot-Lab-NUS, 230), but scored better on average resulting in a better ranking. This perfectly illustrates the two competing goals of the challenge: maximizing the number of agents that reach their targets vs minimising the time needed to bring the agents there.

## Citation

```bibtex
@misc{laurent2021flatland,
  title={Flatland Competition 2020: MAPF and MARL for Efficient Train Coordination on a Grid World},
  author={Laurent et al. (2021)},
  year={2021},
  note={arXiv:2103.16511}
}
```

- arXiv: 2103.16511