Competitive Pokemon Singles Eval

---
name: competitive-pokemon-singles-eval
description: Evaluates an agent's ability to play competitive Pokémon Singles under partial observability and long-horizon uncertainty. It measures strategic decision-making, team building, and adaptation against heuristic opponents, search-based engines, LLM agents, and human players on a ranked ladder. Use when the user wants to benchmark on Competitive Pokémon Singles (CPS) on Pokémon Showdown, or asks about evaluating this task. Reports win rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.04395
  bibtex_key: grigsby2025competitivepokemon
  confidence: high
---

# competitive-pokemon-singles-eval

> Human-Level Competitive Pok\'emon via Scalable Offline Reinforcement Learning with Transformers — Grigsby et al. (2025) (arXiv:2504.04395, 2025)

## What this evaluates

Evaluates an agent's ability to play competitive Pokémon Singles under partial observability and long-horizon uncertainty. It measures strategic decision-making, team building, and adaptation against heuristic opponents, search-based engines, LLM agents, and human players on a ranked ladder.

## Datasets

- **Competitive Pokémon Singles (CPS) on Pokémon Showdown** — total ?; splits: Variety Set (1000), Replay Set (-1), Competitive Set (-1)

## Metrics

- `win rate` **(primary)** — range: percent
  - Calculated as the number of battles won divided by the total number of battles played against a specific opponent or set of opponents.
- `Glicko-1 / GXE` — range: rating / percent
  - Glicko-1 rating system and Glicko-1 Expected Win rate (GXE) computed by Pokémon Showdown based on match outcomes against human players on the ranked ladder.

## Input / output format

**Input**: First-person battle state representation including current team, opponent's revealed Pokémon, move history, and turn-by-turn actions. Teams are provided as prompts from the Variety, Replay, or Competitive sets.

**Output**: A single action per turn: choose a move, switch a Pokémon, or use an item/field effect, given the current game state.

## Scoring recipe

```python
def compute_win_rate(battles):
    wins = sum(1 for b in battles if b.result == 'win')
    return wins / len(battles)

def compute_glicko_gxe(battles):
    ratings = [b.glicko1 for b in battles]
    gxes = [b.gxe for b in battles]
    return sum(ratings) / len(ratings), sum(gxes) / len(gxes)
```

## Common pitfalls

- Win rates are highly sensitive to the team set used for prompting (Variety vs. Replay vs. Competitive), making cross-experiment comparisons difficult without standardizing the set.
- Ladder-based metrics like Glicko-1 and GXE are noisy and can be artificially deflated during tournaments when top players create new accounts, requiring careful percentile estimation.
- Partial observability requires inferring opponent teams from battle logs; imperfect replay reconstruction can lead to inconsistent evaluation conditions across generations.

## Evidence (verbatim from paper)

> Win rates are measured over large samples of hundreds or thousands of battles unless otherwise noted. The average win rate against 6 of these heuristics on the Variety Set forms a “Heuristic Composite Score”. Models’ Glicko-1 and GXE stats at the end of their final battle are shown in Figure [12].

## Citation

```bibtex
@misc{grigsby2025competitivepokemon,
  title={Human-Level Competitive Pok\'emon via Scalable Offline Reinforcement Learning with Transformers},
  author={Grigsby et al. (2025)},
  year={2025},
  note={arXiv:2504.04395}
}
```

- arXiv: 2504.04395