Streetfighter Eval

---
name: streetfighter-eval
description: Tests an LLM agent's real-time decision-making and combat strategy in a video game environment, prioritizing low latency while maintaining competitive win rates. The benchmark probes the model's ability to make timely character actions under a hard frame-rate limit. Use when the user wants to benchmark on StreetFighter, or asks about evaluating this task. Reports ELO Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19481
  bibtex_key: kang2025winfast
  confidence: high
---

# streetfighter-eval

> Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs — Kang et al. (2025) (arXiv:2505.19481, 2025)

## What this evaluates

Tests an LLM agent's real-time decision-making and combat strategy in a video game environment, prioritizing low latency while maintaining competitive win rates. The benchmark probes the model's ability to make timely character actions under a hard frame-rate limit.

## Datasets

- **StreetFighter** — total ?; splits: test (-1); repo https://github.com/HaoKang-Timmy/LatencySensitiveBench

## Metrics

- `ELO Score` **(primary)** — range: other
  - Standard ELO rating derived from win rates across 40 matches between model pairs. Higher scores indicate better competitive performance.
- `Latency (ms)` — range: ms
  - Inference time per action in milliseconds.

## Input / output format

**Input**: Game state observations and opponent actions in a fighting game environment.

**Output**: Character action/move selection.

## Scoring recipe

```python
matches = 40
results = []
for i in range(matches):
    state = get_game_state()
    action = model(state)
    outcome = play_action(action)
    results.append(outcome)
elo = compute_elo(results)
latency = measure_inference_time_ms()
# Note: latency < 200ms yields no benefit due to 5 actions/sec game limit
```

## Common pitfalls

- Optimizing for latency below the game's processing limit (~200ms per action), which provides no competitive advantage.
- Prioritizing speed over decision quality; the fastest models (e.g., 1.5B FP8) perform poorly due to limited strategic capability.

## Evidence (verbatim from paper)

> For the gaming benchmark, we conduct 40 matches between model pairs and compute win rates to derive ElO ratings. ... In StreetFighter, each character action takes a fixed amount of in-game time to complete, with an effective frame rate of around 5 actions per second (i.e., 200ms per action). Any optimization that reduces model latency beyond this threshold yields no further benefit...

## Citation

```bibtex
@misc{kang2025winfast,
  title={Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of LLMs},
  author={Kang et al. (2025)},
  year={2025},
  note={arXiv:2505.19481}
}
```

- arXiv: 2505.19481