Orak Eval

---
name: orak-eval
description: Evaluates LLM agents' long-horizon decision-making and gameplay capabilities across 12 diverse video games spanning six genres. It probes the effectiveness of different agentic strategies (zero-shot, reflection, planning, skill-management) and the impact of multimodal inputs (text vs. visual) on action inference. The benchmark also assesses generalization to unseen in-game scenarios, out-of-distribution games, and non-game tasks like math and web navigation. Use when the user wants to benchmark on Orak, or asks about evaluating this task. Reports normalization score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.03610
  bibtex_key: park2025orak
  confidence: high
---

# orak-eval

> Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games — Park et al. (2025) (arXiv:2506.03610, 2025)

## What this evaluates

Evaluates LLM agents' long-horizon decision-making and gameplay capabilities across 12 diverse video games spanning six genres. It probes the effectiveness of different agentic strategies (zero-shot, reflection, planning, skill-management) and the impact of multimodal inputs (text vs. visual) on action inference. The benchmark also assesses generalization to unseen in-game scenarios, out-of-distribution games, and non-game tasks like math and web navigation.

## Datasets

- **Orak** — total 12; splits: test (12); repo https://github.com/krafton-ai/Orak

## Metrics

- `normalization score` **(primary)** — range: [0, 100] percent
  - The raw game score divided by the maximum possible game score, expressed as a percentage. Average is computed over 3 to 20 independent trials per game.
- `Elo rating` — range: other
  - Pairwise competitive win rates converted to Elo ratings for two-player games (Street Fighter III, StarCraft II). Agents play multiple rounds against each other under controlled conditions.

## Input / output format

**Input**: Textual game state description, optionally supplemented with visual screenshots. Inputs are provided in three modalities: Text-only, Image-only, or Both.

**Output**: Discrete game action (e.g., button press, movement command, or menu selection) inferred by the LLM agent at each step.

## Scoring recipe

```python
def compute_normalization_score(raw_score, max_score):
    return (raw_score / max_score) * 100

def evaluate_agent(trial_scores, max_score):
    scores = [compute_normalization_score(s, max_score) for s in trial_scores]
    return sum(scores) / len(scores)

# For competitive games:
# Compute pairwise win rates across 3 rounds, then convert to Elo rating.
```

## Common pitfalls

- Using absolute game scores instead of the required normalization score makes cross-game comparison invalid.
- Applying a zero-shot strategy to complex games (e.g., Pokémon, Minecraft) will yield near-zero performance; genre-specific agentic strategies (reflection, planning, skill-management) are required.
- Assuming visual input always improves performance; for narrative-heavy or abstract games where text contains all necessary state, adding images can degrade accuracy.

## Evidence (verbatim from paper)

> For each game, we report the normalization score rather than the absolute score, i.e., the game score is normalized by the maximum game score. We report the average score of 3 to 20 trials for each game.

## Citation

```bibtex
@misc{park2025orak,
  title={Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2506.03610}
}
```

- arXiv: 2506.03610