Kwbench Eval

---
name: kwbench-eval
description: This benchmark evaluates language models' ability to recognize formal game-theoretic structures (e.g., principal-agent conflict, signaling, strategic omission) in real-world knowledge work scenarios without explicit task hints. It measures the gap between a model's theoretical understanding of these concepts and its capacity for unprompted, practical problem framing. Use when the user wants to benchmark on KWBench, or asks about evaluating this task. Reports Pass Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.15760
  bibtex_key: [REDACTED]
  confidence: medium
---

# kwbench-eval

> KWBench: Measuring Unprompted Problem Recognition in Knowledge Work — Maloo (2026) (arXiv:2604.15760, 2026)

## What this evaluates

This benchmark evaluates language models' ability to recognize formal game-theoretic structures (e.g., principal-agent conflict, signaling, strategic omission) in real-world knowledge work scenarios without explicit task hints. It measures the gap between a model's theoretical understanding of these concepts and its capacity for unprompted, practical problem framing.

## Datasets

- **KWBench** — total 223; splits: test (223); repo https://github.com/ankitmaloo/fasteval

## Metrics

- `Pass Rate` **(primary)** — range: percent
  - Calculated as the number of tasks where all mandatory criteria were met divided by the total number of evaluated tasks. Tasks failing the mandatory gate receive a score of zero.
- `Mean Score` — range: percent
  - The average score across all evaluated tasks, including tasks that failed the mandatory gate (which contribute zero to the average).
- `Conditional` — range: percent
  - The mean score computed only on tasks that successfully passed the mandatory gate.

## Input / output format

**Input**: Real-world knowledge work scenario descriptions presented without task hints or explicit prompts asking for structural analysis.

**Output**: Identification/classification of the underlying formal game-theoretic structure governing the scenario.

## Scoring recipe

```python
passed = 0
total_score = 0.0
for task in tasks:
    pred = model.predict(task)
    if meets_mandatory_gate(pred):
        passed += 1
        total_score += get_task_score(pred)
pass_rate = passed / len(tasks)
mean_score = total_score / len(tasks)
conditional_score = total_score / passed if passed > 0 else 0.0
```

## Common pitfalls

- Models frequently correctly articulate game-theoretic structures when explicitly prompted, but fail to recognize them unprompted, creating a false impression of capability.
- API failures or malformed outputs can cause some models to be evaluated on fewer than the full task set, making raw pass counts misleading without normalizing by evaluated tasks.
- Conditional scores converge across models (~72-84%), so ranking differences stem almost entirely from pass rates rather than execution quality on passed tasks.

## Evidence (verbatim from paper)

> Passed: tasks where all mandatory criteria were met. Pass Rate: passed / evaluated. Mean Score: average across all evaluated tasks (including zeros). Conditional: mean score on tasks that passed the gate.

## Citation

```bibtex
@misc{maloo2026kwbench,
  title={KWBench: Measuring Unprompted Problem Recognition in Knowledge Work},
  author={Maloo (2026)},
  year={2026},
  note={arXiv:2604.15760}
}
```

- arXiv: 2604.15760