Tempobench Eval

---
name: tempobench-eval
description: Evaluates large language models' temporal reasoning capabilities by decomposing performance into trace-based (TTE) and causal (TCE) components. It measures how well models handle structured logical specifications with varying complexity, isolating structural factors like horizon depth and information density. Use when the user wants to benchmark on TempoBench, or asks about evaluating this task. Reports exact-match accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.27544
  bibtex_key: holzer2025tempobench
  confidence: high
---

# tempobench-eval

> Mechanics of Learned Reasoning 1: TempoBench, A Benchmark for Interpretable Deconstruction of Reasoning System Performance — Holzer et al. (2025) (arXiv:2510.27544, 2025)

## What this evaluates

Evaluates large language models' temporal reasoning capabilities by decomposing performance into trace-based (TTE) and causal (TCE) components. It measures how well models handle structured logical specifications with varying complexity, isolating structural factors like horizon depth and information density.

## Datasets

- **TempoBench** — total 800; splits: TTE (400), TCE (400); repo https://github.com/nik-hz/tempobench

## Metrics

- `exact-match accuracy` **(primary)** — range: [0, 1]
  - Calculates the proportion of instances where the model's generated JSON output exactly matches the ground truth specification. Evaluated separately for TTE and TCE tasks.

## Input / output format

**Input**: JSON objects describing the atomic propositions of a system, formatted as a dictionary mapping system identifiers to lists of constraints for different states.

**Output**: JSON objects containing exact sentences within the specified structure, matching the ground truth format.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, golds):
    if pred == gold:  # Exact match of JSON structure and content
        correct += 1
return correct / total
```

## Common pitfalls

- Models may struggle with exact JSON formatting, though the authors report no difficulties in practice.
- Performance drops significantly on high-complexity variants despite larger model sizes, indicating negative scaling with problem complexity.
- One-shot prompting with CoT examples is used, which may not reflect zero-shot or few-shot capabilities.

## Evidence (verbatim from paper)

> Testing the models on $800$ samples, $400$ for each of the TTE and TCE tasks, we evaluate each model using one-shot prompting. We include an example of a CoT solution strategy in the prompt and evaluate the models using the evaluation metrics detailed in Section [3]. Our inputs take the form of JSON objects shown in Listing [7]. During evaluation, LLMs must produce exact sentences within the JSON object.

## Citation

```bibtex
@misc{holzer2025tempobench,
  title={Mechanics of Learned Reasoning 1: TempoBench, A Benchmark for Interpretable Deconstruction of Reasoning System Performance},
  author={Holzer et al. (2025)},
  year={2025},
  note={arXiv:2510.27544}
}
```

- arXiv: 2510.27544