Futurex Eval

---
name: futurex-eval
description: Evaluates LLM agents' ability to forecast real-world future events under uncertainty. It probes reasoning depth, tool-use/search capability, and temporal validity by requiring models to answer dynamic, live-updated questions before event resolution. Use when the user wants to benchmark on FutureX, or asks about evaluating this task. Reports overall_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.11987
  bibtex_key: zeng2025futurex
  confidence: high
---

# futurex-eval

> FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction — Zhiyuan Zeng et al. (2025) (arXiv:2508.11987, 2025)

## What this evaluates

Evaluates LLM agents' ability to forecast real-world future events under uncertainty. It probes reasoning depth, tool-use/search capability, and temporal validity by requiring models to answer dynamic, live-updated questions before event resolution.

## Datasets

- **FutureX** — total ?; splits: test (-1)

## Metrics

- `overall_score` **(primary)** — range: [0, 1]
  - Weighted average of tier-level accuracy: 0.1 × Acc(Level 1) + 0.2 × Acc(Level 2) + 0.3 × Acc(Level 3) + 0.4 × Acc(Level 4). Tier accuracy is the proportion of correctly predicted events within that difficulty tier.

## Input / output format

**Input**: Event description or question collected from live web sources, labeled with a difficulty tier (Level 1–4). Prompts may be single/multiple-choice or open-ended.

**Output**: Predicted outcome or selected answer option for the specified future event.

## Scoring recipe

```python
def compute_overall_score(predictions, gold, tier_labels):
    tier_acc = {}
    for tier in ['Level 1', 'Level 2', 'Level 3', 'Level 4']:
        mask = [t == tier for t in tier_labels]
        if sum(mask) == 0: continue
        correct = sum(1 for p, g in zip(predictions, gold) if p == g and mask[i])
        tier_acc[tier] = correct / sum(mask)
    weights = {'Level 1': 0.1, 'Level 2': 0.2, 'Level 3': 0.3, 'Level 4': 0.4}
    return sum(weights[t] * tier_acc.get(t, 0.0) for t in weights)
```

## Common pitfalls

- Predictions must be made prospectively before event resolution; post-hoc answers constitute data leakage and invalidate the live benchmark premise.
- Level 1/2 events are multiple-choice while Level 3/4 are open-ended, requiring different evaluation strategies (exact match vs. semantic/LLM-judge scoring).
- Human baseline comparisons used a different question subset than the automated model tests, so reported performance gaps are approximate and not directly comparable.

## Evidence (verbatim from paper)

> As for the overall score, we combine scores from the 4 difficulty tiers (see Table 3) using weights of 10%, 20%, 30%, and 40%, respectively, with heavier weights assigned to the more challenging tiers.

## Citation

```bibtex
@misc{zeng2025futurex,
  title={FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction},
  author={Zhiyuan Zeng et al. (2025)},
  year={2025},
  note={arXiv:2508.11987}
}
```

- arXiv: 2508.11987