Event Inference Eval

---
name: event-inference-eval
description: Evaluates large language models' ability to infer natural language event sequences from real-valued time series data (specifically win probabilities in sports). It probes causal reasoning, temporal context understanding, and the model's capacity to distinguish underlying time series dynamics from linguistic descriptions. Use when the user wants to benchmark on NBA & NFL Event Inference Benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.14190
  bibtex_key: tan2025inferring
  confidence: high
---

# event-inference-eval

> Inferring Events from Time Series using Language Models — Mingtian Tan et al. (2025) (arXiv:2503.14190, 2025)

## What this evaluates

Evaluates large language models' ability to infer natural language event sequences from real-valued time series data (specifically win probabilities in sports). It probes causal reasoning, temporal context understanding, and the model's capacity to distinguish underlying time series dynamics from linguistic descriptions.

## Datasets

- **NBA & NFL Event Inference Benchmark** — total 200; splits: test (200); repo https://github.com/BennyTMT/GAMETime

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the number of correctly inferred event sequences divided by the total number of questions (200).

## Input / output format

**Input**: A segment of real-valued time series data (Team A's win probabilities), optionally augmented with timestamps, scores, partial events, or team/player names depending on the experimental condition. Multiple-choice options include the ground-truth event sequence and negative sequences sampled from other games.

**Output**: A natural language sequence of events corresponding to the provided time series segment.

## Scoring recipe

```python
def compute_accuracy(predictions, gold, total=200):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / total
# Note: 'Invalid' outputs (failure to return a valid sequence) are counted as incorrect.
```

## Common pitfalls

- Models may rely on memorized team/player names or numerical values in event descriptions rather than true time-series reasoning; the benchmark mitigates this by using generic labels and post-cutoff games.
- Chain-of-Thought prompting improves reasoning but increases the rate of format-invalid outputs, which must be strictly filtered or counted as incorrect.
- Performance varies significantly across domains (e.g., sports vs. crypto/health), so results from one domain do not generalize to others without re-evaluation.

## Evidence (verbatim from paper)

> GPT-4o achieves an accuracy of 41%, and DS-R1-distill-Qwen-32B reaches 68%, while o1 performs the best, with an accuracy of 83%.

## Citation

```bibtex
@misc{tan2025inferring,
  title={Inferring Events from Time Series using Language Models},
  author={Mingtian Tan et al. (2025)},
  year={2025},
  note={arXiv:2503.14190}
}
```

- arXiv: 2503.14190