Time Eval

---
name: time-eval
description: Evaluates large language models' ability to reason about temporal information in real-world scenarios. It probes capabilities ranging from basic time retrieval and localization to complex event ordering, duration comparison, and counterfactual temporal reasoning across knowledge-intensive, dynamic news, and long-form dialogue contexts. Use when the user wants to benchmark on TimE-Wiki, TimE-News, TimE-Dial, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.12891
  bibtex_key: wei2025time
  confidence: high
---

# time-eval

> TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios — Wei et al. (2025) (arXiv:2505.12891, 2025)

## What this evaluates

Evaluates large language models' ability to reason about temporal information in real-world scenarios. It probes capabilities ranging from basic time retrieval and localization to complex event ordering, duration comparison, and counterfactual temporal reasoning across knowledge-intensive, dynamic news, and long-form dialogue contexts.

## Datasets

- **TimE-Wiki** — total ?; splits: test (-1); repo https://github.com/sylvain-wei/TIME
- **TimE-News** — total ?; splits: test (-1); repo https://github.com/sylvain-wei/TIME
- **TimE-Dial** — total ?; splits: test (-1); repo https://github.com/sylvain-wei/TIME

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted answer exactly matches the ground truth string, 0 otherwise.
- `F1 score` — range: [0, 1]
  - Token-level F1 score computed between the predicted free-form answer and the ground truth.
- `Option-level F1 (macro F1)` — range: [0, 1]
  - Macro-averaged F1 score across all options for single-choice and multiple-choice QA tasks.

## Input / output format

**Input**: Contextual passages (encyclopedic knowledge, news articles, or multi-turn dialogues) paired with questions requiring temporal understanding, formatted as free-form or multiple-choice prompts.

**Output**: Free-form text responses or selected option letters corresponding to the question.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_types):
    em_scores, f1_scores = [], []
    for pred, gold, task in zip(predictions, golds, task_types):
        if task == 'timeline':
            em_scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
        else:
            f1_scores.append(token_f1(gold, pred))
    return {'EM': mean(em_scores), 'F1': mean(f1_scores)}
```

## Common pitfalls

- The Extract task is explicitly excluded from evaluations using the RAG framework on TimE-News and TimE-Lite-News.
- Retriever strategy (BM25, Vector, Hybrid) heavily influences TimE-News results, sometimes dominating over model architecture differences.
- The Timeline task requires ordering three events and consistently yields very low accuracy (<30%) across all models, making it a poor standalone metric for basic temporal understanding.

## Evidence (verbatim from paper)

> We evaluate free-form QA tasks with token-level metrics: Exact Match (EM) for the Timeline task, and F1 score for other free-form QA tasks. For single-choice and multiple-choice QA tasks, we use option-level F1 scores, emphasizing macro F1 for a comprehensive evaluation across all options.

## Citation

```bibtex
@misc{wei2025time,
  title={TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2505.12891}
}
```

- arXiv: 2505.12891