Datetime Eval

---
name: datetime-eval
description: Evaluates large language models on their ability to parse, translate, and perform arithmetic reasoning with datetime information. It probes structured string formatting (ISO-8601) and multi-step temporal calculations across diverse linguistic contexts. Use when the user wants to benchmark on DATETIME, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.16155
  bibtex_key: gaere2025datetime
  confidence: medium
---

# datetime-eval

> DATETIME: A new benchmark to measure LLM translation and reasoning capabilities — Gaere et al. (2025) (arXiv:2504.16155, 2025)

## What this evaluates

Evaluates large language models on their ability to parse, translate, and perform arithmetic reasoning with datetime information. It probes structured string formatting (ISO-8601) and multi-step temporal calculations across diverse linguistic contexts.

## Datasets

- **DATETIME** — total ?; splits: test (-1); repo https://github.com/EdGaere/DATETIME

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match accuracy: the fraction of instances where the model's output exactly matches the ground truth datetime string or computed result.

## Input / output format

**Input**: Natural language prompts requesting datetime translation, arithmetic operations (e.g., adding days), or mixed tasks requiring both parsing and computation.

**Output**: ISO-8601 formatted datetime strings or computed datetime values, depending on the task type.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_val in zip(predictions, gold):
        if normalize(pred) == normalize(gold_val):
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Strict ISO-8601 formatting requirements mean minor deviations (e.g., timezone offsets, date separators) cause exact-match failures.
- Arithmetic tasks require handling edge cases like leap years, month boundaries, and timezone conversions correctly.
- Models may hallucinate plausible but incorrect datetime strings if not strictly constrained to the ground truth format.

## Evidence (verbatim from paper)

> DATETIME introduces a synthetic, high-fidelity benchmark to evaluate LLMs on datetime translation and reasoning, revealing significant gaps in arithmetic and structured parsing capabilities. Tasks span translation to ISO-8601 format, datetime arithmetic (e.g., adding days), and mixed tasks requiring both, with synthetic data covering a broad temporal range and linguistic variability. Models are evaluated using accuracy.

## Citation

```bibtex
@misc{gaere2025datetime,
  title={DATETIME: A new benchmark to measure LLM translation and reasoning capabilities},
  author={Gaere et al. (2025)},
  year={2025},
  note={arXiv:2504.16155}
}
```

- arXiv: 2504.16155