Timer Eval

---
name: timer-eval
description: This benchmark evaluates a model's ability to perform temporal reasoning and extract accurate information from longitudinal electronic health records (EHRs). It probes whether models can correctly synthesize evidence across multiple time-stamped clinical visits, adhere to specified temporal boundaries, and maintain accuracy over long patient timelines. Use when the user wants to benchmark on TIMER-Bench, MedAlign, or asks about evaluating this task. Reports Correct.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.04176
  bibtex_key: cui2025timer
  confidence: high
---

# timer-eval

> TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records — Cui et al. (2025) (arXiv:2503.04176, 2025)

## What this evaluates

This benchmark evaluates a model's ability to perform temporal reasoning and extract accurate information from longitudinal electronic health records (EHRs). It probes whether models can correctly synthesize evidence across multiple time-stamped clinical visits, adhere to specified temporal boundaries, and maintain accuracy over long patient timelines.

## Datasets

- **TIMER-Bench** — total ?; splits: test (-1)
- **MedAlign** — total ?; splits: test (-1)

## Metrics

- `Correct` **(primary)** — range: percent
  - LLM-as-Judge (GPT-4o-mini) scores response correctness based on clinician-validated prompts. Correlates at |ρ|=0.94 with human judgment.
- `Complete` — range: percent
  - LLM-as-Judge scores response completeness based on coverage of required temporal evidence. Correlates at |ρ|=0.89 with human judgment.
- `BERTScore` — range: [0, 1]
  - Token-level representation similarity using distilbert-based-uncased embeddings.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence overlap between generated and reference text.

## Input / output format

**Input**: Chunked longitudinal EHR patient timeline (structured data + clinical notes) paired with a natural language instruction/query specifying a temporal scope or trend to analyze.

**Output**: Open-text response containing the model's analysis, trend description, or answer to the clinical query.

## Scoring recipe

```python
# LLM-as-Judge (Correctness/Completeness)
judge = GPT4oMini()
correctness_score = judge.evaluate(prediction, gold, prompt_template="Appendix F")
completeness_score = judge.evaluate(prediction, gold, prompt_template="Appendix F")

# Automated Metrics
bertscore = compute_bertscore(prediction, gold, model="distilbert-base-uncased")
rouge_l = compute_rouge_l(prediction, gold)

# Aggregate (reported as mean ± std over 10,000 bootstrap resamples)
metrics = {
    "Correct": correctness_score,
    "Complete": completeness_score,
    "BERTScore": bertscore,
    "ROUGE-L": rouge_l
}
```

## Common pitfalls

- Truncating long EHRs to fit context windows often removes recent records, severely hurting recency-focused tasks like MedAlign.
- Synthetic data generation exhibits a 'lost-in-the-middle' bias, causing models to default to extreme time points (start/end of timeline) rather than mid-timeline events.
- LLM-as-Judge requires specific prompting (Appendix F) and validation against clinicians; using default prompts may misalign with the reported correlation metrics.

## Evidence (verbatim from paper)

> We evaluate models’ open-text responses using LLM-Judge that assesses response correctness and completeness... We also employ head-to-head comparisons and automated metrics derived from token-level representations, including BertScore (using distilbert-based-uncased), ROUGE-L, CHRF, and METEOR to provide standard assessment of response quality. All LLM-based evaluations use GPT-4o-mini as the judge.

## Citation

```bibtex
@misc{cui2025timer,
  title={TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2503.04176}
}
```

- arXiv: 2503.04176