Timeseries Exam Eval

---
name: timeseries-exam-eval
description: This benchmark evaluates vision-language models' ability to reason over time series data across medical, financial, and meteorological domains. It probes pattern recognition, anomaly detection, and causal reasoning using synthetically generated multiple-choice questions derived from real-world datasets. Use when the user wants to benchmark on PTB-XL, MIT-BIH, MIMIC-IV Waveform, Yahoo Finance, WeatherBench 2, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.10291
  bibtex_key: gwiazda2026timeseriesexamagent
  confidence: high
---

# timeseries-exam-eval

> TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale — Gwiazda et al. (2026) (arXiv:2604.10291, 2026)

## What this evaluates

This benchmark evaluates vision-language models' ability to reason over time series data across medical, financial, and meteorological domains. It probes pattern recognition, anomaly detection, and causal reasoning using synthetically generated multiple-choice questions derived from real-world datasets.

## Datasets

- **PTB-XL** — total 151; splits: test (151)
- **MIT-BIH** — total 197; splits: test (197)
- **MIMIC-IV Waveform** — total 205; splits: test (205)
- **Yahoo Finance** — total 209; splits: test (209)
- **WeatherBench 2** — total 95; splits: test (95)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered multiple-choice questions out of the total number of questions. Calculated as correct predictions divided by total predictions.

## Input / output format

**Input**: Time series visualization (image) or text representation paired with a multiple-choice question.

**Output**: A single letter corresponding to the chosen multiple-choice option (e.g., A, B, C, D), or a structured response containing the option.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers, parsable=False):
    if parsable:
        valid = [(p, g) for p, g in zip(predictions, gold_answers) if is_valid_format(p)]
        return sum(1 for p, g in valid if p == g) / len(valid)
    return sum(1 for p, g in zip(predictions, gold_answers) if p == g) / len(gold_answers)
```

## Common pitfalls

- Confusing the source datasets (e.g., PTB-XL, MIT-BIH) with the actual benchmark, which consists of synthetically generated MCQs rather than raw time series classification tasks.
- Failing to distinguish between 'General' accuracy (strict formatting required) and 'Parsable' accuracy (only valid formats counted), leading to inconsistent performance reporting.
- Assuming the evaluation measures raw time series forecasting; it actually tests reasoning over generated questions about the data.

## Evidence (verbatim from paper)

> In total, we have 209 samples for YFinance, 197 samples for MIT-BIH, 151 samples for PTB-XL, 205 samples for MIMIC-IV Waveform, and 95 samples for WeatherBench 2. ... Even the strongest model achieves only 51.5% average accuracy, which highlights the limitations of current LLMs.

## Citation

```bibtex
@misc{gwiazda2026timeseriesexamagent,
  title={TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale},
  author={Gwiazda et al. (2026)},
  year={2026},
  note={arXiv:2604.10291}
}
```

- arXiv: 2604.10291