Tsaqa Eval

---
name: tsaqa-eval
description: Evaluates large language models' ability to perform time series analysis and reasoning across six tasks (anomaly detection, classification, characterization, comparison, data transformation, and temporal relationship) using three question formats (true-or-false, multiple-choice, and puzzling). Use when the user wants to benchmark on TSAQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.23204
  bibtex_key: jing2026tsaqa
  confidence: high
---

# tsaqa-eval

> TSAQA: Time Series Analysis Question And Answering Benchmark — Jing et al. (2026) (arXiv:2601.23204, 2026)

## What this evaluates

Evaluates large language models' ability to perform time series analysis and reasoning across six tasks (anomaly detection, classification, characterization, comparison, data transformation, and temporal relationship) using three question formats (true-or-false, multiple-choice, and puzzling).

## Datasets

- **TSAQA** — total 210000; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - The proportion of correctly answered questions out of the total number of questions, averaged over all samples rather than per-task averages.

## Input / output format

**Input**: Time series data (often accompanied by domain context as text) paired with a question in one of three formats: true-or-false (TF), multiple-choice (MC), or puzzling (PZ) requiring chronological ordering.

**Output**: A single answer choice (e.g., A/B/C/D for MC, True/False for TF, or a reordered sequence for PZ).

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Models exhibit a 'Smoothness Bias' on PZ questions, incorrectly predicting smoother transitions than ground truth.
- Performance generally declines as input length increases, except for the Temporal Relationship task where longer inputs help.
- Results should be averaged over all samples rather than computing per-task averages first.

## Evidence (verbatim from paper)

> Accuracy Correlate Analysis. Input Lengths. Figure[3] in Appendix) illustrates the relationship between input length and model accuracy. Across all six models and five tasks, excluding the Temporal Relationship task, we observe a consistent trend that performance declines as input length increases, indicating that longer inputs correspond to more difficult questions.

## Citation

```bibtex
@misc{jing2026tsaqa,
  title={TSAQA: Time Series Analysis Question And Answering Benchmark},
  author={Jing et al. (2026)},
  year={2026},
  note={arXiv:2601.23204}
}
```

- arXiv: 2601.23204