Orionbench Eval

---
name: orionbench-eval
description: Evaluates unsupervised time series anomaly detection pipelines across diverse real-world and synthetic datasets. It measures detection accuracy for both point and segment anomalies while tracking computational efficiency and model stability over continuous benchmarking cycles. Use when the user wants to benchmark on OrionBench (NASA, NAB, Yahoo S5, UCR), or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.17748
  bibtex_key: [REDACTED]
  confidence: high
---

# orionbench-eval

> OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User — Alnegheimish et al. (2023) (arXiv:2310.17748, 2023)

## What this evaluates

Evaluates unsupervised time series anomaly detection pipelines across diverse real-world and synthetic datasets. It measures detection accuracy for both point and segment anomalies while tracking computational efficiency and model stability over continuous benchmarking cycles.

## Datasets

- **OrionBench (NASA, NAB, Yahoo S5, UCR)** — total 742; splits: test (-1); repo https://github.com/sintel-dev/Orion

## Metrics

- `F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Computed over predicted anomaly labels versus ground truth labels.
- `Precision` — range: [0, 1]
  - Ratio of correctly predicted anomalies to all predicted anomalies.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted anomalies to all actual anomalies.
- `Runtime` — range: seconds
  - Total elapsed time in seconds to train the pipeline and run inference on a single signal.

## Input / output format

**Input**: Univariate or multivariate time series signals with corresponding ground truth anomaly labels.

**Output**: Per-timestep anomaly scores or binary anomaly flags indicating detected anomalies.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth):
    tp = sum(p == 1 and g == 1 for p, g in zip(predictions, ground_truth))
    fp = sum(p == 1 and g == 0 for p, g in zip(predictions, ground_truth))
    fn = sum(p == 0 and g == 1 for p, g in zip(predictions, ground_truth))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'precision': precision, 'recall': recall, 'f1': f1}
```

## Common pitfalls

- Reconstruction-based models (e.g., LSTM AE, TadGAN) consistently underperform on point anomalies due to smoothed anomaly scores.
- Hyperparameter choices, particularly window size, significantly impact results; static windows may not generalize across signals of varying lengths.
- Cloud-based services (e.g., Azure AD) tend to over-flag anomalies, artificially inflating F1 on high-anomaly datasets while degrading on sparse ones.

## Evidence (verbatim from paper)

> The output result is stored as a detailed .csv file that shows for each pipeline and signals performance metrics such as accuracy, precision, recall, and F1 score.

## Citation

```bibtex
@misc{alnegheimish2023orionbench,
  title={OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User},
  author={Alnegheimish et al. (2023)},
  year={2023},
  note={arXiv:2310.17748}
}
```

- arXiv: 2310.17748