Nab Eval

---
name: nab-eval
description: Evaluates real-time anomaly detection algorithms on streaming time-series data, measuring their ability to detect natural and synthetic anomalies while penalizing false alarms and delayed detections. Use when the user wants to benchmark on NAB 1.0, or asks about evaluating this task. Reports NAB Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1510.03336
  bibtex_key: lavin2015nab
  confidence: medium
---

# nab-eval

> Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark — Lavin et al. (2015) (arXiv:1510.03336, 2015)

## What this evaluates

Evaluates real-time anomaly detection algorithms on streaming time-series data, measuring their ability to detect natural and synthetic anomalies while penalizing false alarms and delayed detections.

## Datasets

- **NAB 1.0** — total 58; splits: (unstated); repo https://github.com/numenta/NAB

## Metrics

- `NAB Score` **(primary)** — range: other
  - A composite score calculated per application profile (Standard, Reward low FP, Reward low FN) that rewards early true positives and penalizes false positives and false negatives. Scores are normalized to a standard range, with higher values indicating better real-time detection performance.

## Input / output format

**Input**: Streaming time-series data files containing labeled natural and synthetic anomalies.

**Output**: Per-timestep anomaly scores or binary anomaly labels produced by the detector.

## Scoring recipe

```python
def compute_nab_score(predictions, gold, profile):
    early_reward = calculate_early_detection_bonus(predictions, gold)
    fp_penalty = calculate_false_positive_penalty(predictions, gold)
    fn_penalty = calculate_false_negative_penalty(predictions, gold)
    if profile == 'Reward low FP':
        score = early_reward - 2 * fp_penalty
    elif profile == 'Reward low FN':
        score = early_reward - 0.5 * fn_penalty
    else:
        score = early_reward - fp_penalty - fn_penalty
    return normalize_to_standard_range(score)
```

## Common pitfalls

- Algorithms' parameters were optimized per profile to yield the best possible NAB scores, which may overstate real-world generalization.
- Random detection baselines do not score near zero due to the NAB score optimization step, making chance performance non-intuitive.
- Scores are reported separately for three application profiles (Standard, Reward low FP, Reward low FN) rather than a single aggregate metric.

## Evidence (verbatim from paper)

> Table 1 summarizes the scores for all algorithms on the three application profiles using the data files in NAB 1.0. The HTM detector achieves the best overall scores, followed by Etsy and Twitter. Although the HTM detector performs markedly better, the Etsy and Twitter algorithms perform significantly better than chance across all three application profiles.

## Citation

```bibtex
@misc{lavin2015nab,
  title={Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark},
  author={Lavin et al. (2015)},
  year={2015},
  note={arXiv:1510.03336}
}
```

- arXiv: 1510.03336