Hilad Anomaly Discovery Eval

---
name: hilad-anomaly-discovery-eval
description: Evaluates the effectiveness of tree-based ensemble anomaly detectors in human-in-the-loop active learning settings. It measures how quickly an algorithm can discover anomalies by querying a limited budget of instances, comparing batch and streaming data paradigms. Use when the user wants to benchmark on Abalone, ANN-Thyroid-1v3, Cardiotocography, KDD-Cup-99, Mammography, Shuttle, Yeast, Covtype, Electricity, Weather, or asks about evaluating this task. Reports anomaly_discovery_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1901.08930
  bibtex_key: das2019effectiveness
  confidence: high
---

# hilad-anomaly-discovery-eval

> Effectiveness of Tree-based Ensembles for Anomaly Discovery: Insights, Batch and Streaming Active Learning — Das et al. (2019) (arXiv:1901.08930, 2019)

## What this evaluates

Evaluates the effectiveness of tree-based ensemble anomaly detectors in human-in-the-loop active learning settings. It measures how quickly an algorithm can discover anomalies by querying a limited budget of instances, comparing batch and streaming data paradigms.

## Datasets

- **Abalone** — total 1920; splits: pool (1920)
- **ANN-Thyroid-1v3** — total 3251; splits: pool (3251)
- **Cardiotocography** — total 1700; splits: pool (1700)
- **KDD-Cup-99** — total 63009; splits: pool (63009)
- **Mammography** — total 11183; splits: pool (11183)
- **Shuttle** — total 12345; splits: pool (12345)
- **Yeast** — total 1191; splits: pool (1191)
- **Covtype** — total 286048; splits: pool (286048)
- **Electricity** — total 27447; splits: pool (27447)
- **Weather** — total 13117; splits: pool (13117)

## Metrics

- `anomaly_discovery_rate` **(primary)** — range: percent
  - Percentage of total anomalies discovered relative to the total number of anomalies in the dataset, plotted against the cumulative number of queried instances. Calculated as (discovered_anomalies / total_anomalies) * 100.

## Input / output format

**Input**: Feature vectors from benchmark datasets, provided either as a static unlabeled pool (batch setting) or sequentially in streaming windows of size K.

**Output**: Anomaly scores for each instance, used to rank candidates and select the top-B instances for human labeling per query round.

## Scoring recipe

```python
def compute_discovery_curve(discovered_set, total_anomalies, query_budget):
    rates = []
    for q in range(1, query_budget + 1):
        discovered_so_far = len(discovered_set[:q])
        rate = (discovered_so_far / total_anomalies) * 100
        rates.append(rate)
    return rates
```

## Common pitfalls

- Budget B is a fixed number of queries (e.g., 300 for batch), not a percentage of the dataset size.
- Streaming experiments require strict adherence to window size K and queries-per-window Q (e.g., Q=20).
- Drift adaptation uses a KL-divergence 95th percentile threshold to replace trees, not a fixed replacement rate.
- Electricity and Weather datasets had their anomaly classes down-sampled to exactly 5% prior to evaluation.

## Evidence (verbatim from paper)

> For each variant of the human-in-the-loop framework HiLAD, we plot the percentage of the total number of anomalies shown to the analyst versus the number of instances queried; this is the most relevant metric for an analyst in any real-world application.

## Citation

```bibtex
@misc{das2019effectiveness,
  title={Effectiveness of Tree-based Ensembles for Anomaly Discovery: Insights, Batch and Streaming Active Learning},
  author={Das et al. (2019)},
  year={2019},
  note={arXiv:1901.08930}
}
```

- arXiv: 1901.08930