Dark Machines Anomaly Score Eval

---
name: dark-machines-anomaly-score-eval
description: Evaluates the ability of unsupervised machine learning models to detect deviations from Standard Model physics in high-energy collider data without assuming specific new physics signatures. It probes model-agnostic anomaly detection by measuring how well density estimation and reconstruction-based methods separate background events from potential signal events. Use when the user wants to benchmark on Dark Machines Anomaly Score Challenge Dataset, or asks about evaluating this task. Reports reconstruction loss.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.14027
  bibtex_key: aarrestad2021darkmachines
  confidence: medium
---

# dark-machines-anomaly-score-eval

> The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider — Aarrestad et al. (2021) (arXiv:2105.14027, 2021)

## What this evaluates

Evaluates the ability of unsupervised machine learning models to detect deviations from Standard Model physics in high-energy collider data without assuming specific new physics signatures. It probes model-agnostic anomaly detection by measuring how well density estimation and reconstruction-based methods separate background events from potential signal events.

## Datasets

- **Dark Machines Anomaly Score Challenge Dataset** — total 1000000000; splits: train (-1), validation (-1), test (-1); repo https://github.com/bostdiek/DarkMachines-UnsupervisedChallenge

## Metrics

- `reconstruction loss` **(primary)** — range: other
  - Computed as the mean squared error (MSE) or Chamfer distance between the input event features and the model's reconstructed output. Optionally combined with a KL divergence term to regularize the latent space. Higher values indicate greater deviation from the background distribution.

## Input / output format

**Input**: Variable-length set of particle 4-vectors ($p_{\mathrm{T}},\eta,\phi,E$) and missing transverse energy, padded to a fixed maximum number of objects (e.g., 8 or 20). Features are standardized using training set statistics.

**Output**: Scalar anomaly score per event.

## Scoring recipe

```python
def compute_anomaly_score(events, model, beta=0.5):
    recon = model(events)
    # Compute reconstruction error (MSE or Chamfer)
    loss = mean_squared_error(events, recon)
    # Add KL divergence term if applicable
    kl = compute_kl_divergence(model.latent_dist)
    score = (1 - beta) * loss + beta * kl
    return score
```

## Common pitfalls

- Failing to respect permutation invariance when processing variable-length particle sets, leading to inconsistent scores across shuffled inputs.
- Standardizing features using statistics from the entire dataset instead of only the training set, causing data leakage and artificially inflated anomaly scores.
- Arbitrarily selecting thresholds on continuous anomaly scores without calibrating against background-only validation data.

## Evidence (verbatim from paper)

> Not only the reconstruction loss can be a good outlier variable. Because the KL-divergence favors inputs that are encoded near the center of the latent space, the radius from the center is another anomaly score definition that can have predictive results.

## Citation

```bibtex
@misc{aarrestad2021darkmachines,
  title={The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider},
  author={Aarrestad et al. (2021)},
  year={2021},
  note={arXiv:2105.14027}
}
```

- arXiv: 2105.14027