Discovery Sensitivity

---
name: discovery-sensitivity
description: Evaluates the ability of machine learning classifiers to distinguish rare signal events from dominant Standard Model backgrounds in simulated high-energy physics collisions, and their capacity to accurately estimate signal fractions and discovery sensitivity via unbinned template fits. Use when the user has predictions and gold and needs to compute discovery-sensitivity.
metadata:
  skill_kind: metric
  source_arxiv: 2108.03125
  bibtex_key: [REDACTED]
  confidence: medium
---

# discovery-sensitivity

> Beyond Cuts in Small Signal Scenarios -- Enhanced Sneutrino Detectability Using Machine Learning — Alvestad et al. (2021) (arXiv:2108.03125, 2021)

## What this evaluates

Evaluates the ability of machine learning classifiers to distinguish rare signal events from dominant Standard Model backgrounds in simulated high-energy physics collisions, and their capacity to accurately estimate signal fractions and discovery sensitivity via unbinned template fits.

## Datasets

- **Simulated SUSY event samples (Points 0-50)** — total ?; splits: train (-1), test (-1)

## Metrics

- `discovery-sensitivity` **(primary)** — range: other
  - Statistical significance in standard deviations derived from unbinned template fits of classifier output distributions. Typically computed as $z = \sqrt{2((s+b)\ln(1+s/b) - s)}$ or via profile likelihood ratio in high-energy physics.
- `estimated signal fraction ($\hat{\alpha}$)` — range: [0, 1]
  - Signal mixture parameter extracted from the template fit, reported with statistical uncertainty.

## Input / output format

**Input**: High-dimensional kinematic observables (event features) from simulated particle collision events, labeled as signal or Standard Model background.

**Output**: Continuous classifier scores/probabilities per event, used as input to an unbinned template fit to yield $\hat{\alpha}$ and $z$.

## Scoring recipe

```python
model = train_classifier(features_point12, labels_point12)
scores = model.predict(test_features)
alpha_hat, alpha_err = unbinned_template_fit(scores, background_model, signal_model)
z = compute_significance(alpha_hat * N_signal_expected, N_background_expected)
return alpha_hat, z
```

## Common pitfalls

- Training and testing on the same parameter point overestimates generalization; cross-point testing is required.
- Using fixed kinematic cuts instead of ML classifier outputs reduces sensitivity in high-background regions.
- Ignoring systematic uncertainties in template shapes can bias $\hat{\alpha}$ and $z$ estimates.

## Evidence (verbatim from paper)

> machine learning (ML) outperforms traditional cut-and-count analyses by capturing non-linear correlations in high-dimensional kinematic observables. XGBoost and deep neural networks are trained on event features, with classifier outputs used in unbinned template fits to estimate signal mixtures, outperforming simple cuts. Shapley decomposition reveals how kinematic variables contribute to model decisions, demonstrating that ML models generalize across parameter space and improve discovery sensitivity, particularly in regions with low signal yields and high background contamination.

## Citation

```bibtex
@misc{alvestad2021sneutrino,
  title={Beyond Cuts in Small Signal Scenarios -- Enhanced Sneutrino Detectability Using Machine Learning},
  author={Alvestad et al. (2021)},
  year={2021},
  note={arXiv:2108.03125}
}
```

- arXiv: 2108.03125