Significance Eval

---
name: significance-eval
description: Evaluates the ability of ML classifiers to distinguish signal from background events in high-energy physics simulations. It measures classification quality via AUC and quantifies discovery potential using a likelihood-ratio-based significance metric optimized over a probability threshold. Use when the user has predictions and gold and needs to compute significance.
metadata:
  skill_kind: metric
  source_arxiv: 2401.03178
  bibtex_key: arganda2024mlhiggsdm
  confidence: high
---

# significance-eval

> Machine-Learning Performance on Higgs-Pair Production Associated with Dark Matter at the LHC — Arganda et al. (2024) (arXiv:2401.03178, 2024)

## What this evaluates

Evaluates the ability of ML classifiers to distinguish signal from background events in high-energy physics simulations. It measures classification quality via AUC and quantifies discovery potential using a likelihood-ratio-based significance metric optimized over a probability threshold.

## Datasets

- **Simulated di-Higgs + DM benchmarks** — total 14; splits: test (-1)

## Metrics

- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, measuring the classifier's overall ability to rank signal events higher than background events across all thresholds.
- `significance` **(primary)** — range: other
  - Computed as S = sqrt(-2((S+B)ln(B/(S+B))+S)), where S and B are the signal and background rates after applying a probability threshold. Optimized to maximize discovery potential, with adjustments for systematic uncertainties via Eq. (10).

## Input / output format

**Input**: Kinematic features of simulated particle collision events (signal and background).

**Output**: Probability score indicating the likelihood of the event being a signal event.

## Scoring recipe

```python
1. Train XGBoost/DNN on simulated benchmarks.
2. Compute AUC on test set.
3. For each probability threshold:
   a. Calculate signal rate S and background rate B after threshold cut.
   b. If no systematics: significance = sqrt(-2 * ((S+B)*ln(B/(S+B)) + S))
   c. If systematics included: use Eq. (10) to compute significance.
4. Select threshold that maximizes significance.
5. Report max significance and corresponding S/B rates.
```

## Common pitfalls

- Optimizing the probability threshold for accuracy or F1-score instead of the significance metric leads to suboptimal discovery potential.
- Ignoring systematic uncertainties on background rates significantly degrades significance and shifts the optimal threshold toward stricter background rejection.
- Assuming higher AUC always yields higher significance; in this benchmark, AUC > 0.9 is common, so significance is primarily driven by effective cross-section and background rejection.

## Evidence (verbatim from paper)

> For each benchmark, the probability threshold is chosen to maximize the significance computed as S = \sqrt{-2\left((S+B)\ln\left(\frac{B}{S+B}\right)+S\right)}, when potential systematic uncertainties are neglected, and using Eq. (10) otherwise. In the above expression, S and B are the signal and background rates after applying the cuts in Eq. (6) and the additional cut on the output of the classifiers.

## Citation

```bibtex
@misc{arganda2024mlhiggsdm,
  title={Machine-Learning Performance on Higgs-Pair Production Associated with Dark Matter at the LHC},
  author={Arganda et al. (2024)},
  year={2024},
  note={arXiv:2401.03178}
}
```

- arXiv: 2401.03178