Shap Diversity Ensemble Eval

---
name: shap-diversity-ensemble-eval
description: Probes how well SHAP-based feature attribution divergence correlates with prediction/output diversity across unsupervised anomaly detection algorithms, and quantifies how this explanation-driven diversity impacts the robustness and accuracy of model ensembles. Use when the user wants to benchmark on ADBench subset (16 datasets: anthyroid, breastw, glass, Hepatitis, Lymphography, mammography, PageBlocks, Pima, Stamps, thyroid, vertebral, vowels, WBC, Wilt, wine, yeast), or asks about evaluating this task. Reports AUCPR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.00208
  bibtex_key: levy2026shap
  confidence: high
---

# shap-diversity-ensemble-eval

> Analyzing Shapley Additive Explanations to Understand Anomaly Detection Algorithm Behaviors and Their Complementarity — Levy et al. (2026) (arXiv:2602.00208, 2026)

## What this evaluates

Probes how well SHAP-based feature attribution divergence correlates with prediction/output diversity across unsupervised anomaly detection algorithms, and quantifies how this explanation-driven diversity impacts the robustness and accuracy of model ensembles.

## Datasets

- **ADBench subset (16 datasets: anthyroid, breastw, glass, Hepatitis, Lymphography, mammography, PageBlocks, Pima, Stamps, thyroid, vertebral, vowels, WBC, Wilt, wine, yeast)** — total ?; splits: train (-1), test (-1)

## Metrics

- `AUCPR` **(primary)** — range: [0, 1]
  - Area Under the Precision-Recall Curve, computed on binary predictions versus ground truth labels to handle class imbalance.
- `ρPS` — range: [-1, 1]
  - Pearson linear correlation between SHAP value vectors of two models across test instances.
- `ρNDCG` — range: [0, 1]
  - NDCG similarity between SHAP feature importance rankings of two models.
- `ρScores` — range: [-1, 1]
  - Pearson linear correlation between anomaly score vectors of two models.
- `J` — range: [0, 1]
  - Jaccard index between binary anomaly prediction sets of two models.
- `Mantel correlation (r_M)` — range: [-1, 1]
  - Statistical correlation between two distance/similarity matrices, assessed via permutation test.

## Input / output format

**Input**: Raw feature vectors (≤20 features) from tabular datasets. Each instance is passed to 14 UAD models to generate anomaly scores and binary predictions, and to Kernel SHAP to compute feature attributions.

**Output**: Per instance: anomaly score, binary prediction (normal/anomalous), and SHAP feature attribution values. Ensembles output aggregated anomaly scores via rank aggregation, yielding a final AUCPR score on the test set.

## Scoring recipe

```python
1. Compute SHAP values per instance using Kernel SHAP (k=50 k-means background).
2. Compute similarity matrices across models:
   rho_PS = pearson_corr(shap_m1, shap_m2)
   rho_NDCG = ndcg_similarity(rank(shap_m1), rank(shap_m2))
   rho_Scores = pearson_corr(scores_m1, scores_m2)
   J = jaccard_index(pred_m1, pred_m2)
3. For ensembles (size 3), apply rank aggregation to anomaly scores.
4. Evaluate ensemble performance: aucpr = compute_aucpr(y_true, ensemble_scores)
5. Assess matrix correlation: r_M = mantel_test(matrix_A, matrix_B, n_perm=999)
6. Predict ensemble gain via linear regression on mean individual AUCPR and diversity (1 - rho_PS).
```

## Common pitfalls

- Default anomaly thresholds are used for all models, which may not be optimal for imbalanced datasets and can skew Jaccard/score correlations.
- SHAP approximations via k-means background summarization (k=50) may introduce attribution noise, affecting ρPS and ρNDCG stability.
- Diversity is not universally beneficial; negative correlations with AUCPR occur when a single dominant model exists or datasets are too complex, requiring careful model quality filtering.

## Evidence (verbatim from paper)

> We computed the four similarity matrices ρPS (linear correlation between SHAP values), ρNDCG (similarity of SHAP feature importance rankings), ρScores (linear correlation between anomaly scores), J (jaccard index between anomaly predictions) for each dataset.

## Citation

```bibtex
@misc{levy2026shap,
  title={Analyzing Shapley Additive Explanations to Understand Anomaly Detection Algorithm Behaviors and Their Complementarity},
  author={Levy et al. (2026)},
  year={2026},
  note={arXiv:2602.00208}
}
```

- arXiv: 2602.00208