Deep Bias Eval

---
name: deep-bias-eval
description: Evaluates the ability of a deep learning model to detect and classify structural bias in heuristic optimization algorithms by analyzing raw performance distributions against a uniform null hypothesis. Use when the user wants to benchmark on BIAS toolbox heuristic pool on $f_0$, or asks about evaluating this task. Reports detection accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.01869
  bibtex_key: vanstein2023deepbias
  confidence: high
---

# deep-bias-eval

> Deep-BIAS: Detecting Structural Bias using Explainable AI — van Stein et al. (2023) (arXiv:2304.01869, 2023)

## What this evaluates

Evaluates the ability of a deep learning model to detect and classify structural bias in heuristic optimization algorithms by analyzing raw performance distributions against a uniform null hypothesis.

## Datasets

- **BIAS toolbox heuristic pool on $f_0$** — total 336; splits: test (336)

## Metrics

- `detection accuracy` **(primary)** — range: [0, 1]
  - Proportion of algorithms correctly classified as biased or non-biased compared to the baseline statistical toolbox or ground truth.
- `classification precision` — range: [0, 1]
  - Precision in predicting the specific type of structural bias (e.g., discretisation, clusters) among algorithms flagged as biased.

## Input / output format

**Input**: Raw performance distributions (point positions in the search space) from executing heuristic optimizers on test function $f_0$ at dimensionality $n=30$.

**Output**: Class probabilities for each bias type (e.g., biased, non-biased, discretisation, clusters) and a final biased/non-biased decision.

## Scoring recipe

```python
correct_biased = sum(1 for p, g in zip(predictions, ground_truth) if p == g)
accuracy = correct_biased / len(predictions)
correct_types = sum(1 for p, g in zip(type_preds, type_truth) if p == g and g != 'non-biased')
precision = correct_types / sum(1 for g in type_truth if g != 'non-biased')
```

## Common pitfalls

- The original statistical toolbox's Random Forest model forces class probabilities to sum to 1, whereas Deep-BIAS reserves probability mass for the uniform class, making direct probability comparison misleading.
- Boundary correction methods like 'saturate' force points to exact bounds, which statistical tests easily flag as non-uniform, but Deep-BIAS may overlook due to lacking explicit boundary context.
- Minor distributional deviations may trigger Deep-BIAS probability shifts but fail statistical p-value corrections, leading to false disagreements on bias presence.

## Evidence (verbatim from paper)

> It replaces 39 statistical tests with a convolutional neural network that directly predicts SB presence and type, outperforming traditional methods in both detection accuracy and classification precision.

## Citation

```bibtex
@misc{vanstein2023deepbias,
  title={Deep-BIAS: Detecting Structural Bias using Explainable AI},
  author={van Stein et al. (2023)},
  year={2023},
  note={arXiv:2304.01869}
}
```

- arXiv: 2304.01869