Mad Ood Eval

---
name: mad-ood-eval
description: Evaluates a model's ability to detect out-of-distribution (OOD) malware variants and classify known malware families without using OOD samples during training. It probes both classification accuracy on in-distribution data and the statistical separation capability between known and novel threats using cluster-driven decision boundaries. Use when the user wants to benchmark on Unspecified malware dataset (25 families), or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.17594
  bibtex_key: ige2025madood
  confidence: medium
---

# mad-ood-eval

> MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification — Ige et al. (2025) (arXiv:2512.17594, 2025)

## What this evaluates

Evaluates a model's ability to detect out-of-distribution (OOD) malware variants and classify known malware families without using OOD samples during training. It probes both classification accuracy on in-distribution data and the statistical separation capability between known and novel threats using cluster-driven decision boundaries.

## Datasets

- **Unspecified malware dataset (25 families)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen in-distribution sample is ranked higher than a randomly chosen OOD sample by the model's confidence or distance score.
- `Accuracy` — range: [0, 1]
  - Standard classification accuracy: the ratio of correctly predicted in-distribution samples (benign or malware family) to the total number of in-distribution test samples.

## Input / output format

**Input**: Image representations of malware samples. For the second-stage classifier, inputs also include the initial cluster analysis prediction and the first model's prediction output.

**Output**: Multi-class classification label (benign or specific malware family) and an OOD detection score/probability derived from Z-score distances to class centroids.

## Scoring recipe

```python
def compute_auroc(y_true, y_scores):
    # y_true: 1 for in-distribution, 0 for OOD
    # y_scores: model confidence or inverse distance for in-distribution class
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_accuracy(y_true, y_pred):
    return sum(y_true == y_pred) / len(y_true)
```

## Common pitfalls

- The Z-score thresholding (±1) is an internal decision rule for flagging outliers during inference, not the evaluation metric itself; evaluation relies on AUROC.
- OOD samples are explicitly excluded from training; using them during training violates the paper's protocol of learning only from in-distribution data.
- The test set contains both in-distribution and OOD samples, so AUROC must be computed on the combined test set, not just on in-distribution data.

## Evidence (verbatim from paper)

> Evaluation Metrics In-Distribution Performance: Accuracy, and Confusion matrix for malware classification. OOD Detection Performance: AUROC (Area Under ROC Curve): Measures separation between in-distribution and OOD

## Citation

```bibtex
@misc{ige2025madood,
  title={MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification},
  author={Ige et al. (2025)},
  year={2025},
  note={arXiv:2512.17594}
}
```

- arXiv: 2512.17594