Ember Eval

---
name: ember-eval
description: Evaluates machine learning models on static malware classification of Windows PE binaries. It probes the effectiveness of engineered static features versus raw binary inputs for distinguishing malicious from benign software. Use when the user wants to benchmark on EMBER, or asks about evaluating this task. Reports ROC AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1804.04637
  bibtex_key: anderson2018ember
  confidence: high
---

# ember-eval

> EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models — Anderson et al. (2018) (arXiv:1804.04637, 2018)

## What this evaluates

Evaluates machine learning models on static malware classification of Windows PE binaries. It probes the effectiveness of engineered static features versus raw binary inputs for distinguishing malicious from benign software.

## Datasets

- **EMBER** — total ?; splits: train (-1), test (-1); repo https://github.com/endgameinc/ember

## Metrics

- `ROC AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the probability that a randomly chosen malicious sample scores higher than a randomly chosen benign sample across all classification thresholds.
- `detection rate` — range: percent
  - True positive rate (recall) at a specified false positive rate threshold, commonly reported at 0.1% and 1% FPR.

## Input / output format

**Input**: Vectorized static features extracted from Windows PE binaries, formatted as a 2351-dimensional vector using raw features and feature hashing.

**Output**: Binary classification label: malicious or benign.

## Scoring recipe

```python
def evaluate(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    roc_auc = auc(fpr, tpr)
    idx_01 = np.argmin(np.abs(fpr - 0.001))
    det_rate_01 = tpr[idx_01]
    idx_1 = np.argmin(np.abs(fpr - 0.01))
    det_rate_1 = tpr[idx_1]
    return roc_auc, det_rate_01, det_rate_1
```

## Common pitfalls

- Using the J48/Adobe Malware Classifier as a baseline without recognizing its dataset bias toward OS classification rather than malware detection.
- Expecting featureless deep learning models (e.g., MalConv) to outperform traditional ML on structured PE features without significant hyperparameter tuning.
- Reporting detection rates without specifying the exact false positive rate threshold, as performance varies significantly between 0.1% and 1% FPR.

## Evidence (verbatim from paper)

> The ROC AUC exceeds 0.99911. A threshold of 0.871 on the model score results in less than 0.1% FP rate at a detection rate exceeding 92.99%. At less than 1% FP rate, the model exceeds 98.2% detection rate.

## Citation

```bibtex
@misc{anderson2018ember,
  title={EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models},
  author={Anderson et al. (2018)},
  year={2018},
  note={arXiv:1804.04637}
}
```

- arXiv: 1804.04637