Evaluates malware classifiers on detection, family identification, and attribute prediction tasks across multiple file formats. It specifically probes robustness against concept drift and novel malware families using a temporally separated test set and a challenge set of evasive samples. Use when the user wants to benchmark on EMBER2024, or asks about evaluating this task. Reports ROC AUC.
Installs into .claude/skills of the current project.
Are you the author of Ember2024 Eval?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/qhjqhj00-ember2024-eval)
---
name: ember2024-eval
description: Evaluates malware classifiers on detection, family identification, and attribute prediction tasks across multiple file formats. It specifically probes robustness against concept drift and novel malware families using a temporally separated test set and a challenge set of evasive samples. Use when the user wants to benchmark on EMBER2024, or asks about evaluating this task. Reports ROC AUC.
metadata:
skill_kind: dataset_eval
source_arxiv: 2506.05074
bibtex_key: [REDACTED]
confidence: high
---
# ember2024-eval
> EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers — Joyce et al. (2025) (arXiv:2506.05074, 2025)
## What this evaluates
Evaluates malware classifiers on detection, family identification, and attribute prediction tasks across multiple file formats. It specifically probes robustness against concept drift and novel malware families using a temporally separated test set and a challenge set of evasive samples.
## Datasets
- **EMBER2024** — total 3200000; splits: train (-1), test (-1), challenge (-1); repo https://github.com/FutureComputing4AI/EMBER2024
## Metrics
- `ROC AUC` **(primary)** — range: [0, 1]
- Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
- `PR AUC` — range: [0, 1]
- Area under the Precision-Recall curve, measuring the trade-off between precision and recall across thresholds. Preferred over ROC AUC for highly imbalanced datasets.
## Input / output format
**Input**: Malware or benign files in PE (Win32/Win64/.NET), APK, ELF, or PDF formats, represented using EMBER feature version 3.
**Output**: Binary label (malicious/benign) or multi-class/multi-label predictions for family/attributes.
## Scoring recipe
```python
def compute_roc_auc(y_true, y_scores):
fpr, tpr, _ = roc_curve(y_true, y_scores)
return auc(fpr, tpr)
def compute_pr_auc(y_true, y_scores):
precision, recall, _ = precision_recall_curve(y_true, y_scores)
return auc(recall, precision)
```
## Common pitfalls
- ROC AUC can be misleading on highly imbalanced datasets (e.g., the challenge set); PR AUC should be reported instead.
- The challenge set contains only malware, so it must be joined with the corresponding benign test partition to compute standard binary classification metrics.
- Temporal train/test splits introduce concept drift, making detection of novel families significantly harder than random splits.
## Evidence (verbatim from paper)
> The ROC AUC and Precision-Recall (PR) AUC scores in Table 5 indicate that the trained LightGBM classifiers are able to accurately detect malicious files in the EMBER2024 test set, which consists of files that appeared in VirusTotal for the first time 1-12 weeks after the most recent file in the training set.
## Citation
```bibtex
@misc{joyce2025ember2024,
title={EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers},
author={Joyce et al. (2025)},
year={2025},
note={arXiv:2506.05074}
}
```
- arXiv: 2506.05074
