Pe Malware Classification Eval

---
name: pe-malware-classification-eval
description: Evaluates learning-based models for PE malware family classification across image, binary, and disassembly input formats. It measures classification accuracy and probes model robustness under concept drift, alongside computational resource overhead. Use when the user wants to benchmark on BIG-15, Malimg, MalwareBazaar, MalwareDrift, or asks about evaluating this task. Reports Macro F1-score ($F1_{macro}$).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.15552
  bibtex_key: [REDACTED]
  confidence: high
---

# pe-malware-classification-eval

> A Comprehensive Study on Learning-Based PE Malware Family Classification Methods — Ma et al. (2021) (arXiv:2110.15552, 2021)

## What this evaluates

Evaluates learning-based models for PE malware family classification across image, binary, and disassembly input formats. It measures classification accuracy and probes model robustness under concept drift, alongside computational resource overhead.

## Datasets

- **BIG-15** — total ?; splits: 10-fold CV (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification
- **Malimg** — total ?; splits: 10-fold CV (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification
- **MalwareBazaar** — total ?; splits: 10-fold CV (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification
- **MalwareDrift** — total ?; splits: pre-drift (-1), post-drift (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification

## Metrics

- `Accuracy (A)` — range: percent
  - Percentage of correctly classified samples out of the total.
- `Macro Precision ($P_{macro}$)` — range: percent
  - Average precision calculated independently for each class and then unweighted mean across all classes.
- `Macro Recall ($R_{macro}$)` — range: percent
  - Average recall calculated independently for each class and then unweighted mean across all classes.
- `Macro F1-score ($F1_{macro}$)` **(primary)** — range: percent
  - Harmonic mean of macro precision and macro recall. Computed as 2 * (P_macro * R_macro) / (P_macro + R_macro).

## Input / output format

**Input**: PE malware files converted to one of three representations: grayscale images, raw binary byte sequences, or disassembly opcode sequences.

**Output**: Predicted malware family label (multi-class classification).

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, num_classes):
    accuracy = sum(1 for t, p in zip(y_true, y_pred) if t == p) / len(y_true)
    precisions, recalls = [], []
    for c in range(num_classes):
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        precisions.append(tp / (tp + fp) if (tp + fp) > 0 else 0.0)
        recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    p_macro = sum(precisions) / num_classes
    r_macro = sum(recalls) / num_classes
    f1_macro = 2 * (p_macro * r_macro) / (p_macro + r_macro) if (p_macro + r_macro) > 0 else 0.0
    return accuracy * 100, p_macro * 100, r_macro * 100, f1_macro * 100
```

## Common pitfalls

- Imbalanced class distributions heavily penalize large models (e.g., VGG-16 on BIG-15 where a minority class has <0.4% of samples).
- Evaluating only on static datasets ignores concept drift, which causes average F1-score drops of over 30% in real-world scenarios.
- GPU memory constraints force small batch sizes for large disassembly files, artificially degrading performance for memory-heavy methods like MAGIC.

## Evidence (verbatim from paper)

> From Table 7, we can observe that all existing methods suffer from a large performance drop while confronting the concept drift in real industry scenarios, where the reduction of F1-score is up to 27.07%-69.62%. The F1-scores for all methods on the post-drift dataset are no more than 45%, reflecting that existing methods fail to consider the scenario of concept drift and there is still much improvement space.

## Citation

```bibtex
@misc{ma2021comprehensive,
  title={A Comprehensive Study on Learning-Based PE Malware Family Classification Methods},
  author={Ma et al. (2021)},
  year={2021},
  note={arXiv:2110.15552}
}
```

- arXiv: 2110.15552