Malware Clustering Eval

---
name: malware-clustering-eval
description: This benchmark evaluates the effectiveness of unsupervised clustering algorithms on large-scale malware binary datasets. It probes how well different feature representations and clustering methods can group malware samples into coherent families while handling real-world noise and benign samples. Use when the user wants to benchmark on Bodmas, Ember, Security, or asks about evaluating this task. Reports Homogeneity.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.23198
  bibtex_key: [REDACTED]
  confidence: high
---

# malware-clustering-eval

> Clustering Malware at Scale: A First Full-Benchmark Study — Mocko et al. (2025) (arXiv:2511.23198, 2025)

## What this evaluates

This benchmark evaluates the effectiveness of unsupervised clustering algorithms on large-scale malware binary datasets. It probes how well different feature representations and clustering methods can group malware samples into coherent families while handling real-world noise and benign samples.

## Datasets

- **Bodmas** — total ?; splits: train (-1), test (-1)
- **Ember** — total ?; splits: train (-1), test (-1)
- **Security** — total ?; splits: train (-1), test (-1)

## Metrics

- `Homogeneity` **(primary)** — range: percent
  - Measures whether each cluster contains only members of a single class. Calculated as 1 - (H(C) + H(K)) / H(C,K), where H(C) is entropy of cluster assignments, H(K) is entropy of class labels, and H(C,K) is joint entropy. Values are reported as percentages in the paper.
- `V-Measure` — range: percent
  - Harmonic mean of homogeneity and completeness. Penalizes over-clustering by incorporating completeness, which measures whether all members of a given class are assigned to the same cluster.

## Input / output format

**Input**: Numerical feature representations (embeddings) of malware binaries, generated using PCA, UMAP, or Autoencoder dimensionality reduction.

**Output**: Cluster assignment labels for each input sample.

## Scoring recipe

```python
def compute_homogeneity(predictions, gold):
    # predictions: cluster labels assigned by algorithm
    # gold: true malware family labels
    from sklearn.metrics import homogeneity_score
    return homogeneity_score(gold, predictions) * 100
```

## Common pitfalls

- High Homogeneity can be artificially inflated by creating too many clusters (e.g., one per sample), which is why V-Measure remains low despite high Homogeneity scores.
- Dataset composition heavily influences clustering quality; Ember underperforms Bodmas likely due to underlying malware family distributions rather than the number of clusters or algorithm choice.
- Including benign samples does not degrade clustering performance, contrary to common assumptions in malware analysis.

## Evidence (verbatim from paper)

> Homogeneity was picked as the primary metric to show us the difference in quality (or lack thereof) in this experiment. Due to time constraints, we trained each representation only once. We also ran K-Means only once on the trained representations.

## Citation

```bibtex
@misc{mocko2025clustering,
  title={Clustering Malware at Scale: A First Full-Benchmark Study},
  author={Mocko et al. (2025)},
  year={2025},
  note={arXiv:2511.23198}
}
```

- arXiv: 2511.23198