Malware Family Classification Eval

---
name: malware-family-classification-eval
description: Evaluates the ability of LLMs and their ensembles to correctly classify malware samples into one of ten canonical families based on their behavior or code semantics. It probes robustness to class imbalance and the effectiveness of hierarchical decision-making under obfuscation. Use when the user wants to benchmark on Gold-standard malware family dataset, or asks about evaluating this task. Reports Macro F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.02490
  bibtex_key: bai2026automated
  confidence: high
---

# malware-family-classification-eval

> Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models — Bai et al. (2026) (arXiv:2604.02490, 2026)

## What this evaluates

Evaluates the ability of LLMs and their ensembles to correctly classify malware samples into one of ten canonical families based on their behavior or code semantics. It probes robustness to class imbalance and the effectiveness of hierarchical decision-making under obfuscation.

## Datasets

- **Gold-standard malware family dataset** — total 200; splits: test (200)

## Metrics

- `Accuracy` — range: [0, 1]
  - Fraction of correctly classified samples out of the total number of samples.
- `Macro Precision` — range: [0, 1]
  - Unweighted mean of precision calculated for each of the ten malware families independently.
- `Macro Recall` — range: [0, 1]
  - Unweighted mean of recall calculated for each of the ten malware families independently.
- `Macro F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of macro precision and macro recall, providing a balanced measure across all classes regardless of imbalance.

## Input / output format

**Input**: Malware samples (code/behavioral representations) to be classified into one of ten canonical malware families.

**Output**: A single canonical malware family label from the predefined set of ten.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, num_classes=10):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    precisions, recalls, f1s = [], [], []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        precisions.append(prec)
        recalls.append(rec)
        f1s.append(f1)
    macro_prec = sum(precisions) / num_classes
    macro_rec = sum(recalls) / num_classes
    macro_f1 = sum(f1s) / num_classes
    return accuracy, macro_prec, macro_rec, macro_f1
```

## Common pitfalls

- The dataset is very small (200 samples), so results lack statistical significance testing and may not generalize.
- Macro-averaging is used to handle class imbalance, which can mask poor performance on minority families.
- Model weights for the ensemble are derived from Macro-F1 scores computed on the same gold-standard dataset, risking overfitting to the evaluation set.

## Evidence (verbatim from paper)

> Performance was measured using Accuracy, Macro Precision, Macro Recall, and Macro F1-score. Macro-averaged metrics were emphasized due to class imbalance and to ensure equal importance across malware families.

## Citation

```bibtex
@misc{bai2026automated,
  title={Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models},
  author={Bai et al. (2026)},
  year={2026},
  note={arXiv:2604.02490}
}
```

- arXiv: 2604.02490